Hardware vs software RAID5 (ext4 vs ZFS)

[Un Nameless]

Hi folks,

I've got a 100 drive JBOD connected to my server via a 12 SAS connection to an LSI 9580 RAID card! I'll use this JBOD for large file storage!

I've got two options as far as I'm aware: either go with the BIOS management of the RAID controller, and create there a couple of RAID5 volumes and let the hardware do the RAID stuff, and thus format in software said volumes with EXT4 (because at this point using ZFS would yield few if any advantages to ext4) or treat the drives in BIOS as JBODs and skip the hardware RAID and go in software with ZFS pools for RAIDZ1.

What are your thoughts? What is the best approach and what are the drawbacks of both?

Thank you!

 

[dswartz]

Whether SW or HW, 100 drives is a hell of a lot to have in only a couple of pools! Just looking to get hosed there!

 

[Un Nameless]

Whether SW or HW, 100 drives is a hell of a lot to have in only a couple of pools! Just looking to get hosed there!

Dully noted but this doesn't address my question!

 

[dswartz]

Well, I'd prefer ZFS over ext4 for 2 reasons: transparent compression and checksumming to protect data. Also makes you less dependent on vendor-specific RAID.

 

[Un Nameless]

Well, I'd prefer ZFS over ext4 for 2 reasons: transparent compression and checksumming to protect data. Also makes you less dependent on vendor-specific RAID.

Do you think there will be significant overhead for software RAID vs hardware?

 

[dswartz]

That would depend on the sw and hw implementations, which is getting way out of my area of expertise...

 

[gea]

On a crash during write a conventional raid or filesystem can become corrupted.
ZFS is immune due copy on write and when a problem happens ZFS can detect and repair the problem on the fly due checksums and redundancy, "Write hole" phenomenon in RAID5, RAID6, RAID1, and other arrays.

ZFS Copy on write offers snaps (versioning and Ransomware protection)

On problems with older filesystems, you need an offline fschk. Without checksums only some structural problems can be repaired, can last days of offline state. ZFS checks (scrubs) are online.

A Raid-5 is in danger of a complete array lost when a disk fails with an additional read error during rebuild. ZFS Z1 will only report a damaged file in same case. With more than say 5 disks per vdev use Z2, with say 10-16 disks per vdev use Z3. With more disks add vdevs.

Software raid calculation is a minor load problem for a modern CPU that is capable for realtime 3D. Only ZFS encryption adds a serious CPU load.

For ZFS use a simple HBA ex LSI 9300 or OEM, best with IT firmware, IR firmware is ok.

 

[Sean Ho]

HBA (or RAID card with IT mode) also gives you flexibility to upgrade the card in the future, or swap in a cheaper HBA as a temporary solution if/when it fails.

Not sure if the 9580 has a true IT mode, though.

 

[ari2asem]

if you have the money and know where to buy this, then look at

GRAID SupremeRAID SR-1000 Review

GRAID is an entirely novel architecture. You see, GRAID SupremeRAID doesn't use a legacy RAID card; it uses a GPU to get the job done.

www.storagereview.com

for me ZFS has one big disadvantage:
you can't expand your pool/array later with new disks, like hardware raid-5 can.
this is the only reason why i don't use ZFS.

beside this disadvantage, is ZFS great

 

[gea]

if you have the money and know where to buy this, then look at

GRAID SupremeRAID SR-1000 Review

GRAID is an entirely novel architecture. You see, GRAID SupremeRAID doesn't use a legacy RAID card; it uses a GPU to get the job done.

www.storagereview.com

for me ZFS has one big disadvantage:
you can't expand your pool/array later with new disks, like hardware raid-5 can.
this is the only reason why i don't use ZFS.

beside this disadvantage, is ZFS great

You can replace all disks in a ZFS vdev to increase capacity or you can add more vdevs similar ro raid-5 -> raid 50
You cannot extend a single vdev ex a 6 disk Z2 to a 7 disk Z2 (there is work to allow this).

The above restriction is more relevant at home for thos wanting to go from 6 disks to 7 disks not for larger setups where you add vdev ex go from 1 x 6 disks z2 to n x 6 disks z2.

 

[Un Nameless]

On a crash during write a conventional raid or filesystem can become corrupted.
ZFS is immune due copy on write and when a problem happens ZFS can detect and repair the problem on the fly due checksums and redundancy, "Write hole" phenomenon in RAID5, RAID6, RAID1, and other arrays.

ZFS Copy on write offers snaps (versioning and Ransomware protection)

On problems with older filesystems, you need an offline fschk. Without checksums only some structural problems can be repaired, can last days of offline state. ZFS checks (scrubs) are online.

A Raid-5 is in danger of a complete array lost when a disk fails with an additional read error during rebuild. ZFS Z1 will only report a damaged file in same case. With more than say 5 disks per vdev use Z2, with say 10-16 disks per vdev use Z3. With more disks add vdevs.

Software raid calculation is a minor load problem for a modern CPU that is capable for realtime 3D. Only ZFS encryption adds a serious CPU load.

For ZFS use a simple HBA ex LSI 9300 or OEM, best with IT firmware, IR firmware is ok.

Thanks for the elaborate input! I was leaning towards ZFS not only for the goodies you mentioned but also more control from software!

I guess though that money was wasted on such an expensive LSI RAID controller, that, I assume, once set as JBOD and the rest done at software, becomes just a pass through device.

 

[Un Nameless]

if you have the money and know where to buy this, then look at

GRAID SupremeRAID SR-1000 Review

GRAID is an entirely novel architecture. You see, GRAID SupremeRAID doesn't use a legacy RAID card; it uses a GPU to get the job done.

www.storagereview.com

for me ZFS has one big disadvantage:
you can't expand your pool/array later with new disks, like hardware raid-5 can.
this is the only reason why i don't use ZFS.

beside this disadvantage, is ZFS great

Isn't this more of a solution for NVME RAIDs or solid state instead of traditional spinners? I can't see what advantage I'd get as long as I still use 100 Seagate Exos X18 7200 rpm drives.