Why do you think that a hardware raid can give a similar performance than a modern 4+ core Xeon with highspeed RAM in the higher gigabyte area. The extra CPU load of a modern ZFS system due data XOR is near zero. Hardware raid is mainly a relict from those ages where CPU performance was limited and server RAM was a few hundred Megabytes and software raid options were slow and not really a professional option.
But the main reason for Sun to develop a complete new way of thinking storage was security and scalability of storage up to the Petabyte/Exabyte ranges. Price was never an item because ZFS was targeted to datacenter use to compete against solutions like NetApp with a very similar feature set..
There are the following problems, ZFS is adressing
- Silent data errors and corruptions due to statistical longterm storage modifications, disk, backplane, cable, controller and driver problems .
The answer of ZFS is end to end checksumming all data and metadata with autorepair on read or scrub.
While you can use a hardware raid for ZFS, this repair option is only available if you use ZFS not only as a filesystem but use its software raid-features.
Inconsistent filesystems. A power outage during a write can damage a filesystem like ntfs or ext4.
The answer is CopyOnWrite where every datablock is written newly (no inline update of data) After the whole datablock is written, pointers are updated so a write action is done successfully or not.
ZFS adds an online scrubbing option, You do not need to offline to check data integrity as this can last days or a week on large arrays. An addition of Copy on Write are Snaps. This are frozen states of former data (or bootable system environments). Done without delay or limited in number of snaps.
Scalability of capacity and performance.
While ZFS should be slower on a single disk than for ex ext4 due the extra security, ZFS scales very well over number of datadisks regarding sequential performance and over number of raid-arrays (ZFS name them vdevs) regarding iops. There is no practical limit for capacity or number of files as ZFS is a 128bit filesystem.
Storage virtualisation
All disks and raid arrays/vdevs are building a pool where you can create filesystems (like partitions) that can grow dynamically up to pool size. You limit or ensure space with quotas and reservations.
ZFS and its ideas is storage for next decade. Even Microsoft is adding ZFS alike features in ReFS. The same is the case with btrfs.
Lets talk about problems of hardware raid.
Main problem beside limited performance and cache is the write hole problem. On a power outage during a write, mirrored disks may be updated different or raid 5/6 stripes may be inconsistent. You can or better must ad a BBU to limit this problem but you will never achieve a similar quality like when you completely avoid the problem with CopyOnWrite.
You are fixed to a special controller. If you do not have a compatible spare controller, you are lost when it fails.
Limited scalability to the real big data.
While you can add disks with espanders, a hardware raif does not really scale to Petabytes.
Additional advantages of ZFS
Superiour ramcache (ARC) that use all free RAM. You can extend with an SSD when needed (L2ARC)
Superiour logging options for secury sync write on external highspeed disks
Easy way to transfer or update filesystems locally or remote
Easy management as ZFS is filesystem, volume management, raid management and share management
Available on Solaris or its free forks like OmniOS where it comes from, on BSD, OSX and Linux
Only a few aspects.
Amazing information, and explanation. I greatly appreciate the time you took to write all that, and it's actually making me think more about using ZFS for the HD RAID6 array
with triple parity, because, well I could do it if I went ZFS. I also have additional RAM, and could run 152gb instead of 128gb for some extra cache. You're opening my eyes more.
Is there any performance increase/decrease when using ZFS and different controllers? Obviously SAS1, SAS2, SAS3 controllers/expanders/backplanes will provide limited throughput based on spec, but is ZFS itself limited by anything?
What about SSD arrays how does ZFS handle that? Still better off using my 12GB/s RAID card for 12GB/s SSD I imagine, if that's the case I assume I can pass through the entire thing to OmniOS and make the iSCSI targets from there?
What about adding additional drives to the array pools?
What about iSCSI targets is that all done in OmniOS w/ZFS as filesystem or is it an add-on?
Do you run your media center/server stuff in OmniOS too or in another VM with another OS?
Has anyone made the onboard LSI 2108/2208 based controller 'pass through' instead of "RAID MODE" for ZFS? I have 7 PCIE slots but plan to cram at-least 4 things in there and as mentioned, would like to avoid the extra HBA/RAID card if not needed.
With the parity drives, do you guys run the same as I would with hardware RAID or is there a benefit using a faster parity drive or parity drive pool?
Just to be 100% clear you are suggestion OmniOS ZFS (File System) + Napp IT all in a VM, correct?
I need to decide, and get this new RAID6 array live, data migrated and going today/tomorrow.