RaidZ(n) with 8 drives (and a lot of RAM)

[voodooFX]

I'm new to ZFS and RaidZ configurations so I need a little help -_-

I have 8 hard drives and I'm wondering what is the best raidz level for balancing performance and safety?
According to (some) best practices, seems that 8 does not fit any of them; I should use (for example) 8+2 if I want a RaidZ2.. is this correct? And if I want to use just my 8 drives?

Another important thing seems to be the RAM amount.
My server has 96GB of ECC Reg (DDR3 1333) RAM, will this somehow give me better performance? And if yes, do I have to set something to have a benefit?

 

[cperalt1]

Some light reading on the subject. It comes down to performance or space. For performance go mirror for space raidz2 as 2 vdevs. So basically a striped mirror.
OpenSolaris ZFS: Mirroring vs. other RAID schemes
Oracle Solaris ZFS: A Closer Look at Vdevs and Performance

 

[Patrick]

I will let some of the ZFS gurus speak on that. With so much ram you could probably stand for a nice SSD for L2ARC. But ZFS is pretty good on just soaking up memory itself and that is a solid amount.

 

[neo]

My server has 96GB of ECC Reg (DDR3 1333) RAM, will this somehow give me better performance? And if yes, do I have to set something to have a benefit?

Deduplication in ZFS is a nice feature, and it requires a large amount of ram which you do have. You are able to enable it on a per vdev or dataset basis.

 

[Deci]

You can do an 8 disk z2, 2x 4 disk z2 is silly, just go for a heap of mirrors instead as it's smart enough to read like a raid0 on mirrors. But it all comes down to how much performance you want/need. Lots of ram is better than less ram and a fast l2 disk as it's still slower than ram, if the application calls for it a fast slog makes the array less random write intensive as the writes are random into the slog but sequential out of the slog to disk.

 

[T_Minus]

@Deci so are you suggesting Raid Z2 or a "heap of mirrors"?

Would the "heap of mirrors" > than Raid z2 performance?

Would it matter that much between the mirors or z2 with a lot of RAM, and a fast SLOG (Zeus, Fusion, NVME)? Or is it less noticeable, and you should go for the 'safest' (Z2 or pool of mirrors) if you have a high-performing slog or lots of RAM?

 

[Deci]

it depends on what the pool needs to do and what level of drive loss to parity/protection you are willing to accept.

4x mirrors would give the best random IO performance, but the write performance of 4 disks
1x 8 drive z2 would give not so great random io performance (more z1/z2/z3 vdevs gives better random io, but at the 4 disk z2 level, you might as well mirror as its still 50% loss to parity and you have the overhead of parity calculations in z2 setups that you dont have in mirrrors) but have fairly reasonable sequential.

if you have the ram/l2arc capacity to soak up most of the common reads then the single z2 is less of an issue, if you need sync writes and have a fast slog to soak up the writes that also becomes less of an issue, but you limit yourself when you add in l2arc and slog drives, writes that force sync like iscsi are limited to what the slog can perform sync writes at, if the array is faster than your l2arc and can satisfy the IO, your l2arc slows the pool down speed wise with the trade off being decreased seek times for cached data.

a big part of it is building the pool to suit what you are doing with it, if you need lots of random io, more smaller vdevs, if you need more sequential, bigger vdevs in z2/z3, or for a mix of both, more vdevs and more disks per vdev.

there is no "one correct pool setup" for everyone and every purpose, while i might sacrifice 12% on one pool to get the performance i want, i might sacrifice 50% on another because its more critical that it stays up or the IO benefits are worth it, the more disks you have the less you need mirrors and the more sense it makes to spend that money on more ram and/or faster L2 and slog devices instead. as a hypothetical example, 50% loss on 30 disks, 4tb @ $200 each being $6000 when you might be better off with 30 smaller cheaper 3tb disks at say $150 each and 33% loss being $4500, both give 60tb raw. option two leaves $1500 to spend on ram/l2/slog that option 1 didnt have to make up for the change from 15x mirrors to 5x 6 disk z2 vdevs.

 

[voodooFX]

Since I will have 4x1G links for iSCSI and single 2x1G links for NFS/SMB I will not be able to benefit of more than 400-450 MB/s.
With that in mind, would z2 (with all 8 drives) be a good choice?
Will be possible something like Raid 50? I mean 2xZ1 with 4 drives each in a single volume?

Also I have not understand the "go for mirror" and "heap of mirrors" suggestion
You mean mirror of what? all the drives? So the available space will be ...? And write performance?

Note: the drives are 2TB each and I need at least 8TB of usable space.

EDIT: 90% of the time this pool will be doing seq. r/w of big files. I will have another pool of two SSDs in mirror for the virtual machines (iSCSI -> esxi)

 

[Deci]

then go for a pool with a single 8 disk z2 vdev in it, that gives you 12tb raw. i wouldnt suggest any z1 setups if you value the data.

some SSD respond to sync writes (iscsi for vm usage) better than others, so if you can use 2x intel s3700 drives for the VM mirror there is no point even thinking about getting a SLOG, and L2 is wasted on an all SSD array.

 

[voodooFX]

I have 2x 730 480G

 

[Entz]

Also I have not understand the "go for mirror" and "heap of mirrors" suggestion
You mean mirror of what? all the drives? So the available space will be ...? And write performance?

Note: the drives are 2TB each and I need at least 8TB of usable space.

Mirrors mean you add the drives in sets of 2 as a mirrored vdev.

It gives you # drives / 2 * size free space (though one should never max out a pool). So in your case it would give you 8/2 * 2TB = 8TB of space.

If you need >= 8TB you will need to go with one large RaidZ2 as Deci mentioned above or get more disks.

 

[voodooFX]

Interesting, so I will have four mirrors and all will be seen as a single volume?

and what seq. r/w performance should I expect with this config?

 

[Entz]

Yes a single volume. in ZFS your volume is the pool and what makes up the pool is up to you. Your pool is made up of one or more vdev and each of those are striped together. In this case it would be 4 mirrors striped together. This is similar to RAID10.

With mirrors your write IOP/Bandwidth per vdev is equal to that of the slowest single disk and your read IOP/Bandwidth will be the aggregate of both disks. Since the vdevs are striped, assuming a large enough block of data that means your writes will be equivalent to 4x a single disk and reads 8x (It doesn't quite scale linearly but you get the idea).

 

[voodooFX]

So in a perfect world 500 MB/s | 1000MB/s and in the real one (I suppose) at least 10-15% less?

..and where is RAM cashing help in all this? I will have more than 5GB for each drive

 

[MatrixMJK]

One thing to keep in mind with mirrors, while they do provide redundancy, a two drive failure could cause a loss of the entire pool if both failed drives are in the same vdev (ie. in the screenshot above if both drives in mirror-1 failed the entire pool is lost). raidz2 can withstand any two drive failures. With the mirrored setup you are talking about (4 two disk mirrors) you could conceivably survive four disk failures if each was on a different vdev, but those are not good odds.

I do think mirrors are safe, but wanted to point that out. Good to have a cold spare for quick swaps.

 

[Entz]

Yeah that sounds about right. Depends a bit on the disks and where you are. On that array above I get 377MB/s writes and 486 MB/s read ( vs 450/900 max, 350/360 average )

The real world isnt linear Ram is going to help absorb random writes and help cache content to make random reading faster. 96GB is likely a bit overkill for what you have. Might be able to use dedup actually. Could put a large SSD in there and cache the heck out of stuff.

 

[Entz]

@MatrixMJK that is a good point. There is a case for 2x 4 RaidZ for the reason you mention. You get a bit better redundancy at the expense of performance and rebuild time. Everything has a trade off

 

[whitey]

Imma put my neck out there and say that I have used raidz's and meh/blah...it's been stripped mirror's (a-la raid 10 in ZFS world). Spinners will do ok in this config in a hybrid setup (ssd's for l2arc/zil) but nothing compares to ent class ssd's....sad (read sarcasm here) to say my days of spinners for anything but bulk storage are over.

The thing I am really debating as of late is replacing my hybrid stripped mirror pools w/ pure ssd pools, do I then fall back to small 3 disk raidz's and call it good? I just found that IOPS were terrible w/ spinners and raidz...like horrific/unusable for my workload anyways...and I know the myth may not be entirely true that you are limited to the throughput of a single disk in a ZFS raidz but it sure as hell felt like it. I forget if I ever tried adding l2arc/zil devices to raidz pools or if I was that 'un-impressed' that I just went to stripe mirrors and was done w/ it/never looked back. Guess I could look at stripped raidz's right (raid-50 sorta???) I know this CAN get cumbersome/don't profess to be an expert on the topic. Just knowledgeable/dangerous enough right...ok ok a little better than that :-D

You get my jist right...w/ spinners/magnetics it was easier/more justifiable to throw mirrors at it stripe them and eat the 1/2 space/cost/investment at the price of sleeping well knowing my data was safe and performant. With pure ssd pools I'd have a hard time justifying coin for mirrors and feel maybe the shift back to raidz (as I wont see the detrimental perf drop, even IF limited to 1 ssd performance across vdev's), the capacity would be there and 400-500MB read/write (5K-75K IOPS depending on ssd disk selection workload IO patterns for R/W) would be more than enough to drive my workloads.

Deci/Gea can probably break this down for me WAY better and clarify my jibberish :-D

 

[Entz]

Makes sense to me SSD only pools do change things. Mine are running in sets of 3 disk RaidZ-1* , for the same reason - cost-vs-space tradeoff and mainly because I don't see the reason to go further. vs Spinners rebuild times are significantly faster, IOPs are 50x+ higher. Your vdevs are really not that limiting anymore. Heck you end up limited by 10GBe real fast. But that is me.

* really should just go with a large RaidZ-2. Something to test one of these days.

 

[whitey]

Makes sense to me SSD only pools do change things. Mine are running in sets of 3 disk RaidZ-1* , for the same reason - cost-vs-space tradeoff and mainly because I don't see the reason to go further. vs Spinners rebuild times are significantly faster, IOPs are 50x+ higher. Your vdevs are really not that limiting anymore. Heck you end up limited by 10GBe real fast. But that is me.

* really should just go with a large RaidZ-2. Something to test one of these days.

Yeah you probably got ballz to the wall 12Gbps SAS ssd drives right...and LOTS of them, I'd imagine 10GbE makes you real sad if so...gotta go infiniband in that world or 40G ethernet right...those gnodal's looked nice but loud as hell, who else had 10G/40G combo switches not to derail but I was drooling over one other brand...now I gotta go hunt to refresh my memory.

What's your take on stripped raidz's pure ssd pools (kinda like raid 50 on ZFS as I don't think there is an official name for this)? Any benefit to that approach keeping in mind an 'all ssd ZFS based approach' or the hell w/ it and just go one large/larger raidz maybe raid2 depending on how many ssd devices you get in that pool/vdev right? Thoughts?