Note: see my post farther down, I no longer think this is accurate.
As with many other folks here I've generally had the understanding that best practice when setting up ZFS vdevs was to keep the number of data disks as a power of 2. It's a pretty constant recommendation and people talk about slight performance boosts and better space efficiency. I saw a great post over at Hard Forum today describing the math so I decided to throw the numbers into a spread sheet for various configs. I was pretty shocked at the results so I figured I'd post here to see if I'm off base and if not create an awareness of how important the power of 2 rule is when using 4k blocks (ashift 12).
When talking disk count I'll refer to data disk count, so 8 data disks would include a 9 disk vdev in raidz1, 10 disk vdev in raidz2, or 11 disk vdev in raidz3.
Under 8 data disks there is a loss of space efficiency when not aligned with power-of-two but you at least get more usable space as you add disks. Above 8 data disks things get really interesting. A vdev with 8 data disks has exactly the same usable space as one with 9 or 10 data disks. The same with 11-15 and again with 16-31.
So from a space perspective there is no difference between 10 disk raidz2 and a 12 disk raidz2, both will yield the same free space once created. Or if you wanted to go with huge vdevs a 32 disk raidz1 would have the exact same free space as a 17 disks raidz1. So there really should never be a reason to create a 4k vdev with 8,9,12-15, or 17-31 data disks unless you're going for sequential speed.
I haven't tested this out for real yet, but that's how the math worked out. I'm curious if this is inline with what others have seen.
As with many other folks here I've generally had the understanding that best practice when setting up ZFS vdevs was to keep the number of data disks as a power of 2. It's a pretty constant recommendation and people talk about slight performance boosts and better space efficiency. I saw a great post over at Hard Forum today describing the math so I decided to throw the numbers into a spread sheet for various configs. I was pretty shocked at the results so I figured I'd post here to see if I'm off base and if not create an awareness of how important the power of 2 rule is when using 4k blocks (ashift 12).
When talking disk count I'll refer to data disk count, so 8 data disks would include a 9 disk vdev in raidz1, 10 disk vdev in raidz2, or 11 disk vdev in raidz3.
Under 8 data disks there is a loss of space efficiency when not aligned with power-of-two but you at least get more usable space as you add disks. Above 8 data disks things get really interesting. A vdev with 8 data disks has exactly the same usable space as one with 9 or 10 data disks. The same with 11-15 and again with 16-31.
So from a space perspective there is no difference between 10 disk raidz2 and a 12 disk raidz2, both will yield the same free space once created. Or if you wanted to go with huge vdevs a 32 disk raidz1 would have the exact same free space as a 17 disks raidz1. So there really should never be a reason to create a 4k vdev with 8,9,12-15, or 17-31 data disks unless you're going for sequential speed.
I haven't tested this out for real yet, but that's how the math worked out. I'm curious if this is inline with what others have seen.
Last edited: