Hey all
I'm in the process of building a new pool on my home Solaris 11.3 server. The final pool will be 16 x 2TB drives, configured as 2 x 8-drive RAIDZ2 for a total of 12 data drives and 4 parity.
One complication is that I have a mixture of 4k and 512 sector drives, and so I've been experimented with different ashift settings and benchmarking and monitoring of various different configs.
When doing so, I noticed something I thought was really weird: when I create a 12 drive pool as 2 x 6 drive RAIDZ2, then do a bonnie++ rewrite test - or just copy data within the pool - I always got four drives (two per VDEV) with 0 reads.
I did further did tests using a single VDEV to simplify things, and here's a quick example. First, a 7 drive RAIDZ2 during bonnie's "Rewrite" test:
That all looks normal to me. Then, a 6 drive RAIDZ2 in the same test:
The difference is clear - with 6 drives, there's always two drives that are reading 0 or close to 0. I see the same during read-only tests: reads on only four disks, not 6.
At first this really confused me, and I thought it might be some kind of problem. But in my other thread about ashift values, @gea mentioned that there is a benefit with VDEVs containing a power-of-two number of data drives. Which is what I have here with my 6-drive RAIDZ2s with 4 data disks.
So, is that the difference? In which case - wow, this at first seems really significant? At first I thought it meant a big performance hit, because I am only using 4 spindles for reading instead of 6. But maybe it does this because it doesn't need to read from those spindles, thereby presumably increasing overall performance - or at least reducing wear on the drives?
But it's hard to compare performance, because if I compare performance of a 7-drive RAIDZ2 with a 6-drive, I have an extra spindle overall which makes it faster. I did do that bonnie test, and the 7 drive test is certainly faster for reading and re-reading because of the extra spindle.
Although if I work out the per-data-disk performance (overall read speed divided by number of data disks), it does seem like I get more performance per-disk from the 6 drive than the 7 drive - it makes better use of its 6 drives, just not so much better that it can out-perform having an extra drive.
So overall I don't know - it definitely doesn't seem like it's worth using fewer disks, because perf will definitely suffer with fewer spindles. But maybe if I had planned to use one-less than a power of two data drives, it should encourage me to add an extra disk, if possible, because I get more than the usual amount of extra perf/value out of it?
In my case I had planned to have 16 disks total in 2 x 8 drive RAIDZ2, so I'd have to add four more drives total and make 2 x 10 drive = 20 drives; not sure I want to spend that much extra. But it is good to think about!
Anyway, I'd be grateful if anyone could confirm that I'm understanding this correctly!
I'm in the process of building a new pool on my home Solaris 11.3 server. The final pool will be 16 x 2TB drives, configured as 2 x 8-drive RAIDZ2 for a total of 12 data drives and 4 parity.
One complication is that I have a mixture of 4k and 512 sector drives, and so I've been experimented with different ashift settings and benchmarking and monitoring of various different configs.
When doing so, I noticed something I thought was really weird: when I create a 12 drive pool as 2 x 6 drive RAIDZ2, then do a bonnie++ rewrite test - or just copy data within the pool - I always got four drives (two per VDEV) with 0 reads.
I did further did tests using a single VDEV to simplify things, and here's a quick example. First, a 7 drive RAIDZ2 during bonnie's "Rewrite" test:
Code:
extended device statistics
device r/s w/s kr/s kw/s wait actv svc_t %w %b
sd22 409.6 38.6 46271.7 34548.0 0.0 0.7 1.5 0 32
sd24 359.2 37.0 46738.0 34547.9 0.0 2.3 5.7 0 98
sd25 361.2 38.4 46531.6 34549.5 0.0 2.1 5.2 0 89
sd26 368.4 38.0 46377.2 34550.3 0.0 1.9 4.6 0 85
sd28 360.2 38.2 46188.4 34549.5 0.0 1.9 4.7 0 88
sd30 355.6 38.4 46422.0 34550.3 0.0 1.9 4.7 0 88
sd35 376.0 38.0 46241.2 34548.7 0.0 2.1 5.0 0 83
That all looks normal to me. Then, a 6 drive RAIDZ2 in the same test:
Code:
extended device statistics
device r/s w/s kr/s kw/s wait actv svc_t %w %b
sd22 109.8 95.4 52777.6 92636.0 0.0 1.3 6.1 0 63
sd24 0.0 62.0 0.0 59208.8 0.0 1.1 17.0 0 53
sd25 0.2 64.0 1.6 61256.8 0.0 1.1 16.6 0 53
sd26 70.6 62.4 52985.6 61436.0 0.0 1.9 14.6 0 94
sd28 64.8 68.8 54267.2 64917.6 0.0 2.0 14.7 0 97
sd35 88.2 65.4 53001.6 62812.0 0.0 1.7 11.1 0 84
The difference is clear - with 6 drives, there's always two drives that are reading 0 or close to 0. I see the same during read-only tests: reads on only four disks, not 6.
At first this really confused me, and I thought it might be some kind of problem. But in my other thread about ashift values, @gea mentioned that there is a benefit with VDEVs containing a power-of-two number of data drives. Which is what I have here with my 6-drive RAIDZ2s with 4 data disks.
So, is that the difference? In which case - wow, this at first seems really significant? At first I thought it meant a big performance hit, because I am only using 4 spindles for reading instead of 6. But maybe it does this because it doesn't need to read from those spindles, thereby presumably increasing overall performance - or at least reducing wear on the drives?
But it's hard to compare performance, because if I compare performance of a 7-drive RAIDZ2 with a 6-drive, I have an extra spindle overall which makes it faster. I did do that bonnie test, and the 7 drive test is certainly faster for reading and re-reading because of the extra spindle.
Although if I work out the per-data-disk performance (overall read speed divided by number of data disks), it does seem like I get more performance per-disk from the 6 drive than the 7 drive - it makes better use of its 6 drives, just not so much better that it can out-perform having an extra drive.
So overall I don't know - it definitely doesn't seem like it's worth using fewer disks, because perf will definitely suffer with fewer spindles. But maybe if I had planned to use one-less than a power of two data drives, it should encourage me to add an extra disk, if possible, because I get more than the usual amount of extra perf/value out of it?
In my case I had planned to have 16 disks total in 2 x 8 drive RAIDZ2, so I'd have to add four more drives total and make 2 x 10 drive = 20 drives; not sure I want to spend that much extra. But it is good to think about!
Anyway, I'd be grateful if anyone could confirm that I'm understanding this correctly!