So I've got a new box, I dropped in 12x18TB, switched off compression, and was attempting to benchmark a 4+2 RaidZ2 setup vs 6 mirrored vdevs.
The results I got back were not as expected though. I ran dd and bonnie++ and then did some read/write testings from my client.
RaidZ2 seemed to perform as I expected.
Similar results on Bonnie++
But then Mirrored vDevs, which expected to shine in a high read situation, didn't...
Read speeds were cut in half.
Over 10GbE to my edit bay, R/W tests were typically around 600 Mb/s no matter what.
zpool iostat results for mirrored vdev during bonnie++
I'm fairly new to this, I was curious if anyone had any tips on better bench marking? if this performance is expected? And what I may have done wrong with mirrored vdevs, or is that just bad testing?
Thanks in advance.
The results I got back were not as expected though. I ran dd and bonnie++ and then did some read/write testings from my client.
RaidZ2 seemed to perform as I expected.
Code:
root@anton[~]# dd if=/dev/zero of=/mnt/tank/Share1/tmp.dat bs=2048k count=50k
51200+0 records in
51200+0 records out
107374182400 bytes transferred in 153.184751 secs (700945632 bytes/sec)
root@anton[~]# dd if=/mnt/tank/Share1/tmp.dat of=/dev/null bs=2048k count=50k
51200+0 records in
51200+0 records out
107374182400 bytes transferred in 122.100346 secs (879392941 bytes/sec)
But then Mirrored vDevs, which expected to shine in a high read situation, didn't...
Code:
root@anton[~]# dd if=/dev/zero of=/mnt/tank/Share1/tmp.dat bs=2048k count=50k
51200+0 records in
51200+0 records out
107374182400 bytes transferred in 107.400316 secs (999756671 bytes/sec)
root@anton[~]# dd if=/mnt/tank/Share1/tmp.dat of=/dev/null bs=2048k count=50k
51200+0 records in
51200+0 records out
107374182400 bytes transferred in 251.934330 secs (426199091 bytes/sec)
Over 10GbE to my edit bay, R/W tests were typically around 600 Mb/s no matter what.
zpool iostat results for mirrored vdev during bonnie++
Code:
---------------------------------------------- ----- ----- ----- ----- ----- -----
boot-pool 2.32G 474G 0 0 0 0
mirror 2.32G 474G 0 0 0 0
ada0p2 - - 0 0 0 0
ada1p2 - - 0 0 0 0
---------------------------------------------- ----- ----- ----- ----- ----- -----
tank 330G 97.8T 0 1.10K 3.19K 980M
mirror 55.3G 16.3T 0 190 408 164M
gptid/9605c4ce-ad76-11eb-ad0b-0cc47a6ea7ec - - 0 96 0 81.9M
gptid/981b5f6b-ad76-11eb-ad0b-0cc47a6ea7ec - - 0 93 408 81.9M
mirror 54.4G 16.3T 0 182 408 163M
gptid/9743487f-ad76-11eb-ad0b-0cc47a6ea7ec - - 0 92 408 81.4M
gptid/98cf3dfb-ad76-11eb-ad0b-0cc47a6ea7ec - - 0 90 0 81.4M
mirror 55.4G 16.3T 0 184 408 166M
gptid/9837e6f4-ad76-11eb-ad0b-0cc47a6ea7ec - - 0 93 408 82.8M
gptid/99438322-ad76-11eb-ad0b-0cc47a6ea7ec - - 0 91 0 82.8M
mirror 55.3G 16.3T 0 184 1.20K 164M
gptid/99dfa9ee-ad76-11eb-ad0b-0cc47a6ea7ec - - 0 95 816 81.8M
gptid/9a03a75d-ad76-11eb-ad0b-0cc47a6ea7ec - - 0 89 408 81.8M
mirror 54.9G 16.3T 0 192 408 160M
gptid/99cc02d0-ad76-11eb-ad0b-0cc47a6ea7ec - - 0 97 408 80.1M
gptid/99aeeb5b-ad76-11eb-ad0b-0cc47a6ea7ec - - 0 95 0 80.1M
mirror 54.9G 16.3T 0 187 408 164M
gptid/9a206a5b-ad76-11eb-ad0b-0cc47a6ea7ec - - 0 93 408 81.9M
gptid/9a4a60f1-ad76-11eb-ad0b-0cc47a6ea7ec - - 0 94 0 81.9M
---------------------------------------------- ----- ----- ----- ----- ----- -----
capacity operations bandwidth
pool alloc free read write read write
---------------------------------------------- ----- ----- ----- ----- ----- -----
boot-pool 2.32G 474G 0 0 0 0
mirror 2.32G 474G 0 0 0 0
ada0p2 - - 0 0 0 0
ada1p2 - - 0 0 0 0
---------------------------------------------- ----- ----- ----- ----- ----- -----
tank 357G 97.8T 3.69K 49 476M 5.03M
mirror 59.9G 16.3T 630 1 79.3M 814K
gptid/9605c4ce-ad76-11eb-ad0b-0cc47a6ea7ec - - 343 0 43.1M 407K
gptid/981b5f6b-ad76-11eb-ad0b-0cc47a6ea7ec - - 286 0 36.2M 407K
mirror 58.9G 16.3T 630 4 79.5M 837K
gptid/9743487f-ad76-11eb-ad0b-0cc47a6ea7ec - - 256 2 32.4M 419K
gptid/98cf3dfb-ad76-11eb-ad0b-0cc47a6ea7ec - - 374 2 47.0M 419K
mirror 59.7G 16.3T 629 8 79.3M 849K
gptid/9837e6f4-ad76-11eb-ad0b-0cc47a6ea7ec - - 271 3 34.1M 425K
gptid/99438322-ad76-11eb-ad0b-0cc47a6ea7ec - - 358 4 45.2M 425K
mirror 59.7G 16.3T 628 14 79.0M 903K
gptid/99dfa9ee-ad76-11eb-ad0b-0cc47a6ea7ec - - 263 7 32.7M 452K
gptid/9a03a75d-ad76-11eb-ad0b-0cc47a6ea7ec - - 365 7 46.3M 452K
mirror 59.3G 16.3T 626 11 78.6M 893K
gptid/99cc02d0-ad76-11eb-ad0b-0cc47a6ea7ec - - 255 5 32.1M 446K
gptid/99aeeb5b-ad76-11eb-ad0b-0cc47a6ea7ec - - 370 6 46.5M 446K
mirror 59.3G 16.3T 635 8 80.0M 855K
gptid/9a206a5b-ad76-11eb-ad0b-0cc47a6ea7ec - - 310 4 39.1M 427K
gptid/9a4a60f1-ad76-11eb-ad0b-0cc47a6ea7ec - - 325 4 40.9M 427K
---------------------------------------------- ----- ----- ----- ----- ----- -----
Thanks in advance.