Benchmarkin RaidZ2 vs vDevs

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

wdp

New Member
May 1, 2021
25
2
3
So I've got a new box, I dropped in 12x18TB, switched off compression, and was attempting to benchmark a 4+2 RaidZ2 setup vs 6 mirrored vdevs.

The results I got back were not as expected though. I ran dd and bonnie++ and then did some read/write testings from my client.

RaidZ2 seemed to perform as I expected.

Code:
root@anton[~]# dd if=/dev/zero of=/mnt/tank/Share1/tmp.dat bs=2048k count=50k
51200+0 records in
51200+0 records out
107374182400 bytes transferred in 153.184751 secs (700945632 bytes/sec)

root@anton[~]# dd if=/mnt/tank/Share1/tmp.dat of=/dev/null bs=2048k count=50k
51200+0 records in
51200+0 records out
107374182400 bytes transferred in 122.100346 secs (879392941 bytes/sec)
Similar results on Bonnie++

But then Mirrored vDevs, which expected to shine in a high read situation, didn't...

Code:
root@anton[~]# dd if=/dev/zero of=/mnt/tank/Share1/tmp.dat bs=2048k count=50k
51200+0 records in
51200+0 records out
107374182400 bytes transferred in 107.400316 secs (999756671 bytes/sec)

root@anton[~]# dd if=/mnt/tank/Share1/tmp.dat of=/dev/null bs=2048k count=50k
51200+0 records in
51200+0 records out
107374182400 bytes transferred in 251.934330 secs (426199091 bytes/sec)
Read speeds were cut in half.

Over 10GbE to my edit bay, R/W tests were typically around 600 Mb/s no matter what.

zpool iostat results for mirrored vdev during bonnie++

Code:
----------------------------------------------  -----  -----  -----  -----  -----  -----
boot-pool                                       2.32G   474G      0      0      0      0
  mirror                                        2.32G   474G      0      0      0      0
    ada0p2                                          -      -      0      0      0      0
    ada1p2                                          -      -      0      0      0      0
----------------------------------------------  -----  -----  -----  -----  -----  -----
tank                                             330G  97.8T      0  1.10K  3.19K   980M
  mirror                                        55.3G  16.3T      0    190    408   164M
    gptid/9605c4ce-ad76-11eb-ad0b-0cc47a6ea7ec      -      -      0     96      0  81.9M
    gptid/981b5f6b-ad76-11eb-ad0b-0cc47a6ea7ec      -      -      0     93    408  81.9M
  mirror                                        54.4G  16.3T      0    182    408   163M
    gptid/9743487f-ad76-11eb-ad0b-0cc47a6ea7ec      -      -      0     92    408  81.4M
    gptid/98cf3dfb-ad76-11eb-ad0b-0cc47a6ea7ec      -      -      0     90      0  81.4M
  mirror                                        55.4G  16.3T      0    184    408   166M
    gptid/9837e6f4-ad76-11eb-ad0b-0cc47a6ea7ec      -      -      0     93    408  82.8M
    gptid/99438322-ad76-11eb-ad0b-0cc47a6ea7ec      -      -      0     91      0  82.8M
  mirror                                        55.3G  16.3T      0    184  1.20K   164M
    gptid/99dfa9ee-ad76-11eb-ad0b-0cc47a6ea7ec      -      -      0     95    816  81.8M
    gptid/9a03a75d-ad76-11eb-ad0b-0cc47a6ea7ec      -      -      0     89    408  81.8M
  mirror                                        54.9G  16.3T      0    192    408   160M
    gptid/99cc02d0-ad76-11eb-ad0b-0cc47a6ea7ec      -      -      0     97    408  80.1M
    gptid/99aeeb5b-ad76-11eb-ad0b-0cc47a6ea7ec      -      -      0     95      0  80.1M
  mirror                                        54.9G  16.3T      0    187    408   164M
    gptid/9a206a5b-ad76-11eb-ad0b-0cc47a6ea7ec      -      -      0     93    408  81.9M
    gptid/9a4a60f1-ad76-11eb-ad0b-0cc47a6ea7ec      -      -      0     94      0  81.9M
----------------------------------------------  -----  -----  -----  -----  -----  -----

                                                  capacity     operations     bandwidth
pool                                            alloc   free   read  write   read  write
----------------------------------------------  -----  -----  -----  -----  -----  -----
boot-pool                                       2.32G   474G      0      0      0      0
  mirror                                        2.32G   474G      0      0      0      0
    ada0p2                                          -      -      0      0      0      0
    ada1p2                                          -      -      0      0      0      0
----------------------------------------------  -----  -----  -----  -----  -----  -----
tank                                             357G  97.8T  3.69K     49   476M  5.03M
  mirror                                        59.9G  16.3T    630      1  79.3M   814K
    gptid/9605c4ce-ad76-11eb-ad0b-0cc47a6ea7ec      -      -    343      0  43.1M   407K
    gptid/981b5f6b-ad76-11eb-ad0b-0cc47a6ea7ec      -      -    286      0  36.2M   407K
  mirror                                        58.9G  16.3T    630      4  79.5M   837K
    gptid/9743487f-ad76-11eb-ad0b-0cc47a6ea7ec      -      -    256      2  32.4M   419K
    gptid/98cf3dfb-ad76-11eb-ad0b-0cc47a6ea7ec      -      -    374      2  47.0M   419K
  mirror                                        59.7G  16.3T    629      8  79.3M   849K
    gptid/9837e6f4-ad76-11eb-ad0b-0cc47a6ea7ec      -      -    271      3  34.1M   425K
    gptid/99438322-ad76-11eb-ad0b-0cc47a6ea7ec      -      -    358      4  45.2M   425K
  mirror                                        59.7G  16.3T    628     14  79.0M   903K
    gptid/99dfa9ee-ad76-11eb-ad0b-0cc47a6ea7ec      -      -    263      7  32.7M   452K
    gptid/9a03a75d-ad76-11eb-ad0b-0cc47a6ea7ec      -      -    365      7  46.3M   452K
  mirror                                        59.3G  16.3T    626     11  78.6M   893K
    gptid/99cc02d0-ad76-11eb-ad0b-0cc47a6ea7ec      -      -    255      5  32.1M   446K
    gptid/99aeeb5b-ad76-11eb-ad0b-0cc47a6ea7ec      -      -    370      6  46.5M   446K
  mirror                                        59.3G  16.3T    635      8  80.0M   855K
    gptid/9a206a5b-ad76-11eb-ad0b-0cc47a6ea7ec      -      -    310      4  39.1M   427K
    gptid/9a4a60f1-ad76-11eb-ad0b-0cc47a6ea7ec      -      -    325      4  40.9M   427K
----------------------------------------------  -----  -----  -----  -----  -----  -----
I'm fairly new to this, I was curious if anyone had any tips on better bench marking? if this performance is expected? And what I may have done wrong with mirrored vdevs, or is that just bad testing?

Thanks in advance.
 
  • Like
Reactions: Patrick

Rand__

Well-Known Member
Mar 6, 2014
6,626
1,767
113
1. dd is not a good test tool, use fio instead
2. mirrors dont scale well in all situations, basically you need to run multiple threads (parallel read/write operations) to make use of them (and dd is single threaded)
 

wdp

New Member
May 1, 2021
25
2
3
Ha, I was actually in the process of using fio when you posted your response. 4+2 raidz2 running fio, I was just watching zpool iostat

Code:
----------  -----  -----  -----  -----  -----  -----
boot-pool   2.32G   474G      1      0  2.25K      0
tank         535G   196T  16.8K     72   563M  2.97M
----------  -----  -----  -----  -----  -----  -----
boot-pool   2.32G   474G      0      0      0      0
tank         535G   196T  18.5K     31   614M   158K
----------  -----  -----  -----  -----  -----  -----
boot-pool   2.32G   474G      0      0    511      0
tank         535G   196T  17.3K     30   578M   158K
----------  -----  -----  -----  -----  -----  -----
boot-pool   2.32G   474G      0      0      0      0
tank         535G   196T  18.2K     27   600M   212K
----------  -----  -----  -----  -----  -----  -----
Code:
----------  -----  -----  -----  -----  -----  -----
boot-pool   2.32G   474G      0      0      0      0
tank         160G   196T      0  2.43K    807   727M
----------  -----  -----  -----  -----  -----  -----
boot-pool   2.32G   474G      0      0      0      0
tank         160G   196T      0  2.53K  1.99K   732M
----------  -----  -----  -----  -----  -----  -----
boot-pool   2.32G   474G      0      0      0      0
tank         160G   196T      0  2.44K  1.99K   733M
----------  -----  -----  -----  -----  -----  -----
boot-pool   2.32G   474G      0      0      0      0
tank         160G   196T      1  2.43K  38.0K   731M
----------  -----  -----  -----  -----  -----  -----
But having just read about fio for the first time an hour ago...any recommendations on a tests to run?
 
  • Like
Reactions: Patrick

i386

Well-Known Member
Mar 18, 2016
4,220
1,540
113
34
Germany
What are you trying to achieve/do with the storage?

Mirrors are for performance, but I wouldn't use hdds for that
Parity ist for storage efficiency, "great" for hdds
 

wdp

New Member
May 1, 2021
25
2
3
What are you trying to achieve/do with the storage?

Mirrors are for performance, but I wouldn't use hdds for that
Parity ist for storage efficiency, "great" for hdds
I do video editing/media production, some photography work. But most of it is timeline scrubbing and media storage. Write speed isn't critically important to me, but I don't want anything abysmally slow. I come home a lot of days with 4-6TB of media to ingest.

Sadly SSD's just don't fit the bill for me, I usually need a mixture of performance to capacity. Mirrored vDevs on a 12 bay already cuts it down to about 90TB usable.

The goal right now is to decided between Mirrored vDevs if they actually perform for multiple edit bays well enough, of RaidZ2 where I get enough performance and a little more capacity in the hopper (about 120TB).
 

wdp

New Member
May 1, 2021
25
2
3
Still trying to formulate how to properly test using fio. Not sure I know what I'm doing at all.

1620289690734.png

1620289904735.png
 

T_Minus

Build. Break. Fix. Repeat
Feb 15, 2015
7,625
2,043
113
Why did you switch off compression? There are tests from various sources that show compression on is yielding > performance.
 

wdp

New Member
May 1, 2021
25
2
3
Why did you switch off compression? There are tests from various sources that show compression on is yielding > performance.
During tests where I was just using lots of zeros, like dd, compression gives you false results.
 

Stephan

Well-Known Member
Apr 21, 2017
920
698
93
Germany
For this many disks RAIDZ2 in any case is the right choice. There is a new mode called "dRAID" coming with ZFS 2.1 but that requires some more ZFS experience imho. Much faster rebuilds. But also some drawbacks. In production always enable LZ4. For testing no compression is alright.

Read speed with 12x18 TB RAIDZ2 should be ~ 1500 MByte/s. Try a network benchmark like iperf2 or iperf3 first, preferably using a recent Linux from USB stick if client machine usually boots Windows. See if that is also stuck at 600 MByte/s. You might have a network problem. Also make sure there are no PCIe bottlenecks. You have to look in mainboard manual how your SATA/SAS ports, network cards and all that possibly connected to the southbridge chip is connected to the CPU. PCH used to have only DMI 3.0 which gets you 3.93 GB/sec for PCH-CPU link. If you got a very stupid mainboard implementation, you might already be pushing this link or chip thermals into some limit.

I am not sure if fio is using sync writes. If it does, since you don't have an SLOG device, like some high-PBW SSD or Optane with PLP, your write speeds will be much slower than expected, maybe 50%.
 

wdp

New Member
May 1, 2021
25
2
3
For this many disks RAIDZ2 in any case is the right choice. There is a new mode called "dRAID" coming with ZFS 2.1 but that requires some more ZFS experience imho. Much faster rebuilds. But also some drawbacks. In production always enable LZ4. For testing no compression is alright.

Read speed with 12x18 TB RAIDZ2 should be ~ 1500 MByte/s. Try a network benchmark like iperf2 or iperf3 first, preferably using a recent Linux from USB stick if client machine usually boots Windows. See if that is also stuck at 600 MByte/s. You might have a network problem. Also make sure there are no PCIe bottlenecks. You have to look in mainboard manual how your SATA/SAS ports, network cards and all that possibly connected to the southbridge chip is connected to the CPU. PCH used to have only DMI 3.0 which gets you 3.93 GB/sec for PCH-CPU link. If you got a very stupid mainboard implementation, you might already be pushing this link or chip thermals into some limit.

I am not sure if fio is using sync writes. If it does, since you don't have an SLOG device, like some high-PBW SSD or Optane with PLP, your write speeds will be much slower than expected, maybe 50%.
Typically, coming from the RAID world, yes. Mirrored vdevs have some appeal for me, since the purpose of 90% of what I do is media read and 10% move / camera card ingestions. I've never had the opportunity to build out my own raid 10 or mirrored vdev and actually benchmark it before. I'm just VERY confused as to why RaidZ2 is outperforming mirrored out of the box.

My edit bay has an onboard aquantia, so not the best or reliable of hardware options. But tests are about what I usually see.

Code:
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  1.09 GBytes  9.35 Gbits/sec    0   1.76 MBytes
[  5]   1.00-2.00   sec  1.09 GBytes  9.34 Gbits/sec    0   1.76 MBytes
[  5]   2.00-3.00   sec  1.09 GBytes  9.40 Gbits/sec    0   1.76 MBytes
[  5]   3.00-4.00   sec  1.09 GBytes  9.40 Gbits/sec    0   1.76 MBytes
[  5]   4.00-5.00   sec  1.09 GBytes  9.38 Gbits/sec    0   1.76 MBytes
[  5]   5.00-6.00   sec  1.09 GBytes  9.41 Gbits/sec    0   1.76 MBytes
[  5]   6.00-7.00   sec  1.09 GBytes  9.40 Gbits/sec    0   1.76 MBytes
[  5]   7.00-8.00   sec  1.09 GBytes  9.40 Gbits/sec    0   1.76 MBytes
[  5]   8.00-9.00   sec  1.09 GBytes  9.39 Gbits/sec    0   1.76 MBytes
[  5]   9.00-10.00  sec  1.09 GBytes  9.39 Gbits/sec    0   1.76 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  10.9 GBytes  9.38 Gbits/sec    0             sender
[  5]   0.00-10.00  sec  10.9 GBytes  9.38 Gbits/sec                  receiver
X10DRH-iT | Motherboards | Products | Super Micro Computer, Inc. is my motherboard, it's pretty much stock, so I have no reason to suspect any conflicts on PCI lanes.
 
Last edited:

Rand__

Well-Known Member
Mar 6, 2014
6,626
1,767
113
I'm just VERY confused as to why RaidZ2 is outperforming mirrored out of the box.
2. mirrors dont scale well in all situations, basically you need to run multiple threads (parallel read/write operations) to make use of them (and dd is single threaded)
I feel ignored;)
Have you run a fio test with multiple parallel threads on the mirror device yet?

The difference is between IOPS and bandwith per vdev,
a mirror has many small vdevs and therefore does IOPS very well [with many parallel requests] but bandwith is limited due to only a single vdev working [for single request];
a raid-zX has the same IOPs (one vdev) but a much higher bandwith it can use in continuous operations (r and w) since that gets sent to many more drives-
 

wdp

New Member
May 1, 2021
25
2
3
I feel ignored;)
Have you run a fio test with multiple parallel threads on the mirror device yet?

The difference is between IOPS and bandwith per vdev,
a mirror has many small vdevs and therefore does IOPS very well [with many parallel requests] but bandwith is limited due to only a single vdev working [for single request];
a raid-zX has the same IOPs (one vdev) but a much higher bandwith it can use in continuous operations (r and w) since that gets sent to many more drives-
Sorry, I haven't gotten around to that yet. I moved the pool back over to RaidZ2 last night for testing and took tonight off from wrenching. Debating if I want to upgrading from these 2603v3's while I sit on ebay...dangerous decisions. Something about seeing a 1.6ghz processor just makes me thing upgrading will help something, I just don't know what.

I will test out mirrroed vdevs in the morning. So far I've only run fio on raidz2 extensively.
 

T_Minus

Build. Break. Fix. Repeat
Feb 15, 2015
7,625
2,043
113
What kind of CPU utilization are you getting with the 2603 V3 ?
What about with compression enabled ?

For what these CPU's cost now I would upgrade, especially relative to this entire system cost. Unless your CPU is barely being used but I'm expecting higher than minima" utilization, curious to see what it is when you benchmark it like @Rand__ has suggested.

How much RAM are you using in here too?
 

wdp

New Member
May 1, 2021
25
2
3
What kind of CPU utilization are you getting with the 2603 V3 ?
What about with compression enabled ?

For what these CPU's cost now I would upgrade, especially relative to this entire system cost. Unless your CPU is barely being used but I'm expecting higher than minima" utilization, curious to see what it is when you benchmark it like @Rand__ has suggested.

How much RAM are you using in here too?
I'd have to go back and look at the CPU while in tests. At times load was hitting 70% though. Budget isn't really a concern, I'd just like to stabilize what I have and learn for a year or so before I go doing something dumb, like getting and AMD Epyc and 36 drive bays.

4x16 64GB 2133 ECC DDR4 SDRAM
 

Rand__

Well-Known Member
Mar 6, 2014
6,626
1,767
113
It sure does look better than dd results;)
On the other hand ideally you identify how your workload looks like (streaming writes due to saving/reading videos or random due to seeking [no idea, not into video], prioritize and then test that - tests from 4k to 4m blocksize with 256QD look nice but will not answer your specific questions.
O/c you can pick an overall winner, but that might not be the best choice for a quite specific workload/user combination.
 

wdp

New Member
May 1, 2021
25
2
3
I do video editing, usually with multiple edit bays (other editors) working at one time. My main goal is really just to calculate if the server will handle it, and at what point does it start to bog down or hit it's cap. And then figure out if I can curb off a few potential bottlenecks. I wanted to take the opportunity to benchmark mirrored vdevs vs 4+2 raidZ2 while the pool is empty.

The ZFS / TrueNAS community RAVES about mirrored vdevs. So I wanted to take a moment and see for myself.

I don't get to play very often. We usually get hard drives, rack them, and start editing. It's been standard raid6 setups for years and years. Sometimes on good hardware that we pay a lot for, sometimes on Synology/QNAP solutions that bog down once more than 1-2 editors hit them hard, but they're practically more cost effective than anything DIY that requires not taking weeks to ebay parts for.

The CPU is 70-100% the entire time through testing anything with a job count in fio. Will I hit that in workloads? Should I upgrade the CPU to say a E5-2680 v4?

I'm at 64Gb of ECC, what does more ram really get me?

Should I look at SIL/SLOG? is L2ARC worth investigating? Right now I have a PCIE 2x 480GB M2 in the chassis with Micron 7300's on it. I don't know if I have a use for it or not, but I had the hardware laying around. I haven't added it to the pool.

Aside from HDD's, all of these parts were in the junk pile at work. This has been a fun project to play around with so far. I feel like I've learned a lot. Chipped off a bunch of rust (haven't been in the command line much the last few years).

I'm honestly a little overloaded trying to decipher what fio is really telling me...

mirrored vdevs...

Code:
=========================

fio -direct=1 -iodepth=128 -rw=randwrite -ioengine=posixaio -bs=4k -size=10G -numjobs=1 -runtime=600 -group_reporting -filename=/mnt/tmp/fiotest1.txt -name=1_Rand_Write_IOPS_Test

1_Rand_Write_IOPS_Test: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=posixaio, iodepth=128
fio-3.26
Starting 1 process
Jobs: 1 (f=1): [w(1)][99.6%][w=135MiB/s][w=34.5k IOPS][eta 00m:01s]
1_Rand_Write_IOPS_Test: (groupid=0, jobs=1): err= 0: pid=21906: Fri May  7 23:59:10 2021
  write: IOPS=10.6k, BW=41.4MiB/s (43.4MB/s)(10.0GiB/247224msec); 0 zone resets
    slat (nsec): min=1145, max=360906k, avg=22355.86, stdev=376136.49
    clat (usec): min=414, max=478286, avg=11280.54, stdev=13700.22
     lat (usec): min=418, max=478294, avg=11302.90, stdev=13720.19
    clat percentiles (usec):
     |  1.00th=[  1237],  5.00th=[  2376], 10.00th=[  3851], 20.00th=[  6259],
     | 30.00th=[  7701], 40.00th=[  8848], 50.00th=[  9896], 60.00th=[ 10945],
     | 70.00th=[ 12125], 80.00th=[ 13435], 90.00th=[ 15926], 95.00th=[ 22938],
     | 99.00th=[ 42206], 99.50th=[ 80217], 99.90th=[223347], 99.95th=[283116],
     | 99.99th=[371196]
   bw (  KiB/s): min= 3460, max=165828, per=100.00%, avg=42481.03, stdev=18719.05, samples=488
   iops        : min=  865, max=41457, avg=10619.95, stdev=4679.78, samples=488
  lat (usec)   : 500=0.01%, 750=0.04%, 1000=0.34%
  lat (msec)   : 2=3.26%, 4=6.90%, 10=40.25%, 20=42.67%, 50=5.69%
  lat (msec)   : 100=0.47%, 250=0.30%, 500=0.08%
  cpu          : usr=8.48%, sys=58.47%, ctx=971949, majf=0, minf=1
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.6%, 16=1.5%, 32=3.9%, >=64=93.9%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=94.2%, 8=3.1%, 16=1.7%, 32=0.7%, 64=0.2%, >=64=0.1%
     issued rwts: total=0,2621440,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=128

Run status group 0 (all jobs):
  WRITE: bw=41.4MiB/s (43.4MB/s), 41.4MiB/s-41.4MiB/s (43.4MB/s-43.4MB/s), io=10.0GiB (10.7GB), run=247224-247224msec

==========================

fio -direct=1 -iodepth=128 -rw=randread -ioengine=posixaio -bs=4k -size=10G -numjobs=1 -runtime=600 -group_reporting -filename=/mnt/tmp/fiotest2.txt -name=2_Rand_Read_IOPS_Test

2_Rand_Read_IOPS_Test: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B,(T) 4096B-4096B, ioengine=posixaio, iodepth=128
fio-3.26
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=285MiB/s][r=72.9k IOPS][eta 00m:00s]
2_Rand_Read_IOPS_Test: (groupid=0, jobs=1): err= 0: pid=22074: Sat May  8 00:03:38 2021
  read: IOPS=47.4k, BW=185MiB/s (194MB/s)(10.0GiB/55305msec)
    slat (nsec): min=1260, max=300542, avg=13430.38, stdev=12814.59
    clat (usec): min=225, max=4303, avg=1803.67, stdev=640.76
     lat (usec): min=229, max=4316, avg=1817.10, stdev=640.55
    clat percentiles (usec):
     |  1.00th=[  807],  5.00th=[  914], 10.00th=[ 1012], 20.00th=[ 1188],
     | 30.00th=[ 1369], 40.00th=[ 1549], 50.00th=[ 1745], 60.00th=[ 1942],
     | 70.00th=[ 2147], 80.00th=[ 2343], 90.00th=[ 2606], 95.00th=[ 2835],
     | 99.00th=[ 3556], 99.50th=[ 3589], 99.90th=[ 3720], 99.95th=[ 3752],
     | 99.99th=[ 3851]
   bw (  KiB/s): min=175035, max=313097, per=100.00%, avg=189608.28, stdev=21324.47, samples=110
   iops        : min=43758, max=78274, avg=47401.75, stdev=5331.09, samples=110
  lat (usec)   : 250=0.01%, 500=0.01%, 750=0.09%, 1000=9.38%
  lat (msec)   : 2=53.51%, 4=37.01%, 10=0.01%
  cpu          : usr=13.61%, sys=67.64%, ctx=674966, majf=0, minf=1
  IO depths    : 1=0.1%, 2=0.1%, 4=1.0%, 8=6.3%, 16=13.1%, 32=26.2%, >=64=53.3%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=99.2%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.8%
     issued rwts: total=2621440,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=128

Run status group 0 (all jobs):
   READ: bw=185MiB/s (194MB/s), 185MiB/s-185MiB/s (194MB/s-194MB/s), io=10.0GiB(10.7GB), run=55305-55305msec

=============================

fio -direct=1 -iodepth=32 -rw=write -ioengine=posixaio -bs=1024k -size=10G -numjobs=1 -runtime=600 -group_reporting -filename=/mnt/tmp/fiotest3.txt -name=3_Write_BandWidth_Test

3_Write_BandWidth_Test: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=posixaio, iodepth=32
fio-3.26
Starting 1 process
Jobs: 1 (f=1): [W(1)][100.0%][w=1242MiB/s][w=1242 IOPS][eta 00m:00s]
3_Write_BandWidth_Test: (groupid=0, jobs=1): err= 0: pid=22175: Sat May  8 00:09:57 2021
  write: IOPS=1273, BW=1274MiB/s (1335MB/s)(10.0GiB/8040msec); 0 zone resets
    slat (usec): min=31, max=65373, avg=441.01, stdev=1293.68
    clat (usec): min=83, max=398966, avg=19932.20, stdev=23130.87
     lat (usec): min=849, max=399068, avg=20373.21, stdev=23321.95
    clat percentiles (usec):
     |  1.00th=[  1319],  5.00th=[  2835], 10.00th=[  4293], 20.00th=[  6718],
     | 30.00th=[  9110], 40.00th=[ 11863], 50.00th=[ 17433], 60.00th=[ 21103],
     | 70.00th=[ 22938], 80.00th=[ 24511], 90.00th=[ 31851], 95.00th=[ 50594],
     | 99.00th=[107480], 99.50th=[152044], 99.90th=[295699], 99.95th=[362808],
     | 99.99th=[392168]
   bw (  MiB/s): min=  297, max= 2336, per=99.90%, avg=1272.31, stdev=571.29, samples=16
   iops        : min=  297, max= 2336, avg=1271.81, stdev=571.29, samples=16
  lat (usec)   : 100=0.02%, 250=0.05%, 500=0.04%, 750=0.12%, 1000=0.23%
  lat (msec)   : 2=2.12%, 4=6.35%, 10=24.51%, 20=22.25%, 50=39.22%
  lat (msec)   : 100=3.94%, 250=1.01%, 500=0.16%
  cpu          : usr=29.59%, sys=2.44%, ctx=12042, majf=0, minf=1
  IO depths    : 1=0.1%, 2=0.6%, 4=4.8%, 8=14.3%, 16=69.6%, 32=10.6%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=94.8%, 8=1.3%, 16=2.0%, 32=1.9%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,10240,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
  WRITE: bw=1274MiB/s (1335MB/s), 1274MiB/s-1274MiB/s (1335MB/s-1335MB/s), io=10.0GiB (10.7GB), run=8040-8040msec

===================================

fio -direct=1 -iodepth=32 -rw=read -ioengine=posixaio -bs=1024k -size=10G -numjobs=1 -runtime=600 -group_reporting -filename=/mnt/tmp/fiotest4.txt -name=4_Read_BandWidth_Test

Jobs: 1 (f=1): [R(1)][100.0%][r=1448MiB/s][r=1447 IOPS][eta 00m:00s]
4_Read_BandWidth_Test: (groupid=0, jobs=1): err= 0: pid=22227: Sat May  8 00:11:08 2021
  read: IOPS=1324, BW=1324MiB/s (1389MB/s)(10.0GiB/7732msec)
    slat (nsec): min=1775, max=18401k, avg=29164.55, stdev=294615.32
    clat (usec): min=49, max=518793, avg=23916.28, stdev=29814.97
     lat (usec): min=822, max=518803, avg=23945.44, stdev=29815.27
    clat percentiles (msec):
     |  1.00th=[    4],  5.00th=[    8], 10.00th=[   11], 20.00th=[   14],
     | 30.00th=[   16], 40.00th=[   18], 50.00th=[   20], 60.00th=[   22],
     | 70.00th=[   25], 80.00th=[   29], 90.00th=[   37], 95.00th=[   47],
     | 99.00th=[   90], 99.50th=[  296], 99.90th=[  443], 99.95th=[  477],
     | 99.99th=[  518]
   bw (  MiB/s): min=  857, max= 1633, per=100.00%, avg=1356.64, stdev=195.01, samples=15
   iops        : min=  857, max= 1633, avg=1356.27, stdev=195.19, samples=15
  lat (usec)   : 50=0.01%, 500=0.01%, 1000=0.02%
  lat (msec)   : 2=0.10%, 4=0.93%, 10=8.55%, 20=43.41%, 50=42.93%
  lat (msec)   : 100=3.15%, 250=0.37%, 500=0.48%, 750=0.04%
  cpu          : usr=1.93%, sys=3.29%, ctx=10357, majf=0, minf=1
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.5%, 16=71.5%, 32=27.9%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=94.0%, 8=4.4%, 16=1.5%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=10240,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: bw=1324MiB/s (1389MB/s), 1324MiB/s-1324MiB/s (1389MB/s-1389MB/s), io=10.0GiB (10.7GB), run=7732-7732msec

=====================================
 

wdp

New Member
May 1, 2021
25
2
3
At this point, I just feel like I'm pasting in tests with no clue what I'm actually looking for...

Code:
fio --name=randrw --ioengine=posixaio --iodepth=1 --rw=randrw --bs=64k --direct=1 --size=2G --numjobs=8 --runtime=300 --group_reporting --time_based --rwmixread=70 -filename=/mnt/tmp/fiotest5.txt

Jobs: 4 (f=4): [m(4)][100.0%][r=1901MiB/s,w=803MiB/s][r=30.4k,w=12.8k IOPS][eta00m:00s]
randrw: (groupid=0, jobs=4): err= 0: pid=22322: Sat May  8 00:20:23 2021
  read: IOPS=24.7k, BW=1544MiB/s (1619MB/s)(452GiB/300001msec)
    slat (nsec): min=1902, max=115538k, avg=10655.50, stdev=122906.10
    clat (nsec): min=1437, max=170579k, avg=94659.54, stdev=641639.23
     lat (usec): min=20, max=170600, avg=105.32, stdev=653.05
    clat percentiles (usec):
     |  1.00th=[    3],  5.00th=[   29], 10.00th=[   32], 20.00th=[   36],
     | 30.00th=[   42], 40.00th=[   47], 50.00th=[   51], 60.00th=[   59],
     | 70.00th=[   75], 80.00th=[  106], 90.00th=[  167], 95.00th=[  249],
     | 99.00th=[  603], 99.50th=[  914], 99.90th=[ 1844], 99.95th=[ 2474],
     | 99.99th=[24249]
   bw (  MiB/s): min=  708, max= 3198, per=100.00%, avg=1545.12, stdev=107.79, samples=2388
   iops        : min=11342, max=51174, avg=24720.28, stdev=1724.66, samples=2388
  write: IOPS=10.6k, BW=662MiB/s (694MB/s)(194GiB/300001msec); 0 zone resets
    slat (usec): min=2, max=103305, avg=17.14, stdev=166.63
    clat (nsec): min=1638, max=174687k, avg=109482.33, stdev=749279.06
     lat (usec): min=29, max=174711, avg=126.62, stdev=767.49
    clat percentiles (usec):
     |  1.00th=[    3],  5.00th=[   43], 10.00th=[   46], 20.00th=[   50],
     | 30.00th=[   53], 40.00th=[   57], 50.00th=[   61], 60.00th=[   69],
     | 70.00th=[   86], 80.00th=[  115], 90.00th=[  178], 95.00th=[  273],
     | 99.00th=[  652], 99.50th=[  979], 99.90th=[ 1942], 99.95th=[ 3032],
     | 99.99th=[35390]
   bw (  KiB/s): min=308965, max=1400971, per=100.00%, avg=678247.92, stdev=47399.48, samples=2388
   iops        : min= 4826, max=21889, avg=10595.97, stdev=740.61, samples=2388
  lat (usec)   : 2=0.85%, 4=1.25%, 10=0.19%, 20=0.07%, 50=37.88%
  lat (usec)   : 100=37.23%, 250=17.31%, 500=3.78%, 750=0.70%, 1000=0.28%
  lat (msec)   : 2=0.36%, 4=0.05%, 10=0.02%, 20=0.01%, 50=0.01%
  lat (msec)   : 100=0.01%, 250=0.01%
  cpu          : usr=5.72%, sys=9.78%, ctx=10787265, majf=0, minf=4
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=7411617,3177182,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=1544MiB/s (1619MB/s), 1544MiB/s-1544MiB/s (1619MB/s-1619MB/s), io=452GiB (486GB), run=300001-300001msec
  WRITE: bw=662MiB/s (694MB/s), 662MiB/s-662MiB/s (694MB/s-694MB/s), io=194GiB (208GB), run=300001-300001msec
 

Rand__

Well-Known Member
Mar 6, 2014
6,626
1,767
113
From my point of view the info you just provided (multiple editors at the same time) basically confirms the need for mirrors.

As mentioned earlier they are good for IOPS with multiple processes opposed to raidZ which is good for one process at a time.


So lets look at the facts - feel free to substitute numbers with more realistic ones if you feel they are off:
Concurrent users: 1-4+
Number of devices: 12 Harddrives , 150 MB r/w, 75 IOPs each
IO Size - depends on client app, mount type etc, but lets say - 128K

The interesting part (since multiple users use the box at the same time) are the IOPS

Raid Z2 with 12 drives means 1 vdev with 10 drives (180T capacity, 75 iops random read/write, 1500 MB/s max read bw [single process], 150 MB/s write [maybe more due to cache and distributed writes]
You need 8 IOPS to read/write a single MB [1 IOP with larger 1M Blocksize]
Question now is what is your average access size (what chunk of a video file is read per step)? watching a video vs actively seeking through it ...
Lets assume its 50 MB/s - that means one user needs 400 IOPs at 128K, or 50 IOPS at 1M.

4 users need 1600 IOPS/s at 128K or 200/s at 1M - you can see where that goes with 75 IOPS...

Mirror:
12 drives = 6 mirror vdevs, 108 TB capacity, 6x 75=450 IOPS, 900 MB/s read/write bw

again looking at " 4 users need 1600 IOPS/s at 128K or 200/s at 1M " looks much better when having 450 IOPS at hand...

-----------------
Memory - is your basic write cache, can turn random io into streaming io, helps a lot with spinners. 64G seems a bit on the low side if you ask me
L2Arc - read cache - if you work a lot on the same videos this can help reduce load on the pool as data does not need to be read from disk but comes from faster cache device. Needs memory which basically takes from your write cache
Slog - Not really needed unless video integrity is paramount. Client needs to run sync writes. Guess you can skip that

CPU - 2603 is basically the entry card - definitely would recommend to upgrade that - higher clock is preferred over many cores in your instance. Think 2+ (1-2 cores/user)

existing M2 drives - l2arc possibly if you have more memory installed, alternative would be special vdev (mirrored) for metadata and such to take load off the spinners