So,
2 years later (https://forums.servethehome.com/index.php?threads/napp-it-not-scaling-well.17154/) still the same problems :/
O/c I have to say its not Napp-It, its ZFS; this time on OmniOS latest (but similar problems on latest FreeNas).
I have a Xeon Gold 6150 QS (2.7GHz base, all core freq 3.4GHz iirc), and 13 HGST SS300 SAS3 drives to play with (800GB). Those drives are attached to two 9305-16i's, 6 drives each; o/c on PCIe3-x8 slots on a single CPU Board (X11SPH-nCTPF); running in a CSE-216 with -A backplane.
I have installed fio to run tests, using the following command to run a pure seq write test:
/opt/csw/bin/fio --refill_buffers --norandommap --randrepeat=0 --group_reporting --ioengine=solarisaio --name="<testname>" --runtime=60 --size=100G --time_based --bs=128k --iodepth=1 --numjobs=1 --rw=write --filename=<outfile>
I have pools with a single disk up to 6 disks (striped) and then 13 disks; a dataset with sync off, no compression, no atime and blocksize 128k.
These tests were explicitly done to see how fast a pool can go for a single thread (which is not very fast at this time for yet unknown reasons).
CPU load (using prstat) peaked at 3.7% (as always its not clear to me whether that's 3.7% of a single core or of total capacity [The physical processor has 18 cores and 36 virtual processors (0-35)]).
So as one can see: a single disk is able to do 900 MB/s writes, adding a second increases by 300 MB/s, a third by another 100 and adding a fourth disk is basically useless.
Below are the detailed testruns for OmniOS
Now the behavior is similar with other blocksizes, see this run from FreeNas with 1M recordsize/ fio testsize
and similar results are there for the other end (recordsize 4k, fio blocksize 4k, ~250 MB/s)
Now of course the question is why ?
Can anyone confirm this behavior or dispute it?
2 years later (https://forums.servethehome.com/index.php?threads/napp-it-not-scaling-well.17154/) still the same problems :/
O/c I have to say its not Napp-It, its ZFS; this time on OmniOS latest (but similar problems on latest FreeNas).
I have a Xeon Gold 6150 QS (2.7GHz base, all core freq 3.4GHz iirc), and 13 HGST SS300 SAS3 drives to play with (800GB). Those drives are attached to two 9305-16i's, 6 drives each; o/c on PCIe3-x8 slots on a single CPU Board (X11SPH-nCTPF); running in a CSE-216 with -A backplane.
I have installed fio to run tests, using the following command to run a pure seq write test:
/opt/csw/bin/fio --refill_buffers --norandommap --randrepeat=0 --group_reporting --ioengine=solarisaio --name="<testname>" --runtime=60 --size=100G --time_based --bs=128k --iodepth=1 --numjobs=1 --rw=write --filename=<outfile>
I have pools with a single disk up to 6 disks (striped) and then 13 disks; a dataset with sync off, no compression, no atime and blocksize 128k.
These tests were explicitly done to see how fast a pool can go for a single thread (which is not very fast at this time for yet unknown reasons).
CPU load (using prstat) peaked at 3.7% (as always its not clear to me whether that's 3.7% of a single core or of total capacity [The physical processor has 18 cores and 36 virtual processors (0-35)]).
So as one can see: a single disk is able to do 900 MB/s writes, adding a second increases by 300 MB/s, a third by another 100 and adding a fourth disk is basically useless.
Below are the detailed testruns for OmniOS
Code:
/opt/csw/bin/fio --refill_buffers --norandommap --randrepeat=0 --group_reporting --ioengine=solarisaio --name="stripe1/ds1_128k_stripe1" --runtime=60 --size=100G --time_based --bs=128k --iodepth=1 --numjobs=1 --rw=write --filename=/stripe1/ds1/fio_1.out
stripe1/ds1_128k: (g=0): rw=write, bs=128K-128K/128K-128K/128K-128K, ioengine=solarisaio, iodepth=1
fio-2.0.14
Starting 1 process
stripe1/ds1_128k: Laying out IO file(s) (1 file(s) / 102400MB)
Jobs: 1 (f=1): [W] [100.0% done] [0K/990.2M/0K /s] [0 /7921 /0 iops] [eta 00m:00s]
stripe1/ds1_128k: (groupid=0, jobs=1): err= 0: pid=5929: Wed Nov 20 22:33:13 2019
write: io=911488KB, bw=923957KB/s, iops=7218 , runt= 60000msec
slat (usec): min=2 , max=366 , avg= 2.84, stdev= 0.79
clat (usec): min=29 , max=258912 , avg=86.68, stdev=487.77
lat (usec): min=33 , max=258915 , avg=89.52, stdev=487.77
clat percentiles (usec):
| 1.00th=[ 33], 5.00th=[ 35], 10.00th=[ 35], 20.00th=[ 36],
| 30.00th=[ 39], 40.00th=[ 51], 50.00th=[ 57], 60.00th=[ 69],
| 70.00th=[ 86], 80.00th=[ 93], 90.00th=[ 147], 95.00th=[ 245],
| 99.00th=[ 470], 99.50th=[ 524], 99.90th=[ 644], 99.95th=[ 724],
| 99.99th=[14400]
bw (KB/s) : min=252672, max=1343488, per=100.00%, avg=926805.27, stdev=248607.24
lat (usec) : 50=39.14%, 100=44.00%, 250=12.02%, 500=4.15%, 750=0.65%
lat (usec) : 1000=0.01%
lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
lat (msec) : 100=0.01%, 500=0.01%
cpu : usr=106.99%, sys=55.18%, ctx=449288, majf=0, minf=0
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued : total=r=0/w=0/d=433105, short=r=0/w=0/d=0
Run status group 0 (all jobs):
WRITE: io=54138MB, aggrb=923957KB/s, minb=923957KB/s, maxb=923957KB/s, mint=60000msec, maxt=60000msec
/opt/csw/bin/fio --refill_buffers --norandommap --randrepeat=0 --group_reporting --ioengine=solarisaio --name="stripe1/ds1_128k_stripe2" --runtime=60 --size=100G --time_based --bs=128k --iodepth=1 --numjobs=1 --rw=write --filename=/stripe1/ds1/fio_1.out
stripe1/ds1_128k: (g=0): rw=write, bs=128K-128K/128K-128K/128K-128K, ioengine=solarisaio, iodepth=1
fio-2.0.14
Starting 1 process
stripe1/ds1_128k: Laying out IO file(s) (1 file(s) / 102400MB)
Jobs: 1 (f=1): [W] [100.0% done] [0K/908.9M/0K /s] [0 /7271 /0 iops] [eta 00m:00s]
stripe1/ds1_128k: (groupid=0, jobs=1): err= 0: pid=6416: Wed Nov 20 22:35:12 2019
write: io=1929.2MB, bw=1192.7MB/s, iops=9541 , runt= 60001msec
slat (usec): min=2 , max=381 , avg= 2.76, stdev= 0.77
clat (usec): min=28 , max=20885 , avg=52.99, stdev=107.18
lat (usec): min=32 , max=20887 , avg=55.75, stdev=107.20
clat percentiles (usec):
| 1.00th=[ 32], 5.00th=[ 34], 10.00th=[ 35], 20.00th=[ 35],
| 30.00th=[ 36], 40.00th=[ 37], 50.00th=[ 38], 60.00th=[ 40],
| 70.00th=[ 45], 80.00th=[ 56], 90.00th=[ 86], 95.00th=[ 91],
| 99.00th=[ 225], 99.50th=[ 438], 99.90th=[ 828], 99.95th=[ 916],
| 99.99th=[ 1608]
bw (MB/s) : min= 677, max= 1398, per=100.00%, avg=1224.04, stdev=229.13
lat (usec) : 50=75.25%, 100=21.38%, 250=2.46%, 500=0.49%, 750=0.26%
lat (usec) : 1000=0.12%
lat (msec) : 2=0.02%, 4=0.01%, 20=0.01%, 50=0.01%
cpu : usr=106.54%, sys=43.56%, ctx=593789, majf=0, minf=0
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued : total=r=0/w=0/d=572489, short=r=0/w=0/d=0
Run status group 0 (all jobs):
WRITE: io=71561MB, aggrb=1192.7MB/s, minb=1192.7MB/s, maxb=1192.7MB/s, mint=60001msec, maxt=60001msec
zpool status stripe1
pool: stripe1
state: ONLINE
scan: none requested
config:
NAME STATE READ WRITE CKSUM
stripe1 ONLINE 0 0 0
c0t5000CCA082001CE4d0 ONLINE 0 0 0
c10t5000CCA082006ED1d0 ONLINE 0 0 0
c11t5000CCA082007151d0 ONLINE 0 0 0
errors: No known data errors
/opt/csw/bin/fio --refill_buffers --norandommap --randrepeat=0 --group_reporting --ioengine=solarisaio --name="stripe1/ds1_128k_stripe3" --runtime=60 --size=100G --time_based --bs=128k --iodepth=1 --numjobs=1 --rw=write --filename=/stripe1/ds1/fio_1.out
stripe1/ds1_128k_stripe3: (g=0): rw=write, bs=128K-128K/128K-128K/128K-128K, ioengine=solarisaio, iodepth=1
fio-2.0.14
Starting 1 process
stripe1/ds1_128k_stripe3: Laying out IO file(s) (1 file(s) / 102400MB)
Jobs: 1 (f=1): [W] [100.0% done] [0K/899.2M/0K /s] [0 /7193 /0 iops] [eta 00m:00s]
stripe1/ds1_128k_stripe3: (groupid=0, jobs=1): err= 0: pid=7028: Wed Nov 20 22:38:02 2019
write: io=126208KB, bw=1299.2MB/s, iops=10392 , runt= 60001msec
slat (usec): min=2 , max=399 , avg= 2.81, stdev= 0.76
clat (usec): min=5 , max=19696 , avg=44.36, stdev=70.88
lat (usec): min=31 , max=19701 , avg=47.16, stdev=70.91
clat percentiles (usec):
| 1.00th=[ 33], 5.00th=[ 35], 10.00th=[ 35], 20.00th=[ 36],
| 30.00th=[ 36], 40.00th=[ 37], 50.00th=[ 37], 60.00th=[ 39],
| 70.00th=[ 41], 80.00th=[ 46], 90.00th=[ 57], 95.00th=[ 84],
| 99.00th=[ 95], 99.50th=[ 101], 99.90th=[ 239], 99.95th=[ 1080],
| 99.99th=[ 2640]
bw (MB/s) : min= 735, max= 1462, per=100.00%, avg=1332.60, stdev=159.71
lat (usec) : 10=0.01%, 50=84.49%, 100=14.92%, 250=0.49%, 500=0.02%
lat (usec) : 750=0.02%, 1000=0.01%
lat (msec) : 2=0.03%, 4=0.02%, 10=0.01%, 20=0.01%
cpu : usr=105.56%, sys=40.03%, ctx=648325, majf=0, minf=0
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued : total=r=0/w=0/d=623578, short=r=0/w=0/d=0
Run status group 0 (all jobs):
WRITE: io=77947MB, aggrb=1299.2MB/s, minb=1299.2MB/s, maxb=1299.2MB/s, mint=60001msec, maxt=60001msec
zpool status stripe1
pool: stripe1
state: ONLINE
scan: none requested
config:
NAME STATE READ WRITE CKSUM
stripe1 ONLINE 0 0 0
c0t5000CCA082001CE4d0 ONLINE 0 0 0
c10t5000CCA082006ED1d0 ONLINE 0 0 0
c11t5000CCA082007151d0 ONLINE 0 0 0
c12t5000CCA082006751d0 ONLINE 0 0 0
errors: No known data errors
root@omniosce:~# /opt/csw/bin/fio --refill_buffers --norandommap --randrepeat=0 --group_reporting --ioengine=solarisaio --name="stripe1/ds1_128k_stripe4" --runtime=60 --size=100G --time_based --bs=128k --iodepth=1 --numjobs=1 --rw=write --filename=/stripe1/ds1/fio_1.out
stripe1/ds1_128k_stripe4: (g=0): rw=write, bs=128K-128K/128K-128K/128K-128K, ioengine=solarisaio, iodepth=1
fio-2.0.14
Starting 1 process
stripe1/ds1_128k_stripe4: Laying out IO file(s) (1 file(s) / 102400MB)
Jobs: 1 (f=1): [W] [100.0% done] [0K/1165M/0K /s] [0 /9322 /0 iops] [eta 00m:00s]
stripe1/ds1_128k_stripe4: (groupid=0, jobs=1): err= 0: pid=7409: Wed Nov 20 22:39:44 2019
write: io=1171.2MB, bw=1316.6MB/s, iops=10532 , runt= 60001msec
slat (usec): min=2 , max=395 , avg= 2.80, stdev= 0.75
clat (usec): min=28 , max=14262 , avg=43.10, stdev=33.82
lat (usec): min=32 , max=14264 , avg=45.90, stdev=33.85
clat percentiles (usec):
| 1.00th=[ 33], 5.00th=[ 34], 10.00th=[ 35], 20.00th=[ 36],
| 30.00th=[ 36], 40.00th=[ 37], 50.00th=[ 37], 60.00th=[ 39],
| 70.00th=[ 42], 80.00th=[ 46], 90.00th=[ 56], 95.00th=[ 69],
| 99.00th=[ 103], 99.50th=[ 129], 99.90th=[ 197], 99.95th=[ 219],
| 99.99th=[ 454]
bw (MB/s) : min= 991, max= 1446, per=100.00%, avg=1353.19, stdev=50.85
lat (usec) : 50=84.09%, 100=14.76%, 250=1.12%, 500=0.02%, 750=0.01%
lat (usec) : 1000=0.01%
lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%
cpu : usr=107.77%, sys=37.08%, ctx=657156, majf=0, minf=0
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued : total=r=0/w=0/d=631961, short=r=0/w=0/d=0
Run status group 0 (all jobs):
WRITE: io=78995MB, aggrb=1316.6MB/s, minb=1316.6MB/s, maxb=1316.6MB/s, mint=60001msec, maxt=60001msec
zpool status stripe1
pool: stripe1
state: ONLINE
scan: none requested
config:
NAME STATE READ WRITE CKSUM
stripe1 ONLINE 0 0 0
c0t5000CCA082001CE4d0 ONLINE 0 0 0
c10t5000CCA082006ED1d0 ONLINE 0 0 0
c11t5000CCA082007151d0 ONLINE 0 0 0
c12t5000CCA082006751d0 ONLINE 0 0 0
c13t5000CCA082007279d0 ONLINE 0 0 0
errors: No known data errors
/opt/csw/bin/fio --refill_buffers --norandommap --randrepeat=0 --group_reporting --ioengine=solarisaio --name="stripe1/ds1_128k_stripe5" --runtime=60 --size=100G --time_based --bs=128k --iodepth=1 --numjobs=1 --rw=write --filename=/stripe1/ds1/fio_1.out
stripe1/ds1_128k_stripe5: (g=0): rw=write, bs=128K-128K/128K-128K/128K-128K, ioengine=solarisaio, iodepth=1
fio-2.0.14
Starting 1 process
stripe1/ds1_128k_stripe5: Laying out IO file(s) (1 file(s) / 102400MB)
Jobs: 1 (f=1): [W] [100.0% done] [0K/873.3M/0K /s] [0 /6986 /0 iops] [eta 00m:00s]
stripe1/ds1_128k_stripe5: (groupid=0, jobs=1): err= 0: pid=7832: Wed Nov 20 22:41:40 2019
write: io=2484.4MB, bw=1338.5MB/s, iops=10707 , runt= 60001msec
slat (usec): min=2 , max=367 , avg= 2.86, stdev= 0.70
clat (usec): min=28 , max=13956 , avg=41.46, stdev=25.17
lat (usec): min=31 , max=13959 , avg=44.32, stdev=25.21
clat percentiles (usec):
| 1.00th=[ 32], 5.00th=[ 34], 10.00th=[ 35], 20.00th=[ 36],
| 30.00th=[ 36], 40.00th=[ 37], 50.00th=[ 37], 60.00th=[ 38],
| 70.00th=[ 40], 80.00th=[ 44], 90.00th=[ 52], 95.00th=[ 62],
| 99.00th=[ 88], 99.50th=[ 95], 99.90th=[ 129], 99.95th=[ 153],
| 99.99th=[ 318]
bw (MB/s) : min= 809, max= 1504, per=100.00%, avg=1373.90, stdev=63.18
lat (usec) : 50=87.35%, 100=12.27%, 250=0.37%, 500=0.01%, 750=0.01%
lat (usec) : 1000=0.01%
lat (msec) : 2=0.01%, 4=0.01%, 20=0.01%
cpu : usr=105.91%, sys=37.94%, ctx=667700, majf=0, minf=0
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued : total=r=0/w=0/d=642467, short=r=0/w=0/d=0
Run status group 0 (all jobs):
WRITE: io=80308MB, aggrb=1338.5MB/s, minb=1338.5MB/s, maxb=1338.5MB/s, mint=60001msec, maxt=60001msec
zpool status stripe1
pool: stripe1
state: ONLINE
scan: none requested
config:
NAME STATE READ WRITE CKSUM
stripe1 ONLINE 0 0 0
c0t5000CCA082001CE4d0 ONLINE 0 0 0
c10t5000CCA082006ED1d0 ONLINE 0 0 0
c11t5000CCA082007151d0 ONLINE 0 0 0
c12t5000CCA082006751d0 ONLINE 0 0 0
c13t5000CCA082007279d0 ONLINE 0 0 0
c14t5000CCA082006749d0 ONLINE 0 0 0
errors: No known data errors
root@omniosce:~# /opt/csw/bin/fio --refill_buffers --norandommap --randrepeat=0 --group_reporting --ioengine=solarisaio --name="stripe1/ds1_128k_stripe6" --runtime=60 --size=100G --time_based --bs=128k --iodepth=1 --numjobs=1 --rw=write --filename=/stripe1/ds1/fio_1.out
stripe1/ds1_128k_stripe6: (g=0): rw=write, bs=128K-128K/128K-128K/128K-128K, ioengine=solarisaio, iodepth=1
fio-2.0.14
Starting 1 process
stripe1/ds1_128k_stripe6: Laying out IO file(s) (1 file(s) / 102400MB)
Jobs: 1 (f=1): [W] [100.0% done] [0K/1356M/0K /s] [0 /10.9K/0 iops] [eta 00m:00s]
stripe1/ds1_128k_stripe6: (groupid=0, jobs=1): err= 0: pid=8263: Wed Nov 20 22:43:36 2019
write: io=2632.8MB, bw=1340.1MB/s, iops=10727 , runt= 60000msec
slat (usec): min=2 , max=367 , avg= 2.80, stdev= 0.72
clat (usec): min=29 , max=619 , avg=41.37, stdev=12.57
lat (usec): min=32 , max=621 , avg=44.17, stdev=12.67
clat percentiles (usec):
| 1.00th=[ 33], 5.00th=[ 35], 10.00th=[ 35], 20.00th=[ 36],
| 30.00th=[ 36], 40.00th=[ 36], 50.00th=[ 37], 60.00th=[ 38],
| 70.00th=[ 40], 80.00th=[ 44], 90.00th=[ 53], 95.00th=[ 63],
| 99.00th=[ 92], 99.50th=[ 107], 99.90th=[ 169], 99.95th=[ 195],
| 99.99th=[ 266]
bw (MB/s) : min= 1327, max= 1429, per=100.00%, avg=1374.82, stdev=20.00
lat (usec) : 50=87.03%, 100=12.29%, 250=0.66%, 500=0.01%, 750=0.01%
cpu : usr=106.23%, sys=37.57%, ctx=670686, majf=0, minf=0
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued : total=r=0/w=0/d=643654, short=r=0/w=0/d=0
Run status group 0 (all jobs):
WRITE: io=80457MB, aggrb=1340.1MB/s, minb=1340.1MB/s, maxb=1340.1MB/s, mint=60000msec, maxt=60000msec
zpool status stripe1
pool: stripe1
state: ONLINE
scan: none requested
config:
NAME STATE READ WRITE CKSUM
stripe1 ONLINE 0 0 0
c0t5000CCA082001CE4d0 ONLINE 0 0 0
c10t5000CCA082006ED1d0 ONLINE 0 0 0
c11t5000CCA082007151d0 ONLINE 0 0 0
c12t5000CCA082006751d0 ONLINE 0 0 0
c13t5000CCA082007279d0 ONLINE 0 0 0
c14t5000CCA082006749d0 ONLINE 0 0 0
c15t5000CCA082006E85d0 ONLINE 0 0 0
c16t5000CCA082007059d0 ONLINE 0 0 0
c17t5000CCA082007339d0 ONLINE 0 0 0
c18t5000CCA082007355d0 ONLINE 0 0 0
c19t5000CCA082007335d0 ONLINE 0 0 0
c8t5000CCA08200733Dd0 ONLINE 0 0 0
c9t5000CCA08200734Dd0 ONLINE 0 0 0
errors: No known data errors
root@omniosce:~# /opt/csw/bin/fio --refill_buffers --norandommap --randrepeat=0 --group_reporting --ioengine=solarisaio --name="stripe1/ds1_128k_stripe13" --runtime=60 --size=100G --time_based --bs=128k --iodepth=1 --numjobs=1 --rw=write --filename=/stripe1/ds1/fio_1.out
stripe1/ds1_128k_stripe13: (g=0): rw=write, bs=128K-128K/128K-128K/128K-128K, ioengine=solarisaio, iodepth=1
fio-2.0.14
Starting 1 process
stripe1/ds1_128k_stripe13: Laying out IO file(s) (1 file(s) / 102400MB)
Jobs: 1 (f=1): [W] [100.0% done] [0K/1345M/0K /s] [0 /10.8K/0 iops] [eta 00m:00s]
stripe1/ds1_128k_stripe13: (groupid=0, jobs=1): err= 0: pid=9091: Wed Nov 20 22:47:57 2019
write: io=75776KB, bw=1298.3MB/s, iops=10386 , runt= 60001msec
slat (usec): min=2 , max=392 , avg= 2.85, stdev= 0.85
clat (usec): min=28 , max=7672 , avg=44.35, stdev=32.81
lat (usec): min=32 , max=7678 , avg=47.19, stdev=33.03
clat percentiles (usec):
| 1.00th=[ 32], 5.00th=[ 34], 10.00th=[ 35], 20.00th=[ 35],
| 30.00th=[ 36], 40.00th=[ 36], 50.00th=[ 37], 60.00th=[ 38],
| 70.00th=[ 40], 80.00th=[ 45], 90.00th=[ 58], 95.00th=[ 78],
| 99.00th=[ 147], 99.50th=[ 195], 99.90th=[ 430], 99.95th=[ 628],
| 99.99th=[ 980]
bw (MB/s) : min= 1244, max= 1498, per=100.00%, avg=1329.82, stdev=42.81
lat (usec) : 50=85.19%, 100=12.07%, 250=2.47%, 500=0.20%, 750=0.04%
lat (usec) : 1000=0.02%
lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%
cpu : usr=107.78%, sys=36.27%, ctx=649328, majf=0, minf=0
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued : total=r=0/w=0/d=623184, short=r=0/w=0/d=0
Run status group 0 (all jobs):
WRITE: io=77898MB, aggrb=1298.3MB/s, minb=1298.3MB/s, maxb=1298.3MB/s, mint=60001msec, maxt=60001msec
Now the behavior is similar with other blocksizes, see this run from FreeNas with 1M recordsize/ fio testsize
and similar results are there for the other end (recordsize 4k, fio blocksize 4k, ~250 MB/s)
Now of course the question is why ?
Can anyone confirm this behavior or dispute it?