SAS3 - Dual Expander vs -A backplane (specific use case)

[Rand__]

Need to do some maintenance on my ZFS filer and while I am at it I was considering moving from the 216A Backplane I currently use to spare a SAS3-EL2.

I have a 9305-16i in a x8 slot (4 ports) + onboard SAS3 HBA of my SM Board also with x8 (2 ports), I am using HGST SS300 SSDs with a peak throughput of 2100MB/s each (wide port & theoretically I assume).

http://www.europ-computer.com/fichiers_constructeurs/WESDD033304_1.pdf

Now usually due to max total bandwith the -A backplane would win, but two conditions apply in my case.

1. I only use 12 drives
At this time I realized the drives are not in an optimal layout as I have 8 on the internal HBA and 4 on the 9305-16i.
Ideally I move those around so that I have 3 drives per HBA channel so I can use max bandwith on the SAS3 lanes and are only limited by the (2x) x8 slots

2. My use case has low parallel access, my current ZFS pool is built on mirrors (6 vdevs) .
Basically I don't think I am using more then 2 vdevs at a time (due to missing parallelism). Now if hat were true, I'd be limited to 4 drives at any time anyway meaning upper bandwith limitations are less my issue than individual drive performance.

That means that I should maximize that and the easiest way to do so would be to use the EL2 backplane with two HBAs so I can utilize the wide port capability of the drives.

Is that sound thinking or utter nonsense ?

 

[Rand__]

Well it seems it was utter nonsense.

Ran a few benches with J1QD1 observing gstat and it still hits all drives (with -A and 4x 3 drives) -
On the other hand its quite slow at this setting (barly hitting 1400 MB/s at 1M blocksize) so maybe I should just give it atry, it doesnt look like I am hitting any bandwith limitations at all, just the usual unexplainable slowness (ie only 30% usage of drives)

Didnt turn off sync properly, so that was 1M sync'ed speed which is not too bad actually

Edit - looks like I am limited to ~8GB with this setup...
Best result on a few quick runs of max throughput test. Will swap to sas3 tomorrow and just give it a try.

fio --filename=/mnt/ss300/nfs/test --direct=1 --rw=rw --refill_buffers --norandommap --randrepeat=0 --ioengine=posixaio --bs=1M --rwmixread=0 --iodepth=64 --numjobs=16 --size=8G --name=test

Run status group 0 (all jobs):
WRITE: bw=7615MiB/s (7985MB/s), 476MiB/s-484MiB/s (499MB/s-508MB/s), io=128GiB (137GB), run=16914-17213msec

fio --filename=/mnt/ss300/nfs/test  --direct=1 --rw=rw --refill_buffers --norandommap --randrepeat=0 --ioengine=posixaio --bs=1M --rwmixread=0 --iodepth=64 --numjobs=16 --size=8G  --name=test

test: (g=0): rw=rw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=posixaio, iodepth=64

...

fio-3.16

Starting 16 processes

Jobs: 16 (f=16): [W(16)][94.7%][w=8146MiB/s][w=8146 IOPS][eta 00m:01s]

test: (groupid=0, jobs=1): err= 0: pid=5069: Sat May 30 22:41:53 2020

  write: IOPS=478, BW=478MiB/s (502MB/s)(8192MiB/17124msec)

    slat (nsec): min=1631, max=109321k, avg=1412305.68, stdev=3148035.37

    clat (usec): min=97, max=344501, avg=66197.74, stdev=51358.50

     lat (usec): min=346, max=345848, avg=67610.05, stdev=51694.78

    clat percentiles (usec):

     |  1.00th=[   570],  5.00th=[  5014], 10.00th=[ 10552], 20.00th=[ 22152],

     | 30.00th=[ 34341], 40.00th=[ 45351], 50.00th=[ 56361], 60.00th=[ 68682],

     | 70.00th=[ 83362], 80.00th=[ 98042], 90.00th=[132645], 95.00th=[168821],

     | 99.00th=[240124], 99.50th=[270533], 99.90th=[308282], 99.95th=[316670],

     | 99.99th=[346031]

   bw (  KiB/s): min=258544, max=772096, per=6.25%, avg=487096.59, stdev=138792.38, samples=34

   iops        : min=  252, max=  754, avg=475.18, stdev=135.48, samples=34

  lat (usec)   : 100=0.02%, 250=0.84%, 500=0.10%, 750=0.18%, 1000=0.20%

  lat (msec)   : 2=0.92%, 4=1.83%, 10=5.41%, 20=8.72%, 50=25.87%

  lat (msec)   : 100=36.43%, 250=18.79%, 500=0.71%

  cpu          : usr=12.36%, sys=0.52%, ctx=46388, majf=0, minf=2

  IO depths    : 1=1.1%, 2=2.8%, 4=6.2%, 8=12.4%, 16=25.1%, 32=50.7%, >=64=1.6%

     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%

     complete  : 0=0.0%, 4=98.4%, 8=0.0%, 16=0.0%, 32=0.1%, 64=1.5%, >=64=0.0%

     issued rwts: total=0,8192,0,0 short=0,0,0,0 dropped=0,0,0,0

     latency   : target=0, window=0, percentile=100.00%, depth=64

test: (groupid=0, jobs=1): err= 0: pid=5070: Sat May 30 22:41:53 2020

  write: IOPS=478, BW=479MiB/s (502MB/s)(8192MiB/17119msec)

    slat (nsec): min=1743, max=146728k, avg=1377917.74, stdev=3717644.39

    clat (usec): min=95, max=561972, avg=66947.27, stdev=63581.73

     lat (usec): min=579, max=562406, avg=68325.19, stdev=63847.89

    clat percentiles (usec):

     |  1.00th=[   644],  5.00th=[  4424], 10.00th=[  9634], 20.00th=[ 20317],

     | 30.00th=[ 30802], 40.00th=[ 41681], 50.00th=[ 53216], 60.00th=[ 65274],

     | 70.00th=[ 79168], 80.00th=[ 95945], 90.00th=[137364], 95.00th=[183501],

     | 99.00th=[287310], 99.50th=[488637], 99.90th=[541066], 99.95th=[549454],

     | 99.99th=[557843]

   bw (  KiB/s): min=128766, max=911584, per=6.24%, avg=486415.56, stdev=193687.39, samples=34

   iops        : min=  125, max=  890, avg=474.50, stdev=189.13, samples=34

  lat (usec)   : 100=0.04%, 250=0.71%, 500=0.18%, 750=0.22%, 1000=0.31%

  lat (msec)   : 2=1.06%, 4=2.10%, 10=5.73%, 20=9.31%, 50=27.12%

  lat (msec)   : 100=34.94%, 250=16.88%, 500=0.96%, 750=0.44%

  cpu          : usr=12.48%, sys=0.54%, ctx=48264, majf=0, minf=2

  IO depths    : 1=1.1%, 2=2.9%, 4=6.2%, 8=12.5%, 16=25.0%, 32=50.7%, >=64=1.6%

     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%

     complete  : 0=0.0%, 4=98.4%, 8=0.0%, 16=0.0%, 32=0.1%, 64=1.5%, >=64=0.0%

     issued rwts: total=0,8192,0,0 short=0,0,0,0 dropped=0,0,0,0

     latency   : target=0, window=0, percentile=100.00%, depth=64

test: (groupid=0, jobs=1): err= 0: pid=5071: Sat May 30 22:41:53 2020

  write: IOPS=476, BW=476MiB/s (500MB/s)(8192MiB/17193msec)

    slat (usec): min=2, max=111967, avg=1382.55, stdev=3382.10

    clat (usec): min=94, max=297226, avg=67147.05, stdev=52781.85

     lat (usec): min=461, max=299158, avg=68529.60, stdev=53120.09

    clat percentiles (usec):

     |  1.00th=[   412],  5.00th=[  4817], 10.00th=[ 10159], 20.00th=[ 20841],

     | 30.00th=[ 32900], 40.00th=[ 44827], 50.00th=[ 56886], 60.00th=[ 70779],

     | 70.00th=[ 84411], 80.00th=[101188], 90.00th=[139461], 95.00th=[177210],

     | 99.00th=[235930], 99.50th=[263193], 99.90th=[291505], 99.95th=[291505],

     | 99.99th=[295699]

   bw (  KiB/s): min=257532, max=784862, per=6.09%, avg=474539.29, stdev=134414.84, samples=34

   iops        : min=  251, max=  766, avg=462.76, stdev=131.15, samples=34

  lat (usec)   : 100=0.04%, 250=0.82%, 500=0.21%, 750=0.21%, 1000=0.22%

  lat (msec)   : 2=0.87%, 4=1.98%, 10=5.58%, 20=9.33%, 50=25.07%

  lat (msec)   : 100=35.05%, 250=19.87%, 500=0.77%

  cpu          : usr=12.52%, sys=0.51%, ctx=51926, majf=0, minf=2

  IO depths    : 1=1.2%, 2=2.9%, 4=6.2%, 8=12.5%, 16=25.1%, 32=50.6%, >=64=1.6%

     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%

     complete  : 0=0.0%, 4=98.4%, 8=0.0%, 16=0.1%, 32=0.1%, 64=1.5%, >=64=0.0%

     issued rwts: total=0,8192,0,0 short=0,0,0,0 dropped=0,0,0,0

     latency   : target=0, window=0, percentile=100.00%, depth=64

test: (groupid=0, jobs=1): err= 0: pid=5072: Sat May 30 22:41:53 2020

  write: IOPS=477, BW=478MiB/s (501MB/s)(8192MiB/17144msec)

    slat (nsec): min=1572, max=146183k, avg=1440571.19, stdev=3108665.34

    clat (usec): min=102, max=280208, avg=66704.30, stdev=52956.80

     lat (usec): min=540, max=281493, avg=68144.87, stdev=53227.87

    clat percentiles (usec):

     |  1.00th=[   594],  5.00th=[  5276], 10.00th=[ 11207], 20.00th=[ 21365],

     | 30.00th=[ 31589], 40.00th=[ 42730], 50.00th=[ 54264], 60.00th=[ 67634],

     | 70.00th=[ 83362], 80.00th=[102237], 90.00th=[141558], 95.00th=[181404],

     | 99.00th=[229639], 99.50th=[240124], 99.90th=[265290], 99.95th=[274727],

     | 99.99th=[278922]

   bw (  KiB/s): min=257532, max=784862, per=6.20%, avg=483200.21, stdev=151605.37, samples=34

   iops        : min=  251, max=  766, avg=471.32, stdev=147.97, samples=34

  lat (usec)   : 250=0.81%, 500=0.17%, 750=0.13%, 1000=0.32%

  lat (msec)   : 2=0.72%, 4=1.82%, 10=5.04%, 20=9.48%, 50=27.78%

  lat (msec)   : 100=32.56%, 250=20.91%, 500=0.26%

  cpu          : usr=12.77%, sys=0.29%, ctx=48109, majf=0, minf=2

  IO depths    : 1=1.1%, 2=2.9%, 4=6.2%, 8=12.3%, 16=25.0%, 32=50.9%, >=64=1.6%

     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%

     complete  : 0=0.0%, 4=98.4%, 8=0.1%, 16=0.1%, 32=0.1%, 64=1.6%, >=64=0.0%

     issued rwts: total=0,8192,0,0 short=0,0,0,0 dropped=0,0,0,0

     latency   : target=0, window=0, percentile=100.00%, depth=64

test: (groupid=0, jobs=1): err= 0: pid=5073: Sat May 30 22:41:53 2020

  write: IOPS=480, BW=480MiB/s (503MB/s)(8192MiB/17061msec)

    slat (nsec): min=1199, max=107901k, avg=1376250.06, stdev=3362845.81

    clat (usec): min=85, max=284445, avg=68146.68, stdev=55295.41

     lat (usec): min=254, max=284453, avg=69522.93, stdev=55558.17

    clat percentiles (usec):

     |  1.00th=[   652],  5.00th=[  4621], 10.00th=[  9634], 20.00th=[ 19792],

     | 30.00th=[ 30802], 40.00th=[ 42730], 50.00th=[ 54789], 60.00th=[ 68682],

     | 70.00th=[ 84411], 80.00th=[107480], 90.00th=[152044], 95.00th=[185598],

     | 99.00th=[227541], 99.50th=[242222], 99.90th=[278922], 99.95th=[283116],

     | 99.99th=[283116]

   bw (  KiB/s): min=258544, max=905537, per=6.24%, avg=486851.88, stdev=176241.15, samples=34

   iops        : min=  252, max=  884, avg=475.00, stdev=172.08, samples=34

  lat (usec)   : 100=0.04%, 250=0.70%, 500=0.16%, 750=0.23%, 1000=0.29%

  lat (msec)   : 2=0.96%, 4=1.99%, 10=6.12%, 20=9.61%, 50=25.61%

  lat (msec)   : 100=31.43%, 250=22.53%, 500=0.33%

  cpu          : usr=12.34%, sys=0.49%, ctx=47439, majf=0, minf=2

  IO depths    : 1=1.1%, 2=2.8%, 4=6.2%, 8=12.4%, 16=25.0%, 32=50.8%, >=64=1.6%

     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%

     complete  : 0=0.0%, 4=98.4%, 8=0.1%, 16=0.1%, 32=0.1%, 64=1.5%, >=64=0.0%

     issued rwts: total=0,8192,0,0 short=0,0,0,0 dropped=0,0,0,0

     latency   : target=0, window=0, percentile=100.00%, depth=64

test: (groupid=0, jobs=1): err= 0: pid=5074: Sat May 30 22:41:53 2020

  write: IOPS=478, BW=479MiB/s (502MB/s)(8192MiB/17110msec)

    slat (nsec): min=1566, max=95436k, avg=1428101.10, stdev=3005383.30

    clat (usec): min=98, max=353646, avg=66899.17, stdev=54374.28

     lat (usec): min=603, max=353652, avg=68327.27, stdev=54666.46

    clat percentiles (usec):

     |  1.00th=[   652],  5.00th=[  4752], 10.00th=[ 10290], 20.00th=[ 20841],

     | 30.00th=[ 31851], 40.00th=[ 43254], 50.00th=[ 55313], 60.00th=[ 68682],

     | 70.00th=[ 82314], 80.00th=[ 99091], 90.00th=[141558], 95.00th=[175113],

     | 99.00th=[254804], 99.50th=[270533], 99.90th=[312476], 99.95th=[337642],

     | 99.99th=[354419]

   bw (  KiB/s): min=180224, max=784862, per=6.18%, avg=482041.44, stdev=145051.92, samples=34

   iops        : min=  176, max=  766, avg=470.56, stdev=141.58, samples=34

  lat (usec)   : 100=0.02%, 250=0.77%, 500=0.09%, 750=0.20%, 1000=0.38%

  lat (msec)   : 2=1.12%, 4=1.78%, 10=5.43%, 20=9.40%, 50=26.56%

  lat (msec)   : 100=34.41%, 250=18.55%, 500=1.28%

  cpu          : usr=12.52%, sys=0.41%, ctx=43345, majf=0, minf=2

  IO depths    : 1=1.1%, 2=2.8%, 4=6.3%, 8=12.6%, 16=25.2%, 32=50.5%, >=64=1.6%

     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%

     complete  : 0=0.0%, 4=98.4%, 8=0.0%, 16=0.0%, 32=0.1%, 64=1.6%, >=64=0.0%

     issued rwts: total=0,8192,0,0 short=0,0,0,0 dropped=0,0,0,0

     latency   : target=0, window=0, percentile=100.00%, depth=64

test: (groupid=0, jobs=1): err= 0: pid=5075: Sat May 30 22:41:53 2020

  write: IOPS=478, BW=478MiB/s (501MB/s)(8192MiB/17132msec)

    slat (nsec): min=1681, max=96736k, avg=1405026.88, stdev=2742498.18

    clat (usec): min=77, max=359643, avg=66577.72, stdev=53415.30

     lat (usec): min=479, max=360851, avg=67982.74, stdev=53723.91

    clat percentiles (usec):

     |  1.00th=[   594],  5.00th=[  5342], 10.00th=[ 10552], 20.00th=[ 21365],

     | 30.00th=[ 32113], 40.00th=[ 43779], 50.00th=[ 55837], 60.00th=[ 67634],

     | 70.00th=[ 82314], 80.00th=[100140], 90.00th=[139461], 95.00th=[175113],

     | 99.00th=[248513], 99.50th=[274727], 99.90th=[320865], 99.95th=[325059],

     | 99.99th=[358613]

   bw (  KiB/s): min=259576, max=826135, per=6.20%, avg=483826.06, stdev=137545.30, samples=34

   iops        : min=  253, max=  806, avg=471.94, stdev=134.17, samples=34

  lat (usec)   : 100=0.04%, 250=0.77%, 500=0.15%, 750=0.18%, 1000=0.23%

  lat (msec)   : 2=0.87%, 4=1.67%, 10=5.51%, 20=9.34%, 50=26.31%

  lat (msec)   : 100=34.85%, 250=19.15%, 500=0.94%

  cpu          : usr=12.39%, sys=0.46%, ctx=46070, majf=0, minf=2

  IO depths    : 1=1.1%, 2=2.9%, 4=6.2%, 8=12.5%, 16=25.2%, 32=50.5%, >=64=1.6%

     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%

     complete  : 0=0.0%, 4=98.4%, 8=0.0%, 16=0.0%, 32=0.1%, 64=1.5%, >=64=0.0%

     issued rwts: total=0,8192,0,0 short=0,0,0,0 dropped=0,0,0,0

     latency   : target=0, window=0, percentile=100.00%, depth=64

test: (groupid=0, jobs=1): err= 0: pid=5076: Sat May 30 22:41:53 2020

  write: IOPS=478, BW=478MiB/s (501MB/s)(8192MiB/17130msec)

    slat (nsec): min=1547, max=195087k, avg=1447910.75, stdev=3872629.30

    clat (usec): min=79, max=398231, avg=64636.99, stdev=53788.06

     lat (usec): min=493, max=405056, avg=66084.90, stdev=54267.81

    clat percentiles (usec):

     |  1.00th=[   611],  5.00th=[  4686], 10.00th=[  9634], 20.00th=[ 20055],

     | 30.00th=[ 30802], 40.00th=[ 41681], 50.00th=[ 53740], 60.00th=[ 65799],

     | 70.00th=[ 79168], 80.00th=[ 96994], 90.00th=[135267], 95.00th=[173016],

     | 99.00th=[231736], 99.50th=[308282], 99.90th=[379585], 99.95th=[387974],

     | 99.99th=[396362]

   bw (  KiB/s): min=255489, max=774642, per=6.18%, avg=481963.91, stdev=145813.64, samples=34

   iops        : min=  249, max=  756, avg=470.12, stdev=142.34, samples=34

  lat (usec)   : 100=0.01%, 250=0.74%, 500=0.20%, 750=0.26%, 1000=0.28%

  lat (msec)   : 2=1.01%, 4=1.86%, 10=6.01%, 20=9.62%, 50=26.67%

  lat (msec)   : 100=34.56%, 250=17.88%, 500=0.90%

  cpu          : usr=12.42%, sys=0.54%, ctx=46495, majf=0, minf=2

  IO depths    : 1=1.1%, 2=2.8%, 4=6.2%, 8=12.4%, 16=25.1%, 32=50.7%, >=64=1.6%

     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%

     complete  : 0=0.0%, 4=98.4%, 8=0.0%, 16=0.0%, 32=0.1%, 64=1.5%, >=64=0.0%

     issued rwts: total=0,8192,0,0 short=0,0,0,0 dropped=0,0,0,0

     latency   : target=0, window=0, percentile=100.00%, depth=64

test: (groupid=0, jobs=1): err= 0: pid=5077: Sat May 30 22:41:53 2020

  write: IOPS=475, BW=476MiB/s (499MB/s)(8192MiB/17213msec)

    slat (nsec): min=1732, max=86452k, avg=1434097.98, stdev=3117623.18

    clat (usec): min=96, max=303664, avg=67389.94, stdev=52828.12

     lat (usec): min=523, max=304852, avg=68824.03, stdev=53067.64

    clat percentiles (usec):

     |  1.00th=[   644],  5.00th=[  5014], 10.00th=[ 10421], 20.00th=[ 21627],

     | 30.00th=[ 33424], 40.00th=[ 45351], 50.00th=[ 58459], 60.00th=[ 71828],

     | 70.00th=[ 85459], 80.00th=[ 99091], 90.00th=[131597], 95.00th=[177210],

     | 99.00th=[250610], 99.50th=[265290], 99.90th=[278922], 99.95th=[287310],

     | 99.99th=[304088]

   bw (  KiB/s): min=257532, max=901365, per=6.13%, avg=477985.12, stdev=147734.73, samples=34

   iops        : min=  251, max=  880, avg=466.26, stdev=144.27, samples=34

  lat (usec)   : 100=0.02%, 250=0.65%, 500=0.21%, 750=0.34%, 1000=0.26%

  lat (msec)   : 2=1.09%, 4=1.65%, 10=5.40%, 20=8.87%, 50=25.26%

  lat (msec)   : 100=36.66%, 250=18.57%, 500=1.04%

  cpu          : usr=12.09%, sys=0.67%, ctx=44743, majf=0, minf=2

  IO depths    : 1=1.0%, 2=2.8%, 4=6.2%, 8=12.4%, 16=25.2%, 32=50.8%, >=64=1.6%

     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%

     complete  : 0=0.0%, 4=98.4%, 8=0.0%, 16=0.1%, 32=0.1%, 64=1.5%, >=64=0.0%

     issued rwts: total=0,8192,0,0 short=0,0,0,0 dropped=0,0,0,0

     latency   : target=0, window=0, percentile=100.00%, depth=64

test: (groupid=0, jobs=1): err= 0: pid=5078: Sat May 30 22:41:53 2020

  write: IOPS=479, BW=480MiB/s (503MB/s)(8192MiB/17068msec)

    slat (nsec): min=1868, max=85229k, avg=1368397.40, stdev=2822634.97

    clat (usec): min=99, max=450288, avg=66144.25, stdev=56028.27

     lat (usec): min=529, max=450297, avg=67512.64, stdev=56381.01

    clat percentiles (usec):

     |  1.00th=[   457],  5.00th=[  4686], 10.00th=[ 10159], 20.00th=[ 20841],

     | 30.00th=[ 31065], 40.00th=[ 42206], 50.00th=[ 53740], 60.00th=[ 66847],

     | 70.00th=[ 80217], 80.00th=[ 98042], 90.00th=[135267], 95.00th=[181404],

     | 99.00th=[263193], 99.50th=[278922], 99.90th=[383779], 99.95th=[413139],

     | 99.99th=[450888]

   bw (  KiB/s): min=258048, max=738909, per=6.21%, avg=483871.15, stdev=143187.66, samples=34

   iops        : min=  252, max=  721, avg=471.94, stdev=139.74, samples=34

  lat (usec)   : 100=0.01%, 250=0.82%, 500=0.17%, 750=0.23%, 1000=0.24%

  lat (msec)   : 2=1.14%, 4=1.82%, 10=5.42%, 20=9.56%, 50=27.16%

  lat (msec)   : 100=34.33%, 250=17.70%, 500=1.40%

  cpu          : usr=12.37%, sys=0.49%, ctx=44377, majf=0, minf=2

  IO depths    : 1=1.2%, 2=2.9%, 4=6.1%, 8=12.3%, 16=25.0%, 32=51.0%, >=64=1.6%

     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%

     complete  : 0=0.0%, 4=98.4%, 8=0.0%, 16=0.0%, 32=0.1%, 64=1.5%, >=64=0.0%

     issued rwts: total=0,8192,0,0 short=0,0,0,0 dropped=0,0,0,0

     latency   : target=0, window=0, percentile=100.00%, depth=64

test: (groupid=0, jobs=1): err= 0: pid=5079: Sat May 30 22:41:53 2020

  write: IOPS=478, BW=478MiB/s (502MB/s)(8192MiB/17126msec)

    slat (nsec): min=1756, max=115832k, avg=1442217.79, stdev=3300830.19

    clat (usec): min=95, max=378651, avg=64871.13, stdev=51286.24

     lat (usec): min=516, max=378661, avg=66313.35, stdev=51712.31

    clat percentiles (usec):

     |  1.00th=[   652],  5.00th=[  4228], 10.00th=[  9634], 20.00th=[ 20579],

     | 30.00th=[ 31589], 40.00th=[ 42206], 50.00th=[ 54789], 60.00th=[ 67634],

     | 70.00th=[ 83362], 80.00th=[101188], 90.00th=[130548], 95.00th=[166724],

     | 99.00th=[231736], 99.50th=[254804], 99.90th=[341836], 99.95th=[371196],

     | 99.99th=[379585]

   bw (  KiB/s): min=256000, max=891145, per=6.12%, avg=477199.65, stdev=133234.28, samples=34

   iops        : min=  250, max=  870, avg=465.91, stdev=130.05, samples=34

  lat (usec)   : 100=0.02%, 250=0.70%, 500=0.18%, 750=0.31%, 1000=0.29%

  lat (msec)   : 2=1.31%, 4=2.05%, 10=5.57%, 20=9.20%, 50=26.65%

  lat (msec)   : 100=33.26%, 250=19.91%, 500=0.55%

  cpu          : usr=12.34%, sys=0.57%, ctx=46758, majf=0, minf=2

  IO depths    : 1=1.0%, 2=2.7%, 4=6.3%, 8=12.6%, 16=25.2%, 32=50.7%, >=64=1.6%

     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%

     complete  : 0=0.0%, 4=98.4%, 8=0.0%, 16=0.0%, 32=0.1%, 64=1.5%, >=64=0.0%

     issued rwts: total=0,8192,0,0 short=0,0,0,0 dropped=0,0,0,0

     latency   : target=0, window=0, percentile=100.00%, depth=64

test: (groupid=0, jobs=1): err= 0: pid=5080: Sat May 30 22:41:53 2020

  write: IOPS=478, BW=478MiB/s (501MB/s)(8192MiB/17129msec)

    slat (nsec): min=1513, max=94671k, avg=1432834.26, stdev=3393386.32

    clat (usec): min=104, max=367902, avg=64974.81, stdev=55722.00

     lat (usec): min=472, max=367906, avg=66407.64, stdev=56137.73

    clat percentiles (usec):

     |  1.00th=[   461],  5.00th=[  4015], 10.00th=[  9110], 20.00th=[ 18744],

     | 30.00th=[ 29754], 40.00th=[ 41681], 50.00th=[ 53740], 60.00th=[ 65799],

     | 70.00th=[ 80217], 80.00th=[ 95945], 90.00th=[135267], 95.00th=[177210],

     | 99.00th=[274727], 99.50th=[295699], 99.90th=[346031], 99.95th=[350225],

     | 99.99th=[367002]

   bw (  KiB/s): min=257532, max=891145, per=6.14%, avg=478971.71, stdev=165556.35, samples=34

   iops        : min=  251, max=  870, avg=467.24, stdev=161.61, samples=34

  lat (usec)   : 250=0.84%, 500=0.20%, 750=0.24%, 1000=0.29%

  lat (msec)   : 2=1.15%, 4=2.23%, 10=5.98%, 20=10.10%, 50=25.99%

  lat (msec)   : 100=34.89%, 250=16.60%, 500=1.49%

  cpu          : usr=12.50%, sys=0.43%, ctx=43842, majf=0, minf=2

  IO depths    : 1=1.2%, 2=2.8%, 4=6.3%, 8=12.6%, 16=25.2%, 32=50.4%, >=64=1.6%

     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%

     complete  : 0=0.0%, 4=98.4%, 8=0.0%, 16=0.0%, 32=0.1%, 64=1.5%, >=64=0.0%

     issued rwts: total=0,8192,0,0 short=0,0,0,0 dropped=0,0,0,0

     latency   : target=0, window=0, percentile=100.00%, depth=64

test: (groupid=0, jobs=1): err= 0: pid=5081: Sat May 30 22:41:53 2020

  write: IOPS=482, BW=482MiB/s (506MB/s)(8192MiB/16979msec)

    slat (usec): min=2, max=90067, avg=1394.90, stdev=2725.88

    clat (usec): min=94, max=321401, avg=67207.05, stdev=52584.18

     lat (usec): min=579, max=321405, avg=68601.95, stdev=52778.20

    clat percentiles (usec):

     |  1.00th=[   553],  5.00th=[  5014], 10.00th=[ 11076], 20.00th=[ 22414],

     | 30.00th=[ 33817], 40.00th=[ 45351], 50.00th=[ 57410], 60.00th=[ 69731],

     | 70.00th=[ 83362], 80.00th=[100140], 90.00th=[133694], 95.00th=[175113],

     | 99.00th=[244319], 99.50th=[274727], 99.90th=[316670], 99.95th=[320865],

     | 99.99th=[320865]

   bw (  KiB/s): min=257532, max=889101, per=6.25%, avg=487636.55, stdev=145227.44, samples=33

   iops        : min=  251, max=  868, avg=475.67, stdev=141.74, samples=33

  lat (usec)   : 100=0.04%, 250=0.74%, 500=0.21%, 750=0.17%, 1000=0.16%

  lat (msec)   : 2=0.92%, 4=2.03%, 10=4.79%, 20=8.72%, 50=25.89%

  lat (msec)   : 100=36.24%, 250=19.24%, 500=0.87%

  cpu          : usr=12.62%, sys=0.32%, ctx=48694, majf=0, minf=2

  IO depths    : 1=1.2%, 2=2.9%, 4=6.2%, 8=12.5%, 16=25.1%, 32=50.5%, >=64=1.6%

     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%

     complete  : 0=0.0%, 4=98.4%, 8=0.0%, 16=0.0%, 32=0.1%, 64=1.6%, >=64=0.0%

     issued rwts: total=0,8192,0,0 short=0,0,0,0 dropped=0,0,0,0

     latency   : target=0, window=0, percentile=100.00%, depth=64

test: (groupid=0, jobs=1): err= 0: pid=5082: Sat May 30 22:41:53 2020

  write: IOPS=477, BW=478MiB/s (501MB/s)(8192MiB/17155msec)

    slat (nsec): min=1398, max=106166k, avg=1399797.15, stdev=2957205.62

    clat (usec): min=97, max=325560, avg=65278.86, stdev=50976.70

     lat (usec): min=405, max=326097, avg=66678.66, stdev=51325.25

    clat percentiles (usec):

     |  1.00th=[   603],  5.00th=[  4621], 10.00th=[  9634], 20.00th=[ 20841],

     | 30.00th=[ 32113], 40.00th=[ 44303], 50.00th=[ 56361], 60.00th=[ 68682],

     | 70.00th=[ 81265], 80.00th=[ 99091], 90.00th=[128451], 95.00th=[168821],

     | 99.00th=[235930], 99.50th=[248513], 99.90th=[270533], 99.95th=[283116],

     | 99.99th=[325059]

   bw (  KiB/s): min=376079, max=649964, per=6.04%, avg=471184.91, stdev=92342.57, samples=34

   iops        : min=  367, max=  634, avg=459.79, stdev=90.05, samples=34

  lat (usec)   : 100=0.02%, 250=0.70%, 500=0.22%, 750=0.21%, 1000=0.26%

  lat (msec)   : 2=1.04%, 4=2.10%, 10=5.77%, 20=8.74%, 50=25.98%

  lat (msec)   : 100=35.29%, 250=19.24%, 500=0.44%

  cpu          : usr=12.55%, sys=0.30%, ctx=46985, majf=0, minf=2

  IO depths    : 1=1.1%, 2=2.8%, 4=6.2%, 8=12.5%, 16=25.0%, 32=50.8%, >=64=1.6%

     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%

     complete  : 0=0.0%, 4=98.4%, 8=0.1%, 16=0.1%, 32=0.1%, 64=1.5%, >=64=0.0%

     issued rwts: total=0,8192,0,0 short=0,0,0,0 dropped=0,0,0,0

     latency   : target=0, window=0, percentile=100.00%, depth=64

test: (groupid=0, jobs=1): err= 0: pid=5083: Sat May 30 22:41:53 2020

  write: IOPS=484, BW=484MiB/s (508MB/s)(8192MiB/16914msec)

    slat (nsec): min=1545, max=99675k, avg=1405340.26, stdev=3337167.62

    clat (usec): min=50, max=353448, avg=66187.42, stdev=52940.41

     lat (usec): min=610, max=353726, avg=67592.76, stdev=53292.51

    clat percentiles (usec):

     |  1.00th=[   570],  5.00th=[  4752], 10.00th=[ 10028], 20.00th=[ 20841],

     | 30.00th=[ 31589], 40.00th=[ 43254], 50.00th=[ 54789], 60.00th=[ 67634],

     | 70.00th=[ 81265], 80.00th=[100140], 90.00th=[143655], 95.00th=[179307],

     | 99.00th=[225444], 99.50th=[240124], 99.90th=[325059], 99.95th=[337642],

     | 99.99th=[354419]

   bw (  KiB/s): min=262144, max=784862, per=6.35%, avg=495348.00, stdev=133182.97, samples=33

   iops        : min=  256, max=  766, avg=483.18, stdev=129.89, samples=33

  lat (usec)   : 100=0.01%, 250=0.87%, 500=0.10%, 750=0.27%, 1000=0.37%

  lat (msec)   : 2=1.05%, 4=1.71%, 10=5.69%, 20=9.12%, 50=26.62%

  lat (msec)   : 100=34.22%, 250=19.64%, 500=0.34%

  cpu          : usr=12.47%, sys=0.46%, ctx=43527, majf=0, minf=2

  IO depths    : 1=1.2%, 2=2.9%, 4=6.3%, 8=12.6%, 16=25.1%, 32=50.4%, >=64=1.6%

     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%

     complete  : 0=0.0%, 4=98.4%, 8=0.0%, 16=0.0%, 32=0.1%, 64=1.5%, >=64=0.0%

     issued rwts: total=0,8192,0,0 short=0,0,0,0 dropped=0,0,0,0

     latency   : target=0, window=0, percentile=100.00%, depth=64

test: (groupid=0, jobs=1): err= 0: pid=5084: Sat May 30 22:41:53 2020

  write: IOPS=477, BW=477MiB/s (501MB/s)(8192MiB/17161msec)

    slat (nsec): min=1619, max=118327k, avg=1403282.69, stdev=3584281.54

    clat (usec): min=95, max=438072, avg=66187.12, stdev=56475.42

     lat (usec): min=180, max=439438, avg=67590.40, stdev=56880.20

    clat percentiles (usec):

     |  1.00th=[   433],  5.00th=[  4047], 10.00th=[  8356], 20.00th=[ 19006],

     | 30.00th=[ 30016], 40.00th=[ 41681], 50.00th=[ 53740], 60.00th=[ 67634],

     | 70.00th=[ 82314], 80.00th=[100140], 90.00th=[137364], 95.00th=[181404],

     | 99.00th=[261096], 99.50th=[283116], 99.90th=[400557], 99.95th=[434111],

     | 99.99th=[438305]

   bw (  KiB/s): min=261620, max=784862, per=5.98%, avg=466111.15, stdev=125114.78, samples=34

   iops        : min=  255, max=  766, avg=454.79, stdev=122.07, samples=34

  lat (usec)   : 100=0.01%, 250=0.83%, 500=0.21%, 750=0.22%, 1000=0.27%

  lat (msec)   : 2=1.06%, 4=2.37%, 10=6.85%, 20=8.96%, 50=26.37%

  lat (msec)   : 100=32.92%, 250=18.60%, 500=1.33%

  cpu          : usr=12.22%, sys=0.65%, ctx=47224, majf=0, minf=2

  IO depths    : 1=1.2%, 2=2.8%, 4=6.2%, 8=12.5%, 16=25.1%, 32=50.7%, >=64=1.6%

     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%

     complete  : 0=0.0%, 4=98.4%, 8=0.0%, 16=0.0%, 32=0.1%, 64=1.5%, >=64=0.0%

     issued rwts: total=0,8192,0,0 short=0,0,0,0 dropped=0,0,0,0

     latency   : target=0, window=0, percentile=100.00%, depth=64


Run status group 0 (all jobs):

  WRITE: bw=7615MiB/s (7985MB/s), 476MiB/s-484MiB/s (499MB/s-508MB/s), io=128GiB (137GB), run=16914-17213msec

 

[Rand__]

Replaced the backplane, dual attached it to the onboard and a second hba and reran the above test (max throughput)


Run status group 0 (all jobs):
WRITE: bw=8458MiB/s (8869MB/s), 529MiB/s-541MiB/s (554MB/s-567MB/s), io=128GiB (137GB), run=15152-15496msec

So a good 10% faster. Interesting.


I do loose some % on my low qd performance though (1J/QD1, 64K)
WRITE: bw=708MiB/s (742MB/s), 708MiB/s-708MiB/s (742MB/s-742MB/s), io=128GiB (137GB), run=185182-185182msec
vs
WRITE: bw=754MiB/s (791MB/s), 754MiB/s-754MiB/s (791MB/s-791MB/s), io=128GiB (137GB), run=173862-173862msec

so thats a bit annoying.

I also gain a slot on the board since I dont need an extra external HBA and can daisy chain instead...

Decisions, decision...