Floored by optane performance

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

whitey

Moderator
Jun 30, 2014
2,766
868
113
41
Well that was an eye opener, I know we have been flooded w/ optane 900p articles lately and I am late to the game (as usual) but holy smokes batman...BEASTMODE...that took me back. 3K/15K iops w/ FIO runs on my ZFS AIO w/ husmm disks...100K+ iops on optane. wowzers. NEVER seen fio runs complete that quick.

optane-crushing-io.png

Still not completely happy abt using optane for AIO w/ slog/l2arc on vdisk w/ fio runs or sVMotions, I mean they are better but not by as shocking a margin. Direct on the device VMFS formatted...fuggggettaboutit/KING!
 
Last edited:

whitey

Moderator
Jun 30, 2014
2,766
868
113
41
What is everyone's favorite fio run settings. Are these good nuff'? Guess I would have to blow up my vdisk on the VM I have parked on the 900p to be able to write out more files. (50GB vdisk ATM)

Seem to recall unless my eyes crossed others reporting near 500K iops w/ fio testing.

Code:
fio --name=randwrite --ioengine=libaio --iodepth=16 --rw=randwrite --bs=4k --direct=1 --size=4G --numjobs=8 --runtime=240 --group_reporting
Code:
fio --name=randread --ioengine=libaio --iodepth=16 --rw=randread --bs=4k --direct=1 --size=4G --numjobs=8 --runtime=240 --group_reporting
 

whitey

Moderator
Jun 30, 2014
2,766
868
113
41
These three options from big players look pretty good. (Shamelessly borrowed from here, forgot about this resource)

Benchmarking - Benchmarking Linux with Sysbench, FIO, Ioping, and UnixBench: Lots of Examples

RANDOM WRITE (Google Compute)
fio --time_based --name=benchmark --size=10G --runtime=30 --filename=$block_dev --ioengine=libaio --randrepeat=0 --iodepth=128 --direct=1 --invalidate=1 --verify=0 --verify_fatal=0 --numjobs=4 --rw=randwrite --blocksize=4k --group_reporting
RANDOM READ (Google Compute)
fio --time_based --name=benchmark --size=10G --runtime=30 --filename=$block_dev --ioengine=libaio --randrepeat=0 --iodepth=128 --direct=1 --invalidate=1 --verify=0 --verify_fatal=0 --numjobs=4 --rw=randread --blocksize=4k --group_reporting

AWS EBS FIO Tests
##The following command performs 16 KB random write operations.

fio --name fio_test_file --direct=1 --rw=randwrite --bs=4k --size=1G --numjobs=16 --time_based --runtime=180 --group_reporting

##The following command performs 16 KB random read operations.

fio --name fio_test_file --direct=1 --rw=randread --bs=4k --size=1G --numjobs=16 --time_based --runtime=180 --group_reporting


RANDOM WRITE (Fio Random Write Test using libaio and direct flags)
fio --name=randwrite --ioengine=libaio --iodepth=16 --rw=randwrite --bs=4k --direct=1 --size=1G --numjobs=8 --runtime=240 --group_reporting

RANDOM READ (Fio Random Read Test using libaio and direct flags)
fio --name=randread --ioengine=libaio --iodepth=16 --rw=randread --bs=4k --direct=1 --size=1G --numjobs=8 --runtime=240 --group_reporting
 

whitey

Moderator
Jun 30, 2014
2,766
868
113
41
Getting more scientific: (first test was lazy current VM on 900p using existing vdisk formatted w/ ext4, LSI Logic SCSI adpater)

Ubuntu LTS 16.04.3 (4k 100% read and write runs)
vdisk1 - vmware paravirt scsi ctrl - 50GB thick prov lazy zero - raw disk /dev/sdb
vdisk2 - vmware paravirt scsi ctrl - 50GB thick prov lazy zero - ext4 formatted /dev/sdc1
vdisk3 - vmware paravirt scsi ctrl - 50GB thick prov lazy zero - xfs formatted /dev/sdd1

Results:
optane-getting-scientific.png

Sure seem seems like ext4 suffered some read bottleneck but all other tests kept about the same pace for the most part. I could micro-analyze the results but may take a few more runs to normalize data/runs and get a better gut feeling abt these results.

Least I am getting some tuning ironed out and doubled my initial results. 100K up to 200K ain't too shabby w/ dedicated vdisks and paravirt scsi adapters. Dunno what else off the top of my head I can do to tweak this for any better results but I am open to enlightenment.

EDIT: FIO cmds included for prosperity sake/throughness :-D

FIO READ TEST (cycled through sdb/sdc1/sdd1 for test runs)
Code:
fio --filename=/dev/sdb --direct=1 --rw=randrw --refill_buffers --norandommap --randrepeat=0 --ioengine=libaio --bs=4k --rwmixread=100 --iodepth=16 --numjobs=16 --runtime=60 --group_reporting --name=4kreadtest
FIO WRITE TEST (cycled through sdb/sdc1/sdd1 for test runs)
Code:
fio --filename=/dev/sdb --direct=1 --rw=randrw --refill_buffers --norandommap --randrepeat=0 --ioengine=libaio --bs=4k --rwmixread=0 --iodepth=16 --numjobs=16 --runtime=60 --group_reporting --name=4kwritetest
 
Last edited:

whitey

Moderator
Jun 30, 2014
2,766
868
113
41
Why ohh why is fio_generate_plots (gnuplot) SUCH a pile of sh|t? lol... Either this device is too fast for it's graphs or something else is going on as damn near none of my graphs rendered properly and I suspect the iops one that did work is mislabled (red/blue legend at bottom) and that should be 'Read'/'Write' in relation to IOPS axis not 'Queue depth'. Is there a better tool I am not aware of?

upload_2017-12-16_0-14-18.png
 

whitey

Moderator
Jun 30, 2014
2,766
868
113
41
Four more screenies to show progress, think that's about all I can squeeze outta her for now. Nearing 300K iops now :-D

optane-getting-scientific-2.png optane-getting-scientific-cpu-constrained.png optane-getting-scientific-cpu-constrained-2.png optane-getting-scientific-3.png
 
Last edited:

whitey

Moderator
Jun 30, 2014
2,766
868
113
41
Sure thing, let me get those raw fio numbers posted here today. I'll take a fresh new run and post here shortly w/in the next hour or two.
 
  • Like
Reactions: T_Minus and vanfawx

T_Minus

Build. Break. Fix. Repeat
Feb 15, 2015
7,625
2,043
113
At least you're making progress!! some bad colds in this house ~month :(

Can't wait to get my 900Ps going!!!
 

whitey

Moderator
Jun 30, 2014
2,766
868
113
41
Raw fio runs:

READ 225K
Code:
root@fio:~# fio --filename=/dev/sdc1 --direct=1 --rw=randrw --refill_buffers --norandommap --randrepeat=0 --ioengine=libaio --bs=4k --rwmixread=100 --iodepth=16 --numjobs=16 --runtime=60 --group_reporting --name=4kreadtest
4kreadtest: (g=0): rw=randrw, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=16
...
fio-2.2.10
Starting 16 processes
Jobs: 16 (f=16): [r(16)] [100.0% done] [832.8MB/0KB/0KB /s] [213K/0/0 iops] [eta 00m:00s]
4kreadtest: (groupid=0, jobs=16): err= 0: pid=12650: Sun Dec 31 16:00:58 2017
  read : io=52966MB, bw=903918KB/s, iops=225979, runt= 60002msec
    slat (usec): min=2, max=86405, avg=32.18, stdev=276.81
    clat (usec): min=2, max=115347, avg=1098.65, stdev=1432.20
     lat (usec): min=86, max=115359, avg=1131.13, stdev=1466.94
    clat percentiles (usec):
     |  1.00th=[  378],  5.00th=[  454], 10.00th=[  502], 20.00th=[  572],
     | 30.00th=[  636], 40.00th=[  700], 50.00th=[  780], 60.00th=[  892],
     | 70.00th=[ 1096], 80.00th=[ 1400], 90.00th=[ 1944], 95.00th=[ 2352],
     | 99.00th=[ 4704], 99.50th=[ 7584], 99.90th=[19840], 99.95th=[26240],
     | 99.99th=[46336]
    bw (KB  /s): min=22856, max=99776, per=6.26%, avg=56550.35, stdev=9746.40
    lat (usec) : 4=0.01%, 100=0.01%, 250=0.48%, 500=9.48%, 750=36.81%
    lat (usec) : 1000=19.16%
    lat (msec) : 2=24.84%, 4=7.93%, 10=0.96%, 20=0.23%, 50=0.09%
    lat (msec) : 100=0.01%, 250=0.01%
  cpu          : usr=2.65%, sys=14.12%, ctx=2375116, majf=0, minf=168
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=13559221/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: io=52966MB, aggrb=903917KB/s, minb=903917KB/s, maxb=903917KB/s, mint=60002msec, maxt=60002msec

Disk stats (read/write):
  sdc: ios=13543508/0, merge=0/0, ticks=8657276/0, in_queue=8807740, util=100.00%
WRITE 242K
Code:
root@fio:~# fio --filename=/dev/sdc1 --direct=1 --rw=randrw --refill_buffers --norandommap --randrepeat=0 --ioengine=libaio --bs=4k --rwmixread=0 --iodepth=16 --numjobs=16 --runtime=60 --group_reporting --name=4kwritetest
4kwritetest: (g=0): rw=randrw, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=16
...
fio-2.2.10
Starting 16 processes
Jobs: 16 (f=16): [w(16)] [100.0% done] [0KB/1021MB/0KB /s] [0/261K/0 iops] [eta 00m:00s]
4kwritetest: (groupid=0, jobs=16): err= 0: pid=12686: Sun Dec 31 16:03:12 2017
  write: io=56737MB, bw=968275KB/s, iops=242068, runt= 60002msec
    slat (usec): min=2, max=65438, avg=28.70, stdev=236.98
    clat (usec): min=1, max=88552, avg=1025.75, stdev=1219.87
     lat (usec): min=47, max=88560, avg=1054.72, stdev=1247.57
    clat percentiles (usec):
     |  1.00th=[  382],  5.00th=[  438], 10.00th=[  494], 20.00th=[  556],
     | 30.00th=[  604], 40.00th=[  660], 50.00th=[  724], 60.00th=[  828],
     | 70.00th=[  988], 80.00th=[ 1240], 90.00th=[ 1752], 95.00th=[ 2288],
     | 99.00th=[ 5088], 99.50th=[ 7392], 99.90th=[15936], 99.95th=[21120],
     | 99.99th=[36608]
    bw (KB  /s): min=31939, max=89648, per=6.25%, avg=60531.67, stdev=7956.90
    lat (usec) : 2=0.01%, 4=0.01%, 20=0.01%, 100=0.01%, 250=0.22%
    lat (usec) : 500=10.78%, 750=41.67%, 1000=18.11%
    lat (msec) : 2=22.02%, 4=5.65%, 10=1.27%, 20=0.22%, 50=0.05%
    lat (msec) : 100=0.01%
  cpu          : usr=3.95%, sys=14.82%, ctx=2091587, majf=0, minf=186
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=14524613/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: io=56737MB, aggrb=968275KB/s, minb=968275KB/s, maxb=968275KB/s, mint=60002msec, maxt=60002msec

Disk stats (read/write):
  sdc: ios=132/14504453, merge=0/0, ticks=0/8581688, in_queue=8728068, util=100.00%
 

whitey

Moderator
Jun 30, 2014
2,766
868
113
41
Dang. All of this and all of the goodies of ZFS wooot!
Sorry not quite this is a Linux VM w/ vdisks placed on 900p datastore VMFS formatted w/ pvscsi adapters. I can take runs w/ that same VM on a slice of NFS that has a 16GB vdisk for SLOG off the 900p for comparision though.
 

gigatexal

I'm here to learn
Nov 25, 2012
2,913
607
113
Portland, Oregon
alexandarnarayan.com
Sorry not quite this is a Linux VM w/ vdisks placed on 900p datastore VMFS formatted w/ pvscsi adapters. I can take runs w/ that same VM on a slice of NFS that has a 16GB vdisk for SLOG off the 900p for comparision though.
I appreciate it! while you're at it what about a vm on KVM where the disk is passed through and formatted with XFS?

if/when you run ZFS can you run lz4 compression and atime off?
 

whitey

Moderator
Jun 30, 2014
2,766
868
113
41
I don't have any KVM physical boxes avail (all vSphere here) but here are the AIO ZFS w/ 900p as SLOG results.

READ
Code:
root@fio:~# fio --filename=/dev/sdc1 --direct=1 --rw=randrw --refill_buffers --norandommap --randrepeat=0 --ioengine=libaio --bs=4k --rwmixread=100 --iodepth=16 --numjobs=16 --runtime=60 --group_reporting --name=4kreadtest
4kreadtest: (g=0): rw=randrw, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=16
...
fio-2.2.10
Starting 16 processes
Jobs: 16 (f=16): [r(16)] [100.0% done] [42844KB/0KB/0KB /s] [10.8K/0/0 iops] [eta 00m:00s]
4kreadtest: (groupid=0, jobs=16): err= 0: pid=13455: Mon Jan  1 14:46:23 2018
  read : io=2397.2MB, bw=40901KB/s, iops=10225, runt= 60016msec
    slat (usec): min=4, max=108179, avg=836.28, stdev=3906.85
    clat (usec): min=619, max=176402, avg=24194.57, stdev=12130.50
     lat (usec): min=660, max=176412, avg=25031.12, stdev=12286.63
    clat percentiles (msec):
     |  1.00th=[   12],  5.00th=[   13], 10.00th=[   14], 20.00th=[   16],
     | 30.00th=[   17], 40.00th=[   18], 50.00th=[   20], 60.00th=[   23],
     | 70.00th=[   28], 80.00th=[   33], 90.00th=[   41], 95.00th=[   50],
     | 99.00th=[   67], 99.50th=[   73], 99.90th=[   89], 99.95th=[   97],
     | 99.99th=[  115]
    bw (KB  /s): min= 1059, max= 3396, per=6.25%, avg=2558.17, stdev=248.01
    lat (usec) : 750=0.01%
    lat (msec) : 2=0.01%, 4=0.01%, 10=0.02%, 20=51.88%, 50=43.44%
    lat (msec) : 100=4.62%, 250=0.04%
  cpu          : usr=0.22%, sys=0.92%, ctx=127697, majf=0, minf=169
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=613681/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: io=2397.2MB, aggrb=40901KB/s, minb=40901KB/s, maxb=40901KB/s, mint=60016msec, maxt=60016msec

Disk stats (read/write):
  sdc: ios=612841/0, merge=0/0, ticks=9937792/0, in_queue=9946700, util=99.92%
zpool iostat while running fio test
Code:
                                           capacity     operations    bandwidth
pool                                    alloc   free   read  write   read  write
--------------------------------------  -----  -----  -----  -----  -----  -----
husmm1640-rz                            1.20T   255G  6.40K      4   810M  77.9K
  raidz1                                1.20T   255G  6.40K      0   810M      0
    gptid/c7ad6be0-e056-11e7-a2f9-0050569a060b      -      -  4.71K      0   206M      0
    gptid/c7de27b9-e056-11e7-a2f9-0050569a060b      -      -  4.75K      0   195M      0
    gptid/c80db279-e056-11e7-a2f9-0050569a060b      -      -  4.86K      0   211M      0
    gptid/c83b669a-e056-11e7-a2f9-0050569a060b      -      -  4.88K      0   198M      0
logs                                        -      -      -      -      -      -
  gptid/c85e0fba-e056-11e7-a2f9-0050569a060b  8.23M  15.9G      0      4      0  77.9K
cache                                       -      -      -      -      -      -
  gptid/c8864a6f-e056-11e7-a2f9-0050569a060b   149G  16.0E  2.51K    380   318M  40.1M
--------------------------------------  -----  -----  -----  -----  -----  -----
WRITE
Code:
root@fio:~# fio --filename=/dev/sdc1 --direct=1 --rw=randrw --refill_buffers --norandommap --randrepeat=0 --ioengine=libaio --bs=4k --rwmixread=0 --iodepth=16 --numjobs=16 --runtime=60 --group_reporting --name=4kwritetest
4kwritetest: (g=0): rw=randrw, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=16
...
fio-2.2.10
Starting 16 processes
Jobs: 16 (f=16): [w(16)] [100.0% done] [0KB/5876KB/0KB /s] [0/1469/0 iops] [eta 00m:00s]
4kwritetest: (groupid=0, jobs=16): err= 0: pid=13488: Mon Jan  1 14:50:04 2018
  write: io=287028KB, bw=4776.9KB/s, iops=1194, runt= 60087msec
    slat (usec): min=4, max=406215, avg=7216.07, stdev=30014.49
    clat (msec): min=2, max=701, avg=207.04, stdev=88.11
     lat (msec): min=2, max=733, avg=214.26, stdev=87.84
    clat percentiles (msec):
     |  1.00th=[   32],  5.00th=[   68], 10.00th=[   90], 20.00th=[  124],
     | 30.00th=[  151], 40.00th=[  188], 50.00th=[  217], 60.00th=[  237],
     | 70.00th=[  255], 80.00th=[  277], 90.00th=[  314], 95.00th=[  351],
     | 99.00th=[  424], 99.50th=[  453], 99.90th=[  553], 99.95th=[  578],
     | 99.99th=[  701]
    bw (KB  /s): min=   80, max=  694, per=6.25%, avg=298.49, stdev=88.08
    lat (msec) : 4=0.01%, 20=0.31%, 50=1.71%, 100=10.82%, 250=54.67%
    lat (msec) : 500=32.24%, 750=0.24%
  cpu          : usr=0.04%, sys=0.11%, ctx=12211, majf=0, minf=180
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.2%, 16=99.7%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=71757/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: io=287028KB, aggrb=4776KB/s, minb=4776KB/s, maxb=4776KB/s, mint=60087msec, maxt=60087msec

Disk stats (read/write):
  sdc: ios=94/71429, merge=0/0, ticks=20/8584992, in_queue=8600092, util=99.93%
zpool iostat while running fio test - note SLOG doesn't seem to be hit[b/]
Code:
                                           capacity     operations    bandwidth
pool                                    alloc   free   read  write   read  write
--------------------------------------  -----  -----  -----  -----  -----  -----
husmm1640-rz                            1.20T   255G    798  5.31K  99.5M   294M
  raidz1                                1.20T   255G    798  4.16K  99.5M   284M
    gptid/c7ad6be0-e056-11e7-a2f9-0050569a060b      -      -    595  2.01K  25.6M  97.4M
    gptid/c7de27b9-e056-11e7-a2f9-0050569a060b      -      -    581  2.08K  23.4M  96.3M
    gptid/c80db279-e056-11e7-a2f9-0050569a060b      -      -    604  2.03K  26.0M  97.4M
    gptid/c83b669a-e056-11e7-a2f9-0050569a060b      -      -    605  2.07K  24.3M  97.2M
logs                                        -      -      -      -      -      -
  gptid/c85e0fba-e056-11e7-a2f9-0050569a060b  15.2M  15.9G      0  1.15K      0  10.1M
cache                                       -      -      -      -      -      -
  gptid/c8864a6f-e056-11e7-a2f9-0050569a060b   151G  16.0E    393    319  48.8M  39.9M
--------------------------------------  -----  -----  -----  -----  -----  -----
 
  • Like
Reactions: gigatexal

whitey

Moderator
Jun 30, 2014
2,766
868
113
41
Yeah so you can see that a 900p SLOG backed SAS3 ssd pool can't nearly keep up iops wise but throughput is still pretty good and sVMotions are certainly faster.