fio --name=seqread \
> --rw=read \
> --direct=0 \
> --iodepth=32 \
> --ioengine=libaio \
> --bs=4k \
> --numjobs=1 \
> --size=128G \
> --group_reporting
seqread: (g=0): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=32
fio-3.1
Starting 1 process
seqread: Laying out IO file (1 file / 131072MiB)
fio: native_fallocate call failed: Operation not supported
Jobs: 1 (f=1): [R(1)][100.0%][r=209MiB/s,w=0KiB/s][r=53.4k,w=0 IOPS][eta 00m:00s]
seqread: (groupid=0, jobs=1): err= 0: pid=6673: Sun Jun 3 21:54:34 2018
read: IOPS=68.8k, BW=269MiB/s (282MB/s)(128GiB/487876msec)
slat (usec): min=4, max=211017, avg=12.32, stdev=224.50
clat (usec): min=2, max=211288, avg=451.67, stdev=1235.02
lat (usec): min=7, max=211294, avg=464.25, stdev=1254.57
clat percentiles (usec):
| 1.00th=[ 269], 5.00th=[ 318], 10.00th=[ 326], 20.00th=[ 330],
| 30.00th=[ 343], 40.00th=[ 371], 50.00th=[ 383], 60.00th=[ 388],
| 70.00th=[ 392], 80.00th=[ 400], 90.00th=[ 412], 95.00th=[ 449],
| 99.00th=[ 693], 99.50th=[ 2638], 99.90th=[20317], 99.95th=[23725],
| 99.99th=[35914]
bw ( KiB/s): min=83464, max=376191, per=100.00%, avg=275458.64, stdev=40776.52, samples=975
iops : min=20866, max=94045, avg=68864.61, stdev=10194.13, samples=975
lat (usec) : 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%, 100=0.01%
lat (usec) : 250=0.85%, 500=95.62%, 750=2.72%, 1000=0.22%
lat (msec) : 2=0.08%, 4=0.04%, 10=0.10%, 20=0.27%, 50=0.10%
lat (msec) : 100=0.01%, 250=0.01%
cpu : usr=13.53%, sys=65.10%, ctx=840988, majf=0, minf=993
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued rwt: total=33554432,0,0, short=0,0,0, dropped=0,0,0
latency : target=0, window=0, percentile=100.00%, depth=32
Run status group 0 (all jobs):
READ: bw=269MiB/s (282MB/s), 269MiB/s-269MiB/s (282MB/s-282MB/s), io=128GiB (137GB), run=487876-487876msec