root@fio:~# fio --filename=/dev/sdc1 --direct=1 --rw=randrw --refill_buffers --norandommap --randrepeat=0 --ioengine=libaio --bs=4k --rwmixread=0 --iodepth=16 --numjobs=16 --runtime=60 --group_reporting --name=4kwritetest
4kwritetest: (g=0): rw=randrw, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=16
...
fio-2.2.10
Starting 16 processes
Jobs: 16 (f=16): [w(16)] [100.0% done] [0KB/1021MB/0KB /s] [0/261K/0 iops] [eta 00m:00s]
4kwritetest: (groupid=0, jobs=16): err= 0: pid=12686: Sun Dec 31 16:03:12 2017
write: io=56737MB, bw=968275KB/s, iops=242068, runt= 60002msec
slat (usec): min=2, max=65438, avg=28.70, stdev=236.98
clat (usec): min=1, max=88552, avg=1025.75, stdev=1219.87
lat (usec): min=47, max=88560, avg=1054.72, stdev=1247.57
clat percentiles (usec):
| 1.00th=[ 382], 5.00th=[ 438], 10.00th=[ 494], 20.00th=[ 556],
| 30.00th=[ 604], 40.00th=[ 660], 50.00th=[ 724], 60.00th=[ 828],
| 70.00th=[ 988], 80.00th=[ 1240], 90.00th=[ 1752], 95.00th=[ 2288],
| 99.00th=[ 5088], 99.50th=[ 7392], 99.90th=[15936], 99.95th=[21120],
| 99.99th=[36608]
bw (KB /s): min=31939, max=89648, per=6.25%, avg=60531.67, stdev=7956.90
lat (usec) : 2=0.01%, 4=0.01%, 20=0.01%, 100=0.01%, 250=0.22%
lat (usec) : 500=10.78%, 750=41.67%, 1000=18.11%
lat (msec) : 2=22.02%, 4=5.65%, 10=1.27%, 20=0.22%, 50=0.05%
lat (msec) : 100=0.01%
cpu : usr=3.95%, sys=14.82%, ctx=2091587, majf=0, minf=186
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
issued : total=r=0/w=14524613/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
latency : target=0, window=0, percentile=100.00%, depth=16
Run status group 0 (all jobs):
WRITE: io=56737MB, aggrb=968275KB/s, minb=968275KB/s, maxb=968275KB/s, mint=60002msec, maxt=60002msec
Disk stats (read/write):
sdc: ios=132/14504453, merge=0/0, ticks=0/8581688, in_queue=8728068, util=100.00%