I didn't do a full drive test, just basically installed Proxmox VE on it. The Micron 7300 had 1PB written on it though (it was a steal, thanks for @ca3y6 again).
Did you mean smart self-test of like
fio ?Impending Drive failure with 4 different Drives of 2 Brands/Models
?I thought that maybe ASPM is interfering with Performance, that could definitively be (and on the Hyve EPYC Server I wouldn't be surprised if PCIe auto-negotiation is SUPER wonky), but that seems to be alright:
Code:
root@pve:~# sudo lspci -s 81:00.0 -vvv | grep -E "LnkCap|LnkCtl|LnkSta"
LnkCap: Port #0, Speed 8GT/s, Width x4, ASPM not supported
LnkCtl: ASPM Disabled; RCB 64 bytes, LnkDisable- CommClk+
LnkSta: Speed 8GT/s, Width x4
LnkCap2: Supported Link Speeds: 2.5-8GT/s, Crosslink- Retimer- 2Retimers- DRS-
LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete+ EqualizationPhase1+
LnkCtl3: LnkEquIntrruptEn- PerformEqu-
root@pve:~# sudo lspci -s c2:00.0 -vvv | grep -E "LnkCap|LnkCtl|LnkSta"
LnkCap: Port #0, Speed 8GT/s, Width x4, ASPM not supported
LnkCtl: ASPM Disabled; RCB 64 bytes, LnkDisable- CommClk+
LnkSta: Speed 8GT/s, Width x4
LnkCap2: Supported Link Speeds: 2.5-8GT/s, Crosslink- Retimer- 2Retimers- DRS-
LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete+ EqualizationPhase1+
LnkCtl3: LnkEquIntrruptEn- PerformEqu-Sector Size:
Code:
root@pve:~# sudo nvme id-ns -H /dev/nvme0n1 | grep -E "LBA Format |Relative Performance"
[6:5] : 0 Most significant 2 bits of Current LBA Format Selected
[3:0] : 0 Least significant 4 bits of Current LBA Format Selected
LBA Format 0 : Metadata Size: 0 bytes - Data Size: 512 bytes - Relative Performance: 0 Best (in use)
LBA Format 1 : Metadata Size: 0 bytes - Data Size: 4096 bytes - Relative Performance: 0 Best
LBA Format 2 : Metadata Size: 8 bytes - Data Size: 512 bytes - Relative Performance: 0 Best
LBA Format 3 : Metadata Size: 8 bytes - Data Size: 4096 bytes - Relative Performance: 0 Best
LBA Format 4 : Metadata Size: 64 bytes - Data Size: 4096 bytes - Relative Performance: 0 Best
root@pve:~# sudo nvme id-ns -H /dev/nvme1n1 | grep -E "LBA Format |Relative Performance"
[6:5] : 0 Most significant 2 bits of Current LBA Format Selected
[3:0] : 0 Least significant 4 bits of Current LBA Format Selected
LBA Format 0 : Metadata Size: 0 bytes - Data Size: 512 bytes - Relative Performance: 0 Best (in use)
LBA Format 1 : Metadata Size: 0 bytes - Data Size: 4096 bytes - Relative Performance: 0 Best
LBA Format 2 : Metadata Size: 8 bytes - Data Size: 512 bytes - Relative Performance: 0 Best
LBA Format 3 : Metadata Size: 8 bytes - Data Size: 4096 bytes - Relative Performance: 0 Best
LBA Format 4 : Metadata Size: 64 bytes - Data Size: 4096 bytes - Relative Performance: 0 Best
Code:
root@pve:~# fio --time_based --runtime=30 --ramp_time=5 --ioengine=io_uring --direct=1 \
--size=2G --filename=/testfile.fio --stonewall \
--name=SEQ1M_Q8T1 --bs=1m --rw=readwrite --iodepth=8 --numjobs=1 \
--name=SEQ1M_Q1T1 --bs=1m --rw=readwrite --iodepth=1 --numjobs=1 \
--name=RND4K_Q32T16 --bs=4k --rw=randrw --iodepth=32 --numjobs=16 \
--name=RND4K_Q1T1 --bs=4k --rw=randrw --iodepth=1 --numjobs=1
SEQ1M_Q8T1: (g=0): rw=rw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=io_uring, iodepth=8
SEQ1M_Q1T1: (g=1): rw=rw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=io_uring, iodepth=1
RND4K_Q32T16: (g=2): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=io_uring, iodepth=32
...
RND4K_Q1T1: (g=3): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=io_uring, iodepth=1
fio-3.39
Starting 19 processes
Jobs: 1 (f=1): [_(18),m(1)][20.7%][r=452KiB/s,w=440KiB/s][r=113,w=110 IOPS][eta 09m:00s]
SEQ1M_Q8T1: (groupid=0, jobs=1): err= 0: pid=3664122: Thu Sep 24 17:13:03 2026
read: IOPS=207, BW=207MiB/s (217MB/s)(6222MiB/30034msec)
slat (nsec): min=1885, max=30800, avg=5537.07, stdev=1875.48
clat (usec): min=1065, max=130075, avg=3672.11, stdev=7562.67
lat (usec): min=1069, max=130081, avg=3677.65, stdev=7562.73
clat percentiles (usec):
| 1.00th=[ 1221], 5.00th=[ 1352], 10.00th=[ 1450], 20.00th=[ 1614],
| 30.00th=[ 1778], 40.00th=[ 1942], 50.00th=[ 2147], 60.00th=[ 2376],
| 70.00th=[ 2737], 80.00th=[ 3228], 90.00th=[ 4113], 95.00th=[ 4883],
| 99.00th=[ 44827], 99.50th=[ 49546], 99.90th=[ 65799], 99.95th=[ 92799],
| 99.99th=[130548]
bw ( KiB/s): min=147751, max=319488, per=100.00%, avg=212509.13, stdev=32473.74, samples=60
iops : min= 144, max= 312, avg=207.40, stdev=31.71, samples=60
write: IOPS=206, BW=207MiB/s (217MB/s)(6224MiB/30034msec); 0 zone resets
slat (usec): min=19, max=633, avg=46.01, stdev=15.23
clat (usec): min=5162, max=63420, avg=34889.68, stdev=5556.12
lat (usec): min=5213, max=63467, avg=34935.69, stdev=5555.36
clat percentiles (usec):
| 1.00th=[14091], 5.00th=[25297], 10.00th=[28967], 20.00th=[31589],
| 30.00th=[33162], 40.00th=[34866], 50.00th=[35390], 60.00th=[36439],
| 70.00th=[38011], 80.00th=[39584], 90.00th=[40633], 95.00th=[42730],
| 99.00th=[43779], 99.50th=[43779], 99.90th=[44303], 99.95th=[50070],
| 99.99th=[63177]
bw ( KiB/s): min=186368, max=231424, per=100.00%, avg=212308.03, stdev=12528.14, samples=60
iops : min= 182, max= 226, avg=207.20, stdev=12.21, samples=60
lat (msec) : 2=21.59%, 4=22.97%, 10=4.04%, 20=1.33%, 50=49.86%
lat (msec) : 100=0.25%, 250=0.02%
cpu : usr=1.13%, sys=0.82%, ctx=12486, majf=0, minf=1077
IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=6222,6217,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
SEQ1M_Q1T1: (groupid=1, jobs=1): err= 0: pid=3665476: Thu Sep 24 17:13:03 2026
read: IOPS=117, BW=118MiB/s (124MB/s)(3536MiB/30003msec)
slat (nsec): min=2794, max=29823, avg=6396.77, stdev=2163.23
clat (usec): min=1083, max=104943, avg=3652.46, stdev=7789.39
lat (usec): min=1087, max=104948, avg=3658.85, stdev=7789.26
clat percentiles (usec):
| 1.00th=[ 1188], 5.00th=[ 1287], 10.00th=[ 1369], 20.00th=[ 1500],
| 30.00th=[ 1631], 40.00th=[ 1745], 50.00th=[ 1909], 60.00th=[ 2114],
| 70.00th=[ 2376], 80.00th=[ 2769], 90.00th=[ 3621], 95.00th=[ 5080],
| 99.00th=[ 40633], 99.50th=[ 45876], 99.90th=[ 84411], 99.95th=[ 91751],
| 99.99th=[105382]
bw ( KiB/s): min=63488, max=161792, per=100.00%, avg=120730.18, stdev=23058.20, samples=60
iops : min= 62, max= 158, avg=117.83, stdev=22.51, samples=60
write: IOPS=121, BW=122MiB/s (128MB/s)(3660MiB/30003msec); 0 zone resets
slat (nsec): min=12292, max=89468, avg=42617.62, stdev=8771.97
clat (usec): min=4253, max=6125, avg=4609.41, stdev=313.60
lat (usec): min=4292, max=6160, avg=4652.03, stdev=314.27
clat percentiles (usec):
| 1.00th=[ 4293], 5.00th=[ 4293], 10.00th=[ 4359], 20.00th=[ 4359],
| 30.00th=[ 4359], 40.00th=[ 4424], 50.00th=[ 4424], 60.00th=[ 4490],
| 70.00th=[ 5014], 80.00th=[ 5014], 90.00th=[ 5080], 95.00th=[ 5080],
| 99.00th=[ 5145], 99.50th=[ 5145], 99.90th=[ 5800], 99.95th=[ 6128],
| 99.99th=[ 6128]
bw ( KiB/s): min=79872, max=167936, per=100.00%, avg=124966.12, stdev=23560.99, samples=60
iops : min= 78, max= 164, avg=121.97, stdev=22.99, samples=60
lat (msec) : 2=26.81%, 4=18.58%, 10=52.40%, 20=0.04%, 50=1.96%
lat (msec) : 100=0.19%, 250=0.01%
cpu : usr=0.66%, sys=0.52%, ctx=7198, majf=0, minf=47
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=3536,3660,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
RND4K_Q32T16: (groupid=2, jobs=1): err= 0: pid=3666586: Thu Sep 24 17:13:03 2026
read: IOPS=3499, BW=13.7MiB/s (14.3MB/s)(411MiB/30045msec)
slat (nsec): min=1886, max=357593, avg=5745.67, stdev=3845.16
clat (nsec): min=1956, max=171199k, avg=214165.86, stdev=2469893.77
lat (usec): min=16, max=171204, avg=219.91, stdev=2469.91
clat percentiles (usec):
| 1.00th=[ 25], 5.00th=[ 34], 10.00th=[ 38], 20.00th=[ 44],
| 30.00th=[ 50], 40.00th=[ 58], 50.00th=[ 71], 60.00th=[ 88],
| 70.00th=[ 111], 80.00th=[ 135], 90.00th=[ 169], 95.00th=[ 245],
| 99.00th=[ 2507], 99.50th=[ 4080], 99.90th=[ 17957], 99.95th=[ 74974],
| 99.99th=[113771]
bw ( KiB/s): min= 3264, max=44168, per=6.40%, avg=14016.98, stdev=9429.80, samples=60
iops : min= 816, max=11042, avg=3504.12, stdev=2357.50, samples=60
write: IOPS=3485, BW=13.6MiB/s (14.3MB/s)(409MiB/30045msec); 0 zone resets
slat (usec): min=2, max=394, avg= 6.17, stdev= 3.70
clat (usec): min=478, max=187077, avg=8946.35, stdev=14988.80
lat (usec): min=485, max=187083, avg=8952.52, stdev=14988.79
clat percentiles (usec):
| 1.00th=[ 725], 5.00th=[ 988], 10.00th=[ 1500], 20.00th=[ 2442],
| 30.00th=[ 2999], 40.00th=[ 3458], 50.00th=[ 3916], 60.00th=[ 5014],
| 70.00th=[ 7242], 80.00th=[ 12387], 90.00th=[ 20317], 95.00th=[ 26084],
| 99.00th=[ 91751], 99.50th=[104334], 99.90th=[143655], 99.95th=[175113],
| 99.99th=[185598]
bw ( KiB/s): min= 3160, max=42877, per=6.37%, avg=13961.23, stdev=9282.05, samples=60
iops : min= 790, max=10719, avg=3490.18, stdev=2320.53, samples=60
lat (usec) : 2=0.01%, 10=0.01%, 20=0.16%, 50=14.95%, 100=17.83%
lat (usec) : 250=14.71%, 500=0.99%, 750=0.97%, 1000=2.17%
lat (msec) : 2=4.84%, 4=18.86%, 10=12.48%, 20=6.80%, 50=4.11%
lat (msec) : 100=0.80%, 250=0.33%
cpu : usr=2.26%, sys=11.21%, ctx=133674, majf=0, minf=228
IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued rwts: total=105132,104721,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=32
RND4K_Q32T16: (groupid=2, jobs=1): err= 0: pid=3666587: Thu Sep 24 17:13:03 2026
read: IOPS=3436, BW=13.4MiB/s (14.1MB/s)(404MiB/30069msec)
slat (nsec): min=1816, max=623064, avg=5699.38, stdev=4942.11
clat (usec): min=3, max=158882, avg=216.32, stdev=2342.68
lat (usec): min=16, max=158889, avg=222.02, stdev=2342.70
clat percentiles (usec):
| 1.00th=[ 25], 5.00th=[ 34], 10.00th=[ 39], 20.00th=[ 45],
| 30.00th=[ 50], 40.00th=[ 59], 50.00th=[ 72], 60.00th=[ 90],
| 70.00th=[ 112], 80.00th=[ 135], 90.00th=[ 174], 95.00th=[ 265],
| 99.00th=[ 2769], 99.50th=[ 4293], 99.90th=[16057], 99.95th=[69731],
| 99.99th=[99091]
bw ( KiB/s): min= 2845, max=33539, per=6.29%, avg=13779.25, stdev=9068.70, samples=60
iops : min= 711, max= 8384, avg=3444.67, stdev=2267.20, samples=60
write: IOPS=3449, BW=13.5MiB/s (14.1MB/s)(405MiB/30069msec); 0 zone resets
slat (usec): min=2, max=1641, avg= 6.11, stdev= 6.82
clat (usec): min=390, max=214204, avg=9040.72, stdev=15173.79
lat (usec): min=399, max=214209, avg=9046.84, stdev=15173.79
clat percentiles (usec):
| 1.00th=[ 742], 5.00th=[ 1057], 10.00th=[ 1614], 20.00th=[ 2474],
| 30.00th=[ 2999], 40.00th=[ 3425], 50.00th=[ 3884], 60.00th=[ 5080],
| 70.00th=[ 7635], 80.00th=[ 12911], 90.00th=[ 20579], 95.00th=[ 25560],
| 99.00th=[ 92799], 99.50th=[107480], 99.90th=[158335], 99.95th=[168821],
| 99.99th=[208667]
bw ( KiB/s): min= 2744, max=34693, per=6.31%, avg=13830.60, stdev=9100.75, samples=60
iops : min= 686, max= 8673, avg=3457.52, stdev=2275.21, samples=60
lat (usec) : 4=0.01%, 10=0.01%, 20=0.18%, 50=14.62%, 100=17.61%
lat (usec) : 250=14.85%, 500=1.04%, 750=0.93%, 1000=1.90%
lat (msec) : 2=4.98%, 4=19.52%, 10=11.95%, 20=7.06%, 50=4.29%
lat (msec) : 100=0.73%, 250=0.36%
cpu : usr=2.09%, sys=11.20%, ctx=134814, majf=0, minf=265
IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued rwts: total=103332,103716,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=32
RND4K_Q32T16: (groupid=2, jobs=1): err= 0: pid=3666588: Thu Sep 24 17:13:03 2026
read: IOPS=3496, BW=13.7MiB/s (14.3MB/s)(410MiB/30018msec)
slat (nsec): min=1816, max=562021, avg=5896.69, stdev=5507.82
clat (usec): min=3, max=161412, avg=204.35, stdev=2080.08
lat (usec): min=16, max=161422, avg=210.25, stdev=2080.09
clat percentiles (usec):
| 1.00th=[ 25], 5.00th=[ 35], 10.00th=[ 39], 20.00th=[ 45],
| 30.00th=[ 51], 40.00th=[ 61], 50.00th=[ 75], 60.00th=[ 93],
| 70.00th=[ 114], 80.00th=[ 139], 90.00th=[ 182], 95.00th=[ 289],
| 99.00th=[ 2606], 99.50th=[ 4113], 99.90th=[11207], 99.95th=[48497],
| 99.99th=[93848]
bw ( KiB/s): min= 2709, max=38192, per=6.39%, avg=13996.33, stdev=9482.97, samples=60
iops : min= 677, max= 9548, avg=3498.93, stdev=2370.81, samples=60
write: IOPS=3503, BW=13.7MiB/s (14.4MB/s)(411MiB/30018msec); 0 zone resets
slat (usec): min=2, max=1118, avg= 6.36, stdev= 7.01
clat (usec): min=362, max=277522, avg=8908.03, stdev=14733.70
lat (usec): min=380, max=277527, avg=8914.39, stdev=14733.69
clat percentiles (usec):
| 1.00th=[ 758], 5.00th=[ 1057], 10.00th=[ 1647], 20.00th=[ 2442],
| 30.00th=[ 2966], 40.00th=[ 3425], 50.00th=[ 3916], 60.00th=[ 5014],
| 70.00th=[ 7242], 80.00th=[ 12518], 90.00th=[ 20579], 95.00th=[ 26084],
| 99.00th=[ 90702], 99.50th=[104334], 99.90th=[139461], 99.95th=[177210],
| 99.99th=[196084]
bw ( KiB/s): min= 2765, max=38440, per=6.40%, avg=14025.95, stdev=9514.17, samples=60
iops : min= 691, max= 9610, avg=3506.35, stdev=2378.60, samples=60
lat (usec) : 4=0.01%, 10=0.01%, 20=0.16%, 50=14.06%, 100=17.67%
lat (usec) : 250=15.08%, 500=1.34%, 750=0.87%, 1000=2.01%
lat (msec) : 2=4.94%, 4=19.36%, 10=12.39%, 20=6.76%, 50=4.23%
lat (msec) : 100=0.81%, 250=0.32%, 500=0.01%
cpu : usr=2.47%, sys=11.18%, ctx=134615, majf=0, minf=246
IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued rwts: total=104969,105164,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=32
RND4K_Q32T16: (groupid=2, jobs=1): err= 0: pid=3666589: Thu Sep 24 17:13:03 2026
read: IOPS=3553, BW=13.9MiB/s (14.6MB/s)(417MiB/30015msec)
slat (nsec): min=1885, max=349212, avg=5618.59, stdev=3389.50
clat (usec): min=4, max=142434, avg=207.14, stdev=2286.33
lat (usec): min=15, max=142440, avg=212.76, stdev=2286.33
clat percentiles (usec):
| 1.00th=[ 24], 5.00th=[ 34], 10.00th=[ 38], 20.00th=[ 44],
| 30.00th=[ 50], 40.00th=[ 59], 50.00th=[ 72], 60.00th=[ 89],
| 70.00th=[ 111], 80.00th=[ 135], 90.00th=[ 172], 95.00th=[ 255],
| 99.00th=[ 2573], 99.50th=[ 4080], 99.90th=[ 9896], 99.95th=[ 70779],
| 99.99th=[101188]
bw ( KiB/s): min= 3248, max=37563, per=6.49%, avg=14223.85, stdev=9746.95, samples=60
iops : min= 812, max= 9390, avg=3555.80, stdev=2436.81, samples=60
write: IOPS=3549, BW=13.9MiB/s (14.5MB/s)(416MiB/30015msec); 0 zone resets
slat (nsec): min=1886, max=567819, avg=6055.89, stdev=3865.05
clat (usec): min=340, max=199801, avg=8785.84, stdev=14626.99
lat (usec): min=362, max=199808, avg=8791.90, stdev=14626.95
clat percentiles (usec):
| 1.00th=[ 693], 5.00th=[ 955], 10.00th=[ 1401], 20.00th=[ 2278],
| 30.00th=[ 2868], 40.00th=[ 3359], 50.00th=[ 3785], 60.00th=[ 4817],
| 70.00th=[ 7046], 80.00th=[ 12387], 90.00th=[ 20579], 95.00th=[ 26084],
| 99.00th=[ 89654], 99.50th=[102237], 99.90th=[139461], 99.95th=[149947],
| 99.99th=[191890]
bw ( KiB/s): min= 3088, max=36817, per=6.48%, avg=14213.55, stdev=9605.88, samples=60
iops : min= 772, max= 9204, avg=3553.27, stdev=2401.53, samples=60
lat (usec) : 10=0.01%, 20=0.19%, 50=14.72%, 100=17.87%, 250=14.70%
lat (usec) : 500=1.07%, 750=1.14%, 1000=2.30%
lat (msec) : 2=5.46%, 4=19.10%, 10=11.72%, 20=6.30%, 50=4.34%
lat (msec) : 100=0.81%, 250=0.31%
cpu : usr=2.26%, sys=11.27%, ctx=134023, majf=0, minf=256
IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued rwts: total=106661,106552,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=32
RND4K_Q32T16: (groupid=2, jobs=1): err= 0: pid=3666590: Thu Sep 24 17:13:03 2026
read: IOPS=3207, BW=12.5MiB/s (13.1MB/s)(376MiB/30042msec)
slat (nsec): min=1886, max=656727, avg=5848.89, stdev=4586.04
clat (usec): min=3, max=189538, avg=209.78, stdev=2332.41
lat (usec): min=16, max=189542, avg=215.63, stdev=2332.41
clat percentiles (usec):
| 1.00th=[ 25], 5.00th=[ 34], 10.00th=[ 39], 20.00th=[ 45],
| 30.00th=[ 51], 40.00th=[ 60], 50.00th=[ 74], 60.00th=[ 92],
| 70.00th=[ 114], 80.00th=[ 137], 90.00th=[ 174], 95.00th=[ 262],
| 99.00th=[ 2540], 99.50th=[ 4047], 99.90th=[ 11469], 99.95th=[ 66323],
| 99.99th=[102237]
bw ( KiB/s): min= 3656, max=33912, per=5.87%, avg=12850.42, stdev=8451.76, samples=60
iops : min= 914, max= 8478, avg=3212.45, stdev=2112.91, samples=60
write: IOPS=3206, BW=12.5MiB/s (13.1MB/s)(376MiB/30042msec); 0 zone resets
slat (nsec): min=1956, max=2787.6k, avg=6283.83, stdev=9885.63
clat (usec): min=380, max=270625, avg=9753.35, stdev=16849.45
lat (usec): min=387, max=270629, avg=9759.63, stdev=16849.41
clat percentiles (usec):
| 1.00th=[ 742], 5.00th=[ 1004], 10.00th=[ 1631], 20.00th=[ 2540],
| 30.00th=[ 3064], 40.00th=[ 3490], 50.00th=[ 3916], 60.00th=[ 5080],
| 70.00th=[ 7635], 80.00th=[ 13960], 90.00th=[ 21627], 95.00th=[ 27919],
| 99.00th=[ 99091], 99.50th=[108528], 99.90th=[164627], 99.95th=[177210],
| 99.99th=[270533]
bw ( KiB/s): min= 3616, max=33504, per=5.86%, avg=12844.68, stdev=8443.95, samples=60
iops : min= 904, max= 8376, avg=3211.02, stdev=2111.03, samples=60
lat (usec) : 4=0.01%, 10=0.01%, 20=0.16%, 50=14.27%, 100=17.59%
lat (usec) : 250=15.35%, 500=1.09%, 750=0.91%, 1000=2.16%
lat (msec) : 2=4.34%, 4=19.47%, 10=11.80%, 20=6.76%, 50=4.64%
lat (msec) : 100=0.98%, 250=0.47%, 500=0.01%
cpu : usr=1.93%, sys=10.61%, ctx=123744, majf=0, minf=245
IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued rwts: total=96359,96318,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=32
RND4K_Q32T16: (groupid=2, jobs=1): err= 0: pid=3666591: Thu Sep 24 17:13:03 2026
read: IOPS=3560, BW=13.9MiB/s (14.6MB/s)(417MiB/30015msec)
slat (nsec): min=1886, max=544561, avg=5744.24, stdev=4511.81
clat (usec): min=3, max=141924, avg=204.56, stdev=2166.70
lat (usec): min=17, max=141931, avg=210.31, stdev=2166.70
clat percentiles (usec):
| 1.00th=[ 25], 5.00th=[ 35], 10.00th=[ 39], 20.00th=[ 45],
| 30.00th=[ 51], 40.00th=[ 59], 50.00th=[ 73], 60.00th=[ 90],
| 70.00th=[ 112], 80.00th=[ 135], 90.00th=[ 174], 95.00th=[ 262],
| 99.00th=[ 2540], 99.50th=[ 4080], 99.90th=[11469], 99.95th=[60556],
| 99.99th=[99091]
bw ( KiB/s): min= 2597, max=40304, per=6.51%, avg=14250.35, stdev=9755.99, samples=60
iops : min= 649, max=10076, avg=3562.45, stdev=2439.00, samples=60
write: IOPS=3556, BW=13.9MiB/s (14.6MB/s)(417MiB/30015msec); 0 zone resets
slat (nsec): min=1816, max=424362, avg=6123.52, stdev=3690.32
clat (usec): min=335, max=253125, avg=8790.95, stdev=15033.96
lat (usec): min=340, max=253130, avg=8797.07, stdev=15033.97
clat percentiles (usec):
| 1.00th=[ 693], 5.00th=[ 1004], 10.00th=[ 1598], 20.00th=[ 2409],
| 30.00th=[ 2933], 40.00th=[ 3392], 50.00th=[ 3851], 60.00th=[ 4948],
| 70.00th=[ 7308], 80.00th=[ 12518], 90.00th=[ 20317], 95.00th=[ 25035],
| 99.00th=[ 89654], 99.50th=[111674], 99.90th=[164627], 99.95th=[198181],
| 99.99th=[217056]
bw ( KiB/s): min= 2300, max=39296, per=6.50%, avg=14238.85, stdev=9682.15, samples=60
iops : min= 575, max= 9824, avg=3559.60, stdev=2420.55, samples=60
lat (usec) : 4=0.01%, 10=0.01%, 20=0.15%, 50=14.30%, 100=18.04%
lat (usec) : 250=14.91%, 500=1.11%, 750=1.09%, 1000=1.99%
lat (msec) : 2=4.93%, 4=19.38%, 10=12.03%, 20=6.91%, 50=4.16%
lat (msec) : 100=0.66%, 250=0.34%, 500=0.01%
cpu : usr=2.15%, sys=11.69%, ctx=138269, majf=0, minf=242
IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued rwts: total=106864,106747,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=32
RND4K_Q32T16: (groupid=2, jobs=1): err= 0: pid=3666592: Thu Sep 24 17:13:03 2026
read: IOPS=3221, BW=12.6MiB/s (13.2MB/s)(378MiB/30016msec)
slat (nsec): min=1466, max=966548, avg=5798.37, stdev=4876.93
clat (usec): min=3, max=140069, avg=216.76, stdev=2260.75
lat (usec): min=14, max=140077, avg=222.56, stdev=2260.75
clat percentiles (usec):
| 1.00th=[ 25], 5.00th=[ 35], 10.00th=[ 39], 20.00th=[ 45],
| 30.00th=[ 51], 40.00th=[ 60], 50.00th=[ 75], 60.00th=[ 94],
| 70.00th=[ 116], 80.00th=[ 139], 90.00th=[ 180], 95.00th=[ 273],
| 99.00th=[ 2704], 99.50th=[ 4146], 99.90th=[18220], 99.95th=[68682],
| 99.99th=[98042]
bw ( KiB/s): min= 3672, max=30152, per=5.89%, avg=12896.05, stdev=7609.87, samples=60
iops : min= 918, max= 7538, avg=3223.87, stdev=1902.50, samples=60
write: IOPS=3254, BW=12.7MiB/s (13.3MB/s)(382MiB/30016msec); 0 zone resets
slat (usec): min=2, max=641, avg= 6.22, stdev= 4.63
clat (usec): min=332, max=242928, avg=9606.26, stdev=16131.99
lat (usec): min=336, max=242932, avg=9612.47, stdev=16131.97
clat percentiles (usec):
| 1.00th=[ 701], 5.00th=[ 1106], 10.00th=[ 1844], 20.00th=[ 2606],
| 30.00th=[ 3163], 40.00th=[ 3556], 50.00th=[ 3982], 60.00th=[ 5276],
| 70.00th=[ 8225], 80.00th=[ 13960], 90.00th=[ 20841], 95.00th=[ 26608],
| 99.00th=[ 94897], 99.50th=[108528], 99.90th=[154141], 99.95th=[181404],
| 99.99th=[238027]
bw ( KiB/s): min= 3672, max=29170, per=5.94%, avg=13029.57, stdev=7634.26, samples=60
iops : min= 918, max= 7292, avg=3257.23, stdev=1908.59, samples=60
lat (usec) : 4=0.01%, 20=0.15%, 50=14.01%, 100=17.18%, 250=15.61%
lat (usec) : 500=1.23%, 750=1.06%, 1000=1.61%
lat (msec) : 2=3.97%, 4=19.89%, 10=12.14%, 20=7.44%, 50=4.43%
lat (msec) : 100=0.92%, 250=0.37%
cpu : usr=2.03%, sys=10.74%, ctx=127725, majf=0, minf=274
IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued rwts: total=96711,97689,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=32
RND4K_Q32T16: (groupid=2, jobs=1): err= 0: pid=3666593: Thu Sep 24 17:13:03 2026
read: IOPS=3367, BW=13.2MiB/s (13.8MB/s)(396MiB/30082msec)
slat (nsec): min=1815, max=1963.4k, avg=6034.93, stdev=9442.41
clat (nsec): min=1886, max=156086k, avg=217625.39, stdev=2334139.10
lat (usec): min=14, max=156089, avg=223.66, stdev=2334.15
clat percentiles (usec):
| 1.00th=[ 25], 5.00th=[ 35], 10.00th=[ 39], 20.00th=[ 45],
| 30.00th=[ 52], 40.00th=[ 61], 50.00th=[ 76], 60.00th=[ 95],
| 70.00th=[ 117], 80.00th=[ 141], 90.00th=[ 186], 95.00th=[ 297],
| 99.00th=[ 2769], 99.50th=[ 4228], 99.90th=[ 8979], 99.95th=[ 63701],
| 99.99th=[101188]
bw ( KiB/s): min= 3920, max=35448, per=6.17%, avg=13506.83, stdev=8590.06, samples=60
iops : min= 980, max= 8862, avg=3376.57, stdev=2147.55, samples=60
write: IOPS=3351, BW=13.1MiB/s (13.7MB/s)(394MiB/30082msec); 0 zone resets
slat (nsec): min=1886, max=2305.6k, avg=6418.36, stdev=8800.13
clat (usec): min=276, max=258723, avg=9331.98, stdev=15020.92
lat (usec): min=281, max=258728, avg=9338.40, stdev=15020.89
clat percentiles (usec):
| 1.00th=[ 750], 5.00th=[ 1156], 10.00th=[ 1811], 20.00th=[ 2540],
| 30.00th=[ 3064], 40.00th=[ 3523], 50.00th=[ 4015], 60.00th=[ 5538],
| 70.00th=[ 8455], 80.00th=[ 13960], 90.00th=[ 21103], 95.00th=[ 26346],
| 99.00th=[ 88605], 99.50th=[102237], 99.90th=[175113], 99.95th=[191890],
| 99.99th=[256902]
bw ( KiB/s): min= 4200, max=35088, per=6.13%, avg=13443.90, stdev=8532.64, samples=60
iops : min= 1050, max= 8772, avg=3360.85, stdev=2133.23, samples=60
lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.15%, 50=13.85%
lat (usec) : 100=17.46%, 250=15.57%, 500=1.37%, 750=0.90%, 1000=1.54%
lat (msec) : 2=4.69%, 4=19.04%, 10=12.21%, 20=7.41%, 50=4.68%
lat (msec) : 100=0.85%, 250=0.26%, 500=0.02%
cpu : usr=2.34%, sys=11.07%, ctx=133450, majf=0, minf=341
IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued rwts: total=101292,100822,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=32
RND4K_Q32T16: (groupid=2, jobs=1): err= 0: pid=3666594: Thu Sep 24 17:13:03 2026
read: IOPS=3475, BW=13.6MiB/s (14.2MB/s)(407MiB/30011msec)
slat (nsec): min=1956, max=562511, avg=5756.10, stdev=4235.63
clat (usec): min=3, max=273602, avg=229.76, stdev=2693.05
lat (usec): min=17, max=273607, avg=235.52, stdev=2693.04
clat percentiles (usec):
| 1.00th=[ 25], 5.00th=[ 34], 10.00th=[ 38], 20.00th=[ 44],
| 30.00th=[ 50], 40.00th=[ 58], 50.00th=[ 70], 60.00th=[ 88],
| 70.00th=[ 111], 80.00th=[ 135], 90.00th=[ 176], 95.00th=[ 265],
| 99.00th=[ 2802], 99.50th=[ 4228], 99.90th=[ 33162], 99.95th=[ 76022],
| 99.99th=[106431]
bw ( KiB/s): min= 3624, max=38472, per=6.35%, avg=13908.55, stdev=9154.12, samples=60
iops : min= 906, max= 9618, avg=3477.02, stdev=2288.57, samples=60
write: IOPS=3485, BW=13.6MiB/s (14.3MB/s)(409MiB/30011msec); 0 zone resets
slat (usec): min=2, max=2318, avg= 6.20, stdev= 8.56
clat (usec): min=382, max=154913, avg=8939.56, stdev=14337.25
lat (usec): min=392, max=154918, avg=8945.76, stdev=14337.21
clat percentiles (usec):
| 1.00th=[ 693], 5.00th=[ 988], 10.00th=[ 1483], 20.00th=[ 2474],
| 30.00th=[ 3032], 40.00th=[ 3458], 50.00th=[ 3916], 60.00th=[ 5080],
| 70.00th=[ 7701], 80.00th=[ 13042], 90.00th=[ 20317], 95.00th=[ 25822],
| 99.00th=[ 89654], 99.50th=[102237], 99.90th=[143655], 99.95th=[147850],
| 99.99th=[154141]
bw ( KiB/s): min= 3256, max=37712, per=6.36%, avg=13949.73, stdev=9153.46, samples=60
iops : min= 814, max= 9428, avg=3487.30, stdev=2288.41, samples=60
lat (usec) : 4=0.01%, 10=0.01%, 20=0.18%, 50=14.92%, 100=17.70%
lat (usec) : 250=14.46%, 500=1.15%, 750=1.12%, 1000=2.04%
lat (msec) : 2=4.90%, 4=18.96%, 10=12.07%, 20=7.24%, 50=4.18%
lat (msec) : 100=0.81%, 250=0.29%, 500=0.01%
cpu : usr=2.09%, sys=11.43%, ctx=134473, majf=0, minf=279
IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued rwts: total=104317,104598,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=32
RND4K_Q32T16: (groupid=2, jobs=1): err= 0: pid=3666595: Thu Sep 24 17:13:03 2026
read: IOPS=3286, BW=12.8MiB/s (13.5MB/s)(385MiB/30021msec)
slat (usec): min=2, max=396, avg= 5.79, stdev= 4.05
clat (usec): min=3, max=183173, avg=221.68, stdev=2441.54
lat (usec): min=16, max=183177, avg=227.47, stdev=2441.54
clat percentiles (usec):
| 1.00th=[ 25], 5.00th=[ 35], 10.00th=[ 39], 20.00th=[ 45],
| 30.00th=[ 51], 40.00th=[ 60], 50.00th=[ 75], 60.00th=[ 93],
| 70.00th=[ 115], 80.00th=[ 139], 90.00th=[ 180], 95.00th=[ 265],
| 99.00th=[ 2835], 99.50th=[ 4293], 99.90th=[ 20841], 99.95th=[ 67634],
| 99.99th=[109577]
bw ( KiB/s): min= 3062, max=36736, per=6.01%, avg=13155.65, stdev=9094.15, samples=60
iops : min= 765, max= 9184, avg=3288.77, stdev=2273.60, samples=60
write: IOPS=3302, BW=12.9MiB/s (13.5MB/s)(387MiB/30021msec); 0 zone resets
slat (usec): min=2, max=670, avg= 6.22, stdev= 4.77
clat (usec): min=187, max=203795, avg=9445.49, stdev=15555.00
lat (usec): min=191, max=203800, avg=9451.71, stdev=15555.00
clat percentiles (usec):
| 1.00th=[ 775], 5.00th=[ 1156], 10.00th=[ 1696], 20.00th=[ 2540],
| 30.00th=[ 3064], 40.00th=[ 3458], 50.00th=[ 3884], 60.00th=[ 5145],
| 70.00th=[ 7898], 80.00th=[ 13566], 90.00th=[ 21103], 95.00th=[ 27395],
| 99.00th=[ 95945], 99.50th=[105382], 99.90th=[137364], 99.95th=[164627],
| 99.99th=[200279]
bw ( KiB/s): min= 3575, max=37520, per=6.03%, avg=13226.02, stdev=9135.50, samples=60
iops : min= 893, max= 9380, avg=3306.40, stdev=2283.91, samples=60
lat (usec) : 4=0.01%, 10=0.01%, 20=0.16%, 50=14.09%, 100=17.35%
lat (usec) : 250=15.57%, 500=1.18%, 750=0.71%, 1000=1.71%
lat (msec) : 2=5.11%, 4=19.70%, 10=11.42%, 20=7.21%, 50=4.50%
lat (msec) : 100=0.93%, 250=0.37%
cpu : usr=2.24%, sys=10.72%, ctx=130150, majf=0, minf=241
IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued rwts: total=98657,99157,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=32
RND4K_Q32T16: (groupid=2, jobs=1): err= 0: pid=3666596: Thu Sep 24 17:13:03 2026
read: IOPS=3341, BW=13.1MiB/s (13.7MB/s)(392MiB/30010msec)
slat (nsec): min=1886, max=3315.2k, avg=5997.87, stdev=11998.10
clat (nsec): min=1885, max=258671k, avg=216319.22, stdev=2554218.14
lat (usec): min=16, max=258680, avg=222.32, stdev=2554.29
clat percentiles (usec):
| 1.00th=[ 25], 5.00th=[ 35], 10.00th=[ 39], 20.00th=[ 45],
| 30.00th=[ 51], 40.00th=[ 61], 50.00th=[ 75], 60.00th=[ 94],
| 70.00th=[ 116], 80.00th=[ 141], 90.00th=[ 184], 95.00th=[ 289],
| 99.00th=[ 2573], 99.50th=[ 4080], 99.90th=[ 8979], 99.95th=[ 64750],
| 99.99th=[109577]
bw ( KiB/s): min= 3664, max=35664, per=6.11%, avg=13375.27, stdev=8555.43, samples=60
iops : min= 916, max= 8916, avg=3343.73, stdev=2138.88, samples=60
write: IOPS=3356, BW=13.1MiB/s (13.8MB/s)(394MiB/30010msec); 0 zone resets
slat (usec): min=2, max=995, avg= 6.36, stdev= 6.18
clat (usec): min=361, max=268754, avg=9321.04, stdev=14916.18
lat (usec): min=368, max=268759, avg=9327.40, stdev=14916.19
clat percentiles (usec):
| 1.00th=[ 832], 5.00th=[ 1237], 10.00th=[ 1811], 20.00th=[ 2573],
| 30.00th=[ 3097], 40.00th=[ 3556], 50.00th=[ 4015], 60.00th=[ 5473],
| 70.00th=[ 8586], 80.00th=[ 14091], 90.00th=[ 21103], 95.00th=[ 26346],
| 99.00th=[ 87557], 99.50th=[106431], 99.90th=[154141], 99.95th=[189793],
| 99.99th=[265290]
bw ( KiB/s): min= 3944, max=35504, per=6.13%, avg=13438.78, stdev=8656.23, samples=60
iops : min= 986, max= 8876, avg=3359.57, stdev=2164.08, samples=60
lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.17%, 50=13.89%
lat (usec) : 100=17.29%, 250=15.49%, 500=1.34%, 750=0.65%, 1000=1.47%
lat (msec) : 2=5.00%, 4=19.15%, 10=11.99%, 20=7.79%, 50=4.70%
lat (msec) : 100=0.72%, 250=0.31%, 500=0.01%
cpu : usr=2.19%, sys=11.13%, ctx=133484, majf=0, minf=231
IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued rwts: total=100290,100734,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=32
RND4K_Q32T16: (groupid=2, jobs=1): err= 0: pid=3666597: Thu Sep 24 17:13:03 2026
read: IOPS=3522, BW=13.8MiB/s (14.4MB/s)(413MiB/30010msec)
slat (usec): min=2, max=633, avg= 5.76, stdev= 5.80
clat (usec): min=2, max=266659, avg=223.97, stdev=2696.10
lat (usec): min=14, max=266663, avg=229.74, stdev=2696.09
clat percentiles (usec):
| 1.00th=[ 25], 5.00th=[ 35], 10.00th=[ 39], 20.00th=[ 45],
| 30.00th=[ 50], 40.00th=[ 59], 50.00th=[ 72], 60.00th=[ 90],
| 70.00th=[ 111], 80.00th=[ 135], 90.00th=[ 169], 95.00th=[ 255],
| 99.00th=[ 2606], 99.50th=[ 4015], 99.90th=[ 28705], 99.95th=[ 74974],
| 99.99th=[107480]
bw ( KiB/s): min= 3959, max=37370, per=6.44%, avg=14099.33, stdev=9243.02, samples=60
iops : min= 989, max= 9342, avg=3524.70, stdev=2310.77, samples=60
write: IOPS=3545, BW=13.9MiB/s (14.5MB/s)(416MiB/30010msec); 0 zone resets
slat (usec): min=2, max=515, avg= 6.17, stdev= 4.35
clat (usec): min=239, max=284389, avg=8836.06, stdev=14304.55
lat (usec): min=242, max=284394, avg=8842.23, stdev=14304.55
clat percentiles (usec):
| 1.00th=[ 709], 5.00th=[ 1004], 10.00th=[ 1565], 20.00th=[ 2442],
| 30.00th=[ 2999], 40.00th=[ 3458], 50.00th=[ 3916], 60.00th=[ 5145],
| 70.00th=[ 7832], 80.00th=[ 12911], 90.00th=[ 20579], 95.00th=[ 25560],
| 99.00th=[ 85459], 99.50th=[100140], 99.90th=[139461], 99.95th=[173016],
| 99.99th=[235930]
bw ( KiB/s): min= 4344, max=39006, per=6.48%, avg=14195.52, stdev=9347.34, samples=60
iops : min= 1086, max= 9751, avg=3548.73, stdev=2336.87, samples=60
lat (usec) : 4=0.01%, 10=0.01%, 20=0.16%, 50=14.52%, 100=17.88%
lat (usec) : 250=14.72%, 500=1.04%, 750=0.97%, 1000=2.08%
lat (msec) : 2=4.95%, 4=19.10%, 10=11.88%, 20=7.30%, 50=4.40%
lat (msec) : 100=0.73%, 250=0.26%, 500=0.01%
cpu : usr=2.19%, sys=11.60%, ctx=139155, majf=0, minf=282
IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued rwts: total=105712,106401,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=32
RND4K_Q32T16: (groupid=2, jobs=1): err= 0: pid=3666598: Thu Sep 24 17:13:03 2026
read: IOPS=3442, BW=13.4MiB/s (14.1MB/s)(404MiB/30012msec)
slat (nsec): min=1886, max=1131.2k, avg=5791.94, stdev=5181.83
clat (usec): min=4, max=140355, avg=213.46, stdev=2339.42
lat (usec): min=16, max=140360, avg=219.25, stdev=2339.42
clat percentiles (usec):
| 1.00th=[ 25], 5.00th=[ 34], 10.00th=[ 38], 20.00th=[ 44],
| 30.00th=[ 50], 40.00th=[ 58], 50.00th=[ 71], 60.00th=[ 89],
| 70.00th=[ 112], 80.00th=[ 135], 90.00th=[ 176], 95.00th=[ 262],
| 99.00th=[ 2606], 99.50th=[ 4047], 99.90th=[ 18482], 99.95th=[ 63177],
| 99.99th=[107480]
bw ( KiB/s): min= 3264, max=33928, per=6.29%, avg=13779.63, stdev=8850.53, samples=60
iops : min= 816, max= 8482, avg=3444.77, stdev=2212.64, samples=60
write: IOPS=3435, BW=13.4MiB/s (14.1MB/s)(403MiB/30012msec); 0 zone resets
slat (usec): min=2, max=332, avg= 6.17, stdev= 3.68
clat (usec): min=375, max=213354, avg=9100.29, stdev=15111.25
lat (usec): min=390, max=213362, avg=9106.46, stdev=15111.25
clat percentiles (usec):
| 1.00th=[ 701], 5.00th=[ 979], 10.00th=[ 1614], 20.00th=[ 2474],
| 30.00th=[ 3032], 40.00th=[ 3490], 50.00th=[ 3982], 60.00th=[ 5342],
| 70.00th=[ 7767], 80.00th=[ 12911], 90.00th=[ 20579], 95.00th=[ 26346],
| 99.00th=[ 91751], 99.50th=[105382], 99.90th=[158335], 99.95th=[164627],
| 99.99th=[210764]
bw ( KiB/s): min= 3448, max=34368, per=6.28%, avg=13756.97, stdev=8715.76, samples=60
iops : min= 862, max= 8592, avg=3439.08, stdev=2178.95, samples=60
lat (usec) : 10=0.01%, 20=0.17%, 50=15.02%, 100=17.51%, 250=14.69%
lat (usec) : 500=1.14%, 750=1.07%, 1000=2.14%
lat (msec) : 2=4.52%, 4=18.65%, 10=12.57%, 20=7.09%, 50=4.36%
lat (msec) : 100=0.71%, 250=0.37%
cpu : usr=2.28%, sys=11.11%, ctx=133267, majf=0, minf=192
IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued rwts: total=103317,103115,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=32
RND4K_Q32T16: (groupid=2, jobs=1): err= 0: pid=3666599: Thu Sep 24 17:13:03 2026
read: IOPS=3359, BW=13.1MiB/s (13.8MB/s)(394MiB/30008msec)
slat (nsec): min=1885, max=424921, avg=5739.68, stdev=4584.59
clat (nsec): min=1816, max=180381k, avg=217406.73, stdev=2380796.54
lat (usec): min=15, max=180386, avg=223.15, stdev=2380.80
clat percentiles (usec):
| 1.00th=[ 25], 5.00th=[ 35], 10.00th=[ 39], 20.00th=[ 45],
| 30.00th=[ 51], 40.00th=[ 60], 50.00th=[ 75], 60.00th=[ 93],
| 70.00th=[ 115], 80.00th=[ 139], 90.00th=[ 178], 95.00th=[ 273],
| 99.00th=[ 2769], 99.50th=[ 4178], 99.90th=[ 18220], 99.95th=[ 66847],
| 99.99th=[104334]
bw ( KiB/s): min= 2840, max=34536, per=6.14%, avg=13445.93, stdev=8391.27, samples=60
iops : min= 710, max= 8634, avg=3361.37, stdev=2097.87, samples=60
write: IOPS=3353, BW=13.1MiB/s (13.7MB/s)(393MiB/30008msec); 0 zone resets
slat (usec): min=2, max=754, avg= 6.15, stdev= 4.82
clat (usec): min=505, max=186449, avg=9343.92, stdev=14897.01
lat (usec): min=515, max=186453, avg=9350.07, stdev=14896.96
clat percentiles (usec):
| 1.00th=[ 783], 5.00th=[ 1188], 10.00th=[ 1844], 20.00th=[ 2573],
| 30.00th=[ 3064], 40.00th=[ 3490], 50.00th=[ 3916], 60.00th=[ 5342],
| 70.00th=[ 8586], 80.00th=[ 13698], 90.00th=[ 21103], 95.00th=[ 26608],
| 99.00th=[ 89654], 99.50th=[107480], 99.90th=[145753], 99.95th=[149947],
| 99.99th=[183501]
bw ( KiB/s): min= 2968, max=33568, per=6.12%, avg=13424.22, stdev=8298.43, samples=60
iops : min= 742, max= 8392, avg=3355.95, stdev=2074.66, samples=60
lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.15%, 50=14.15%
lat (usec) : 100=17.47%, 250=15.47%, 500=1.22%, 750=0.69%, 1000=1.60%
lat (msec) : 2=4.46%, 4=20.11%, 10=11.19%, 20=7.81%, 50=4.56%
lat (msec) : 100=0.81%, 250=0.33%
cpu : usr=2.14%, sys=11.05%, ctx=134928, majf=0, minf=207
IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued rwts: total=100818,100623,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=32
RND4K_Q32T16: (groupid=2, jobs=1): err= 0: pid=3666600: Thu Sep 24 17:13:03 2026
read: IOPS=3488, BW=13.6MiB/s (14.3MB/s)(409MiB/30013msec)
slat (nsec): min=1886, max=560136, avg=5739.44, stdev=4559.03
clat (usec): min=2, max=202697, avg=213.21, stdev=2461.10
lat (usec): min=16, max=202701, avg=218.95, stdev=2461.10
clat percentiles (usec):
| 1.00th=[ 25], 5.00th=[ 34], 10.00th=[ 38], 20.00th=[ 45],
| 30.00th=[ 51], 40.00th=[ 60], 50.00th=[ 74], 60.00th=[ 92],
| 70.00th=[ 114], 80.00th=[ 137], 90.00th=[ 176], 95.00th=[ 262],
| 99.00th=[ 2507], 99.50th=[ 4146], 99.90th=[ 15270], 99.95th=[ 68682],
| 99.99th=[114820]
bw ( KiB/s): min= 2925, max=37120, per=6.38%, avg=13965.20, stdev=9522.25, samples=60
iops : min= 731, max= 9280, avg=3491.17, stdev=2380.62, samples=60
write: IOPS=3456, BW=13.5MiB/s (14.2MB/s)(405MiB/30013msec); 0 zone resets
slat (nsec): min=1885, max=1330.4k, avg=6170.97, stdev=6417.24
clat (usec): min=377, max=213422, avg=9029.65, stdev=14853.32
lat (usec): min=383, max=213426, avg=9035.82, stdev=14853.31
clat percentiles (usec):
| 1.00th=[ 717], 5.00th=[ 1012], 10.00th=[ 1516], 20.00th=[ 2442],
| 30.00th=[ 3032], 40.00th=[ 3458], 50.00th=[ 3884], 60.00th=[ 4948],
| 70.00th=[ 7308], 80.00th=[ 13435], 90.00th=[ 20841], 95.00th=[ 26084],
| 99.00th=[ 92799], 99.50th=[101188], 99.90th=[137364], 99.95th=[170918],
| 99.99th=[212861]
bw ( KiB/s): min= 2917, max=35472, per=6.31%, avg=13838.30, stdev=9315.22, samples=60
iops : min= 729, max= 8868, avg=3459.43, stdev=2328.85, samples=60
lat (usec) : 4=0.01%, 10=0.01%, 20=0.18%, 50=14.55%, 100=17.48%
lat (usec) : 250=15.38%, 500=1.12%, 750=0.99%, 1000=2.03%
lat (msec) : 2=5.12%, 4=18.98%, 10=11.79%, 20=6.76%, 50=4.59%
lat (msec) : 100=0.76%, 250=0.29%
cpu : usr=2.19%, sys=11.23%, ctx=134628, majf=0, minf=257
IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued rwts: total=104705,103727,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=32
RND4K_Q32T16: (groupid=2, jobs=1): err= 0: pid=3666601: Thu Sep 24 17:13:03 2026
read: IOPS=3601, BW=14.1MiB/s (14.8MB/s)(422MiB/30008msec)
slat (nsec): min=1816, max=424362, avg=5629.31, stdev=4001.49
clat (usec): min=3, max=164130, avg=209.60, stdev=2259.24
lat (usec): min=15, max=164134, avg=215.23, stdev=2259.24
clat percentiles (usec):
| 1.00th=[ 25], 5.00th=[ 34], 10.00th=[ 38], 20.00th=[ 44],
| 30.00th=[ 50], 40.00th=[ 58], 50.00th=[ 71], 60.00th=[ 88],
| 70.00th=[ 109], 80.00th=[ 133], 90.00th=[ 172], 95.00th=[ 251],
| 99.00th=[ 2573], 99.50th=[ 4113], 99.90th=[ 17695], 99.95th=[ 64226],
| 99.99th=[100140]
bw ( KiB/s): min= 2492, max=36176, per=6.58%, avg=14418.30, stdev=9848.89, samples=60
iops : min= 623, max= 9044, avg=3604.47, stdev=2462.28, samples=60
write: IOPS=3593, BW=14.0MiB/s (14.7MB/s)(421MiB/30008msec); 0 zone resets
slat (usec): min=2, max=695, avg= 6.06, stdev= 4.44
clat (usec): min=412, max=256891, avg=8700.35, stdev=14373.71
lat (usec): min=435, max=256895, avg=8706.41, stdev=14373.74
clat percentiles (usec):
| 1.00th=[ 750], 5.00th=[ 1029], 10.00th=[ 1516], 20.00th=[ 2376],
| 30.00th=[ 2933], 40.00th=[ 3425], 50.00th=[ 3884], 60.00th=[ 5080],
| 70.00th=[ 7570], 80.00th=[ 12911], 90.00th=[ 19792], 95.00th=[ 25035],
| 99.00th=[ 88605], 99.50th=[103285], 99.90th=[154141], 99.95th=[179307],
| 99.99th=[252707]
bw ( KiB/s): min= 2837, max=36224, per=6.56%, avg=14383.68, stdev=9781.13, samples=60
iops : min= 709, max= 9056, avg=3595.78, stdev=2445.36, samples=60
lat (usec) : 4=0.01%, 10=0.01%, 20=0.15%, 50=15.04%, 100=17.95%
lat (usec) : 250=14.40%, 500=1.00%, 750=0.82%, 1000=2.09%
lat (msec) : 2=5.47%, 4=18.85%, 10=11.89%, 20=7.38%, 50=4.11%
lat (msec) : 100=0.59%, 250=0.29%, 500=0.01%
cpu : usr=2.30%, sys=11.53%, ctx=142187, majf=0, minf=261
IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued rwts: total=108080,107821,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=32
RND4K_Q1T1: (groupid=3, jobs=1): err= 0: pid=3668147: Thu Sep 24 17:13:03 2026
read: IOPS=107, BW=431KiB/s (441kB/s)(12.6MiB/30001msec)
slat (nsec): min=2444, max=33106, avg=6092.80, stdev=2137.71
clat (usec): min=32, max=138466, avg=4609.26, stdev=12047.26
lat (usec): min=35, max=138472, avg=4615.35, stdev=12047.30
clat percentiles (usec):
| 1.00th=[ 39], 5.00th=[ 48], 10.00th=[ 58], 20.00th=[ 685],
| 30.00th=[ 742], 40.00th=[ 816], 50.00th=[ 898], 60.00th=[ 971],
| 70.00th=[ 1221], 80.00th=[ 1565], 90.00th=[ 4752], 95.00th=[ 37487],
| 99.00th=[ 50594], 99.50th=[ 58983], 99.90th=[ 92799], 99.95th=[125305],
| 99.99th=[139461]
bw ( KiB/s): min= 152, max= 1202, per=99.85%, avg=430.73, stdev=213.53, samples=60
iops : min= 38, max= 300, avg=107.67, stdev=53.34, samples=60
write: IOPS=104, BW=417KiB/s (427kB/s)(12.2MiB/30001msec); 0 zone resets
slat (nsec): min=2374, max=43512, avg=6370.66, stdev=2309.50
clat (usec): min=35, max=97575, avg=4811.75, stdev=11855.18
lat (usec): min=43, max=97581, avg=4818.12, stdev=11855.27
clat percentiles (usec):
| 1.00th=[ 51], 5.00th=[ 63], 10.00th=[ 82], 20.00th=[ 676],
| 30.00th=[ 734], 40.00th=[ 799], 50.00th=[ 889], 60.00th=[ 963],
| 70.00th=[ 1205], 80.00th=[ 1598], 90.00th=[15926], 95.00th=[38011],
| 99.00th=[47449], 99.50th=[55837], 99.90th=[73925], 99.95th=[87557],
| 99.99th=[98042]
bw ( KiB/s): min= 160, max= 1416, per=99.84%, avg=416.63, stdev=223.01, samples=60
iops : min= 40, max= 354, avg=104.13, stdev=55.72, samples=60
lat (usec) : 50=3.54%, 100=7.63%, 250=0.36%, 750=20.68%, 1000=29.99%
lat (msec) : 2=23.07%, 4=4.20%, 10=0.72%, 20=0.33%, 50=8.51%
lat (msec) : 100=0.91%, 250=0.05%
cpu : usr=0.13%, sys=0.51%, ctx=6356, majf=0, minf=37
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=3230,3125,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=207MiB/s (217MB/s), 207MiB/s-207MiB/s (217MB/s-217MB/s), io=6222MiB (6524MB), run=30034-30034msec
WRITE: bw=207MiB/s (217MB/s), 207MiB/s-207MiB/s (217MB/s-217MB/s), io=6224MiB (6526MB), run=30034-30034msec
Run status group 1 (all jobs):
READ: bw=118MiB/s (124MB/s), 118MiB/s-118MiB/s (124MB/s-124MB/s), io=3536MiB (3708MB), run=30003-30003msec
WRITE: bw=122MiB/s (128MB/s), 122MiB/s-122MiB/s (128MB/s-128MB/s), io=3660MiB (3838MB), run=30003-30003msec
Run status group 2 (all jobs):
READ: bw=214MiB/s (224MB/s), 12.5MiB/s-14.1MiB/s (13.1MB/s-14.8MB/s), io=6434MiB (6747MB), run=30008-30082msec
WRITE: bw=214MiB/s (224MB/s), 12.5MiB/s-14.0MiB/s (13.1MB/s-14.7MB/s), io=6439MiB (6752MB), run=30008-30082msec
Run status group 3 (all jobs):
READ: bw=431KiB/s (441kB/s), 431KiB/s-431KiB/s (441kB/s-441kB/s), io=12.6MiB (13.2MB), run=30001-30001msec
WRITE: bw=417KiB/s (427kB/s), 417KiB/s-417KiB/s (427kB/s-427kB/s), io=12.2MiB (12.8MB), run=30001-30001msec
