[Promox VE] LUKS + ZFS - Extremely slow NVMe Root Pool

luckylinux · Sep 25, 2026

Continuing here the Previos Conversation which was getting quite out of Topic (https://forums.servethehome.com/index.php?threads/enterprise-ssd-small-deals.48343/post-512115)

I have a couple 3.2TB Micron 7300 Max NVMe Drives (PCIe 3.0 x4 each) which are extremely slow under Proxmox VE.

My Stack is:
Disk <--> LUKS Cryptdevice <--> ZFS Pool (Mirror of 2 x Micron 7300 Max)

I still quite/very bad performance (after converting to LBA Format 1 = 4096 bytes Sector Size)

Code:

root@pve:~# fio --time_based --runtime=30 --ramp_time=5 --ioengine=io_uring --direct=1 \
--size=2G --filename=/testfile.fio --stonewall \
--name=SEQ1M_Q8T1 --bs=1m --rw=readwrite --iodepth=8 --numjobs=1 \
--name=SEQ1M_Q1T1 --bs=1m --rw=readwrite --iodepth=1 --numjobs=1 \
--name=RND4K_Q32T16 --bs=4k --rw=randrw --iodepth=32 --numjobs=16 \
--name=RND4K_Q1T1 --bs=4k --rw=randrw --iodepth=1 --numjobs=1
SEQ1M_Q8T1: (g=0): rw=rw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=io_uring, iodepth=8
SEQ1M_Q1T1: (g=1): rw=rw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=io_uring, iodepth=1
RND4K_Q32T16: (g=2): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=io_uring, iodepth=32
...
RND4K_Q1T1: (g=3): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=io_uring, iodepth=1
fio-3.39
Starting 19 processes
SEQ1M_Q8T1: Laying out IO file (1 file / 2048MiB)
Jobs: 1 (f=1): [_(18),m(1)][23.2%][r=2312KiB/s,w=2220KiB/s][r=578,w=555 IOPS][eta 07m:46s]          
SEQ1M_Q8T1: (groupid=0, jobs=1): err= 0: pid=9251: Fri Sep 25 19:39:33 2026
  read: IOPS=234, BW=235MiB/s (246MB/s)(7050MiB/30029msec)
    slat (nsec): min=1730, max=274414, avg=4098.74, stdev=3857.10
    clat (usec): min=723, max=159424, avg=5916.62, stdev=14217.84
     lat (usec): min=726, max=159428, avg=5920.72, stdev=14218.05
    clat percentiles (usec):
     |  1.00th=[   807],  5.00th=[   865], 10.00th=[   906], 20.00th=[   979],
     | 30.00th=[  1037], 40.00th=[  1106], 50.00th=[  1319], 60.00th=[  1647],
     | 70.00th=[  2089], 80.00th=[  3064], 90.00th=[  6456], 95.00th=[ 43254],
     | 99.00th=[ 60031], 99.50th=[ 83362], 99.90th=[117965], 99.95th=[130548],
     | 99.99th=[160433]
   bw (  KiB/s): min=167936, max=323584, per=100.00%, avg=240694.70, stdev=33411.87, samples=60
   iops        : min=  164, max=  316, avg=234.97, stdev=32.61, samples=60
  write: IOPS=235, BW=235MiB/s (247MB/s)(7066MiB/30029msec); 0 zone resets
    slat (usec): min=21, max=2166, avg=47.44, stdev=59.17
    clat (usec): min=8435, max=67782, avg=28050.63, stdev=5159.53
     lat (usec): min=8522, max=67875, avg=28098.06, stdev=5159.45
    clat percentiles (usec):
     |  1.00th=[15139],  5.00th=[19268], 10.00th=[20841], 20.00th=[23987],
     | 30.00th=[25560], 40.00th=[27395], 50.00th=[28443], 60.00th=[29492],
     | 70.00th=[31065], 80.00th=[32375], 90.00th=[33817], 95.00th=[35390],
     | 99.00th=[37487], 99.50th=[40109], 99.90th=[53216], 99.95th=[63701],
     | 99.99th=[67634]
   bw (  KiB/s): min=227328, max=253952, per=100.00%, avg=241035.03, stdev=7686.19, samples=60
   iops        : min=  222, max=  248, avg=235.30, stdev= 7.52, samples=60
  lat (usec)   : 750=0.07%, 1000=11.90%
  lat (msec)   : 2=22.19%, 4=8.77%, 10=2.29%, 20=3.23%, 50=50.37%
  lat (msec)   : 100=1.13%, 250=0.11%
  cpu          : usr=1.33%, sys=0.50%, ctx=14118, majf=0, minf=1077
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=7050,7059,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8
SEQ1M_Q1T1: (groupid=1, jobs=1): err= 0: pid=9832: Fri Sep 25 19:39:33 2026
  read: IOPS=65, BW=65.5MiB/s (68.7MB/s)(1965MiB/30002msec)
    slat (nsec): min=2170, max=20391, avg=5225.34, stdev=1971.06
    clat (usec): min=692, max=109681, avg=10761.39, stdev=16916.34
     lat (usec): min=695, max=109685, avg=10766.61, stdev=16916.67
    clat percentiles (usec):
     |  1.00th=[   824],  5.00th=[   881], 10.00th=[   963], 20.00th=[  1074],
     | 30.00th=[  1385], 40.00th=[  1614], 50.00th=[  1893], 60.00th=[  2343],
     | 70.00th=[  3490], 80.00th=[ 31589], 90.00th=[ 40109], 95.00th=[ 44303],
     | 99.00th=[ 57410], 99.50th=[ 72877], 99.90th=[103285], 99.95th=[109577],
     | 99.99th=[109577]
   bw (  KiB/s): min=30720, max=124928, per=100.00%, avg=67089.12, stdev=23331.32, samples=60
   iops        : min=   30, max=  122, avg=65.50, stdev=22.78, samples=60
  write: IOPS=68, BW=68.6MiB/s (71.9MB/s)(2058MiB/30002msec); 0 zone resets
    slat (usec): min=23, max=267, avg=47.44, stdev=10.67
    clat (usec): min=3904, max=6351, avg=4245.21, stdev=201.44
     lat (usec): min=3961, max=6399, avg=4292.65, stdev=203.39
    clat percentiles (usec):
     |  1.00th=[ 3949],  5.00th=[ 4015], 10.00th=[ 4047], 20.00th=[ 4113],
     | 30.00th=[ 4146], 40.00th=[ 4178], 50.00th=[ 4228], 60.00th=[ 4293],
     | 70.00th=[ 4293], 80.00th=[ 4359], 90.00th=[ 4424], 95.00th=[ 4424],
     | 99.00th=[ 5080], 99.50th=[ 5538], 99.90th=[ 6063], 99.95th=[ 6063],
     | 99.99th=[ 6325]
   bw (  KiB/s): min=24576, max=126976, per=99.98%, avg=70227.12, stdev=24881.07, samples=60
   iops        : min=   24, max=  124, avg=68.57, stdev=24.30, samples=60
  lat (usec)   : 750=0.05%, 1000=7.33%
  lat (msec)   : 2=18.44%, 4=11.58%, 10=51.16%, 20=0.50%, 50=10.07%
  lat (msec)   : 100=0.82%, 250=0.05%
  cpu          : usr=0.42%, sys=0.16%, ctx=4034, majf=0, minf=58
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1965,2058,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
RND4K_Q32T16: (groupid=2, jobs=1): err= 0: pid=10071: Fri Sep 25 19:39:33 2026
  read: IOPS=3046, BW=11.9MiB/s (12.5MB/s)(357MiB/30018msec)
    slat (nsec): min=1570, max=429837, avg=3555.76, stdev=2970.47
    clat (nsec): min=1440, max=178827k, avg=241516.67, stdev=2515530.55
     lat (usec): min=15, max=178830, avg=245.07, stdev=2515.53
    clat percentiles (usec):
     |  1.00th=[    23],  5.00th=[    28], 10.00th=[    31], 20.00th=[    36],
     | 30.00th=[    42], 40.00th=[    55], 50.00th=[    76], 60.00th=[    98],
     | 70.00th=[   119], 80.00th=[   137], 90.00th=[   176], 95.00th=[   424],
     | 99.00th=[  3556], 99.50th=[  4883], 99.90th=[  9241], 99.95th=[ 49546],
     | 99.99th=[143655]
   bw (  KiB/s): min= 1017, max=32952, per=6.29%, avg=12345.93, stdev=9678.45, samples=59
   iops        : min=  254, max= 8238, avg=3086.42, stdev=2419.63, samples=59
  write: IOPS=3037, BW=11.9MiB/s (12.4MB/s)(356MiB/30018msec); 0 zone resets
    slat (nsec): min=1640, max=1207.4k, avg=3994.21, stdev=5907.53
    clat (usec): min=456, max=476407, avg=10325.34, stdev=17843.18
     lat (usec): min=467, max=476410, avg=10329.33, stdev=17843.16
    clat percentiles (usec):
     |  1.00th=[   840],  5.00th=[  1827], 10.00th=[  2540], 20.00th=[  3097],
     | 30.00th=[  3425], 40.00th=[  3687], 50.00th=[  4178], 60.00th=[  5866],
     | 70.00th=[  9241], 80.00th=[ 16057], 90.00th=[ 24249], 95.00th=[ 29492],
     | 99.00th=[ 83362], 99.50th=[109577], 99.90th=[217056], 99.95th=[254804],
     | 99.99th=[471860]
   bw (  KiB/s): min= 1052, max=33552, per=6.27%, avg=12310.15, stdev=9574.23, samples=59
   iops        : min=  263, max= 8388, avg=3077.46, stdev=2393.55, samples=59
  lat (usec)   : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.18%, 50=18.65%
  lat (usec)   : 100=11.79%, 250=16.28%, 500=0.76%, 750=0.75%, 1000=0.89%
  lat (msec)   : 2=2.75%, 4=21.53%, 10=12.13%, 20=6.69%, 50=6.67%
  lat (msec)   : 100=0.62%, 250=0.30%, 500=0.03%
  cpu          : usr=1.66%, sys=5.66%, ctx=153863, majf=0, minf=196
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=91448,91193,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
RND4K_Q32T16: (groupid=2, jobs=1): err= 0: pid=10072: Fri Sep 25 19:39:33 2026
  read: IOPS=3031, BW=11.8MiB/s (12.4MB/s)(355MiB/30013msec)
    slat (nsec): min=1560, max=367016, avg=3579.85, stdev=2752.46
    clat (nsec): min=1580, max=208958k, avg=233309.61, stdev=2493142.96
     lat (usec): min=15, max=208960, avg=236.89, stdev=2493.15
    clat percentiles (usec):
     |  1.00th=[    24],  5.00th=[    28], 10.00th=[    31], 20.00th=[    36],
     | 30.00th=[    43], 40.00th=[    56], 50.00th=[    77], 60.00th=[    98],
     | 70.00th=[   118], 80.00th=[   137], 90.00th=[   174], 95.00th=[   359],
     | 99.00th=[  3425], 99.50th=[  4752], 99.90th=[  8848], 99.95th=[ 44827],
     | 99.99th=[135267]
   bw (  KiB/s): min=  464, max=31384, per=6.25%, avg=12259.19, stdev=9439.85, samples=59
   iops        : min=  116, max= 7846, avg=3064.75, stdev=2359.98, samples=59
  write: IOPS=3034, BW=11.9MiB/s (12.4MB/s)(356MiB/30013msec); 0 zone resets
    slat (nsec): min=1610, max=464827, avg=3982.96, stdev=3082.15
    clat (usec): min=503, max=306627, avg=10353.08, stdev=17265.53
     lat (usec): min=506, max=306630, avg=10357.06, stdev=17265.59
    clat percentiles (usec):
     |  1.00th=[   898],  5.00th=[  1893], 10.00th=[  2540], 20.00th=[  3130],
     | 30.00th=[  3425], 40.00th=[  3687], 50.00th=[  4228], 60.00th=[  5866],
     | 70.00th=[  9372], 80.00th=[ 15926], 90.00th=[ 23725], 95.00th=[ 28967],
     | 99.00th=[ 87557], 99.50th=[115868], 99.90th=[210764], 99.95th=[238027],
     | 99.99th=[299893]
   bw (  KiB/s): min=  504, max=31680, per=6.25%, avg=12276.20, stdev=9490.49, samples=59
   iops        : min=  126, max= 7920, avg=3068.98, stdev=2372.61, samples=59
  lat (usec)   : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.15%, 50=18.22%
  lat (usec)   : 100=12.21%, 250=16.35%, 500=0.75%, 750=0.57%, 1000=0.97%
  lat (msec)   : 2=2.62%, 4=21.44%, 10=12.16%, 20=7.02%, 50=6.61%
  lat (msec)   : 100=0.53%, 250=0.39%, 500=0.02%
  cpu          : usr=1.68%, sys=5.73%, ctx=153893, majf=0, minf=233
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=90988,91088,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
RND4K_Q32T16: (groupid=2, jobs=1): err= 0: pid=10073: Fri Sep 25 19:39:33 2026
  read: IOPS=3070, BW=12.0MiB/s (12.6MB/s)(360MiB/30024msec)
    slat (nsec): min=1510, max=720671, avg=3592.82, stdev=3486.94
    clat (usec): min=2, max=163381, avg=227.52, stdev=2170.89
     lat (usec): min=12, max=163385, avg=231.11, stdev=2170.90
    clat percentiles (usec):
     |  1.00th=[    23],  5.00th=[    28], 10.00th=[    31], 20.00th=[    36],
     | 30.00th=[    42], 40.00th=[    55], 50.00th=[    76], 60.00th=[    97],
     | 70.00th=[   119], 80.00th=[   137], 90.00th=[   178], 95.00th=[   420],
     | 99.00th=[  3425], 99.50th=[  4752], 99.90th=[  7898], 99.95th=[ 39584],
     | 99.99th=[121111]
   bw (  KiB/s): min=  760, max=34920, per=6.34%, avg=12439.14, stdev=9596.62, samples=59
   iops        : min=  190, max= 8730, avg=3109.71, stdev=2399.18, samples=59
  write: IOPS=3077, BW=12.0MiB/s (12.6MB/s)(361MiB/30024msec); 0 zone resets
    slat (nsec): min=1680, max=144002, avg=3956.28, stdev=1959.84
    clat (usec): min=140, max=470982, avg=10165.12, stdev=17686.55
     lat (usec): min=152, max=470986, avg=10169.08, stdev=17686.55
    clat percentiles (usec):
     |  1.00th=[   832],  5.00th=[  1860], 10.00th=[  2540], 20.00th=[  3130],
     | 30.00th=[  3425], 40.00th=[  3720], 50.00th=[  4359], 60.00th=[  5932],
     | 70.00th=[  9372], 80.00th=[ 15401], 90.00th=[ 23462], 95.00th=[ 29230],
     | 99.00th=[ 70779], 99.50th=[110625], 99.90th=[283116], 99.95th=[312476],
     | 99.99th=[375391]
   bw (  KiB/s): min=  768, max=35640, per=6.35%, avg=12463.34, stdev=9581.26, samples=59
   iops        : min=  192, max= 8910, avg=3115.76, stdev=2395.32, samples=59
  lat (usec)   : 4=0.01%, 10=0.01%, 20=0.22%, 50=18.52%, 100=11.91%
  lat (usec)   : 250=16.13%, 500=0.81%, 750=0.73%, 1000=0.94%
  lat (msec)   : 2=2.64%, 4=20.80%, 10=12.86%, 20=7.37%, 50=6.32%
  lat (msec)   : 100=0.41%, 250=0.30%, 500=0.05%
  cpu          : usr=1.72%, sys=5.72%, ctx=155670, majf=0, minf=214
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=92191,92387,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
RND4K_Q32T16: (groupid=2, jobs=1): err= 0: pid=10074: Fri Sep 25 19:39:33 2026
  read: IOPS=3095, BW=12.1MiB/s (12.7MB/s)(363MiB/30022msec)
    slat (nsec): min=1550, max=279804, avg=3628.48, stdev=2786.90
    clat (nsec): min=1470, max=213406k, avg=218680.71, stdev=2165822.63
     lat (usec): min=13, max=213409, avg=222.31, stdev=2165.82
    clat percentiles (usec):
     |  1.00th=[    23],  5.00th=[    28], 10.00th=[    31], 20.00th=[    36],
     | 30.00th=[    43], 40.00th=[    57], 50.00th=[    79], 60.00th=[    99],
     | 70.00th=[   119], 80.00th=[   137], 90.00th=[   176], 95.00th=[   330],
     | 99.00th=[  3228], 99.50th=[  4490], 99.90th=[  7767], 99.95th=[ 37487],
     | 99.99th=[100140]
   bw (  KiB/s): min=  936, max=36416, per=6.39%, avg=12536.36, stdev=10150.03, samples=59
   iops        : min=  234, max= 9104, avg=3134.03, stdev=2537.50, samples=59
  write: IOPS=3087, BW=12.1MiB/s (12.6MB/s)(362MiB/30022msec); 0 zone resets
    slat (nsec): min=1650, max=349265, avg=4018.27, stdev=2718.80
    clat (usec): min=122, max=338023, avg=10138.96, stdev=17727.06
     lat (usec): min=137, max=338029, avg=10142.98, stdev=17727.08
    clat percentiles (usec):
     |  1.00th=[   840],  5.00th=[  1811], 10.00th=[  2474], 20.00th=[  2999],
     | 30.00th=[  3326], 40.00th=[  3621], 50.00th=[  4080], 60.00th=[  5669],
     | 70.00th=[  8979], 80.00th=[ 15664], 90.00th=[ 23462], 95.00th=[ 28705],
     | 99.00th=[ 86508], 99.50th=[131597], 99.90th=[246416], 99.95th=[258999],
     | 99.99th=[337642]
   bw (  KiB/s): min=  936, max=35808, per=6.36%, avg=12498.88, stdev=10096.82, samples=59
   iops        : min=  234, max= 8952, avg=3124.68, stdev=2524.20, samples=59
  lat (usec)   : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.17%, 50=18.02%
  lat (usec)   : 100=12.12%, 250=16.69%, 500=0.89%, 750=0.82%, 1000=0.71%
  lat (msec)   : 2=2.85%, 4=21.86%, 10=11.81%, 20=6.85%, 50=6.43%
  lat (msec)   : 100=0.42%, 250=0.33%, 500=0.04%
  cpu          : usr=1.84%, sys=5.73%, ctx=154863, majf=0, minf=199
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=92927,92679,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
RND4K_Q32T16: (groupid=2, jobs=1): err= 0: pid=10075: Fri Sep 25 19:39:33 2026
  read: IOPS=3008, BW=11.8MiB/s (12.3MB/s)(353MiB/30019msec)
    slat (nsec): min=1520, max=16070k, avg=3876.83, stdev=54197.99
    clat (nsec): min=1670, max=207003k, avg=259227.42, stdev=2878904.24
     lat (usec): min=14, max=207007, avg=263.10, stdev=2879.41
    clat percentiles (usec):
     |  1.00th=[    23],  5.00th=[    28], 10.00th=[    31], 20.00th=[    36],
     | 30.00th=[    41], 40.00th=[    53], 50.00th=[    75], 60.00th=[    96],
     | 70.00th=[   118], 80.00th=[   137], 90.00th=[   178], 95.00th=[   502],
     | 99.00th=[  3589], 99.50th=[  4817], 99.90th=[ 16319], 99.95th=[ 71828],
     | 99.99th=[149947]
   bw (  KiB/s): min= 2104, max=34512, per=6.20%, avg=12170.25, stdev=9026.75, samples=59
   iops        : min=  526, max= 8628, avg=3042.51, stdev=2256.68, samples=59
  write: IOPS=3008, BW=11.8MiB/s (12.3MB/s)(353MiB/30019msec); 0 zone resets
    slat (nsec): min=1670, max=945885, avg=4066.76, stdev=4013.95
    clat (usec): min=135, max=260194, avg=10373.01, stdev=15619.03
     lat (usec): min=152, max=260198, avg=10377.08, stdev=15619.07
    clat percentiles (usec):
     |  1.00th=[   783],  5.00th=[  1778], 10.00th=[  2442], 20.00th=[  3130],
     | 30.00th=[  3425], 40.00th=[  3720], 50.00th=[  4228], 60.00th=[  5932],
     | 70.00th=[  9765], 80.00th=[ 16450], 90.00th=[ 24773], 95.00th=[ 30016],
     | 99.00th=[ 86508], 99.50th=[109577], 99.90th=[187696], 99.95th=[191890],
     | 99.99th=[204473]
   bw (  KiB/s): min= 2181, max=33632, per=6.20%, avg=12170.88, stdev=9026.25, samples=59
   iops        : min=  545, max= 8408, avg=3042.68, stdev=2256.57, samples=59
  lat (usec)   : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.19%, 50=19.01%
  lat (usec)   : 100=11.78%, 250=15.72%, 500=0.79%, 750=0.88%, 1000=1.14%
  lat (msec)   : 2=2.81%, 4=20.91%, 10=11.98%, 20=7.05%, 50=6.77%
  lat (msec)   : 100=0.64%, 250=0.33%, 500=0.01%
  cpu          : usr=1.79%, sys=5.67%, ctx=151315, majf=0, minf=202
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=90320,90299,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
RND4K_Q32T16: (groupid=2, jobs=1): err= 0: pid=10076: Fri Sep 25 19:39:33 2026
  read: IOPS=3146, BW=12.3MiB/s (12.9MB/s)(369MiB/30021msec)
    slat (nsec): min=1550, max=951295, avg=3677.04, stdev=4584.10
    clat (usec): min=2, max=184404, avg=227.61, stdev=2211.12
     lat (usec): min=15, max=184409, avg=231.28, stdev=2211.14
    clat percentiles (usec):
     |  1.00th=[    23],  5.00th=[    28], 10.00th=[    31], 20.00th=[    36],
     | 30.00th=[    43], 40.00th=[    57], 50.00th=[    78], 60.00th=[    98],
     | 70.00th=[   119], 80.00th=[   139], 90.00th=[   180], 95.00th=[   400],
     | 99.00th=[  3523], 99.50th=[  4752], 99.90th=[  8291], 99.95th=[ 39060],
     | 99.99th=[106431]
   bw (  KiB/s): min=  759, max=41496, per=6.49%, avg=12736.59, stdev=10262.54, samples=59
   iops        : min=  189, max=10374, avg=3184.07, stdev=2565.64, samples=59
  write: IOPS=3147, BW=12.3MiB/s (12.9MB/s)(369MiB/30021msec); 0 zone resets
    slat (nsec): min=1651, max=361926, avg=4058.38, stdev=3002.08
    clat (usec): min=239, max=421038, avg=9942.92, stdev=16702.27
     lat (usec): min=242, max=421042, avg=9946.98, stdev=16702.31
    clat percentiles (usec):
     |  1.00th=[   840],  5.00th=[  1811], 10.00th=[  2474], 20.00th=[  3064],
     | 30.00th=[  3392], 40.00th=[  3654], 50.00th=[  4113], 60.00th=[  5800],
     | 70.00th=[  9241], 80.00th=[ 15401], 90.00th=[ 23462], 95.00th=[ 28705],
     | 99.00th=[ 79168], 99.50th=[109577], 99.90th=[223347], 99.95th=[299893],
     | 99.99th=[392168]
   bw (  KiB/s): min=  836, max=40776, per=6.49%, avg=12745.20, stdev=10277.30, samples=59
   iops        : min=  209, max=10194, avg=3186.24, stdev=2569.31, samples=59
  lat (usec)   : 4=0.01%, 10=0.01%, 20=0.21%, 50=18.03%, 100=12.20%
  lat (usec)   : 250=16.30%, 500=0.92%, 750=0.72%, 1000=1.15%
  lat (msec)   : 2=2.67%, 4=21.65%, 10=11.86%, 20=7.14%, 50=6.48%
  lat (msec)   : 100=0.36%, 250=0.29%, 500=0.03%
  cpu          : usr=1.68%, sys=6.03%, ctx=157094, majf=0, minf=221
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=94447,94493,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
RND4K_Q32T16: (groupid=2, jobs=1): err= 0: pid=10077: Fri Sep 25 19:39:33 2026
  read: IOPS=2984, BW=11.7MiB/s (12.2MB/s)(350MiB/30022msec)
    slat (nsec): min=1570, max=1018.2k, avg=3976.22, stdev=7083.71
    clat (nsec): min=340, max=176103k, avg=227333.45, stdev=2213288.93
     lat (usec): min=13, max=176106, avg=231.31, stdev=2213.28
    clat percentiles (usec):
     |  1.00th=[    23],  5.00th=[    28], 10.00th=[    31], 20.00th=[    36],
     | 30.00th=[    43], 40.00th=[    57], 50.00th=[    80], 60.00th=[   103],
     | 70.00th=[   123], 80.00th=[   143], 90.00th=[   192], 95.00th=[   416],
     | 99.00th=[  3392], 99.50th=[  4752], 99.90th=[  8029], 99.95th=[ 29492],
     | 99.99th=[111674]
   bw (  KiB/s): min=  768, max=33384, per=6.16%, avg=12083.69, stdev=9539.63, samples=59
   iops        : min=  192, max= 8346, avg=3020.86, stdev=2384.88, samples=59
  write: IOPS=3012, BW=11.8MiB/s (12.3MB/s)(353MiB/30022msec); 0 zone resets
    slat (nsec): min=1690, max=1364.9k, avg=4377.88, stdev=7942.42
    clat (usec): min=125, max=437673, avg=10426.76, stdev=18831.17
     lat (usec): min=129, max=437675, avg=10431.13, stdev=18831.14
    clat percentiles (usec):
     |  1.00th=[   791],  5.00th=[  1778], 10.00th=[  2540], 20.00th=[  3097],
     | 30.00th=[  3392], 40.00th=[  3654], 50.00th=[  4146], 60.00th=[  5866],
     | 70.00th=[  9634], 80.00th=[ 16188], 90.00th=[ 23725], 95.00th=[ 29230],
     | 99.00th=[ 87557], 99.50th=[115868], 99.90th=[244319], 99.95th=[287310],
     | 99.99th=[421528]
   bw (  KiB/s): min=  888, max=34544, per=6.21%, avg=12193.69, stdev=9502.57, samples=59
   iops        : min=  222, max= 8636, avg=3048.34, stdev=2375.61, samples=59
  lat (nsec)   : 500=0.01%, 750=0.01%, 1000=0.01%
  lat (usec)   : 4=0.01%, 10=0.01%, 20=0.22%, 50=17.90%, 100=11.16%
  lat (usec)   : 250=17.04%, 500=1.13%, 750=0.89%, 1000=1.17%
  lat (msec)   : 2=2.39%, 4=21.88%, 10=11.46%, 20=7.35%, 50=6.57%
  lat (msec)   : 100=0.53%, 250=0.28%, 500=0.04%
  cpu          : usr=2.06%, sys=5.70%, ctx=147518, majf=0, minf=270
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=89589,90451,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
RND4K_Q32T16: (groupid=2, jobs=1): err= 0: pid=10078: Fri Sep 25 19:39:33 2026
  read: IOPS=3004, BW=11.7MiB/s (12.3MB/s)(352MiB/30021msec)
    slat (nsec): min=1580, max=290755, avg=3590.56, stdev=2251.93
    clat (nsec): min=1520, max=175621k, avg=239898.72, stdev=2548917.28
     lat (usec): min=11, max=175623, avg=243.49, stdev=2548.92
    clat percentiles (usec):
     |  1.00th=[    23],  5.00th=[    28], 10.00th=[    31], 20.00th=[    36],
     | 30.00th=[    42], 40.00th=[    56], 50.00th=[    77], 60.00th=[    99],
     | 70.00th=[   119], 80.00th=[   139], 90.00th=[   178], 95.00th=[   379],
     | 99.00th=[  3294], 99.50th=[  4752], 99.90th=[  8979], 99.95th=[ 67634],
     | 99.99th=[115868]
   bw (  KiB/s): min=  176, max=31600, per=6.19%, avg=12151.27, stdev=9370.32, samples=59
   iops        : min=   44, max= 7900, avg=3037.78, stdev=2342.60, samples=59
  write: IOPS=2988, BW=11.7MiB/s (12.2MB/s)(351MiB/30021msec); 0 zone resets
    slat (nsec): min=1640, max=407847, avg=3978.44, stdev=2866.08
    clat (usec): min=412, max=796411, avg=10459.33, stdev=18571.16
     lat (usec): min=417, max=796415, avg=10463.31, stdev=18571.18
    clat percentiles (usec):
     |  1.00th=[   930],  5.00th=[  1991], 10.00th=[  2638], 20.00th=[  3163],
     | 30.00th=[  3490], 40.00th=[  3752], 50.00th=[  4490], 60.00th=[  6194],
     | 70.00th=[  9634], 80.00th=[ 16581], 90.00th=[ 24249], 95.00th=[ 30016],
     | 99.00th=[ 83362], 99.50th=[101188], 99.90th=[170918], 99.95th=[250610],
     | 99.99th=[650118]
   bw (  KiB/s): min=  144, max=31152, per=6.16%, avg=12090.93, stdev=9244.00, samples=59
   iops        : min=   36, max= 7788, avg=3022.68, stdev=2311.02, samples=59
  lat (usec)   : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.15%, 50=18.50%
  lat (usec)   : 100=11.78%, 250=16.52%, 500=0.84%, 750=0.67%, 1000=0.83%
  lat (msec)   : 2=2.49%, 4=20.66%, 10=12.92%, 20=6.91%, 50=6.83%
  lat (msec)   : 100=0.64%, 250=0.25%, 500=0.01%, 750=0.02%, 1000=0.01%
  cpu          : usr=1.82%, sys=5.57%, ctx=151982, majf=0, minf=209
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=90184,89728,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
RND4K_Q32T16: (groupid=2, jobs=1): err= 0: pid=10079: Fri Sep 25 19:39:33 2026
  read: IOPS=3062, BW=12.0MiB/s (12.5MB/s)(359MiB/30022msec)
    slat (nsec): min=1600, max=329665, avg=3646.07, stdev=2914.92
    clat (nsec): min=430, max=169477k, avg=230869.27, stdev=2264721.00
     lat (usec): min=13, max=169479, avg=234.52, stdev=2264.73
    clat percentiles (usec):
     |  1.00th=[    23],  5.00th=[    28], 10.00th=[    31], 20.00th=[    36],
     | 30.00th=[    42], 40.00th=[    55], 50.00th=[    75], 60.00th=[    97],
     | 70.00th=[   119], 80.00th=[   139], 90.00th=[   182], 95.00th=[   429],
     | 99.00th=[  3392], 99.50th=[  4621], 99.90th=[  8717], 99.95th=[ 55837],
     | 99.99th=[102237]
   bw (  KiB/s): min=  848, max=36968, per=6.32%, avg=12400.14, stdev=9724.70, samples=59
   iops        : min=  212, max= 9242, avg=3099.98, stdev=2431.17, samples=59
  write: IOPS=3069, BW=12.0MiB/s (12.6MB/s)(360MiB/30022msec); 0 zone resets
    slat (nsec): min=1680, max=584909, avg=4033.73, stdev=3960.38
    clat (usec): min=121, max=433551, avg=10186.81, stdev=16747.21
     lat (usec): min=124, max=433553, avg=10190.84, stdev=16747.20
    clat percentiles (usec):
     |  1.00th=[   807],  5.00th=[  1582], 10.00th=[  2409], 20.00th=[  3064],
     | 30.00th=[  3425], 40.00th=[  3720], 50.00th=[  4359], 60.00th=[  5866],
     | 70.00th=[  9241], 80.00th=[ 15533], 90.00th=[ 23725], 95.00th=[ 29754],
     | 99.00th=[ 92799], 99.50th=[108528], 99.90th=[210764], 99.95th=[217056],
     | 99.99th=[346031]
   bw (  KiB/s): min=  960, max=37088, per=6.33%, avg=12422.08, stdev=9683.46, samples=59
   iops        : min=  240, max= 9272, avg=3105.46, stdev=2420.87, samples=59
  lat (nsec)   : 500=0.01%, 750=0.01%, 1000=0.01%
  lat (usec)   : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.21%, 50=18.62%
  lat (usec)   : 100=11.91%, 250=15.94%, 500=0.89%, 750=0.83%, 1000=1.20%
  lat (msec)   : 2=3.08%, 4=20.18%, 10=12.81%, 20=7.05%, 50=6.46%
  lat (msec)   : 100=0.45%, 250=0.38%, 500=0.02%
  cpu          : usr=1.78%, sys=5.72%, ctx=153959, majf=0, minf=204
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=91942,92149,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
RND4K_Q32T16: (groupid=2, jobs=1): err= 0: pid=10080: Fri Sep 25 19:39:33 2026
  read: IOPS=3102, BW=12.1MiB/s (12.7MB/s)(364MiB/30016msec)
    slat (nsec): min=1600, max=262064, avg=3580.24, stdev=2585.16
    clat (nsec): min=1620, max=199242k, avg=215830.19, stdev=2264817.86
     lat (usec): min=12, max=199245, avg=219.41, stdev=2264.83
    clat percentiles (usec):
     |  1.00th=[    23],  5.00th=[    28], 10.00th=[    31], 20.00th=[    36],
     | 30.00th=[    41], 40.00th=[    55], 50.00th=[    77], 60.00th=[    98],
     | 70.00th=[   118], 80.00th=[   137], 90.00th=[   169], 95.00th=[   285],
     | 99.00th=[  3097], 99.50th=[  4424], 99.90th=[  8291], 99.95th=[ 48497],
     | 99.99th=[121111]
   bw (  KiB/s): min=  352, max=31096, per=6.40%, avg=12558.05, stdev=9711.34, samples=59
   iops        : min=   88, max= 7772, avg=3139.44, stdev=2427.79, samples=59
  write: IOPS=3120, BW=12.2MiB/s (12.8MB/s)(366MiB/30016msec); 0 zone resets
    slat (nsec): min=1680, max=712001, avg=3966.04, stdev=3678.05
    clat (usec): min=325, max=568463, avg=10083.76, stdev=18104.04
     lat (usec): min=343, max=568467, avg=10087.73, stdev=18104.05
    clat percentiles (usec):
     |  1.00th=[   832],  5.00th=[  2024], 10.00th=[  2606], 20.00th=[  3130],
     | 30.00th=[  3425], 40.00th=[  3687], 50.00th=[  4293], 60.00th=[  5866],
     | 70.00th=[  8979], 80.00th=[ 15270], 90.00th=[ 23725], 95.00th=[ 28705],
     | 99.00th=[ 76022], 99.50th=[124257], 99.90th=[250610], 99.95th=[283116],
     | 99.99th=[392168]
   bw (  KiB/s): min=  432, max=31864, per=6.43%, avg=12632.24, stdev=9751.56, samples=59
   iops        : min=  108, max= 7966, avg=3158.02, stdev=2437.91, samples=59
  lat (usec)   : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.19%, 50=18.56%
  lat (usec)   : 100=11.81%, 250=16.50%, 500=0.77%, 750=0.72%, 1000=0.89%
  lat (msec)   : 2=2.10%, 4=21.45%, 10=13.22%, 20=6.70%, 50=6.44%
  lat (msec)   : 100=0.27%, 250=0.36%, 500=0.05%, 750=0.01%
  cpu          : usr=1.86%, sys=5.69%, ctx=157189, majf=0, minf=212
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=93128,93665,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
RND4K_Q32T16: (groupid=2, jobs=1): err= 0: pid=10081: Fri Sep 25 19:39:33 2026
  read: IOPS=3104, BW=12.1MiB/s (12.7MB/s)(364MiB/30016msec)
    slat (nsec): min=1580, max=1091.1k, avg=3783.54, stdev=4968.12
    clat (nsec): min=1450, max=210604k, avg=236339.18, stdev=2712466.98
     lat (usec): min=13, max=210608, avg=240.12, stdev=2712.47
    clat percentiles (usec):
     |  1.00th=[    23],  5.00th=[    28], 10.00th=[    31], 20.00th=[    36],
     | 30.00th=[    42], 40.00th=[    55], 50.00th=[    76], 60.00th=[    97],
     | 70.00th=[   119], 80.00th=[   139], 90.00th=[   176], 95.00th=[   351],
     | 99.00th=[  3261], 99.50th=[  4752], 99.90th=[  8586], 99.95th=[ 64750],
     | 99.99th=[158335]
   bw (  KiB/s): min=  168, max=37824, per=6.33%, avg=12429.78, stdev=10047.26, samples=60
   iops        : min=   42, max= 9456, avg=3107.37, stdev=2511.81, samples=60
  write: IOPS=3116, BW=12.2MiB/s (12.8MB/s)(366MiB/30016msec); 0 zone resets
    slat (nsec): min=1680, max=4203.3k, avg=4183.20, stdev=13946.60
    clat (usec): min=141, max=477210, avg=10049.70, stdev=17242.74
     lat (usec): min=146, max=477213, avg=10053.89, stdev=17242.77
    clat percentiles (usec):
     |  1.00th=[   816],  5.00th=[  1762], 10.00th=[  2474], 20.00th=[  3097],
     | 30.00th=[  3425], 40.00th=[  3687], 50.00th=[  4228], 60.00th=[  5604],
     | 70.00th=[  8848], 80.00th=[ 16057], 90.00th=[ 23725], 95.00th=[ 28967],
     | 99.00th=[ 72877], 99.50th=[109577], 99.90th=[227541], 99.95th=[283116],
     | 99.99th=[467665]
   bw (  KiB/s): min=  304, max=37232, per=6.35%, avg=12476.97, stdev=10032.99, samples=60
   iops        : min=   76, max= 9308, avg=3119.17, stdev=2508.25, samples=60
  lat (usec)   : 2=0.01%, 4=0.02%, 10=0.01%, 20=0.19%, 50=18.49%
  lat (usec)   : 100=11.92%, 250=16.23%, 500=0.81%, 750=0.75%, 1000=1.10%
  lat (msec)   : 2=2.68%, 4=20.85%, 10=12.87%, 20=6.84%, 50=6.48%
  lat (msec)   : 100=0.47%, 250=0.30%, 500=0.03%
  cpu          : usr=1.83%, sys=5.99%, ctx=155939, majf=0, minf=194
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=93193,93548,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
RND4K_Q32T16: (groupid=2, jobs=1): err= 0: pid=10082: Fri Sep 25 19:39:33 2026
  read: IOPS=3094, BW=12.1MiB/s (12.7MB/s)(363MiB/30010msec)
    slat (nsec): min=1600, max=181343, avg=3589.54, stdev=2340.94
    clat (nsec): min=1820, max=195093k, avg=230946.33, stdev=2498920.08
     lat (usec): min=14, max=195096, avg=234.54, stdev=2498.92
    clat percentiles (usec):
     |  1.00th=[    23],  5.00th=[    28], 10.00th=[    31], 20.00th=[    35],
     | 30.00th=[    41], 40.00th=[    53], 50.00th=[    74], 60.00th=[    95],
     | 70.00th=[   117], 80.00th=[   137], 90.00th=[   176], 95.00th=[   396],
     | 99.00th=[  3359], 99.50th=[  4621], 99.90th=[  8029], 99.95th=[ 43779],
     | 99.99th=[141558]
   bw (  KiB/s): min= 1432, max=31528, per=6.31%, avg=12386.85, stdev=9527.98, samples=60
   iops        : min=  358, max= 7882, avg=3096.63, stdev=2382.03, samples=60
  write: IOPS=3111, BW=12.2MiB/s (12.7MB/s)(365MiB/30010msec); 0 zone resets
    slat (nsec): min=1670, max=1021.9k, avg=3989.15, stdev=4439.60
    clat (usec): min=184, max=230150, avg=10076.88, stdev=15261.92
     lat (usec): min=187, max=230152, avg=10080.86, stdev=15261.98
    clat percentiles (usec):
     |  1.00th=[   807],  5.00th=[  1713], 10.00th=[  2507], 20.00th=[  3097],
     | 30.00th=[  3425], 40.00th=[  3752], 50.00th=[  4359], 60.00th=[  5997],
     | 70.00th=[  9241], 80.00th=[ 15795], 90.00th=[ 24249], 95.00th=[ 29230],
     | 99.00th=[ 71828], 99.50th=[111674], 99.90th=[187696], 99.95th=[202376],
     | 99.99th=[221250]
   bw (  KiB/s): min= 1352, max=31760, per=6.34%, avg=12458.10, stdev=9601.12, samples=60
   iops        : min=  338, max= 7940, avg=3114.45, stdev=2400.30, samples=60
  lat (usec)   : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.21%, 50=19.01%
  lat (usec)   : 100=11.78%, 250=15.69%, 500=0.82%, 750=0.82%, 1000=1.06%
  lat (msec)   : 2=2.79%, 4=20.24%, 10=13.14%, 20=7.03%, 50=6.63%
  lat (msec)   : 100=0.49%, 250=0.31%
  cpu          : usr=1.84%, sys=5.72%, ctx=156355, majf=0, minf=204
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=92876,93379,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
RND4K_Q32T16: (groupid=2, jobs=1): err= 0: pid=10083: Fri Sep 25 19:39:33 2026
  read: IOPS=3098, BW=12.1MiB/s (12.7MB/s)(363MiB/30011msec)
    slat (nsec): min=1560, max=1787.0k, avg=3680.72, stdev=8283.09
    clat (nsec): min=1730, max=169512k, avg=234237.21, stdev=2421427.89
     lat (usec): min=13, max=169515, avg=237.92, stdev=2421.45
    clat percentiles (usec):
     |  1.00th=[    23],  5.00th=[    28], 10.00th=[    31], 20.00th=[    36],
     | 30.00th=[    42], 40.00th=[    56], 50.00th=[    77], 60.00th=[    98],
     | 70.00th=[   119], 80.00th=[   139], 90.00th=[   178], 95.00th=[   379],
     | 99.00th=[  3392], 99.50th=[  4621], 99.90th=[ 11731], 99.95th=[ 64226],
     | 99.99th=[112722]
   bw (  KiB/s): min=  424, max=34080, per=6.32%, avg=12402.65, stdev=9735.09, samples=60
   iops        : min=  106, max= 8520, avg=3100.58, stdev=2433.78, samples=60
  write: IOPS=3101, BW=12.1MiB/s (12.7MB/s)(364MiB/30011msec); 0 zone resets
    slat (nsec): min=1680, max=638940, avg=4036.88, stdev=3094.42
    clat (usec): min=191, max=424508, avg=10107.67, stdev=17103.14
     lat (usec): min=204, max=424511, avg=10111.71, stdev=17103.16
    clat percentiles (usec):
     |  1.00th=[   832],  5.00th=[  1876], 10.00th=[  2540], 20.00th=[  3097],
     | 30.00th=[  3425], 40.00th=[  3687], 50.00th=[  4359], 60.00th=[  5997],
     | 70.00th=[  9372], 80.00th=[ 15795], 90.00th=[ 23462], 95.00th=[ 28443],
     | 99.00th=[ 74974], 99.50th=[115868], 99.90th=[217056], 99.95th=[325059],
     | 99.99th=[350225]
   bw (  KiB/s): min=  624, max=34208, per=6.32%, avg=12417.30, stdev=9801.97, samples=60
   iops        : min=  156, max= 8552, avg=3104.25, stdev=2450.51, samples=60
  lat (usec)   : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.19%, 50=18.39%
  lat (usec)   : 100=11.84%, 250=16.39%, 500=0.85%, 750=0.83%, 1000=0.90%
  lat (msec)   : 2=2.62%, 4=21.06%, 10=12.46%, 20=7.22%, 50=6.52%
  lat (msec)   : 100=0.35%, 250=0.33%, 500=0.05%
  cpu          : usr=1.80%, sys=5.84%, ctx=156120, majf=0, minf=188
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=92995,93074,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
RND4K_Q32T16: (groupid=2, jobs=1): err= 0: pid=10084: Fri Sep 25 19:39:33 2026
  read: IOPS=3053, BW=11.9MiB/s (12.5MB/s)(358MiB/30059msec)
    slat (nsec): min=1590, max=494338, avg=3574.64, stdev=2635.02
    clat (nsec): min=840, max=202994k, avg=237713.23, stdev=2677589.42
     lat (usec): min=14, max=202997, avg=241.29, stdev=2677.59
    clat percentiles (usec):
     |  1.00th=[    24],  5.00th=[    29], 10.00th=[    31], 20.00th=[    36],
     | 30.00th=[    43], 40.00th=[    59], 50.00th=[    80], 60.00th=[   100],
     | 70.00th=[   119], 80.00th=[   137], 90.00th=[   174], 95.00th=[   359],
     | 99.00th=[  3228], 99.50th=[  4686], 99.90th=[  8586], 99.95th=[ 62653],
     | 99.99th=[147850]
   bw (  KiB/s): min=  708, max=32576, per=6.24%, avg=12239.80, stdev=9711.18, samples=60
   iops        : min=  177, max= 8144, avg=3059.87, stdev=2427.76, samples=60
  write: IOPS=3051, BW=11.9MiB/s (12.5MB/s)(358MiB/30059msec); 0 zone resets
    slat (nsec): min=1671, max=182953, avg=3989.66, stdev=2357.00
    clat (usec): min=147, max=462166, avg=10261.74, stdev=17283.61
     lat (usec): min=185, max=462171, avg=10265.73, stdev=17283.63
    clat percentiles (usec):
     |  1.00th=[   906],  5.00th=[  2040], 10.00th=[  2606], 20.00th=[  3130],
     | 30.00th=[  3425], 40.00th=[  3720], 50.00th=[  4293], 60.00th=[  5669],
     | 70.00th=[  9110], 80.00th=[ 15664], 90.00th=[ 23987], 95.00th=[ 30016],
     | 99.00th=[ 87557], 99.50th=[105382], 99.90th=[198181], 99.95th=[295699],
     | 99.99th=[383779]
   bw (  KiB/s): min=  708, max=34400, per=6.23%, avg=12232.02, stdev=9745.56, samples=60
   iops        : min=  177, max= 8600, avg=3057.95, stdev=2436.40, samples=60
  lat (nsec)   : 1000=0.01%
  lat (usec)   : 4=0.01%, 10=0.01%, 20=0.16%, 50=17.68%, 100=12.22%
  lat (usec)   : 250=16.94%, 500=0.72%, 750=0.74%, 1000=0.75%
  lat (msec)   : 2=2.34%, 4=21.64%, 10=12.65%, 20=6.89%, 50=6.38%
  lat (msec)   : 100=0.56%, 250=0.30%, 500=0.04%
  cpu          : usr=1.64%, sys=5.84%, ctx=155528, majf=0, minf=269
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=91771,91711,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
RND4K_Q32T16: (groupid=2, jobs=1): err= 0: pid=10085: Fri Sep 25 19:39:33 2026
  read: IOPS=3177, BW=12.4MiB/s (13.0MB/s)(373MiB/30018msec)
    slat (nsec): min=1570, max=411806, avg=3570.76, stdev=3852.90
    clat (nsec): min=1490, max=202991k, avg=250574.77, stdev=2819963.24
     lat (usec): min=13, max=202992, avg=254.15, stdev=2819.98
    clat percentiles (usec):
     |  1.00th=[    23],  5.00th=[    28], 10.00th=[    31], 20.00th=[    36],
     | 30.00th=[    42], 40.00th=[    56], 50.00th=[    76], 60.00th=[    97],
     | 70.00th=[   118], 80.00th=[   137], 90.00th=[   178], 95.00th=[   445],
     | 99.00th=[  3359], 99.50th=[  4686], 99.90th=[ 11994], 99.95th=[ 76022],
     | 99.99th=[147850]
   bw (  KiB/s): min= 1203, max=34440, per=6.56%, avg=12868.07, stdev=10200.23, samples=59
   iops        : min=  300, max= 8610, avg=3216.97, stdev=2550.07, samples=59
  write: IOPS=3141, BW=12.3MiB/s (12.9MB/s)(368MiB/30018msec); 0 zone resets
    slat (nsec): min=1630, max=731901, avg=3921.89, stdev=3585.22
    clat (usec): min=256, max=252867, avg=9924.83, stdev=15631.58
     lat (usec): min=267, max=252871, avg=9928.75, stdev=15631.64
    clat percentiles (usec):
     |  1.00th=[   832],  5.00th=[  1745], 10.00th=[  2442], 20.00th=[  3064],
     | 30.00th=[  3425], 40.00th=[  3687], 50.00th=[  4228], 60.00th=[  5604],
     | 70.00th=[  8586], 80.00th=[ 15008], 90.00th=[ 23725], 95.00th=[ 28967],
     | 99.00th=[ 82314], 99.50th=[113771], 99.90th=[181404], 99.95th=[210764],
     | 99.99th=[248513]
   bw (  KiB/s): min= 1106, max=32952, per=6.48%, avg=12719.07, stdev=9995.43, samples=59
   iops        : min=  276, max= 8238, avg=3179.71, stdev=2498.82, samples=59
  lat (usec)   : 2=0.03%, 4=0.01%, 10=0.01%, 20=0.22%, 50=18.44%
  lat (usec)   : 100=12.28%, 250=15.99%, 500=0.90%, 750=0.80%, 1000=1.14%
  lat (msec)   : 2=2.81%, 4=20.92%, 10=12.88%, 20=6.36%, 50=6.39%
  lat (msec)   : 100=0.48%, 250=0.39%, 500=0.01%
  cpu          : usr=1.68%, sys=5.83%, ctx=158675, majf=0, minf=202
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=95376,94300,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
RND4K_Q32T16: (groupid=2, jobs=1): err= 0: pid=10086: Fri Sep 25 19:39:33 2026
  read: IOPS=3053, BW=11.9MiB/s (12.5MB/s)(358MiB/30018msec)
    slat (nsec): min=1590, max=527348, avg=4114.97, stdev=5917.49
    clat (nsec): min=310, max=196648k, avg=238123.26, stdev=2541195.34
     lat (usec): min=13, max=196652, avg=242.24, stdev=2541.19
    clat percentiles (usec):
     |  1.00th=[    23],  5.00th=[    28], 10.00th=[    31], 20.00th=[    36],
     | 30.00th=[    43], 40.00th=[    57], 50.00th=[    78], 60.00th=[   100],
     | 70.00th=[   122], 80.00th=[   143], 90.00th=[   196], 95.00th=[   453],
     | 99.00th=[  3228], 99.50th=[  4490], 99.90th=[  8848], 99.95th=[ 58459],
     | 99.99th=[127402]
   bw (  KiB/s): min=  905, max=34808, per=6.29%, avg=12352.24, stdev=10047.64, samples=59
   iops        : min=  226, max= 8702, avg=3087.98, stdev=2511.89, samples=59
  write: IOPS=3038, BW=11.9MiB/s (12.4MB/s)(356MiB/30018msec); 0 zone resets
    slat (nsec): min=1670, max=796562, avg=4554.34, stdev=7018.03
    clat (usec): min=177, max=337480, avg=10282.46, stdev=18093.87
     lat (usec): min=180, max=337484, avg=10287.02, stdev=18093.89
    clat percentiles (usec):
     |  1.00th=[   832],  5.00th=[  1696], 10.00th=[  2474], 20.00th=[  3097],
     | 30.00th=[  3392], 40.00th=[  3687], 50.00th=[  4359], 60.00th=[  5866],
     | 70.00th=[  9110], 80.00th=[ 15664], 90.00th=[ 23725], 95.00th=[ 28967],
     | 99.00th=[ 93848], 99.50th=[132645], 99.90th=[227541], 99.95th=[299893],
     | 99.99th=[337642]
   bw (  KiB/s): min=  720, max=34560, per=6.26%, avg=12291.05, stdev=9907.14, samples=59
   iops        : min=  180, max= 8640, avg=3072.69, stdev=2476.80, samples=59
  lat (nsec)   : 500=0.01%, 750=0.01%, 1000=0.01%
  lat (usec)   : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.19%, 50=18.01%
  lat (usec)   : 100=11.78%, 250=16.48%, 500=1.24%, 750=0.83%, 1000=1.13%
  lat (msec)   : 2=2.98%, 4=20.53%, 10=12.60%, 20=7.08%, 50=6.31%
  lat (msec)   : 100=0.41%, 250=0.38%, 500=0.04%
  cpu          : usr=2.12%, sys=5.97%, ctx=148840, majf=0, minf=213
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=91651,91209,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
RND4K_Q1T1: (groupid=3, jobs=1): err= 0: pid=10644: Fri Sep 25 19:39:33 2026
  read: IOPS=536, BW=2146KiB/s (2197kB/s)(62.9MiB/30001msec)
    slat (nsec): min=1830, max=20161, avg=3244.62, stdev=1404.85
    clat (usec): min=19, max=81860, avg=926.67, stdev=4032.14
     lat (usec): min=21, max=81863, avg=929.91, stdev=4032.18
    clat percentiles (usec):
     |  1.00th=[   26],  5.00th=[   31], 10.00th=[   34], 20.00th=[   43],
     | 30.00th=[  482], 40.00th=[  515], 50.00th=[  562], 60.00th=[  586],
     | 70.00th=[  685], 80.00th=[  725], 90.00th=[  758], 95.00th=[  922],
     | 99.00th=[20579], 99.50th=[37487], 99.90th=[49021], 99.95th=[65274],
     | 99.99th=[76022]
   bw (  KiB/s): min=  496, max= 3302, per=100.00%, avg=2146.52, stdev=510.20, samples=60
   iops        : min=  124, max=  825, avg=536.57, stdev=127.57, samples=60
  write: IOPS=543, BW=2174KiB/s (2226kB/s)(63.7MiB/30001msec); 0 zone resets
    slat (nsec): min=2130, max=24011, avg=3507.51, stdev=1504.30
    clat (usec): min=26, max=90810, avg=915.79, stdev=4046.45
     lat (usec): min=28, max=90813, avg=919.29, stdev=4046.45
    clat percentiles (usec):
     |  1.00th=[   33],  5.00th=[   38], 10.00th=[   42], 20.00th=[   59],
     | 30.00th=[  478], 40.00th=[  506], 50.00th=[  553], 60.00th=[  578],
     | 70.00th=[  676], 80.00th=[  717], 90.00th=[  750], 95.00th=[  898],
     | 99.00th=[ 8291], 99.50th=[38536], 99.90th=[47973], 99.95th=[53740],
     | 99.99th=[86508]
   bw (  KiB/s): min=  504, max= 3358, per=100.00%, avg=2174.03, stdev=539.05, samples=60
   iops        : min=  126, max=  839, avg=543.45, stdev=134.76, samples=60
  lat (usec)   : 20=0.01%, 50=19.30%, 100=2.29%, 250=0.03%, 500=15.61%
  lat (usec)   : 750=52.33%, 1000=5.99%
  lat (msec)   : 2=2.41%, 4=0.77%, 10=0.26%, 20=0.03%, 50=0.89%
  lat (msec)   : 100=0.09%
  cpu          : usr=0.37%, sys=1.23%, ctx=32401, majf=0, minf=169
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=16095,16302,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=235MiB/s (246MB/s), 235MiB/s-235MiB/s (246MB/s-246MB/s), io=7050MiB (7392MB), run=30029-30029msec
  WRITE: bw=235MiB/s (247MB/s), 235MiB/s-235MiB/s (247MB/s-247MB/s), io=7066MiB (7409MB), run=30029-30029msec

Run status group 1 (all jobs):
   READ: bw=65.5MiB/s (68.7MB/s), 65.5MiB/s-65.5MiB/s (68.7MB/s-68.7MB/s), io=1965MiB (2060MB), run=30002-30002msec
  WRITE: bw=68.6MiB/s (71.9MB/s), 68.6MiB/s-68.6MiB/s (71.9MB/s-71.9MB/s), io=2058MiB (2158MB), run=30002-30002msec

Run status group 2 (all jobs):
   READ: bw=192MiB/s (201MB/s), 11.7MiB/s-12.4MiB/s (12.2MB/s-13.0MB/s), io=5762MiB (6042MB), run=30010-30059msec
  WRITE: bw=192MiB/s (201MB/s), 11.7MiB/s-12.3MiB/s (12.2MB/s-12.9MB/s), io=5765MiB (6045MB), run=30010-30059msec

Run status group 3 (all jobs):
   READ: bw=2146KiB/s (2197kB/s), 2146KiB/s-2146KiB/s (2197kB/s-2197kB/s), io=62.9MiB (65.9MB), run=30001-30001msec
  WRITE: bw=2174KiB/s (2226kB/s), 2174KiB/s-2174KiB/s (2226kB/s-2226kB/s), io=63.7MiB (66.8MB), run=30001-30001msec

Trying a single test at the Time now shows much better Results:

Code:

fio --time_based --runtime=30 --ramp_time=5 --ioengine=io_uring \
--size=4G --filename=/seq_test.fio \
--name=PURE_SEQ --bs=1m --rw=write --iodepth=8 --numjobs=1
PURE_SEQ: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=io_uring, iodepth=8
fio-3.39
Starting 1 process
PURE_SEQ: Laying out IO file (1 file / 4096MiB)
Jobs: 1 (f=1): [W(1)][100.0%][w=1291MiB/s][w=1291 IOPS][eta 00m:00s]
PURE_SEQ: (groupid=0, jobs=1): err= 0: pid=34627: Fri Sep 25 20:02:03 2026
  write: IOPS=1454, BW=1455MiB/s (1525MB/s)(42.6GiB/30009msec); 0 zone resets
    slat (usec): min=18, max=1705, avg=33.86, stdev=12.23
    clat (usec): min=591, max=51276, avg=5457.70, stdev=1945.79
     lat (usec): min=645, max=51313, avg=5491.56, stdev=1945.76
    clat percentiles (usec):
     |  1.00th=[ 3916],  5.00th=[ 4228], 10.00th=[ 4359], 20.00th=[ 4555],
     | 30.00th=[ 4752], 40.00th=[ 4948], 50.00th=[ 5145], 60.00th=[ 5407],
     | 70.00th=[ 5735], 80.00th=[ 6128], 90.00th=[ 6718], 95.00th=[ 7242],
     | 99.00th=[ 8225], 99.50th=[11338], 99.90th=[35914], 99.95th=[39060],
     | 99.99th=[46400]
   bw (  MiB/s): min=  946, max= 1894, per=100.00%, avg=1455.07, stdev=222.33, samples=60
   iops        : min=  946, max= 1894, avg=1455.02, stdev=222.35, samples=60
  lat (usec)   : 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.03%, 4=2.16%, 10=97.26%, 20=0.21%, 50=0.35%
  lat (msec)   : 100=0.01%
  cpu          : usr=5.29%, sys=2.46%, ctx=43664, majf=0, minf=55
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,43642,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
  WRITE: bw=1455MiB/s (1525MB/s), 1455MiB/s-1455MiB/s (1525MB/s-1525MB/s), io=42.6GiB (45.8GB), run=30009-30009msec

The biggest Performance Difference is due to the different fio Command.

I also added this to /etc/crypttab for each Drive though:

Code:

no-read-workqueue,no-write-workqueue

But ... If I do a simple zfs send | zfs receive (locally, from that ZFS Pool to within that same ZFS Pool), I am getting around 50 MiB/s.

SATA Drives are WAY faster than this.

ashift=12, LUKS Sector Size 4096, NVMe Formatted with LBA 4096.

TrevorH · Sep 25, 2026

I'd run a more basic check at the drive level first and see if there is a problem underlying everything. Try dd if=/dev/nvme0n1 of=/dev/null iflag=direct bs=16M status=progress (nvme0n1 may vary) and see if it gets decent throughput.

luckylinux · Sep 25, 2026

TrevorH said:
I'd run a more basic check at the drive level first and see if there is a problem underlying everything. Try dd if=/dev/nvme0n1 of=/dev/null iflag=direct bs=16M status=progress (nvme0n1 may vary) and see if it gets decent throughput.

That's not possible without destroying the Drive ...

If you can be satisfied by ext4 or vfat I have a 2GB /boot and 1GB EFI (individual EFI and mdadm RAID-1 ext4 /boot), maybe that can be done non-destructively ?

I feel I had similar Problems on all NVMe Drives I installed Proxmox VE / Ubuntu on, though I don't really have the objective Numbers to back the "it feels slower than a SATA SSD" Claim up

.

jhartbarger · Sep 25, 2026

luckylinux said:

Trying a single test at the Time now shows much better Results:

Code:

fio --time_based --runtime=30 --ramp_time=5 --ioengine=io_uring \
--size=4G --filename=/seq_test.fio \
--name=PURE_SEQ --bs=1m --rw=write --iodepth=8 --numjobs=1
PURE_SEQ: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=io_uring, iodepth=8
fio-3.39
Starting 1 process
PURE_SEQ: Laying out IO file (1 file / 4096MiB)
Jobs: 1 (f=1): [W(1)][100.0%][w=1291MiB/s][w=1291 IOPS][eta 00m:00s]
PURE_SEQ: (groupid=0, jobs=1): err= 0: pid=34627: Fri Sep 25 20:02:03 2026
  write: IOPS=1454, BW=1455MiB/s (1525MB/s)(42.6GiB/30009msec); 0 zone resets
    slat (usec): min=18, max=1705, avg=33.86, stdev=12.23
    clat (usec): min=591, max=51276, avg=5457.70, stdev=1945.79
     lat (usec): min=645, max=51313, avg=5491.56, stdev=1945.76
    clat percentiles (usec):
     |  1.00th=[ 3916],  5.00th=[ 4228], 10.00th=[ 4359], 20.00th=[ 4555],
     | 30.00th=[ 4752], 40.00th=[ 4948], 50.00th=[ 5145], 60.00th=[ 5407],
     | 70.00th=[ 5735], 80.00th=[ 6128], 90.00th=[ 6718], 95.00th=[ 7242],
     | 99.00th=[ 8225], 99.50th=[11338], 99.90th=[35914], 99.95th=[39060],
     | 99.99th=[46400]
   bw (  MiB/s): min=  946, max= 1894, per=100.00%, avg=1455.07, stdev=222.33, samples=60
   iops        : min=  946, max= 1894, avg=1455.02, stdev=222.35, samples=60
  lat (usec)   : 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.03%, 4=2.16%, 10=97.26%, 20=0.21%, 50=0.35%
  lat (msec)   : 100=0.01%
  cpu          : usr=5.29%, sys=2.46%, ctx=43664, majf=0, minf=55
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,43642,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
  WRITE: bw=1455MiB/s (1525MB/s), 1455MiB/s-1455MiB/s (1525MB/s-1525MB/s), io=42.6GiB (45.8GB), run=30009-30009msec

The biggest Performance Difference is due to the different fio Command.

I also added this to /etc/crypttab for each Drive though:

Code:

no-read-workqueue,no-write-workqueue

there is an error in this command, run the below commands and send the output

run this test instead:
fio --time_based --runtime=30 --ramp_time=5 --ioengine=io_uring --direct=1 \ --size=2G --stonewall \ --name=SEQ1M_Read_Q8T1 --bs=1m --rw=read --iodepth=8 --numjobs=1 --filename=/testfile_seq.fio \ --name=SEQ1M_Write_Q8T1 --bs=1m --rw=write --iodepth=8 --numjobs=1 --filename=/testfile_seq_w.fio \ --name=RND4K_Read_Q32T16 --bs=4k --rw=randread --iodepth=32 --numjobs=16 --filename=/testfile_rnd.fio \ --name=RND4K_Write_Q32T16 --bs=4k --rw=randwrite --iodepth=32 --numjobs=16 --filename=/testfile_rnd_w.fio

and then run:
fio --time_based --runtime=30 --ramp_time=5 --ioengine=io_uring --direct=1 \ --size=4G --filename=/seq_test.fio \ --name=PURE_SEQ --bs=1m --rw=write --iodepth=8 --numjobs=1

TrevorH · Sep 25, 2026

That's not possible without destroying the Drive ...

The command I gave reads from the drive and writes to /dev/null. It does not write to the drive at all. I would not post a destructive command without also posting major warnings that it was going to overwrite things.

luckylinux · Sep 25, 2026

TrevorH said:
The command I gave reads from the drive and writes to /dev/null. It does not write to the drive at all. I would not post a destructive command without also posting major warnings that it was going to overwrite things.

I was quite tired yesterday apparently.

Indeed this was read-only.

It starts at around 400-500 MB/s (SATA-like speeds) for a Second or so, then quickly drops to around 200MB/s.

luckylinux · Sep 25, 2026

jhartbarger said:
there is an error in this command, run the below commands and send the output

run this test instead:
fio --time_based --runtime=30 --ramp_time=5 --ioengine=io_uring --direct=1 \ --size=2G --stonewall \ --name=SEQ1M_Read_Q8T1 --bs=1m --rw=read --iodepth=8 --numjobs=1 --filename=/testfile_seq.fio \ --name=SEQ1M_Write_Q8T1 --bs=1m --rw=write --iodepth=8 --numjobs=1 --filename=/testfile_seq_w.fio \ --name=RND4K_Read_Q32T16 --bs=4k --rw=randread --iodepth=32 --numjobs=16 --filename=/testfile_rnd.fio \ --name=RND4K_Write_Q32T16 --bs=4k --rw=randwrite --iodepth=32 --numjobs=16 --filename=/testfile_rnd_w.fio

and then run:
fio --time_based --runtime=30 --ramp_time=5 --ioengine=io_uring --direct=1 \ --size=4G --filename=/seq_test.fio \ --name=PURE_SEQ --bs=1m --rw=write --iodepth=8 --numjobs=1

There is some Syntax Error and I just woke up

- Fixed (probably Backslash \ is some wrong Place).

Code:

root@pve:/#
fio --time_based \
    --runtime=30 \
    --ramp_time=5 \
    --ioengine=io_uring \
    --direct=1 \
    --size=2G \
    --stonewall \
    --name=SEQ1M_Read_Q8T1 --bs=1m --rw=read --iodepth=8 --numjobs=1 --filename=/testfile_seq.fio \
    --name=SEQ1M_Write_Q8T1 --bs=1m --rw=write --iodepth=8 --numjobs=1 --filename=/testfile_seq_w.fio \
    --name=RND4K_Read_Q32T16 --bs=4k --rw=randread --iodepth=32 --numjobs=16 --filename=/testfile_rnd.fio \
    --name=RND4K_Write_Q32T16 --bs=4k --rw=randwrite --iodepth=32 --numjobs=16 --filename=/testfile_rnd_w.fio

SEQ1M_Read_Q8T1: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=io_uring, iodepth=8
SEQ1M_Write_Q8T1: (g=1): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=io_uring, iodepth=8
RND4K_Read_Q32T16: (g=2): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=io_uring, iodepth=32
...
RND4K_Write_Q32T16: (g=3): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=io_uring, iodepth=32
...
fio-3.39
Starting 34 processes
SEQ1M_Read_Q8T1: Laying out IO file (1 file / 2048MiB)
SEQ1M_Write_Q8T1: Laying out IO file (1 file / 2048MiB)
RND4K_Read_Q32T16: Laying out IO file (1 file / 2048MiB)
RND4K_Write_Q32T16: Laying out IO file (1 file / 2048MiB)
Jobs: 16 (f=16): [_(18),w(16)][78.5%][w=575MiB/s][w=147k IOPS][eta 00m:39s]     
SEQ1M_Read_Q8T1: (groupid=0, jobs=1): err= 0: pid=1854264: Sat Sep 26 07:23:42 2026
  read: IOPS=2831, BW=2831MiB/s (2969MB/s)(83.0GiB/30005msec)
    slat (nsec): min=1620, max=307155, avg=2895.26, stdev=1413.93
    clat (usec): min=834, max=121178, avg=2822.12, stdev=2287.47
     lat (usec): min=836, max=121180, avg=2825.02, stdev=2287.44
    clat percentiles (usec):
     |  1.00th=[ 1074],  5.00th=[ 1172], 10.00th=[ 1237], 20.00th=[ 1352],
     | 30.00th=[ 1532], 40.00th=[ 1860], 50.00th=[ 2638], 60.00th=[ 3163],
     | 70.00th=[ 3621], 80.00th=[ 4080], 90.00th=[ 4686], 95.00th=[ 5145],
     | 99.00th=[ 6063], 99.50th=[ 6587], 99.90th=[40109], 99.95th=[51119],
     | 99.99th=[80217]
   bw (  MiB/s): min= 2608, max= 2966, per=100.00%, avg=2832.31, stdev=74.96, samples=60
   iops        : min= 2608, max= 2966, avg=2832.22, stdev=74.95, samples=60
  lat (usec)   : 1000=0.20%
  lat (msec)   : 2=41.68%, 4=36.69%, 10=21.29%, 20=0.03%, 50=0.08%
  lat (msec)   : 100=0.05%, 250=0.01%
  cpu          : usr=0.61%, sys=2.85%, ctx=83817, majf=0, minf=73
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=84945,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8
SEQ1M_Write_Q8T1: (groupid=1, jobs=1): err= 0: pid=1855874: Sat Sep 26 07:23:42 2026
  write: IOPS=185, BW=185MiB/s (194MB/s)(5567MiB/30039msec); 0 zone resets
    slat (usec): min=24, max=537, avg=44.86, stdev=19.06
    clat (usec): min=41940, max=76287, avg=43139.16, stdev=1027.92
     lat (usec): min=41979, max=76326, avg=43184.01, stdev=1027.62
    clat percentiles (usec):
     |  1.00th=[42206],  5.00th=[42206], 10.00th=[42206], 20.00th=[42206],
     | 30.00th=[42730], 40.00th=[42730], 50.00th=[42730], 60.00th=[43254],
     | 70.00th=[43254], 80.00th=[43779], 90.00th=[44303], 95.00th=[44827],
     | 99.00th=[45351], 99.50th=[45351], 99.90th=[47973], 99.95th=[65274],
     | 99.99th=[76022]
   bw (  KiB/s): min=184320, max=192897, per=100.00%, avg=189816.58, stdev=1863.22, samples=60
   iops        : min=  180, max=  188, avg=185.30, stdev= 1.83, samples=60
  lat (msec)   : 50=100.04%, 100=0.09%
  cpu          : usr=0.96%, sys=0.25%, ctx=5570, majf=0, minf=2100
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,5560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8
RND4K_Read_Q32T16: (groupid=2, jobs=1): err= 0: pid=1857263: Sat Sep 26 07:23:42 2026
  read: IOPS=866, BW=3470KiB/s (3553kB/s)(104MiB/30642msec)
    slat (nsec): min=1641, max=83482, avg=3401.43, stdev=1944.96
    clat (usec): min=366, max=1306.0k, avg=37272.11, stdev=154640.59
     lat (usec): min=369, max=1306.1k, avg=37275.51, stdev=154640.62
    clat percentiles (usec):
     |  1.00th=[    441],  5.00th=[    486], 10.00th=[    515],
     | 20.00th=[    562], 30.00th=[    603], 40.00th=[    644],
     | 50.00th=[    701], 60.00th=[    758], 70.00th=[    824],
     | 80.00th=[    938], 90.00th=[   1188], 95.00th=[ 557843],
     | 99.00th=[ 734004], 99.50th=[ 784335], 99.90th=[ 943719],
     | 99.95th=[ 994051], 99.99th=[1283458]
   bw (  KiB/s): min= 1544, max= 5808, per=6.07%, avg=3532.45, stdev=911.70, samples=60
   iops        : min=  386, max= 1452, avg=883.07, stdev=227.93, samples=60
  lat (usec)   : 500=7.74%, 750=50.25%, 1000=25.62%
  lat (msec)   : 2=11.06%, 4=0.05%, 500=0.05%, 750=4.57%, 1000=0.73%
  lat (msec)   : 2000=0.05%
  cpu          : usr=0.27%, sys=1.00%, ctx=26667, majf=0, minf=37
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=26550,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
RND4K_Read_Q32T16: (groupid=2, jobs=1): err= 0: pid=1857265: Sat Sep 26 07:23:42 2026
  read: IOPS=915, BW=3668KiB/s (3756kB/s)(110MiB/30610msec)
    slat (nsec): min=1650, max=85361, avg=3332.99, stdev=1502.51
    clat (usec): min=370, max=1233.0k, avg=35208.55, stdev=150717.57
     lat (usec): min=373, max=1233.0k, avg=35211.88, stdev=150717.61
    clat percentiles (usec):
     |  1.00th=[    437],  5.00th=[    482], 10.00th=[    510],
     | 20.00th=[    553], 30.00th=[    603], 40.00th=[    644],
     | 50.00th=[    701], 60.00th=[    758], 70.00th=[    824],
     | 80.00th=[    938], 90.00th=[   1156], 95.00th=[ 501220],
     | 99.00th=[ 734004], 99.50th=[ 784335], 99.90th=[ 943719],
     | 99.95th=[1035994], 99.99th=[1233126]
   bw (  KiB/s): min= 1920, max= 5904, per=6.42%, avg=3732.90, stdev=972.23, samples=60
   iops        : min=  480, max= 1476, avg=933.17, stdev=243.05, samples=60
  lat (usec)   : 500=7.81%, 750=50.30%, 1000=25.74%
  lat (msec)   : 2=11.14%, 4=0.04%, 10=0.01%, 500=0.07%, 750=4.25%
  lat (msec)   : 1000=0.70%, 2000=0.06%
  cpu          : usr=0.29%, sys=1.04%, ctx=28132, majf=0, minf=36
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=28037,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
RND4K_Read_Q32T16: (groupid=2, jobs=1): err= 0: pid=1857267: Sat Sep 26 07:23:42 2026
  read: IOPS=938, BW=3760KiB/s (3850kB/s)(113MiB/30646msec)
    slat (nsec): min=1630, max=81921, avg=3294.75, stdev=1618.20
    clat (usec): min=376, max=1329.0k, avg=34336.93, stdev=148881.13
     lat (usec): min=378, max=1329.0k, avg=34340.23, stdev=148881.16
    clat percentiles (usec):
     |  1.00th=[    437],  5.00th=[    482], 10.00th=[    510],
     | 20.00th=[    553], 30.00th=[    594], 40.00th=[    644],
     | 50.00th=[    701], 60.00th=[    758], 70.00th=[    824],
     | 80.00th=[    930], 90.00th=[   1156], 95.00th=[   1975],
     | 99.00th=[ 725615], 99.50th=[ 775947], 99.90th=[ 960496],
     | 99.95th=[1061159], 99.99th=[1266680]
   bw (  KiB/s): min= 2320, max= 6533, per=6.59%, avg=3833.95, stdev=979.73, samples=60
   iops        : min=  580, max= 1633, avg=958.42, stdev=244.90, samples=60
  lat (usec)   : 500=8.02%, 750=50.17%, 1000=25.99%
  lat (msec)   : 2=10.92%, 4=0.05%, 10=0.01%, 500=0.07%, 750=4.16%
  lat (msec)   : 1000=0.64%, 2000=0.08%
  cpu          : usr=0.31%, sys=1.03%, ctx=28897, majf=0, minf=36
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=28775,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
RND4K_Read_Q32T16: (groupid=2, jobs=1): err= 0: pid=1857268: Sat Sep 26 07:23:42 2026
  read: IOPS=944, BW=3783KiB/s (3874kB/s)(113MiB/30629msec)
    slat (nsec): min=1660, max=77321, avg=3365.43, stdev=1443.69
    clat (usec): min=367, max=1307.0k, avg=34180.10, stdev=148567.40
     lat (usec): min=370, max=1307.0k, avg=34183.46, stdev=148567.42
    clat percentiles (usec):
     |  1.00th=[    437],  5.00th=[    482], 10.00th=[    515],
     | 20.00th=[    562], 30.00th=[    603], 40.00th=[    644],
     | 50.00th=[    701], 60.00th=[    758], 70.00th=[    824],
     | 80.00th=[    930], 90.00th=[   1172], 95.00th=[   1942],
     | 99.00th=[ 725615], 99.50th=[ 775947], 99.90th=[ 952108],
     | 99.95th=[1069548], 99.99th=[1233126]
   bw (  KiB/s): min= 2544, max= 5480, per=6.62%, avg=3850.78, stdev=841.85, samples=60
   iops        : min=  636, max= 1370, avg=962.63, stdev=210.46, samples=60
  lat (usec)   : 500=7.64%, 750=50.85%, 1000=25.47%
  lat (msec)   : 2=11.15%, 4=0.06%, 500=0.08%, 750=4.15%, 1000=0.62%
  lat (msec)   : 2000=0.09%
  cpu          : usr=0.33%, sys=1.04%, ctx=29057, majf=0, minf=36
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=28939,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
RND4K_Read_Q32T16: (groupid=2, jobs=1): err= 0: pid=1857269: Sat Sep 26 07:23:42 2026
  read: IOPS=861, BW=3449KiB/s (3532kB/s)(103MiB/30641msec)
    slat (nsec): min=1660, max=72271, avg=3381.55, stdev=1796.98
    clat (usec): min=358, max=1305.3k, avg=37372.79, stdev=155461.38
     lat (usec): min=389, max=1305.3k, avg=37376.18, stdev=155461.40
    clat percentiles (usec):
     |  1.00th=[    441],  5.00th=[    482], 10.00th=[    510],
     | 20.00th=[    553], 30.00th=[    594], 40.00th=[    644],
     | 50.00th=[    701], 60.00th=[    758], 70.00th=[    824],
     | 80.00th=[    938], 90.00th=[   1172], 95.00th=[ 557843],
     | 99.00th=[ 734004], 99.50th=[ 784335], 99.90th=[ 985662],
     | 99.95th=[1098908], 99.99th=[1283458]
   bw (  KiB/s): min= 1608, max= 6456, per=6.04%, avg=3510.53, stdev=1062.86, samples=60
   iops        : min=  402, max= 1614, avg=877.57, stdev=265.71, samples=60
  lat (usec)   : 500=7.82%, 750=50.36%, 1000=25.43%
  lat (msec)   : 2=11.06%, 4=0.05%, 500=0.08%, 750=4.57%, 1000=0.66%
  lat (msec)   : 2000=0.09%
  cpu          : usr=0.31%, sys=0.95%, ctx=26524, majf=0, minf=37
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=26390,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
RND4K_Read_Q32T16: (groupid=2, jobs=1): err= 0: pid=1857270: Sat Sep 26 07:23:42 2026
  read: IOPS=865, BW=3467KiB/s (3550kB/s)(104MiB/30587msec)
    slat (nsec): min=1660, max=71151, avg=3284.46, stdev=1375.51
    clat (usec): min=379, max=1231.7k, avg=37289.95, stdev=155042.90
     lat (usec): min=383, max=1231.7k, avg=37293.23, stdev=155042.94
    clat percentiles (usec):
     |  1.00th=[    437],  5.00th=[    482], 10.00th=[    510],
     | 20.00th=[    562], 30.00th=[    603], 40.00th=[    644],
     | 50.00th=[    709], 60.00th=[    766], 70.00th=[    824],
     | 80.00th=[    938], 90.00th=[   1172], 95.00th=[ 557843],
     | 99.00th=[ 734004], 99.50th=[ 784335], 99.90th=[ 985662],
     | 99.95th=[1044382], 99.99th=[1216349]
   bw (  KiB/s): min= 1512, max= 5819, per=6.05%, avg=3519.45, stdev=955.33, samples=60
   iops        : min=  378, max= 1454, avg=879.80, stdev=238.78, samples=60
  lat (usec)   : 500=7.90%, 750=49.49%, 1000=26.47%
  lat (msec)   : 2=10.82%, 4=0.05%, 500=0.09%, 750=4.51%, 1000=0.71%
  lat (msec)   : 2000=0.08%
  cpu          : usr=0.28%, sys=0.95%, ctx=26597, majf=0, minf=36
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=26480,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
RND4K_Read_Q32T16: (groupid=2, jobs=1): err= 0: pid=1857271: Sat Sep 26 07:23:42 2026
  read: IOPS=904, BW=3622KiB/s (3709kB/s)(108MiB/30636msec)
    slat (nsec): min=1640, max=81571, avg=3527.30, stdev=1774.80
    clat (usec): min=352, max=1269.8k, avg=35595.40, stdev=151404.28
     lat (usec): min=355, max=1269.8k, avg=35598.92, stdev=151404.34
    clat percentiles (usec):
     |  1.00th=[    437],  5.00th=[    482], 10.00th=[    510],
     | 20.00th=[    553], 30.00th=[    603], 40.00th=[    644],
     | 50.00th=[    709], 60.00th=[    766], 70.00th=[    824],
     | 80.00th=[    938], 90.00th=[   1172], 95.00th=[ 517997],
     | 99.00th=[ 725615], 99.50th=[ 775947], 99.90th=[ 960496],
     | 99.95th=[1044382], 99.99th=[1233126]
   bw (  KiB/s): min= 1960, max= 7144, per=6.34%, avg=3689.17, stdev=1029.86, samples=60
   iops        : min=  490, max= 1786, avg=922.22, stdev=257.47, samples=60
  lat (usec)   : 500=7.90%, 750=50.04%, 1000=25.68%
  lat (msec)   : 2=11.29%, 4=0.06%, 500=0.07%, 750=4.37%, 1000=0.62%
  lat (msec)   : 2000=0.08%
  cpu          : usr=0.34%, sys=1.00%, ctx=27833, majf=0, minf=37
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=27711,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
RND4K_Read_Q32T16: (groupid=2, jobs=1): err= 0: pid=1857272: Sat Sep 26 07:23:42 2026
  read: IOPS=886, BW=3548KiB/s (3633kB/s)(106MiB/30669msec)
    slat (nsec): min=1650, max=75541, avg=3336.07, stdev=1567.03
    clat (usec): min=343, max=1270.3k, avg=36415.79, stdev=153154.60
     lat (usec): min=345, max=1270.3k, avg=36419.12, stdev=153154.65
    clat percentiles (usec):
     |  1.00th=[    437],  5.00th=[    482], 10.00th=[    510],
     | 20.00th=[    553], 30.00th=[    594], 40.00th=[    644],
     | 50.00th=[    709], 60.00th=[    766], 70.00th=[    824],
     | 80.00th=[    938], 90.00th=[   1172], 95.00th=[ 541066],
     | 99.00th=[ 725615], 99.50th=[ 784335], 99.90th=[ 968885],
     | 99.95th=[1098908], 99.99th=[1249903]
   bw (  KiB/s): min= 1912, max= 5800, per=6.22%, avg=3616.80, stdev=783.63, samples=60
   iops        : min=  478, max= 1450, avg=904.13, stdev=195.93, samples=60
  lat (usec)   : 500=8.10%, 750=49.71%, 1000=25.65%
  lat (msec)   : 2=11.36%, 4=0.03%, 500=0.08%, 750=4.45%, 1000=0.66%
  lat (msec)   : 2000=0.08%
  cpu          : usr=0.25%, sys=1.02%, ctx=27313, majf=0, minf=38
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=27174,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
RND4K_Read_Q32T16: (groupid=2, jobs=1): err= 0: pid=1857273: Sat Sep 26 07:23:42 2026
  read: IOPS=927, BW=3713KiB/s (3802kB/s)(111MiB/30682msec)
    slat (nsec): min=1650, max=78612, avg=3326.07, stdev=1423.95
    clat (usec): min=369, max=1302.4k, avg=34782.52, stdev=150058.24
     lat (usec): min=372, max=1302.4k, avg=34785.85, stdev=150058.28
    clat percentiles (usec):
     |  1.00th=[    437],  5.00th=[    482], 10.00th=[    510],
     | 20.00th=[    562], 30.00th=[    594], 40.00th=[    644],
     | 50.00th=[    701], 60.00th=[    758], 70.00th=[    824],
     | 80.00th=[    938], 90.00th=[   1156], 95.00th=[ 446694],
     | 99.00th=[ 734004], 99.50th=[ 784335], 99.90th=[ 985662],
     | 99.95th=[1069548], 99.99th=[1283458]
   bw (  KiB/s): min= 1968, max= 6632, per=6.51%, avg=3784.13, stdev=972.79, samples=60
   iops        : min=  492, max= 1658, avg=945.97, stdev=243.21, samples=60
  lat (usec)   : 500=7.81%, 750=50.33%, 1000=25.64%
  lat (msec)   : 2=11.27%, 4=0.05%, 500=0.06%, 750=4.23%, 1000=0.64%
  lat (msec)   : 2000=0.08%
  cpu          : usr=0.33%, sys=1.01%, ctx=28541, majf=0, minf=38
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=28451,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
RND4K_Read_Q32T16: (groupid=2, jobs=1): err= 0: pid=1857274: Sat Sep 26 07:23:42 2026
  read: IOPS=921, BW=3691KiB/s (3780kB/s)(111MiB/30666msec)
    slat (nsec): min=1640, max=70031, avg=3366.10, stdev=1438.95
    clat (usec): min=382, max=1318.6k, avg=34991.05, stdev=150140.89
     lat (usec): min=386, max=1318.6k, avg=34994.42, stdev=150140.91
    clat percentiles (usec):
     |  1.00th=[    437],  5.00th=[    482], 10.00th=[    515],
     | 20.00th=[    562], 30.00th=[    603], 40.00th=[    644],
     | 50.00th=[    701], 60.00th=[    758], 70.00th=[    824],
     | 80.00th=[    938], 90.00th=[   1156], 95.00th=[ 488637],
     | 99.00th=[ 734004], 99.50th=[ 775947], 99.90th=[ 977273],
     | 99.95th=[1044382], 99.99th=[1266680]
   bw (  KiB/s): min= 1928, max= 6240, per=6.47%, avg=3760.13, stdev=897.99, samples=60
   iops        : min=  482, max= 1560, avg=939.97, stdev=224.51, samples=60
  lat (usec)   : 500=7.82%, 750=50.29%, 1000=25.88%
  lat (msec)   : 2=11.02%, 4=0.04%, 500=0.08%, 750=4.23%, 1000=0.67%
  lat (msec)   : 2000=0.08%
  cpu          : usr=0.33%, sys=1.01%, ctx=28405, majf=0, minf=36
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=28269,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
RND4K_Read_Q32T16: (groupid=2, jobs=1): err= 0: pid=1857275: Sat Sep 26 07:23:42 2026
  read: IOPS=956, BW=3829KiB/s (3921kB/s)(115MiB/30641msec)
    slat (nsec): min=1630, max=79841, avg=3322.07, stdev=1566.34
    clat (usec): min=348, max=1245.7k, avg=33707.66, stdev=147381.48
     lat (usec): min=351, max=1245.7k, avg=33710.99, stdev=147381.52
    clat percentiles (usec):
     |  1.00th=[    437],  5.00th=[    482], 10.00th=[    510],
     | 20.00th=[    553], 30.00th=[    594], 40.00th=[    644],
     | 50.00th=[    701], 60.00th=[    758], 70.00th=[    824],
     | 80.00th=[    930], 90.00th=[   1156], 95.00th=[   1795],
     | 99.00th=[ 725615], 99.50th=[ 775947], 99.90th=[ 943719],
     | 99.95th=[1019216], 99.99th=[1233126]
   bw (  KiB/s): min= 1699, max= 5531, per=6.70%, avg=3899.83, stdev=980.41, samples=60
   iops        : min=  424, max= 1382, avg=974.87, stdev=245.10, samples=60
  lat (usec)   : 500=8.00%, 750=50.83%, 1000=25.51%
  lat (msec)   : 2=10.83%, 4=0.06%, 500=0.04%, 750=4.13%, 1000=0.64%
  lat (msec)   : 2000=0.05%
  cpu          : usr=0.31%, sys=1.07%, ctx=29396, majf=0, minf=36
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=29301,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
RND4K_Read_Q32T16: (groupid=2, jobs=1): err= 0: pid=1857276: Sat Sep 26 07:23:42 2026
  read: IOPS=899, BW=3600KiB/s (3687kB/s)(107MiB/30562msec)
    slat (nsec): min=1680, max=76022, avg=3403.44, stdev=1504.09
    clat (usec): min=384, max=1239.1k, avg=35845.82, stdev=151980.65
     lat (usec): min=387, max=1239.1k, avg=35849.22, stdev=151980.67
    clat percentiles (usec):
     |  1.00th=[    437],  5.00th=[    482], 10.00th=[    510],
     | 20.00th=[    553], 30.00th=[    594], 40.00th=[    644],
     | 50.00th=[    701], 60.00th=[    766], 70.00th=[    832],
     | 80.00th=[    938], 90.00th=[   1172], 95.00th=[ 526386],
     | 99.00th=[ 725615], 99.50th=[ 775947], 99.90th=[ 985662],
     | 99.95th=[1044382], 99.99th=[1233126]
   bw (  KiB/s): min=   16, max= 7118, per=6.20%, avg=3604.80, stdev=1234.83, samples=61
   iops        : min=    4, max= 1779, avg=901.13, stdev=308.69, samples=61
  lat (usec)   : 500=7.92%, 750=50.13%, 1000=26.00%
  lat (msec)   : 2=10.81%, 4=0.07%, 500=0.10%, 750=4.35%, 1000=0.65%
  lat (msec)   : 2000=0.08%
  cpu          : usr=0.29%, sys=1.02%, ctx=27565, majf=0, minf=36
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=27478,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
RND4K_Read_Q32T16: (groupid=2, jobs=1): err= 0: pid=1857277: Sat Sep 26 07:23:42 2026
  read: IOPS=925, BW=3707KiB/s (3796kB/s)(111MiB/30623msec)
    slat (nsec): min=1640, max=92552, avg=3339.54, stdev=1683.50
    clat (usec): min=385, max=1242.4k, avg=34814.05, stdev=149717.91
     lat (usec): min=388, max=1242.4k, avg=34817.39, stdev=149717.93
    clat percentiles (usec):
     |  1.00th=[    433],  5.00th=[    482], 10.00th=[    510],
     | 20.00th=[    553], 30.00th=[    594], 40.00th=[    644],
     | 50.00th=[    701], 60.00th=[    758], 70.00th=[    824],
     | 80.00th=[    930], 90.00th=[   1156], 95.00th=[ 488637],
     | 99.00th=[ 725615], 99.50th=[ 784335], 99.90th=[ 943719],
     | 99.95th=[ 994051], 99.99th=[1233126]
   bw (  KiB/s): min= 1008, max= 6060, per=6.49%, avg=3775.65, stdev=1035.35, samples=60
   iops        : min=  252, max= 1515, avg=943.87, stdev=258.84, samples=60
  lat (usec)   : 500=8.27%, 750=50.23%, 1000=25.71%
  lat (msec)   : 2=10.82%, 4=0.05%, 500=0.06%, 750=4.21%, 1000=0.71%
  lat (msec)   : 2000=0.05%
  cpu          : usr=0.32%, sys=1.01%, ctx=28446, majf=0, minf=39
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=28347,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
RND4K_Read_Q32T16: (groupid=2, jobs=1): err= 0: pid=1857279: Sat Sep 26 07:23:42 2026
  read: IOPS=923, BW=3698KiB/s (3787kB/s)(111MiB/30631msec)
    slat (nsec): min=1640, max=72131, avg=3351.52, stdev=1614.94
    clat (usec): min=385, max=1270.2k, avg=34937.58, stdev=150167.06
     lat (usec): min=388, max=1270.2k, avg=34940.93, stdev=150167.08
    clat percentiles (usec):
     |  1.00th=[    437],  5.00th=[    482], 10.00th=[    510],
     | 20.00th=[    553], 30.00th=[    594], 40.00th=[    644],
     | 50.00th=[    701], 60.00th=[    766], 70.00th=[    824],
     | 80.00th=[    938], 90.00th=[   1156], 95.00th=[ 471860],
     | 99.00th=[ 734004], 99.50th=[ 784335], 99.90th=[ 935330],
     | 99.95th=[1010828], 99.99th=[1266680]
   bw (  KiB/s): min= 1528, max= 7288, per=6.47%, avg=3762.98, stdev=1145.95, samples=60
   iops        : min=  382, max= 1822, avg=940.70, stdev=286.50, samples=60
  lat (usec)   : 500=7.85%, 750=49.87%, 1000=26.24%
  lat (msec)   : 2=11.05%, 4=0.06%, 500=0.07%, 750=4.26%, 1000=0.63%
  lat (msec)   : 2000=0.07%
  cpu          : usr=0.26%, sys=1.06%, ctx=28409, majf=0, minf=36
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=28289,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
RND4K_Read_Q32T16: (groupid=2, jobs=1): err= 0: pid=1857280: Sat Sep 26 07:23:42 2026
  read: IOPS=901, BW=3610KiB/s (3696kB/s)(108MiB/30641msec)
    slat (nsec): min=1660, max=87412, avg=3378.31, stdev=1778.06
    clat (usec): min=370, max=1245.2k, avg=35752.48, stdev=151829.19
     lat (usec): min=373, max=1245.2k, avg=35755.86, stdev=151829.24
    clat percentiles (usec):
     |  1.00th=[    437],  5.00th=[    482], 10.00th=[    510],
     | 20.00th=[    553], 30.00th=[    594], 40.00th=[    644],
     | 50.00th=[    701], 60.00th=[    758], 70.00th=[    824],
     | 80.00th=[    938], 90.00th=[   1156], 95.00th=[ 522191],
     | 99.00th=[ 734004], 99.50th=[ 784335], 99.90th=[ 968885],
     | 99.95th=[1035994], 99.99th=[1233126]
   bw (  KiB/s): min= 2160, max= 5960, per=6.32%, avg=3676.28, stdev=872.87, samples=60
   iops        : min=  540, max= 1490, avg=919.00, stdev=218.20, samples=60
  lat (usec)   : 500=7.97%, 750=50.03%, 1000=26.12%
  lat (msec)   : 2=10.78%, 4=0.05%, 500=0.07%, 750=4.35%, 1000=0.67%
  lat (msec)   : 2000=0.07%
  cpu          : usr=0.34%, sys=0.97%, ctx=27759, majf=0, minf=36
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=27619,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
RND4K_Read_Q32T16: (groupid=2, jobs=1): err= 0: pid=1857281: Sat Sep 26 07:23:42 2026
  read: IOPS=906, BW=3630KiB/s (3717kB/s)(109MiB/30621msec)
    slat (nsec): min=1650, max=88351, avg=3302.43, stdev=1422.18
    clat (usec): min=375, max=1214.6k, avg=35645.40, stdev=151300.34
     lat (usec): min=378, max=1214.6k, avg=35648.71, stdev=151300.38
    clat percentiles (usec):
     |  1.00th=[    437],  5.00th=[    482], 10.00th=[    510],
     | 20.00th=[    553], 30.00th=[    594], 40.00th=[    644],
     | 50.00th=[    701], 60.00th=[    766], 70.00th=[    824],
     | 80.00th=[    938], 90.00th=[   1172], 95.00th=[ 526386],
     | 99.00th=[ 734004], 99.50th=[ 784335], 99.90th=[ 952108],
     | 99.95th=[1019216], 99.99th=[1115685]
   bw (  KiB/s): min= 1608, max= 6116, per=6.34%, avg=3689.03, stdev=1076.00, samples=60
   iops        : min=  402, max= 1529, avg=922.20, stdev=268.99, samples=60
  lat (usec)   : 500=8.08%, 750=49.91%, 1000=25.95%
  lat (msec)   : 2=10.97%, 4=0.04%, 500=0.08%, 750=4.30%, 1000=0.72%
  lat (msec)   : 2000=0.06%
  cpu          : usr=0.31%, sys=0.99%, ctx=27913, majf=0, minf=36
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=27759,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
RND4K_Write_Q32T16: (groupid=3, jobs=1): err= 0: pid=1859251: Sat Sep 26 07:23:42 2026
  write: IOPS=7659, BW=29.9MiB/s (31.4MB/s)(898MiB/30022msec); 0 zone resets
    slat (nsec): min=1690, max=510288, avg=3791.81, stdev=3244.77
    clat (usec): min=109, max=39520, avg=4172.73, stdev=5156.68
     lat (usec): min=126, max=39522, avg=4176.52, stdev=5156.86
    clat percentiles (usec):
     |  1.00th=[ 1254],  5.00th=[ 1516], 10.00th=[ 1549], 20.00th=[ 1598],
     | 30.00th=[ 1614], 40.00th=[ 1647], 50.00th=[ 1680], 60.00th=[ 1745],
     | 70.00th=[ 2573], 80.00th=[ 5342], 90.00th=[12256], 95.00th=[17433],
     | 99.00th=[22676], 99.50th=[23987], 99.90th=[25560], 99.95th=[26346],
     | 99.99th=[27395]
   bw (  KiB/s): min= 6736, max=60960, per=6.22%, avg=30658.97, stdev=18440.14, samples=60
   iops        : min= 1684, max=15240, avg=7664.65, stdev=4610.07, samples=60
  lat (usec)   : 250=0.05%, 500=0.11%, 750=0.18%, 1000=0.25%
  lat (msec)   : 2=65.23%, 4=9.35%, 10=12.52%, 20=9.50%, 50=2.84%
  cpu          : usr=2.44%, sys=7.00%, ctx=228057, majf=0, minf=294
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,229962,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
RND4K_Write_Q32T16: (groupid=3, jobs=1): err= 0: pid=1859252: Sat Sep 26 07:23:42 2026
  write: IOPS=7737, BW=30.2MiB/s (31.7MB/s)(907MiB/30020msec); 0 zone resets
    slat (nsec): min=1680, max=808442, avg=3572.04, stdev=3637.03
    clat (usec): min=118, max=38016, avg=4131.35, stdev=5101.08
     lat (usec): min=121, max=38021, avg=4134.92, stdev=5101.37
    clat percentiles (usec):
     |  1.00th=[ 1319],  5.00th=[ 1500], 10.00th=[ 1532], 20.00th=[ 1565],
     | 30.00th=[ 1598], 40.00th=[ 1614], 50.00th=[ 1647], 60.00th=[ 1713],
     | 70.00th=[ 2507], 80.00th=[ 5342], 90.00th=[12125], 95.00th=[17171],
     | 99.00th=[22676], 99.50th=[23725], 99.90th=[25560], 99.95th=[26084],
     | 99.99th=[27132]
   bw (  KiB/s): min= 6800, max=60736, per=6.28%, avg=30967.10, stdev=18696.36, samples=60
   iops        : min= 1700, max=15184, avg=7741.68, stdev=4674.13, samples=60
  lat (usec)   : 250=0.03%, 500=0.08%, 750=0.12%, 1000=0.16%
  lat (msec)   : 2=65.77%, 4=8.84%, 10=12.70%, 20=9.62%, 50=2.69%
  cpu          : usr=2.23%, sys=6.92%, ctx=232514, majf=0, minf=131
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,232272,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
RND4K_Write_Q32T16: (groupid=3, jobs=1): err= 0: pid=1859253: Sat Sep 26 07:23:42 2026
  write: IOPS=7723, BW=30.2MiB/s (31.6MB/s)(906MiB/30022msec); 0 zone resets
    slat (nsec): min=1740, max=246193, avg=3432.15, stdev=2160.25
    clat (usec): min=129, max=40515, avg=4138.94, stdev=5131.28
     lat (usec): min=133, max=40518, avg=4142.37, stdev=5131.49
    clat percentiles (usec):
     |  1.00th=[ 1369],  5.00th=[ 1500], 10.00th=[ 1532], 20.00th=[ 1582],
     | 30.00th=[ 1598], 40.00th=[ 1631], 50.00th=[ 1663], 60.00th=[ 1729],
     | 70.00th=[ 2507], 80.00th=[ 5342], 90.00th=[12256], 95.00th=[17433],
     | 99.00th=[22676], 99.50th=[23987], 99.90th=[26084], 99.95th=[26608],
     | 99.99th=[30540]
   bw (  KiB/s): min= 6776, max=63552, per=6.27%, avg=30913.55, stdev=18642.22, samples=60
   iops        : min= 1694, max=15888, avg=7728.28, stdev=4660.56, samples=60
  lat (usec)   : 250=0.01%, 500=0.04%, 750=0.08%, 1000=0.11%
  lat (msec)   : 2=66.39%, 4=8.50%, 10=12.56%, 20=9.65%, 50=2.67%
  cpu          : usr=2.31%, sys=6.86%, ctx=231871, majf=0, minf=143
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,231868,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
RND4K_Write_Q32T16: (groupid=3, jobs=1): err= 0: pid=1859254: Sat Sep 26 07:23:42 2026
  write: IOPS=7743, BW=30.2MiB/s (31.7MB/s)(908MiB/30022msec); 0 zone resets
    slat (nsec): min=1720, max=450756, avg=3195.18, stdev=1865.84
    clat (usec): min=98, max=40882, avg=4128.62, stdev=5125.91
     lat (usec): min=110, max=40886, avg=4131.81, stdev=5126.19
    clat percentiles (usec):
     |  1.00th=[ 1401],  5.00th=[ 1500], 10.00th=[ 1516], 20.00th=[ 1565],
     | 30.00th=[ 1582], 40.00th=[ 1614], 50.00th=[ 1647], 60.00th=[ 1696],
     | 70.00th=[ 2507], 80.00th=[ 5211], 90.00th=[12256], 95.00th=[17433],
     | 99.00th=[22676], 99.50th=[23725], 99.90th=[26084], 99.95th=[26608],
     | 99.99th=[27132]
   bw (  KiB/s): min= 6808, max=61808, per=6.28%, avg=30991.78, stdev=18861.15, samples=60
   iops        : min= 1702, max=15452, avg=7747.85, stdev=4715.31, samples=60
  lat (usec)   : 100=0.01%, 250=0.02%, 500=0.06%, 750=0.08%, 1000=0.12%
  lat (msec)   : 2=66.26%, 4=8.38%, 10=12.72%, 20=9.62%, 50=2.74%
  cpu          : usr=2.22%, sys=6.61%, ctx=230657, majf=0, minf=171
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,232461,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
RND4K_Write_Q32T16: (groupid=3, jobs=1): err= 0: pid=1859255: Sat Sep 26 07:23:42 2026
  write: IOPS=7663, BW=29.9MiB/s (31.4MB/s)(899MiB/30023msec); 0 zone resets
    slat (nsec): min=1690, max=1400.8k, avg=3566.53, stdev=3899.17
    clat (usec): min=113, max=79104, avg=4170.99, stdev=5209.87
     lat (usec): min=123, max=79120, avg=4174.56, stdev=5210.03
    clat percentiles (usec):
     |  1.00th=[ 1188],  5.00th=[ 1516], 10.00th=[ 1549], 20.00th=[ 1582],
     | 30.00th=[ 1614], 40.00th=[ 1647], 50.00th=[ 1680], 60.00th=[ 1745],
     | 70.00th=[ 2540], 80.00th=[ 5276], 90.00th=[12387], 95.00th=[17433],
     | 99.00th=[22938], 99.50th=[23987], 99.90th=[25560], 99.95th=[26084],
     | 99.99th=[72877]
   bw (  KiB/s): min= 7208, max=59336, per=6.22%, avg=30673.60, stdev=18458.71, samples=60
   iops        : min= 1802, max=14834, avg=7668.30, stdev=4614.70, samples=60
  lat (usec)   : 250=0.04%, 500=0.12%, 750=0.19%, 1000=0.29%
  lat (msec)   : 2=65.28%, 4=9.21%, 10=12.47%, 20=9.49%, 50=2.91%
  lat (msec)   : 100=0.01%
  cpu          : usr=2.31%, sys=7.00%, ctx=225166, majf=0, minf=172
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,230073,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
RND4K_Write_Q32T16: (groupid=3, jobs=1): err= 0: pid=1859257: Sat Sep 26 07:23:42 2026
  write: IOPS=7644, BW=29.9MiB/s (31.3MB/s)(897MiB/30022msec); 0 zone resets
    slat (nsec): min=1670, max=485867, avg=3769.78, stdev=2796.81
    clat (usec): min=117, max=40729, avg=4181.28, stdev=5175.19
     lat (usec): min=121, max=40732, avg=4185.05, stdev=5175.29
    clat percentiles (usec):
     |  1.00th=[ 1156],  5.00th=[ 1516], 10.00th=[ 1549], 20.00th=[ 1598],
     | 30.00th=[ 1614], 40.00th=[ 1647], 50.00th=[ 1696], 60.00th=[ 1762],
     | 70.00th=[ 2606], 80.00th=[ 5342], 90.00th=[12256], 95.00th=[17433],
     | 99.00th=[23200], 99.50th=[24249], 99.90th=[26084], 99.95th=[26608],
     | 99.99th=[27132]
   bw (  KiB/s): min= 6736, max=59232, per=6.20%, avg=30596.28, stdev=18486.36, samples=60
   iops        : min= 1684, max=14808, avg=7648.98, stdev=4621.61, samples=60
  lat (usec)   : 250=0.05%, 500=0.14%, 750=0.21%, 1000=0.32%
  lat (msec)   : 2=64.62%, 4=9.74%, 10=12.60%, 20=9.33%, 50=2.99%
  cpu          : usr=2.28%, sys=7.18%, ctx=225589, majf=0, minf=239
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,229493,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
RND4K_Write_Q32T16: (groupid=3, jobs=1): err= 0: pid=1859258: Sat Sep 26 07:23:42 2026
  write: IOPS=7793, BW=30.4MiB/s (31.9MB/s)(914MiB/30023msec); 0 zone resets
    slat (nsec): min=1691, max=256544, avg=3343.85, stdev=2471.71
    clat (usec): min=116, max=40619, avg=4101.53, stdev=5105.94
     lat (usec): min=126, max=40622, avg=4104.88, stdev=5106.26
    clat percentiles (usec):
     |  1.00th=[ 1336],  5.00th=[ 1500], 10.00th=[ 1532], 20.00th=[ 1565],
     | 30.00th=[ 1582], 40.00th=[ 1614], 50.00th=[ 1647], 60.00th=[ 1696],
     | 70.00th=[ 2409], 80.00th=[ 5211], 90.00th=[12125], 95.00th=[17171],
     | 99.00th=[22676], 99.50th=[23987], 99.90th=[25822], 99.95th=[26608],
     | 99.99th=[27395]
   bw (  KiB/s): min= 6944, max=64568, per=6.32%, avg=31195.95, stdev=18977.68, samples=60
   iops        : min= 1736, max=16142, avg=7798.90, stdev=4744.47, samples=60
  lat (usec)   : 250=0.03%, 500=0.07%, 750=0.08%, 1000=0.13%
  lat (msec)   : 2=66.71%, 4=8.30%, 10=12.34%, 20=9.66%, 50=2.69%
  cpu          : usr=2.13%, sys=6.90%, ctx=234166, majf=0, minf=260
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,233991,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
RND4K_Write_Q32T16: (groupid=3, jobs=1): err= 0: pid=1859259: Sat Sep 26 07:23:42 2026
  write: IOPS=7705, BW=30.1MiB/s (31.6MB/s)(904MiB/30022msec); 0 zone resets
    slat (nsec): min=1720, max=7003.0k, avg=3514.96, stdev=14908.07
    clat (usec): min=119, max=47595, avg=4148.12, stdev=5179.35
     lat (usec): min=129, max=47611, avg=4151.63, stdev=5179.84
    clat percentiles (usec):
     |  1.00th=[ 1287],  5.00th=[ 1500], 10.00th=[ 1532], 20.00th=[ 1582],
     | 30.00th=[ 1598], 40.00th=[ 1631], 50.00th=[ 1663], 60.00th=[ 1729],
     | 70.00th=[ 2507], 80.00th=[ 5276], 90.00th=[11994], 95.00th=[17433],
     | 99.00th=[23200], 99.50th=[24249], 99.90th=[25822], 99.95th=[26870],
     | 99.99th=[46400]
   bw (  KiB/s): min= 6632, max=60576, per=6.25%, avg=30842.73, stdev=18638.84, samples=60
   iops        : min= 1658, max=15144, avg=7710.58, stdev=4659.73, samples=60
  lat (usec)   : 250=0.03%, 500=0.09%, 750=0.14%, 1000=0.21%
  lat (msec)   : 2=65.76%, 4=8.88%, 10=12.59%, 20=9.35%, 50=2.95%
  cpu          : usr=2.21%, sys=6.90%, ctx=228695, majf=0, minf=110
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,231345,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
RND4K_Write_Q32T16: (groupid=3, jobs=1): err= 0: pid=1859260: Sat Sep 26 07:23:42 2026
  write: IOPS=7673, BW=30.0MiB/s (31.4MB/s)(900MiB/30022msec); 0 zone resets
    slat (nsec): min=1730, max=1716.9k, avg=3902.28, stdev=4757.41
    clat (usec): min=119, max=40440, avg=4165.08, stdev=5156.59
     lat (usec): min=130, max=40448, avg=4168.99, stdev=5156.62
    clat percentiles (usec):
     |  1.00th=[ 1237],  5.00th=[ 1532], 10.00th=[ 1565], 20.00th=[ 1598],
     | 30.00th=[ 1614], 40.00th=[ 1647], 50.00th=[ 1680], 60.00th=[ 1745],
     | 70.00th=[ 2507], 80.00th=[ 5407], 90.00th=[12256], 95.00th=[17695],
     | 99.00th=[22938], 99.50th=[23987], 99.90th=[25822], 99.95th=[26346],
     | 99.99th=[27395]
   bw (  KiB/s): min= 6680, max=59928, per=6.23%, avg=30715.27, stdev=18561.47, samples=60
   iops        : min= 1670, max=14982, avg=7678.72, stdev=4640.41, samples=60
  lat (usec)   : 250=0.04%, 500=0.11%, 750=0.18%, 1000=0.26%
  lat (msec)   : 2=65.41%, 4=8.98%, 10=12.74%, 20=9.33%, 50=2.97%
  cpu          : usr=2.41%, sys=7.04%, ctx=230861, majf=0, minf=232
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,230385,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
RND4K_Write_Q32T16: (groupid=3, jobs=1): err= 0: pid=1859261: Sat Sep 26 07:23:42 2026
  write: IOPS=7674, BW=30.0MiB/s (31.4MB/s)(900MiB/30019msec); 0 zone resets
    slat (nsec): min=1650, max=388786, avg=3649.93, stdev=3237.33
    clat (usec): min=123, max=35640, avg=4165.08, stdev=5185.69
     lat (usec): min=135, max=35649, avg=4168.73, stdev=5185.81
    clat percentiles (usec):
     |  1.00th=[ 1254],  5.00th=[ 1516], 10.00th=[ 1549], 20.00th=[ 1582],
     | 30.00th=[ 1614], 40.00th=[ 1631], 50.00th=[ 1663], 60.00th=[ 1745],
     | 70.00th=[ 2540], 80.00th=[ 5276], 90.00th=[12125], 95.00th=[17433],
     | 99.00th=[23200], 99.50th=[24249], 99.90th=[26346], 99.95th=[26870],
     | 99.99th=[27395]
   bw (  KiB/s): min= 6656, max=60968, per=6.23%, avg=30714.55, stdev=18532.43, samples=60
   iops        : min= 1664, max=15242, avg=7678.53, stdev=4633.25, samples=60
  lat (usec)   : 250=0.03%, 500=0.10%, 750=0.16%, 1000=0.25%
  lat (msec)   : 2=65.27%, 4=9.37%, 10=12.52%, 20=9.29%, 50=3.02%
  cpu          : usr=2.25%, sys=7.04%, ctx=228107, majf=0, minf=267
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,230377,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
RND4K_Write_Q32T16: (groupid=3, jobs=1): err= 0: pid=1859262: Sat Sep 26 07:23:42 2026
  write: IOPS=7781, BW=30.4MiB/s (31.9MB/s)(913MiB/30020msec); 0 zone resets
    slat (nsec): min=1700, max=3822.1k, avg=3410.46, stdev=9800.83
    clat (usec): min=142, max=35421, avg=4107.72, stdev=5107.81
     lat (usec): min=155, max=35424, avg=4111.14, stdev=5108.10
    clat percentiles (usec):
     |  1.00th=[ 1401],  5.00th=[ 1500], 10.00th=[ 1532], 20.00th=[ 1565],
     | 30.00th=[ 1582], 40.00th=[ 1614], 50.00th=[ 1647], 60.00th=[ 1713],
     | 70.00th=[ 2442], 80.00th=[ 5211], 90.00th=[12125], 95.00th=[17171],
     | 99.00th=[22938], 99.50th=[23987], 99.90th=[25560], 99.95th=[26346],
     | 99.99th=[27395]
   bw (  KiB/s): min= 6712, max=60544, per=6.31%, avg=31144.60, stdev=18741.19, samples=60
   iops        : min= 1678, max=15136, avg=7786.07, stdev=4685.33, samples=60
  lat (usec)   : 250=0.01%, 500=0.03%, 750=0.05%, 1000=0.09%
  lat (msec)   : 2=66.87%, 4=8.24%, 10=12.58%, 20=9.43%, 50=2.72%
  cpu          : usr=2.34%, sys=6.87%, ctx=232078, majf=0, minf=279
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,233607,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
RND4K_Write_Q32T16: (groupid=3, jobs=1): err= 0: pid=1859263: Sat Sep 26 07:23:42 2026
  write: IOPS=7717, BW=30.2MiB/s (31.6MB/s)(905MiB/30021msec); 0 zone resets
    slat (nsec): min=1710, max=480287, avg=3438.05, stdev=2557.15
    clat (usec): min=128, max=40535, avg=4141.93, stdev=5155.73
     lat (usec): min=138, max=40539, avg=4145.37, stdev=5156.02
    clat percentiles (usec):
     |  1.00th=[ 1319],  5.00th=[ 1516], 10.00th=[ 1549], 20.00th=[ 1582],
     | 30.00th=[ 1598], 40.00th=[ 1631], 50.00th=[ 1663], 60.00th=[ 1729],
     | 70.00th=[ 2474], 80.00th=[ 5276], 90.00th=[12256], 95.00th=[17433],
     | 99.00th=[22938], 99.50th=[24249], 99.90th=[26346], 99.95th=[26870],
     | 99.99th=[39060]
   bw (  KiB/s): min= 6672, max=59912, per=6.26%, avg=30890.27, stdev=18672.71, samples=60
   iops        : min= 1668, max=14978, avg=7722.47, stdev=4668.21, samples=60
  lat (usec)   : 250=0.03%, 500=0.08%, 750=0.14%, 1000=0.21%
  lat (msec)   : 2=65.90%, 4=8.78%, 10=12.57%, 20=9.53%, 50=2.77%
  cpu          : usr=2.29%, sys=6.95%, ctx=228655, majf=0, minf=102
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,231695,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
RND4K_Write_Q32T16: (groupid=3, jobs=1): err= 0: pid=1859264: Sat Sep 26 07:23:42 2026
  write: IOPS=7694, BW=30.1MiB/s (31.5MB/s)(902MiB/30022msec); 0 zone resets
    slat (nsec): min=1710, max=40979k, avg=3745.24, stdev=90355.74
    clat (usec): min=100, max=73209, avg=4154.31, stdev=5163.05
     lat (usec): min=110, max=73211, avg=4158.06, stdev=5164.60
    clat percentiles (usec):
     |  1.00th=[ 1221],  5.00th=[ 1516], 10.00th=[ 1549], 20.00th=[ 1582],
     | 30.00th=[ 1614], 40.00th=[ 1631], 50.00th=[ 1663], 60.00th=[ 1729],
     | 70.00th=[ 2573], 80.00th=[ 5342], 90.00th=[12256], 95.00th=[17171],
     | 99.00th=[22414], 99.50th=[23725], 99.90th=[25560], 99.95th=[26608],
     | 99.99th=[67634]
   bw (  KiB/s): min= 6768, max=58792, per=6.24%, avg=30796.57, stdev=18444.21, samples=60
   iops        : min= 1692, max=14698, avg=7699.03, stdev=4611.06, samples=60
  lat (usec)   : 250=0.05%, 500=0.12%, 750=0.19%, 1000=0.26%
  lat (msec)   : 2=65.42%, 4=9.18%, 10=12.37%, 20=9.72%, 50=2.69%
  lat (msec)   : 100=0.01%
  cpu          : usr=2.35%, sys=6.84%, ctx=226767, majf=0, minf=131
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,230993,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
RND4K_Write_Q32T16: (groupid=3, jobs=1): err= 0: pid=1859265: Sat Sep 26 07:23:42 2026
  write: IOPS=7656, BW=29.9MiB/s (31.4MB/s)(898MiB/30023msec); 0 zone resets
    slat (nsec): min=1650, max=2155.9k, avg=3800.16, stdev=5914.12
    clat (usec): min=111, max=39868, avg=4174.41, stdev=5158.65
     lat (usec): min=114, max=39872, avg=4178.21, stdev=5158.86
    clat percentiles (usec):
     |  1.00th=[ 1303],  5.00th=[ 1516], 10.00th=[ 1549], 20.00th=[ 1582],
     | 30.00th=[ 1614], 40.00th=[ 1647], 50.00th=[ 1680], 60.00th=[ 1745],
     | 70.00th=[ 2573], 80.00th=[ 5342], 90.00th=[12256], 95.00th=[17695],
     | 99.00th=[22938], 99.50th=[23987], 99.90th=[26084], 99.95th=[26608],
     | 99.99th=[27132]
   bw (  KiB/s): min= 6648, max=58976, per=6.21%, avg=30645.75, stdev=18382.05, samples=60
   iops        : min= 1662, max=14744, avg=7661.35, stdev=4595.54, samples=60
  lat (usec)   : 250=0.04%, 500=0.10%, 750=0.15%, 1000=0.23%
  lat (msec)   : 2=65.40%, 4=9.14%, 10=12.55%, 20=9.50%, 50=2.90%
  cpu          : usr=2.47%, sys=6.95%, ctx=228819, majf=0, minf=145
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,229863,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
RND4K_Write_Q32T16: (groupid=3, jobs=1): err= 0: pid=1859266: Sat Sep 26 07:23:42 2026
  write: IOPS=7719, BW=30.2MiB/s (31.6MB/s)(905MiB/30022msec); 0 zone resets
    slat (nsec): min=1710, max=1018.3k, avg=3424.11, stdev=2921.01
    clat (usec): min=110, max=40835, avg=4140.90, stdev=5134.53
     lat (usec): min=128, max=40839, avg=4144.32, stdev=5134.63
    clat percentiles (usec):
     |  1.00th=[ 1401],  5.00th=[ 1516], 10.00th=[ 1532], 20.00th=[ 1582],
     | 30.00th=[ 1598], 40.00th=[ 1631], 50.00th=[ 1647], 60.00th=[ 1713],
     | 70.00th=[ 2540], 80.00th=[ 5276], 90.00th=[12256], 95.00th=[17171],
     | 99.00th=[22938], 99.50th=[23987], 99.90th=[26084], 99.95th=[26608],
     | 99.99th=[27395]
   bw (  KiB/s): min= 7032, max=61072, per=6.26%, avg=30898.65, stdev=18618.82, samples=60
   iops        : min= 1758, max=15268, avg=7724.58, stdev=4654.73, samples=60
  lat (usec)   : 250=0.03%, 500=0.09%, 750=0.12%, 1000=0.17%
  lat (msec)   : 2=65.96%, 4=8.63%, 10=12.68%, 20=9.52%, 50=2.81%
  cpu          : usr=2.12%, sys=6.99%, ctx=229867, majf=0, minf=76
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,231757,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
RND4K_Write_Q32T16: (groupid=3, jobs=1): err= 0: pid=1859268: Sat Sep 26 07:23:42 2026
  write: IOPS=7717, BW=30.1MiB/s (31.6MB/s)(905MiB/30022msec); 0 zone resets
    slat (nsec): min=1670, max=200733, avg=3491.59, stdev=2324.11
    clat (usec): min=126, max=40147, avg=4142.14, stdev=5123.09
     lat (usec): min=138, max=40154, avg=4145.64, stdev=5123.22
    clat percentiles (usec):
     |  1.00th=[ 1434],  5.00th=[ 1516], 10.00th=[ 1549], 20.00th=[ 1582],
     | 30.00th=[ 1598], 40.00th=[ 1631], 50.00th=[ 1663], 60.00th=[ 1713],
     | 70.00th=[ 2507], 80.00th=[ 5342], 90.00th=[12256], 95.00th=[17171],
     | 99.00th=[22938], 99.50th=[23987], 99.90th=[25822], 99.95th=[26346],
     | 99.99th=[27132]
   bw (  KiB/s): min= 7064, max=60232, per=6.26%, avg=30888.37, stdev=18593.01, samples=60
   iops        : min= 1766, max=15058, avg=7722.00, stdev=4648.30, samples=60
  lat (usec)   : 250=0.02%, 500=0.05%, 750=0.09%, 1000=0.12%
  lat (msec)   : 2=66.23%, 4=8.64%, 10=12.46%, 20=9.67%, 50=2.74%
  cpu          : usr=2.20%, sys=6.97%, ctx=232295, majf=0, minf=110
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,231680,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: bw=2831MiB/s (2969MB/s), 2831MiB/s-2831MiB/s (2969MB/s-2969MB/s), io=83.0GiB (89.1GB), run=30005-30005msec

Run status group 1 (all jobs):
  WRITE: bw=185MiB/s (194MB/s), 185MiB/s-185MiB/s (194MB/s-194MB/s), io=5567MiB (5837MB), run=30039-30039msec

Run status group 2 (all jobs):
   READ: bw=56.8MiB/s (59.5MB/s), 3449KiB/s-3829KiB/s (3532kB/s-3921kB/s), io=1742MiB (1827MB), run=30562-30682msec

Run status group 3 (all jobs):
  WRITE: bw=482MiB/s (505MB/s), 29.9MiB/s-30.4MiB/s (31.3MB/s-31.9MB/s), io=14.1GiB (15.2GB), run=30019-30023msec

And:

Code:

root@pve:/#

fio --time_based \
    --runtime=30 \
    --ramp_time=5 \
    --ioengine=io_uring \
    --direct=1 \
    --size=4G \
    --filename=/seq_test.fio \
    --name=PURE_SEQ \
    --bs=1m \
    --rw=write \
    --iodepth=8 \
    --numjobs=1

PURE_SEQ: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=io_uring, iodepth=8
fio-3.39
Starting 1 process
Jobs: 1 (f=1): [W(1)][100.0%][w=184MiB/s][w=184 IOPS][eta 00m:00s]
PURE_SEQ: (groupid=0, jobs=1): err= 0: pid=1863138: Sat Sep 26 07:25:11 2026
  write: IOPS=183, BW=184MiB/s (192MB/s)(5515MiB/30043msec); 0 zone resets
    slat (usec): min=23, max=551, avg=43.24, stdev=18.75
    clat (usec): min=41814, max=74670, avg=43553.94, stdev=1009.52
     lat (usec): min=41854, max=74711, avg=43597.17, stdev=1009.46
    clat percentiles (usec):
     |  1.00th=[42206],  5.00th=[42206], 10.00th=[42730], 20.00th=[42730],
     | 30.00th=[43254], 40.00th=[43254], 50.00th=[43254], 60.00th=[43779],
     | 70.00th=[43779], 80.00th=[44303], 90.00th=[44827], 95.00th=[44827],
     | 99.00th=[45351], 99.50th=[45876], 99.90th=[48497], 99.95th=[64226],
     | 99.99th=[74974]
   bw (  KiB/s): min=184320, max=192512, per=100.00%, avg=187994.28, stdev=1800.21, samples=60
   iops        : min=  180, max=  188, avg=183.53, stdev= 1.78, samples=60
  lat (msec)   : 50=100.04%, 100=0.09%
  cpu          : usr=0.89%, sys=0.27%, ctx=5524, majf=0, minf=2099
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,5508,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
  WRITE: bw=184MiB/s (192MB/s), 184MiB/s-184MiB/s (192MB/s-192MB/s), io=5515MiB (5783MB), run=30043-30043msec

Extremely slow. In the first test the Value in Brackets (Time Estimator) sometimes "stalls" and becomes
bs: 1 (f=0): [/(1),P(33)][-.-%][eta -] so slow it is.

Not sure if CPU is relevant for this but Server is the Hyve AMD EPYC 7642 48 Core Server Server from https://forums.servethehome.com/index.php?threads/hyve-edge-metal-g10-epyc-7642-250-offer.50844/:

Code:

root@pve:/# cpupower frequency-info
analyzing CPU 7:
  driver: amd-pstate
  CPUs which run at the same hardware frequency: 7
  CPUs which need to have their frequency coordinated by software: 7
  maximum transition latency: 4.00 ms
  hardware limits: 403 MHz - 3.32 GHz
  available cpufreq governors: conservative ondemand userspace powersave performance schedutil
  current policy: frequency should be within 1000 MHz and 2.10 GHz.
                  The governor "performance" may decide which speed to use
                  within this range.
  current CPU frequency: Unable to call hardware
  current CPU frequency: 2.08 GHz (asserted by call to kernel)
  boost state support:
    Supported: yes
    Active: yes
    AMD PSTATE Highest Performance: 255. Maximum Frequency: 3.32 GHz.
    AMD PSTATE Nominal Performance: 177. Nominal Frequency: 2.30 GHz.
    AMD PSTATE Lowest Non-linear Performance: 116. Lowest Non-linear Frequency: 1.51 GHz.
    AMD PSTATE Lowest Performance: 31. Lowest Frequency: 400 MHz.
  
root@pve:/# cat /proc/cpuinfo | grep MHz
cpu MHz        : 2038.418
cpu MHz        : 2074.761
cpu MHz        : 2074.829
cpu MHz        : 2075.009
cpu MHz        : 2100.000
cpu MHz        : 2074.707
cpu MHz        : 2100.000
cpu MHz        : 2100.000
cpu MHz        : 2100.000
cpu MHz        : 2025.901
cpu MHz        : 2100.000
cpu MHz        : 2100.000
cpu MHz        : 2100.000
cpu MHz        : 2100.000
cpu MHz        : 2100.000
cpu MHz        : 2100.000
cpu MHz        : 2100.000
cpu MHz        : 2100.000
cpu MHz        : 2100.000
cpu MHz        : 2100.000
cpu MHz        : 2100.000
cpu MHz        : 2100.000
cpu MHz        : 2100.000
cpu MHz        : 2100.000
cpu MHz        : 2100.000
cpu MHz        : 2100.000
cpu MHz        : 2100.000
cpu MHz        : 2100.000
cpu MHz        : 2100.000
cpu MHz        : 2100.000
cpu MHz        : 2100.000
cpu MHz        : 2100.000
cpu MHz        : 2100.000
cpu MHz        : 2100.000
cpu MHz        : 2100.000
cpu MHz        : 2100.000
cpu MHz        : 2100.000
cpu MHz        : 2100.000
cpu MHz        : 2100.000
cpu MHz        : 2100.000
cpu MHz        : 2074.724
cpu MHz        : 2100.000
cpu MHz        : 2072.161
cpu MHz        : 2100.000
cpu MHz        : 2100.000
cpu MHz        : 2100.000
cpu MHz        : 2100.000
cpu MHz        : 2100.000
cpu MHz        : 2100.000
cpu MHz        : 2074.806
cpu MHz        : 2100.000
cpu MHz        : 2100.000
cpu MHz        : 2100.000
cpu MHz        : 2100.000
cpu MHz        : 2074.619
cpu MHz        : 2100.000
cpu MHz        : 2074.997
cpu MHz        : 2100.000
cpu MHz        : 2100.000
cpu MHz        : 2100.000
cpu MHz        : 2100.000
cpu MHz        : 2100.000
cpu MHz        : 2100.000
cpu MHz        : 2100.000
cpu MHz        : 2100.000
cpu MHz        : 2100.000
cpu MHz        : 2100.000
cpu MHz        : 2073.461
cpu MHz        : 2073.432
cpu MHz        : 2100.000
cpu MHz        : 2100.000
cpu MHz        : 2100.000
cpu MHz        : 2100.000
cpu MHz        : 2100.000
cpu MHz        : 2100.000
cpu MHz        : 2100.000
cpu MHz        : 2100.000
cpu MHz        : 2100.000
cpu MHz        : 2100.000
cpu MHz        : 2100.000
cpu MHz        : 2100.000
cpu MHz        : 2100.000
cpu MHz        : 2100.000
cpu MHz        : 2100.000
cpu MHz        : 2100.000
cpu MHz        : 2100.000
cpu MHz        : 2100.000
cpu MHz        : 2074.847
cpu MHz        : 2100.000
cpu MHz        : 2100.000
cpu MHz        : 2100.000
cpu MHz        : 2074.734
cpu MHz        : 2100.000
cpu MHz        : 2074.528
cpu MHz        : 2100.000
cpu MHz        : 2074.737

Single Threaded Performance is surely not that great, if relevant, though for Performance to be this bad, I don't think the CPU is solely to blame.

jhartbarger · Sep 25, 2026

luckylinux said:
But ... If I do a simple zfs send | zfs receive (locally, from that ZFS Pool to within that same ZFS Pool), I am getting around 50 MiB/s.

If that's the case what is your ZFS Record Size for that dataset?

luckylinux · Sep 25, 2026

jhartbarger said:
If that's the case what is your ZFS Record Size for that dataset?

Lately these were LXC "Templates". They are not Proxmox VE Templates as Proxmox intends, just "regular" LXC Containers that I use as templates, so that they can updated automatically instead of being frozen like a normal Template would.

LXC Templates recorsize=128k, complete Info:

Code:

NAME                               PROPERTY                VALUE                         SOURCE
rpool/data/subvol-98998006-disk-0  type                    filesystem                    -
rpool/data/subvol-98998006-disk-0  creation                Fri Sep 25 18:27 2026         -
rpool/data/subvol-98998006-disk-0  used                    1.73G                         -
rpool/data/subvol-98998006-disk-0  available               2.40T                         -
rpool/data/subvol-98998006-disk-0  referenced              1.06G                         -
rpool/data/subvol-98998006-disk-0  compressratio           1.99x                         -
rpool/data/subvol-98998006-disk-0  mounted                 yes                           -
rpool/data/subvol-98998006-disk-0  quota                   none                          default
rpool/data/subvol-98998006-disk-0  reservation             none                          default
rpool/data/subvol-98998006-disk-0  recordsize              128K                          default
rpool/data/subvol-98998006-disk-0  mountpoint              /data/subvol-98998006-disk-0  inherited from rpool
rpool/data/subvol-98998006-disk-0  sharenfs                off                           default
rpool/data/subvol-98998006-disk-0  checksum                on                            default
rpool/data/subvol-98998006-disk-0  compression             lz4                           inherited from rpool
rpool/data/subvol-98998006-disk-0  atime                   on                            default
rpool/data/subvol-98998006-disk-0  devices                 off                           inherited from rpool
rpool/data/subvol-98998006-disk-0  exec                    on                            default
rpool/data/subvol-98998006-disk-0  setuid                  on                            default
rpool/data/subvol-98998006-disk-0  readonly                off                           inherited from rpool
rpool/data/subvol-98998006-disk-0  zoned                   off                           default
rpool/data/subvol-98998006-disk-0  snapdir                 hidden                        default
rpool/data/subvol-98998006-disk-0  aclmode                 discard                       default
rpool/data/subvol-98998006-disk-0  aclinherit              restricted                    default
rpool/data/subvol-98998006-disk-0  createtxg               1099                          -
rpool/data/subvol-98998006-disk-0  canmount                on                            default
rpool/data/subvol-98998006-disk-0  xattr                   sa                            inherited from rpool
rpool/data/subvol-98998006-disk-0  copies                  1                             default
rpool/data/subvol-98998006-disk-0  version                 5                             -
rpool/data/subvol-98998006-disk-0  utf8only                on                            -
rpool/data/subvol-98998006-disk-0  normalization           formD                         -
rpool/data/subvol-98998006-disk-0  casesensitivity         sensitive                     -
rpool/data/subvol-98998006-disk-0  vscan                   off                           default
rpool/data/subvol-98998006-disk-0  nbmand                  off                           default
rpool/data/subvol-98998006-disk-0  sharesmb                off                           default
rpool/data/subvol-98998006-disk-0  refquota                none                          default
rpool/data/subvol-98998006-disk-0  refreservation          none                          default
rpool/data/subvol-98998006-disk-0  guid                    4471001476560274912           -
rpool/data/subvol-98998006-disk-0  primarycache            all                           default
rpool/data/subvol-98998006-disk-0  secondarycache          all                           default
rpool/data/subvol-98998006-disk-0  usedbysnapshots         685M                          -
rpool/data/subvol-98998006-disk-0  usedbydataset           1.06G                         -
rpool/data/subvol-98998006-disk-0  usedbychildren          0B                            -
rpool/data/subvol-98998006-disk-0  usedbyrefreservation    0B                            -
rpool/data/subvol-98998006-disk-0  logbias                 latency                       default
rpool/data/subvol-98998006-disk-0  objsetid                6774                          -
rpool/data/subvol-98998006-disk-0  dedup                   off                           default
rpool/data/subvol-98998006-disk-0  mlslabel                none                          default
rpool/data/subvol-98998006-disk-0  sync                    standard                      default
rpool/data/subvol-98998006-disk-0  dnodesize               auto                          inherited from rpool
rpool/data/subvol-98998006-disk-0  refcompressratio        2.04x                         -
rpool/data/subvol-98998006-disk-0  written                 156M                          -
rpool/data/subvol-98998006-disk-0  logicalused             3.29G                         -
rpool/data/subvol-98998006-disk-0  logicalreferenced       2.06G                         -
rpool/data/subvol-98998006-disk-0  volmode                 default                       default
rpool/data/subvol-98998006-disk-0  filesystem_limit        none                          default
rpool/data/subvol-98998006-disk-0  snapshot_limit          none                          default
rpool/data/subvol-98998006-disk-0  filesystem_count        none                          default
rpool/data/subvol-98998006-disk-0  snapshot_count          none                          default
rpool/data/subvol-98998006-disk-0  snapdev                 hidden                        default
rpool/data/subvol-98998006-disk-0  acltype                 posix                         inherited from rpool
rpool/data/subvol-98998006-disk-0  context                 none                          default
rpool/data/subvol-98998006-disk-0  fscontext               none                          default
rpool/data/subvol-98998006-disk-0  defcontext              none                          default
rpool/data/subvol-98998006-disk-0  rootcontext             none                          default
rpool/data/subvol-98998006-disk-0  relatime                on                            inherited from rpool
rpool/data/subvol-98998006-disk-0  redundant_metadata      all                           default
rpool/data/subvol-98998006-disk-0  overlay                 on                            default
rpool/data/subvol-98998006-disk-0  encryption              off                           default
rpool/data/subvol-98998006-disk-0  keylocation             none                          default
rpool/data/subvol-98998006-disk-0  keyformat               none                          default
rpool/data/subvol-98998006-disk-0  pbkdf2iters             0                             default
rpool/data/subvol-98998006-disk-0  special_small_blocks    0                             default
rpool/data/subvol-98998006-disk-0  snapshots_changed       Fri Sep 25 20:10:45 2026      -
rpool/data/subvol-98998006-disk-0  prefetch                all                           default
rpool/data/subvol-98998006-disk-0  direct                  standard                      default
rpool/data/subvol-98998006-disk-0  longname                off                           default
rpool/data/subvol-98998006-disk-0  defaultuserquota        0                             -
rpool/data/subvol-98998006-disk-0  defaultgroupquota       0                             -
rpool/data/subvol-98998006-disk-0  defaultprojectquota     0                             -
rpool/data/subvol-98998006-disk-0  defaultuserobjquota     0                             -
rpool/data/subvol-98998006-disk-0  defaultgroupobjquota    0                             -
rpool/data/subvol-98998006-disk-0  defaultprojectobjquota  0                             -

Pretty sure I was having the same Issue with ZFS ZVOLS for KVM Virtual Machines though.
KVM Virtual Machine Templates ZFS volblocksize=16k, complete Info:

Code:

NAME                           PROPERTY                VALUE                     SOURCE
rpool/data/vm-99998004-disk-0  type                    volume                    -
rpool/data/vm-99998004-disk-0  creation                Fri Sep 25 18:13 2026     -
rpool/data/vm-99998004-disk-0  used                    7.87G                     -
rpool/data/vm-99998004-disk-0  available               2.40T                     -
rpool/data/vm-99998004-disk-0  referenced              3.37G                     -
rpool/data/vm-99998004-disk-0  compressratio           1.47x                     -
rpool/data/vm-99998004-disk-0  reservation             none                      default
rpool/data/vm-99998004-disk-0  volsize                 64G                       local
rpool/data/vm-99998004-disk-0  volblocksize            16K                       default
rpool/data/vm-99998004-disk-0  checksum                on                        default
rpool/data/vm-99998004-disk-0  compression             lz4                       inherited from rpool
rpool/data/vm-99998004-disk-0  readonly                off                       inherited from rpool
rpool/data/vm-99998004-disk-0  createtxg               748                       -
rpool/data/vm-99998004-disk-0  copies                  1                         default
rpool/data/vm-99998004-disk-0  refreservation          none                      default
rpool/data/vm-99998004-disk-0  guid                    1778882710462063220       -
rpool/data/vm-99998004-disk-0  primarycache            all                       default
rpool/data/vm-99998004-disk-0  secondarycache          all                       default
rpool/data/vm-99998004-disk-0  usedbysnapshots         4.49G                     -
rpool/data/vm-99998004-disk-0  usedbydataset           3.37G                     -
rpool/data/vm-99998004-disk-0  usedbychildren          0B                        -
rpool/data/vm-99998004-disk-0  usedbyrefreservation    0B                        -
rpool/data/vm-99998004-disk-0  logbias                 latency                   default
rpool/data/vm-99998004-disk-0  objsetid                6693                      -
rpool/data/vm-99998004-disk-0  dedup                   off                       default
rpool/data/vm-99998004-disk-0  mlslabel                none                      default
rpool/data/vm-99998004-disk-0  sync                    standard                  default
rpool/data/vm-99998004-disk-0  refcompressratio        1.46x                     -
rpool/data/vm-99998004-disk-0  written                 163M                      -
rpool/data/vm-99998004-disk-0  logicalused             11.5G                     -
rpool/data/vm-99998004-disk-0  logicalreferenced       4.92G                     -
rpool/data/vm-99998004-disk-0  volmode                 default                   default
rpool/data/vm-99998004-disk-0  snapshot_limit          none                      default
rpool/data/vm-99998004-disk-0  snapshot_count          none                      default
rpool/data/vm-99998004-disk-0  snapdev                 hidden                    default
rpool/data/vm-99998004-disk-0  context                 none                      default
rpool/data/vm-99998004-disk-0  fscontext               none                      default
rpool/data/vm-99998004-disk-0  defcontext              none                      default
rpool/data/vm-99998004-disk-0  rootcontext             none                      default
rpool/data/vm-99998004-disk-0  redundant_metadata      all                       default
rpool/data/vm-99998004-disk-0  encryption              off                       default
rpool/data/vm-99998004-disk-0  keylocation             none                      default
rpool/data/vm-99998004-disk-0  keyformat               none                      default
rpool/data/vm-99998004-disk-0  pbkdf2iters             0                         default
rpool/data/vm-99998004-disk-0  special_small_blocks    0                         default
rpool/data/vm-99998004-disk-0  snapshots_changed       Fri Sep 25 18:14:08 2026  -
rpool/data/vm-99998004-disk-0  prefetch                all                       default
rpool/data/vm-99998004-disk-0  volthreading            on                        default

luckylinux · Sep 25, 2026

For what it's worth here are some Figures from a comparable Hyve Server, same CPU & Motherboard, this Time with 2 x Micron 9100 MAX 1.2TB (to be fair this Micron 9100 MAX System is still on 512 Sector Size, not 4096).

Same Bad Performance in my View.

Code:

root@pve:~# fio --time_based \
    --runtime=30 \
    --ramp_time=5 \
    --ioengine=io_uring \
    --direct=1 \
    --size=2G \
    --stonewall \
    --name=SEQ1M_Read_Q8T1 --bs=1m --rw=read --iodepth=8 --numjobs=1 --filename=/testfile_seq.fio \
    --name=SEQ1M_Write_Q8T1 --bs=1m --rw=write --iodepth=8 --numjobs=1 --filename=/testfile_seq_w.fio \
    --name=RND4K_Read_Q32T16 --bs=4k --rw=randread --iodepth=32 --numjobs=16 --filename=/testfile_rnd.fio \
    --name=RND4K_Write_Q32T16 --bs=4k --rw=randwrite --iodepth=32 --numjobs=16 --filename=/testfile_rnd_w.fio
SEQ1M_Read_Q8T1: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=io_uring, iodepth=8
SEQ1M_Write_Q8T1: (g=1): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=io_uring, iodepth=8
RND4K_Read_Q32T16: (g=2): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=io_uring, iodepth=32
...
RND4K_Write_Q32T16: (g=3): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=io_uring, iodepth=32
...
fio-3.39
Starting 34 processes
SEQ1M_Read_Q8T1: Laying out IO file (1 file / 2048MiB)
SEQ1M_Write_Q8T1: Laying out IO file (1 file / 2048MiB)
RND4K_Read_Q32T16: Laying out IO file (1 file / 2048MiB)
RND4K_Write_Q32T16: Laying out IO file (1 file / 2048MiB)
Jobs: 16 (f=16): [_(18),w(16)][71.6%][w=44.0MiB/s][w=11.3k IOPS][eta 00m:56s]    
SEQ1M_Read_Q8T1: (groupid=0, jobs=1): err= 0: pid=2893064: Sat Sep 26 07:56:23 2026
  read: IOPS=3157, BW=3158MiB/s (3312MB/s)(92.5GiB/30003msec)
    slat (nsec): min=1620, max=64810, avg=2969.87, stdev=832.83
    clat (usec): min=1165, max=7980, avg=2529.61, stdev=429.31
     lat (usec): min=1169, max=7983, avg=2532.58, stdev=429.28
    clat percentiles (usec):
     |  1.00th=[ 1532],  5.00th=[ 1893], 10.00th=[ 2073], 20.00th=[ 2245],
     | 30.00th=[ 2343], 40.00th=[ 2442], 50.00th=[ 2507], 60.00th=[ 2573],
     | 70.00th=[ 2671], 80.00th=[ 2769], 90.00th=[ 2999], 95.00th=[ 3195],
     | 99.00th=[ 3982], 99.50th=[ 4424], 99.90th=[ 5211], 99.95th=[ 5669],
     | 99.99th=[ 6456]
   bw (  MiB/s): min= 2860, max= 3294, per=100.00%, avg=3158.90, stdev=101.81, samples=60
   iops        : min= 2860, max= 3294, avg=3158.85, stdev=101.83, samples=60
  lat (msec)   : 2=7.69%, 4=91.32%, 10=0.99%
  cpu          : usr=0.80%, sys=3.17%, ctx=94905, majf=0, minf=66
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=94747,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8
SEQ1M_Write_Q8T1: (groupid=1, jobs=1): err= 0: pid=2893364: Sat Sep 26 07:56:23 2026
  write: IOPS=214, BW=215MiB/s (225MB/s)(6460MiB/30041msec); 0 zone resets
    slat (usec): min=29, max=523, avg=40.56, stdev=16.77
    clat (usec): min=33263, max=71700, avg=37172.20, stdev=1921.97
     lat (usec): min=33301, max=71737, avg=37212.76, stdev=1923.34
    clat percentiles (usec):
     |  1.00th=[33424],  5.00th=[34866], 10.00th=[35914], 20.00th=[36439],
     | 30.00th=[36439], 40.00th=[36439], 50.00th=[36963], 60.00th=[36963],
     | 70.00th=[36963], 80.00th=[37487], 90.00th=[39584], 95.00th=[41681],
     | 99.00th=[42730], 99.50th=[44303], 99.90th=[47449], 99.95th=[57934],
     | 99.99th=[71828]
   bw (  KiB/s): min=196608, max=231424, per=100.00%, avg=220286.90, stdev=6312.57, samples=60
   iops        : min=  192, max=  226, avg=215.07, stdev= 6.17, samples=60
  lat (msec)   : 50=100.03%, 100=0.08%
  cpu          : usr=1.04%, sys=0.27%, ctx=6474, majf=0, minf=2101
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,6453,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8
RND4K_Read_Q32T16: (groupid=2, jobs=1): err= 0: pid=2893613: Sat Sep 26 07:56:23 2026
  read: IOPS=1395, BW=5586KiB/s (5720kB/s)(164MiB/30032msec)
    slat (nsec): min=1650, max=63560, avg=3790.28, stdev=1385.09
    clat (usec): min=706, max=64087, avg=22922.55, stdev=16863.63
     lat (usec): min=709, max=64093, avg=22926.34, stdev=16863.63
    clat percentiles (usec):
     |  1.00th=[  988],  5.00th=[ 1336], 10.00th=[ 1795], 20.00th=[ 3818],
     | 30.00th=[ 8160], 40.00th=[11338], 50.00th=[22938], 60.00th=[33817],
     | 70.00th=[36963], 80.00th=[42206], 90.00th=[44303], 95.00th=[45351],
     | 99.00th=[48497], 99.50th=[49546], 99.90th=[51643], 99.95th=[52167],
     | 99.99th=[57934]
   bw (  KiB/s): min= 5107, max= 5920, per=6.23%, avg=5589.62, stdev=168.74, samples=60
   iops        : min= 1276, max= 1480, avg=1397.27, stdev=42.18, samples=60
  lat (usec)   : 750=0.01%, 1000=1.09%
  lat (msec)   : 2=10.68%, 4=8.68%, 10=15.56%, 20=11.80%, 50=51.83%
  lat (msec)   : 100=0.42%
  cpu          : usr=0.61%, sys=1.57%, ctx=42048, majf=0, minf=38
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=41910,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
RND4K_Read_Q32T16: (groupid=2, jobs=1): err= 0: pid=2893614: Sat Sep 26 07:56:23 2026
  read: IOPS=1415, BW=5667KiB/s (5803kB/s)(166MiB/30033msec)
    slat (nsec): min=1640, max=33490, avg=3770.43, stdev=1394.16
    clat (usec): min=711, max=66559, avg=22595.40, stdev=16909.74
     lat (usec): min=715, max=66562, avg=22599.17, stdev=16909.74
    clat percentiles (usec):
     |  1.00th=[  996],  5.00th=[ 1319], 10.00th=[ 1745], 20.00th=[ 3523],
     | 30.00th=[ 7767], 40.00th=[10945], 50.00th=[22152], 60.00th=[33817],
     | 70.00th=[36963], 80.00th=[41681], 90.00th=[44303], 95.00th=[45351],
     | 99.00th=[48497], 99.50th=[49546], 99.90th=[51643], 99.95th=[52167],
     | 99.99th=[57410]
   bw (  KiB/s): min= 5144, max= 6072, per=6.32%, avg=5670.32, stdev=186.26, samples=60
   iops        : min= 1286, max= 1518, avg=1417.43, stdev=46.55, samples=60
  lat (usec)   : 750=0.01%, 1000=1.06%
  lat (msec)   : 2=11.25%, 4=8.98%, 10=15.75%, 20=11.76%, 50=50.86%
  lat (msec)   : 100=0.41%
  cpu          : usr=0.50%, sys=1.70%, ctx=42626, majf=0, minf=37
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=42515,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
RND4K_Read_Q32T16: (groupid=2, jobs=1): err= 0: pid=2893615: Sat Sep 26 07:56:23 2026
  read: IOPS=1404, BW=5621KiB/s (5756kB/s)(165MiB/30035msec)
    slat (nsec): min=1640, max=31010, avg=3664.20, stdev=1316.75
    clat (usec): min=758, max=64734, avg=22781.80, stdev=16908.34
     lat (usec): min=762, max=64739, avg=22785.47, stdev=16908.34
    clat percentiles (usec):
     |  1.00th=[ 1004],  5.00th=[ 1352], 10.00th=[ 1745], 20.00th=[ 3654],
     | 30.00th=[ 7898], 40.00th=[11076], 50.00th=[22676], 60.00th=[33817],
     | 70.00th=[36963], 80.00th=[42206], 90.00th=[44303], 95.00th=[45351],
     | 99.00th=[48497], 99.50th=[49546], 99.90th=[51643], 99.95th=[52167],
     | 99.99th=[56886]
   bw (  KiB/s): min= 5176, max= 6076, per=6.27%, avg=5625.55, stdev=186.42, samples=60
   iops        : min= 1294, max= 1519, avg=1406.27, stdev=46.62, samples=60
  lat (usec)   : 1000=0.97%
  lat (msec)   : 2=11.24%, 4=8.86%, 10=15.65%, 20=11.32%, 50=51.64%
  lat (msec)   : 100=0.40%
  cpu          : usr=0.51%, sys=1.65%, ctx=42284, majf=0, minf=37
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=42179,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
RND4K_Read_Q32T16: (groupid=2, jobs=1): err= 0: pid=2893616: Sat Sep 26 07:56:23 2026
  read: IOPS=1397, BW=5592KiB/s (5726kB/s)(164MiB/30035msec)
    slat (nsec): min=1660, max=21620, avg=3871.08, stdev=1369.57
    clat (usec): min=759, max=68356, avg=22899.83, stdev=16880.68
     lat (usec): min=762, max=68359, avg=22903.70, stdev=16880.69
    clat percentiles (usec):
     |  1.00th=[  996],  5.00th=[ 1369], 10.00th=[ 1811], 20.00th=[ 3818],
     | 30.00th=[ 8029], 40.00th=[11338], 50.00th=[22938], 60.00th=[33817],
     | 70.00th=[36963], 80.00th=[42206], 90.00th=[44303], 95.00th=[45351],
     | 99.00th=[47973], 99.50th=[49546], 99.90th=[52167], 99.95th=[52691],
     | 99.99th=[62129]
   bw (  KiB/s): min= 5024, max= 5872, per=6.24%, avg=5596.45, stdev=172.40, samples=60
   iops        : min= 1256, max= 1468, avg=1398.98, stdev=43.10, samples=60
  lat (usec)   : 1000=1.03%
  lat (msec)   : 2=10.60%, 4=8.88%, 10=15.71%, 20=11.67%, 50=51.77%
  lat (msec)   : 100=0.42%
  cpu          : usr=0.47%, sys=1.71%, ctx=42070, majf=0, minf=37
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=41960,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
RND4K_Read_Q32T16: (groupid=2, jobs=1): err= 0: pid=2893617: Sat Sep 26 07:56:23 2026
  read: IOPS=1390, BW=5566KiB/s (5700kB/s)(163MiB/30034msec)
    slat (nsec): min=1690, max=79850, avg=3875.30, stdev=1438.78
    clat (usec): min=711, max=64318, avg=23004.68, stdev=16865.69
     lat (usec): min=715, max=64322, avg=23008.56, stdev=16865.70
    clat percentiles (usec):
     |  1.00th=[  996],  5.00th=[ 1369], 10.00th=[ 1795], 20.00th=[ 3884],
     | 30.00th=[ 8160], 40.00th=[11469], 50.00th=[23200], 60.00th=[33817],
     | 70.00th=[36963], 80.00th=[42206], 90.00th=[44303], 95.00th=[45351],
     | 99.00th=[48497], 99.50th=[49546], 99.90th=[51643], 99.95th=[52691],
     | 99.99th=[59507]
   bw (  KiB/s): min= 4772, max= 5952, per=6.21%, avg=5569.92, stdev=201.77, samples=60
   iops        : min= 1193, max= 1488, avg=1392.35, stdev=50.41, samples=60
  lat (usec)   : 750=0.01%, 1000=1.00%
  lat (msec)   : 2=10.82%, 4=8.43%, 10=15.68%, 20=11.64%, 50=52.06%
  lat (msec)   : 100=0.43%
  cpu          : usr=0.50%, sys=1.68%, ctx=41931, majf=0, minf=38
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=41763,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
RND4K_Read_Q32T16: (groupid=2, jobs=1): err= 0: pid=2893618: Sat Sep 26 07:56:23 2026
  read: IOPS=1399, BW=5600KiB/s (5735kB/s)(164MiB/30033msec)
    slat (nsec): min=1670, max=22580, avg=3907.26, stdev=1389.93
    clat (usec): min=733, max=64517, avg=22867.06, stdev=16863.43
     lat (usec): min=739, max=64523, avg=22870.96, stdev=16863.43
    clat percentiles (usec):
     |  1.00th=[ 1004],  5.00th=[ 1385], 10.00th=[ 1827], 20.00th=[ 3851],
     | 30.00th=[ 8029], 40.00th=[11207], 50.00th=[22676], 60.00th=[33817],
     | 70.00th=[36963], 80.00th=[42206], 90.00th=[44303], 95.00th=[45351],
     | 99.00th=[48497], 99.50th=[49546], 99.90th=[51643], 99.95th=[52167],
     | 99.99th=[53216]
   bw (  KiB/s): min= 5056, max= 5944, per=6.24%, avg=5604.17, stdev=181.70, samples=60
   iops        : min= 1264, max= 1486, avg=1400.92, stdev=45.41, samples=60
  lat (usec)   : 750=0.01%, 1000=0.94%
  lat (msec)   : 2=10.59%, 4=8.89%, 10=16.01%, 20=11.56%, 50=51.66%
  lat (msec)   : 100=0.41%
  cpu          : usr=0.56%, sys=1.64%, ctx=42185, majf=0, minf=37
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=42018,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
RND4K_Read_Q32T16: (groupid=2, jobs=1): err= 0: pid=2893619: Sat Sep 26 07:56:23 2026
  read: IOPS=1404, BW=5622KiB/s (5757kB/s)(165MiB/30034msec)
    slat (nsec): min=1650, max=21590, avg=3773.75, stdev=1366.87
    clat (usec): min=728, max=67777, avg=22771.97, stdev=16905.39
     lat (usec): min=731, max=67780, avg=22775.74, stdev=16905.40
    clat percentiles (usec):
     |  1.00th=[  988],  5.00th=[ 1352], 10.00th=[ 1762], 20.00th=[ 3687],
     | 30.00th=[ 7898], 40.00th=[11076], 50.00th=[22676], 60.00th=[33817],
     | 70.00th=[36963], 80.00th=[42206], 90.00th=[44303], 95.00th=[45351],
     | 99.00th=[48497], 99.50th=[49546], 99.90th=[51643], 99.95th=[52691],
     | 99.99th=[64226]
   bw (  KiB/s): min= 5059, max= 5947, per=6.27%, avg=5626.33, stdev=187.72, samples=60
   iops        : min= 1264, max= 1486, avg=1406.43, stdev=46.92, samples=60
  lat (usec)   : 750=0.01%, 1000=1.13%
  lat (msec)   : 2=10.89%, 4=8.95%, 10=15.61%, 20=11.62%, 50=51.47%
  lat (msec)   : 100=0.39%
  cpu          : usr=0.56%, sys=1.62%, ctx=42325, majf=0, minf=37
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=42185,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
RND4K_Read_Q32T16: (groupid=2, jobs=1): err= 0: pid=2893620: Sat Sep 26 07:56:23 2026
  read: IOPS=1404, BW=5623KiB/s (5758kB/s)(165MiB/30035msec)
    slat (nsec): min=1670, max=36860, avg=3820.06, stdev=1404.45
    clat (usec): min=668, max=69419, avg=22773.56, stdev=16896.15
     lat (usec): min=671, max=69422, avg=22777.38, stdev=16896.17
    clat percentiles (usec):
     |  1.00th=[  996],  5.00th=[ 1352], 10.00th=[ 1762], 20.00th=[ 3752],
     | 30.00th=[ 7898], 40.00th=[11076], 50.00th=[22676], 60.00th=[33817],
     | 70.00th=[36963], 80.00th=[42206], 90.00th=[44303], 95.00th=[45351],
     | 99.00th=[48497], 99.50th=[49546], 99.90th=[51643], 99.95th=[51643],
     | 99.99th=[66323]
   bw (  KiB/s): min= 5072, max= 5992, per=6.27%, avg=5626.78, stdev=179.13, samples=60
   iops        : min= 1268, max= 1498, avg=1406.57, stdev=44.80, samples=60
  lat (usec)   : 750=0.01%, 1000=1.02%
  lat (msec)   : 2=11.13%, 4=8.61%, 10=15.93%, 20=11.46%, 50=51.52%
  lat (msec)   : 100=0.39%
  cpu          : usr=0.54%, sys=1.64%, ctx=42354, majf=0, minf=37
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=42189,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
RND4K_Read_Q32T16: (groupid=2, jobs=1): err= 0: pid=2893621: Sat Sep 26 07:56:23 2026
  read: IOPS=1415, BW=5667KiB/s (5803kB/s)(166MiB/30034msec)
    slat (nsec): min=1640, max=65250, avg=3896.70, stdev=1447.36
    clat (usec): min=740, max=64127, avg=22601.25, stdev=16896.03
     lat (usec): min=746, max=64131, avg=22605.15, stdev=16896.03
    clat percentiles (usec):
     |  1.00th=[ 1004],  5.00th=[ 1352], 10.00th=[ 1778], 20.00th=[ 3621],
     | 30.00th=[ 7767], 40.00th=[10814], 50.00th=[22152], 60.00th=[33424],
     | 70.00th=[36439], 80.00th=[41681], 90.00th=[44303], 95.00th=[45351],
     | 99.00th=[48497], 99.50th=[49546], 99.90th=[51643], 99.95th=[52167],
     | 99.99th=[55313]
   bw (  KiB/s): min= 5080, max= 6232, per=6.32%, avg=5670.47, stdev=206.95, samples=60
   iops        : min= 1270, max= 1558, avg=1417.48, stdev=51.73, samples=60
  lat (usec)   : 750=0.01%, 1000=1.00%
  lat (msec)   : 2=10.87%, 4=9.32%, 10=15.86%, 20=11.69%, 50=50.97%
  lat (msec)   : 100=0.37%
  cpu          : usr=0.55%, sys=1.67%, ctx=42695, majf=0, minf=38
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=42516,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
RND4K_Read_Q32T16: (groupid=2, jobs=1): err= 0: pid=2893622: Sat Sep 26 07:56:23 2026
  read: IOPS=1396, BW=5590KiB/s (5725kB/s)(164MiB/30032msec)
    slat (nsec): min=1670, max=83810, avg=3928.60, stdev=1456.11
    clat (usec): min=749, max=66145, avg=22905.15, stdev=16870.62
     lat (usec): min=753, max=66150, avg=22909.08, stdev=16870.63
    clat percentiles (usec):
     |  1.00th=[  996],  5.00th=[ 1369], 10.00th=[ 1795], 20.00th=[ 3785],
     | 30.00th=[ 8029], 40.00th=[11338], 50.00th=[22938], 60.00th=[33817],
     | 70.00th=[36963], 80.00th=[42206], 90.00th=[44303], 95.00th=[45351],
     | 99.00th=[48497], 99.50th=[49546], 99.90th=[51643], 99.95th=[52167],
     | 99.99th=[58983]
   bw (  KiB/s): min= 5184, max= 5896, per=6.23%, avg=5593.85, stdev=163.68, samples=60
   iops        : min= 1296, max= 1474, avg=1398.33, stdev=40.90, samples=60
  lat (usec)   : 750=0.01%, 1000=1.07%
  lat (msec)   : 2=10.62%, 4=8.86%, 10=15.72%, 20=11.61%, 50=51.77%
  lat (msec)   : 100=0.41%
  cpu          : usr=0.61%, sys=1.59%, ctx=42077, majf=0, minf=38
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=41942,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
RND4K_Read_Q32T16: (groupid=2, jobs=1): err= 0: pid=2893623: Sat Sep 26 07:56:23 2026
  read: IOPS=1398, BW=5600KiB/s (5735kB/s)(164MiB/30033msec)
    slat (nsec): min=1660, max=33890, avg=3821.61, stdev=1413.81
    clat (usec): min=734, max=65501, avg=22868.13, stdev=16887.05
     lat (usec): min=737, max=65505, avg=22871.96, stdev=16887.05
    clat percentiles (usec):
     |  1.00th=[  996],  5.00th=[ 1352], 10.00th=[ 1778], 20.00th=[ 3818],
     | 30.00th=[ 7963], 40.00th=[11076], 50.00th=[22938], 60.00th=[33817],
     | 70.00th=[36963], 80.00th=[42206], 90.00th=[44303], 95.00th=[45351],
     | 99.00th=[48497], 99.50th=[49546], 99.90th=[51643], 99.95th=[52167],
     | 99.99th=[58459]
   bw (  KiB/s): min= 4912, max= 6032, per=6.24%, avg=5603.97, stdev=221.03, samples=60
   iops        : min= 1228, max= 1508, avg=1400.87, stdev=55.29, samples=60
  lat (usec)   : 750=0.01%, 1000=1.03%
  lat (msec)   : 2=10.84%, 4=8.68%, 10=15.91%, 20=11.44%, 50=51.75%
  lat (msec)   : 100=0.43%
  cpu          : usr=0.56%, sys=1.62%, ctx=42170, majf=0, minf=37
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=42016,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
RND4K_Read_Q32T16: (groupid=2, jobs=1): err= 0: pid=2893624: Sat Sep 26 07:56:23 2026
  read: IOPS=1402, BW=5613KiB/s (5747kB/s)(165MiB/30035msec)
    slat (nsec): min=1650, max=21490, avg=3824.02, stdev=1369.00
    clat (usec): min=717, max=68157, avg=22814.60, stdev=16884.90
     lat (usec): min=720, max=68163, avg=22818.42, stdev=16884.91
    clat percentiles (usec):
     |  1.00th=[  996],  5.00th=[ 1352], 10.00th=[ 1778], 20.00th=[ 3752],
     | 30.00th=[ 7963], 40.00th=[11076], 50.00th=[22676], 60.00th=[33817],
     | 70.00th=[36963], 80.00th=[42206], 90.00th=[44303], 95.00th=[45351],
     | 99.00th=[48497], 99.50th=[49546], 99.90th=[51643], 99.95th=[52167],
     | 99.99th=[60556]
   bw (  KiB/s): min= 5042, max= 5896, per=6.26%, avg=5616.55, stdev=187.93, samples=60
   iops        : min= 1260, max= 1474, avg=1404.00, stdev=46.99, samples=60
  lat (usec)   : 750=0.01%, 1000=1.03%
  lat (msec)   : 2=10.90%, 4=8.82%, 10=15.81%, 20=11.54%, 50=51.57%
  lat (msec)   : 100=0.39%
  cpu          : usr=0.66%, sys=1.53%, ctx=42258, majf=0, minf=37
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=42112,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
RND4K_Read_Q32T16: (groupid=2, jobs=1): err= 0: pid=2893625: Sat Sep 26 07:56:23 2026
  read: IOPS=1402, BW=5615KiB/s (5750kB/s)(165MiB/30035msec)
    slat (nsec): min=1640, max=21130, avg=3675.32, stdev=1313.94
    clat (usec): min=748, max=67801, avg=22804.99, stdev=16878.07
     lat (usec): min=753, max=67805, avg=22808.66, stdev=16878.06
    clat percentiles (usec):
     |  1.00th=[ 1004],  5.00th=[ 1369], 10.00th=[ 1795], 20.00th=[ 3720],
     | 30.00th=[ 8094], 40.00th=[11207], 50.00th=[22676], 60.00th=[33817],
     | 70.00th=[36963], 80.00th=[42206], 90.00th=[44303], 95.00th=[45351],
     | 99.00th=[48497], 99.50th=[49546], 99.90th=[51643], 99.95th=[52167],
     | 99.99th=[64750]
   bw (  KiB/s): min= 5187, max= 6152, per=6.26%, avg=5618.83, stdev=205.57, samples=60
   iops        : min= 1296, max= 1538, avg=1404.58, stdev=51.42, samples=60
  lat (usec)   : 750=0.01%, 1000=1.00%
  lat (msec)   : 2=10.66%, 4=9.09%, 10=15.65%, 20=11.79%, 50=51.49%
  lat (msec)   : 100=0.39%
  cpu          : usr=0.63%, sys=1.53%, ctx=42268, majf=0, minf=38
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=42129,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
RND4K_Read_Q32T16: (groupid=2, jobs=1): err= 0: pid=2893626: Sat Sep 26 07:56:23 2026
  read: IOPS=1402, BW=5614KiB/s (5749kB/s)(165MiB/30033msec)
    slat (nsec): min=1670, max=22450, avg=3833.28, stdev=1393.99
    clat (usec): min=742, max=63635, avg=22807.36, stdev=16881.41
     lat (usec): min=746, max=63639, avg=22811.19, stdev=16881.42
    clat percentiles (usec):
     |  1.00th=[ 1012],  5.00th=[ 1369], 10.00th=[ 1795], 20.00th=[ 3785],
     | 30.00th=[ 8029], 40.00th=[11207], 50.00th=[22676], 60.00th=[33817],
     | 70.00th=[36963], 80.00th=[42206], 90.00th=[44303], 95.00th=[45351],
     | 99.00th=[48497], 99.50th=[49546], 99.90th=[51643], 99.95th=[52167],
     | 99.99th=[61604]
   bw (  KiB/s): min= 5104, max= 5992, per=6.26%, avg=5617.73, stdev=195.88, samples=60
   iops        : min= 1276, max= 1498, avg=1404.30, stdev=48.98, samples=60
  lat (usec)   : 750=0.01%, 1000=0.90%
  lat (msec)   : 2=11.01%, 4=8.75%, 10=15.79%, 20=11.74%, 50=51.44%
  lat (msec)   : 100=0.44%
  cpu          : usr=0.50%, sys=1.68%, ctx=42307, majf=0, minf=37
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=42121,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
RND4K_Read_Q32T16: (groupid=2, jobs=1): err= 0: pid=2893627: Sat Sep 26 07:56:23 2026
  read: IOPS=1394, BW=5581KiB/s (5715kB/s)(164MiB/30034msec)
    slat (nsec): min=1670, max=57040, avg=3832.46, stdev=1420.15
    clat (usec): min=727, max=66857, avg=22944.55, stdev=16883.17
     lat (usec): min=731, max=66861, avg=22948.38, stdev=16883.17
    clat percentiles (usec):
     |  1.00th=[ 1004],  5.00th=[ 1352], 10.00th=[ 1795], 20.00th=[ 3818],
     | 30.00th=[ 8029], 40.00th=[11338], 50.00th=[22938], 60.00th=[33817],
     | 70.00th=[36963], 80.00th=[42206], 90.00th=[44303], 95.00th=[45351],
     | 99.00th=[48497], 99.50th=[49546], 99.90th=[51643], 99.95th=[52167],
     | 99.99th=[60031]
   bw (  KiB/s): min= 5048, max= 6064, per=6.22%, avg=5585.10, stdev=209.45, samples=60
   iops        : min= 1262, max= 1516, avg=1396.13, stdev=52.32, samples=60
  lat (usec)   : 750=0.01%, 1000=0.99%
  lat (msec)   : 2=10.86%, 4=8.70%, 10=15.62%, 20=11.42%, 50=52.03%
  lat (msec)   : 100=0.43%
  cpu          : usr=0.53%, sys=1.65%, ctx=42025, majf=0, minf=37
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=41876,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
RND4K_Read_Q32T16: (groupid=2, jobs=1): err= 0: pid=2893628: Sat Sep 26 07:56:23 2026
  read: IOPS=1397, BW=5593KiB/s (5727kB/s)(164MiB/30034msec)
    slat (nsec): min=1670, max=57490, avg=3852.39, stdev=1434.98
    clat (usec): min=723, max=68300, avg=22892.59, stdev=16890.86
     lat (usec): min=727, max=68305, avg=22896.44, stdev=16890.86
    clat percentiles (usec):
     |  1.00th=[ 1004],  5.00th=[ 1369], 10.00th=[ 1811], 20.00th=[ 3720],
     | 30.00th=[ 8029], 40.00th=[11207], 50.00th=[22938], 60.00th=[33817],
     | 70.00th=[36963], 80.00th=[42206], 90.00th=[44303], 95.00th=[45351],
     | 99.00th=[48497], 99.50th=[49546], 99.90th=[51643], 99.95th=[52167],
     | 99.99th=[63701]
   bw (  KiB/s): min= 5096, max= 5899, per=6.24%, avg=5597.00, stdev=181.10, samples=60
   iops        : min= 1274, max= 1474, avg=1399.12, stdev=45.24, samples=60
  lat (usec)   : 750=0.01%, 1000=0.99%
  lat (msec)   : 2=10.78%, 4=9.04%, 10=15.56%, 20=11.39%, 50=51.91%
  lat (msec)   : 100=0.39%
  cpu          : usr=0.58%, sys=1.59%, ctx=42114, majf=0, minf=37
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=41964,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
RND4K_Write_Q32T16: (groupid=3, jobs=1): err= 0: pid=2894451: Sat Sep 26 07:56:23 2026
  write: IOPS=6078, BW=23.7MiB/s (24.9MB/s)(714MiB/30050msec); 0 zone resets
    slat (nsec): min=1930, max=215370, avg=3506.25, stdev=1396.70
    clat (usec): min=143, max=96641, avg=5258.78, stdev=10010.39
     lat (usec): min=145, max=96645, avg=5262.29, stdev=10010.64
    clat percentiles (usec):
     |  1.00th=[ 1303],  5.00th=[ 1598], 10.00th=[ 1663], 20.00th=[ 1713],
     | 30.00th=[ 1745], 40.00th=[ 1762], 50.00th=[ 1795], 60.00th=[ 1827],
     | 70.00th=[ 1876], 80.00th=[ 3261], 90.00th=[12256], 95.00th=[30278],
     | 99.00th=[48497], 99.50th=[53740], 99.90th=[69731], 99.95th=[73925],
     | 99.99th=[74974]
   bw (  KiB/s): min= 1800, max=74864, per=6.12%, avg=24362.75, stdev=26270.19, samples=60
   iops        : min=  450, max=18716, avg=6090.60, stdev=6567.59, samples=60
  lat (usec)   : 250=0.01%, 500=0.02%, 750=0.02%, 1000=0.03%
  lat (msec)   : 2=76.44%, 4=4.93%, 10=7.36%, 20=3.04%, 50=7.38%
  lat (msec)   : 100=0.81%
  cpu          : usr=1.54%, sys=5.63%, ctx=182196, majf=0, minf=136
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,182660,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
RND4K_Write_Q32T16: (groupid=3, jobs=1): err= 0: pid=2894452: Sat Sep 26 07:56:23 2026
  write: IOPS=6208, BW=24.3MiB/s (25.4MB/s)(729MiB/30050msec); 0 zone resets
    slat (nsec): min=1820, max=177620, avg=3329.67, stdev=1326.86
    clat (usec): min=181, max=96626, avg=5149.30, stdev=9928.95
     lat (usec): min=192, max=96630, avg=5152.63, stdev=9929.27
    clat percentiles (usec):
     |  1.00th=[ 1352],  5.00th=[ 1500], 10.00th=[ 1532], 20.00th=[ 1582],
     | 30.00th=[ 1631], 40.00th=[ 1696], 50.00th=[ 1762], 60.00th=[ 1811],
     | 70.00th=[ 1860], 80.00th=[ 3621], 90.00th=[11863], 95.00th=[29492],
     | 99.00th=[48497], 99.50th=[53740], 99.90th=[69731], 99.95th=[73925],
     | 99.99th=[74974]
   bw (  KiB/s): min= 1808, max=81048, per=6.25%, avg=24881.98, stdev=27433.99, samples=60
   iops        : min=  452, max=20262, avg=6220.42, stdev=6858.55, samples=60
  lat (usec)   : 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
  lat (msec)   : 2=77.43%, 4=3.51%, 10=8.01%, 20=3.02%, 50=7.23%
  lat (msec)   : 100=0.79%
  cpu          : usr=1.48%, sys=5.55%, ctx=186273, majf=0, minf=140
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,186552,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
RND4K_Write_Q32T16: (groupid=3, jobs=1): err= 0: pid=2894453: Sat Sep 26 07:56:23 2026
  write: IOPS=6486, BW=25.3MiB/s (26.6MB/s)(762MiB/30050msec); 0 zone resets
    slat (nsec): min=1690, max=258231, avg=2930.07, stdev=1288.84
    clat (usec): min=112, max=96656, avg=4928.56, stdev=9773.47
     lat (usec): min=131, max=96659, avg=4931.49, stdev=9773.70
    clat percentiles (usec):
     |  1.00th=[ 1270],  5.00th=[ 1467], 10.00th=[ 1500], 20.00th=[ 1532],
     | 30.00th=[ 1549], 40.00th=[ 1582], 50.00th=[ 1598], 60.00th=[ 1614],
     | 70.00th=[ 1647], 80.00th=[ 3621], 90.00th=[10814], 95.00th=[29230],
     | 99.00th=[47973], 99.50th=[53740], 99.90th=[69731], 99.95th=[72877],
     | 99.99th=[74974]
   bw (  KiB/s): min= 1792, max=81410, per=6.53%, avg=25997.55, stdev=28818.42, samples=60
   iops        : min=  448, max=20352, avg=6499.30, stdev=7204.63, samples=60
  lat (usec)   : 250=0.01%, 500=0.02%, 750=0.03%, 1000=0.03%
  lat (msec)   : 2=77.60%, 4=2.84%, 10=9.08%, 20=2.79%, 50=6.89%
  lat (msec)   : 100=0.75%
  cpu          : usr=1.74%, sys=5.16%, ctx=194520, majf=0, minf=116
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,194916,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
RND4K_Write_Q32T16: (groupid=3, jobs=1): err= 0: pid=2894454: Sat Sep 26 07:56:23 2026
  write: IOPS=6273, BW=24.5MiB/s (25.7MB/s)(737MiB/30049msec); 0 zone resets
    slat (nsec): min=1720, max=203141, avg=3400.17, stdev=2037.48
    clat (usec): min=133, max=96877, avg=5095.31, stdev=9938.46
     lat (usec): min=136, max=96880, avg=5098.71, stdev=9938.59
    clat percentiles (usec):
     |  1.00th=[ 1369],  5.00th=[ 1532], 10.00th=[ 1565], 20.00th=[ 1598],
     | 30.00th=[ 1614], 40.00th=[ 1631], 50.00th=[ 1647], 60.00th=[ 1663],
     | 70.00th=[ 1696], 80.00th=[ 4015], 90.00th=[11207], 95.00th=[30278],
     | 99.00th=[48497], 99.50th=[53740], 99.90th=[69731], 99.95th=[73925],
     | 99.99th=[74974]
   bw (  KiB/s): min= 1800, max=78412, per=6.32%, avg=25143.67, stdev=27907.29, samples=60
   iops        : min=  450, max=19603, avg=6285.83, stdev=6976.86, samples=60
  lat (usec)   : 250=0.01%, 500=0.01%, 750=0.02%, 1000=0.03%
  lat (msec)   : 2=77.80%, 4=2.14%, 10=9.32%, 20=2.84%, 50=7.08%
  lat (msec)   : 100=0.78%
  cpu          : usr=1.80%, sys=5.49%, ctx=190642, majf=0, minf=114
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,188515,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
RND4K_Write_Q32T16: (groupid=3, jobs=1): err= 0: pid=2894455: Sat Sep 26 07:56:23 2026
  write: IOPS=6446, BW=25.2MiB/s (26.4MB/s)(757MiB/30049msec); 0 zone resets
    slat (nsec): min=1700, max=143900, avg=3038.80, stdev=1105.80
    clat (usec): min=142, max=96584, avg=4958.87, stdev=9778.36
     lat (usec): min=151, max=96588, avg=4961.91, stdev=9778.63
    clat percentiles (usec):
     |  1.00th=[ 1385],  5.00th=[ 1483], 10.00th=[ 1516], 20.00th=[ 1549],
     | 30.00th=[ 1565], 40.00th=[ 1598], 50.00th=[ 1614], 60.00th=[ 1647],
     | 70.00th=[ 1696], 80.00th=[ 3130], 90.00th=[10945], 95.00th=[29230],
     | 99.00th=[47973], 99.50th=[53740], 99.90th=[69731], 99.95th=[72877],
     | 99.99th=[74974]
   bw (  KiB/s): min= 1792, max=81080, per=6.49%, avg=25837.78, stdev=28545.57, samples=60
   iops        : min=  448, max=20270, avg=6459.37, stdev=7136.38, samples=60
  lat (usec)   : 250=0.01%, 500=0.02%, 750=0.03%, 1000=0.04%
  lat (msec)   : 2=77.77%, 4=3.69%, 10=7.92%, 20=2.80%, 50=6.99%
  lat (msec)   : 100=0.74%
  cpu          : usr=1.63%, sys=5.34%, ctx=193002, majf=0, minf=169
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,193719,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
RND4K_Write_Q32T16: (groupid=3, jobs=1): err= 0: pid=2894456: Sat Sep 26 07:56:23 2026
  write: IOPS=6306, BW=24.6MiB/s (25.8MB/s)(740MiB/30049msec); 0 zone resets
    slat (nsec): min=1620, max=74160, avg=2860.42, stdev=1066.42
    clat (usec): min=131, max=96750, avg=5069.59, stdev=9872.37
     lat (usec): min=136, max=96753, avg=5072.45, stdev=9872.66
    clat percentiles (usec):
     |  1.00th=[ 1450],  5.00th=[ 1500], 10.00th=[ 1532], 20.00th=[ 1565],
     | 30.00th=[ 1582], 40.00th=[ 1598], 50.00th=[ 1631], 60.00th=[ 1647],
     | 70.00th=[ 1680], 80.00th=[ 4146], 90.00th=[11469], 95.00th=[29492],
     | 99.00th=[48497], 99.50th=[53740], 99.90th=[69731], 99.95th=[73925],
     | 99.99th=[74974]
   bw (  KiB/s): min= 1800, max=81712, per=6.35%, avg=25274.55, stdev=27991.67, samples=60
   iops        : min=  450, max=20428, avg=6318.57, stdev=6998.00, samples=60
  lat (usec)   : 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
  lat (msec)   : 2=77.19%, 4=2.35%, 10=9.56%, 20=3.00%, 50=7.07%
  lat (msec)   : 100=0.79%
  cpu          : usr=1.40%, sys=5.28%, ctx=189228, majf=0, minf=111
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,189496,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
RND4K_Write_Q32T16: (groupid=3, jobs=1): err= 0: pid=2894457: Sat Sep 26 07:56:23 2026
  write: IOPS=6123, BW=23.9MiB/s (25.1MB/s)(719MiB/30048msec); 0 zone resets
    slat (nsec): min=1700, max=420260, avg=3154.62, stdev=1665.56
    clat (usec): min=165, max=95381, avg=5220.82, stdev=9984.14
     lat (usec): min=181, max=95384, avg=5223.98, stdev=9984.37
    clat percentiles (usec):
     |  1.00th=[ 1483],  5.00th=[ 1549], 10.00th=[ 1582], 20.00th=[ 1614],
     | 30.00th=[ 1647], 40.00th=[ 1680], 50.00th=[ 1696], 60.00th=[ 1729],
     | 70.00th=[ 1778], 80.00th=[ 4228], 90.00th=[12125], 95.00th=[30016],
     | 99.00th=[48497], 99.50th=[53740], 99.90th=[69731], 99.95th=[73925],
     | 99.99th=[74974]
   bw (  KiB/s): min= 1800, max=80288, per=6.17%, avg=24540.53, stdev=26852.86, samples=60
   iops        : min=  450, max=20072, avg=6135.03, stdev=6713.22, samples=60
  lat (usec)   : 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.02%
  lat (msec)   : 2=76.16%, 4=3.12%, 10=9.60%, 20=2.98%, 50=7.30%
  lat (msec)   : 100=0.80%
  cpu          : usr=1.60%, sys=5.21%, ctx=184023, majf=0, minf=139
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,183993,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
RND4K_Write_Q32T16: (groupid=3, jobs=1): err= 0: pid=2894458: Sat Sep 26 07:56:23 2026
  write: IOPS=6218, BW=24.3MiB/s (25.5MB/s)(730MiB/30049msec); 0 zone resets
    slat (nsec): min=1730, max=191671, avg=3018.04, stdev=1184.55
    clat (usec): min=156, max=96927, avg=5141.03, stdev=9932.40
     lat (usec): min=168, max=96930, avg=5144.05, stdev=9932.62
    clat percentiles (usec):
     |  1.00th=[ 1287],  5.00th=[ 1483], 10.00th=[ 1532], 20.00th=[ 1565],
     | 30.00th=[ 1598], 40.00th=[ 1631], 50.00th=[ 1663], 60.00th=[ 1729],
     | 70.00th=[ 1795], 80.00th=[ 4178], 90.00th=[11731], 95.00th=[30016],
     | 99.00th=[47973], 99.50th=[54264], 99.90th=[69731], 99.95th=[73925],
     | 99.99th=[74974]
   bw (  KiB/s): min= 1808, max=81320, per=6.26%, avg=24922.65, stdev=27283.51, samples=60
   iops        : min=  452, max=20330, avg=6230.58, stdev=6820.90, samples=60
  lat (usec)   : 250=0.01%, 500=0.02%, 750=0.02%, 1000=0.02%
  lat (msec)   : 2=76.98%, 4=1.99%, 10=10.07%, 20=2.93%, 50=7.21%
  lat (msec)   : 100=0.78%
  cpu          : usr=1.46%, sys=5.23%, ctx=186660, majf=0, minf=114
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,186857,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
RND4K_Write_Q32T16: (groupid=3, jobs=1): err= 0: pid=2894459: Sat Sep 26 07:56:23 2026
  write: IOPS=6028, BW=23.6MiB/s (24.7MB/s)(708MiB/30049msec); 0 zone resets
    slat (nsec): min=1940, max=241830, avg=3367.43, stdev=1498.17
    clat (usec): min=161, max=96855, avg=5303.03, stdev=10039.07
     lat (usec): min=172, max=96859, avg=5306.40, stdev=10039.35
    clat percentiles (usec):
     |  1.00th=[ 1369],  5.00th=[ 1631], 10.00th=[ 1663], 20.00th=[ 1696],
     | 30.00th=[ 1729], 40.00th=[ 1745], 50.00th=[ 1778], 60.00th=[ 1795],
     | 70.00th=[ 1844], 80.00th=[ 4113], 90.00th=[12649], 95.00th=[30278],
     | 99.00th=[48497], 99.50th=[54264], 99.90th=[69731], 99.95th=[73925],
     | 99.99th=[74974]
   bw (  KiB/s): min= 1800, max=74560, per=6.07%, avg=24159.92, stdev=26168.61, samples=60
   iops        : min=  450, max=18640, avg=6039.88, stdev=6542.19, samples=60
  lat (usec)   : 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.02%
  lat (msec)   : 2=76.51%, 4=3.15%, 10=9.08%, 20=2.98%, 50=7.44%
  lat (msec)   : 100=0.81%
  cpu          : usr=1.55%, sys=5.35%, ctx=180857, majf=0, minf=167
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,181139,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
RND4K_Write_Q32T16: (groupid=3, jobs=1): err= 0: pid=2894460: Sat Sep 26 07:56:23 2026
  write: IOPS=6031, BW=23.6MiB/s (24.7MB/s)(708MiB/30049msec); 0 zone resets
    slat (nsec): min=1710, max=199740, avg=3407.00, stdev=1260.66
    clat (usec): min=153, max=96833, avg=5300.34, stdev=10033.11
     lat (usec): min=157, max=96840, avg=5303.75, stdev=10033.36
    clat percentiles (usec):
     |  1.00th=[ 1434],  5.00th=[ 1549], 10.00th=[ 1598], 20.00th=[ 1680],
     | 30.00th=[ 1729], 40.00th=[ 1745], 50.00th=[ 1778], 60.00th=[ 1811],
     | 70.00th=[ 1860], 80.00th=[ 4080], 90.00th=[12649], 95.00th=[30016],
     | 99.00th=[48497], 99.50th=[54264], 99.90th=[69731], 99.95th=[73925],
     | 99.99th=[74974]
   bw (  KiB/s): min= 1800, max=81296, per=6.08%, avg=24172.45, stdev=26217.22, samples=60
   iops        : min=  450, max=20322, avg=6043.03, stdev=6554.29, samples=60
  lat (usec)   : 250=0.01%, 500=0.01%, 750=0.02%, 1000=0.03%
  lat (msec)   : 2=75.61%, 4=4.19%, 10=8.84%, 20=3.02%, 50=7.49%
  lat (msec)   : 100=0.81%
  cpu          : usr=1.55%, sys=5.35%, ctx=180728, majf=0, minf=168
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,181233,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
RND4K_Write_Q32T16: (groupid=3, jobs=1): err= 0: pid=2894461: Sat Sep 26 07:56:23 2026
  write: IOPS=6203, BW=24.2MiB/s (25.4MB/s)(728MiB/30050msec); 0 zone resets
    slat (nsec): min=1690, max=436890, avg=3304.35, stdev=2074.05
    clat (usec): min=165, max=96609, avg=5153.25, stdev=9961.19
     lat (usec): min=177, max=96615, avg=5156.55, stdev=9961.37
    clat percentiles (usec):
     |  1.00th=[ 1467],  5.00th=[ 1532], 10.00th=[ 1565], 20.00th=[ 1598],
     | 30.00th=[ 1614], 40.00th=[ 1647], 50.00th=[ 1663], 60.00th=[ 1696],
     | 70.00th=[ 1762], 80.00th=[ 4146], 90.00th=[11731], 95.00th=[30016],
     | 99.00th=[48497], 99.50th=[53740], 99.90th=[69731], 99.95th=[73925],
     | 99.99th=[74974]
   bw (  KiB/s): min= 1792, max=80360, per=6.25%, avg=24862.25, stdev=27604.61, samples=60
   iops        : min=  448, max=20090, avg=6215.48, stdev=6901.17, samples=60
  lat (usec)   : 250=0.01%, 500=0.01%, 750=0.02%, 1000=0.03%
  lat (msec)   : 2=76.99%, 4=2.71%, 10=9.32%, 20=2.92%, 50=7.22%
  lat (msec)   : 100=0.79%
  cpu          : usr=1.64%, sys=5.51%, ctx=186923, majf=0, minf=180
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,186404,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
RND4K_Write_Q32T16: (groupid=3, jobs=1): err= 0: pid=2894462: Sat Sep 26 07:56:23 2026
  write: IOPS=6225, BW=24.3MiB/s (25.5MB/s)(731MiB/30050msec); 0 zone resets
    slat (nsec): min=1680, max=197120, avg=3216.94, stdev=1525.47
    clat (usec): min=146, max=96713, avg=5135.19, stdev=9937.89
     lat (usec): min=156, max=96717, avg=5138.41, stdev=9938.11
    clat percentiles (usec):
     |  1.00th=[ 1467],  5.00th=[ 1532], 10.00th=[ 1549], 20.00th=[ 1582],
     | 30.00th=[ 1598], 40.00th=[ 1631], 50.00th=[ 1663], 60.00th=[ 1713],
     | 70.00th=[ 1795], 80.00th=[ 4113], 90.00th=[11731], 95.00th=[30016],
     | 99.00th=[48497], 99.50th=[53740], 99.90th=[69731], 99.95th=[73925],
     | 99.99th=[74974]
   bw (  KiB/s): min= 1792, max=80448, per=6.27%, avg=24950.28, stdev=27373.06, samples=60
   iops        : min=  448, max=20112, avg=6237.48, stdev=6843.30, samples=60
  lat (usec)   : 250=0.01%, 500=0.04%, 750=0.06%, 1000=0.06%
  lat (msec)   : 2=76.42%, 4=3.16%, 10=9.40%, 20=2.90%, 50=7.19%
  lat (msec)   : 100=0.79%
  cpu          : usr=1.68%, sys=5.40%, ctx=186621, majf=0, minf=183
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,187065,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
RND4K_Write_Q32T16: (groupid=3, jobs=1): err= 0: pid=2894463: Sat Sep 26 07:56:23 2026
  write: IOPS=6314, BW=24.7MiB/s (25.9MB/s)(741MiB/30049msec); 0 zone resets
    slat (nsec): min=1660, max=419330, avg=3185.78, stdev=2118.79
    clat (usec): min=146, max=96927, avg=5062.15, stdev=9876.19
     lat (usec): min=161, max=96930, avg=5065.33, stdev=9876.39
    clat percentiles (usec):
     |  1.00th=[ 1418],  5.00th=[ 1516], 10.00th=[ 1549], 20.00th=[ 1582],
     | 30.00th=[ 1598], 40.00th=[ 1614], 50.00th=[ 1631], 60.00th=[ 1647],
     | 70.00th=[ 1680], 80.00th=[ 4113], 90.00th=[11338], 95.00th=[29492],
     | 99.00th=[48497], 99.50th=[53740], 99.90th=[69731], 99.95th=[73925],
     | 99.99th=[74974]
   bw (  KiB/s): min= 1800, max=79784, per=6.36%, avg=25309.67, stdev=28000.89, samples=60
   iops        : min=  450, max=19946, avg=6327.35, stdev=7000.24, samples=60
  lat (usec)   : 250=0.01%, 500=0.03%, 750=0.04%, 1000=0.05%
  lat (msec)   : 2=77.24%, 4=2.42%, 10=9.49%, 20=2.96%, 50=7.01%
  lat (msec)   : 100=0.78%
  cpu          : usr=1.61%, sys=5.42%, ctx=190737, majf=0, minf=189
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,189759,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
RND4K_Write_Q32T16: (groupid=3, jobs=1): err= 0: pid=2894464: Sat Sep 26 07:56:23 2026
  write: IOPS=6233, BW=24.4MiB/s (25.5MB/s)(732MiB/30049msec); 0 zone resets
    slat (nsec): min=1670, max=314951, avg=3058.84, stdev=1712.73
    clat (usec): min=143, max=96859, avg=5128.66, stdev=9922.74
     lat (usec): min=153, max=96861, avg=5131.72, stdev=9923.07
    clat percentiles (usec):
     |  1.00th=[ 1467],  5.00th=[ 1532], 10.00th=[ 1565], 20.00th=[ 1598],
     | 30.00th=[ 1614], 40.00th=[ 1647], 50.00th=[ 1663], 60.00th=[ 1680],
     | 70.00th=[ 1729], 80.00th=[ 4113], 90.00th=[11731], 95.00th=[29754],
     | 99.00th=[48497], 99.50th=[53740], 99.90th=[69731], 99.95th=[73925],
     | 99.99th=[77071]
   bw (  KiB/s): min= 1792, max=79928, per=6.28%, avg=24982.62, stdev=27529.97, samples=60
   iops        : min=  448, max=19982, avg=6245.60, stdev=6882.53, samples=60
  lat (usec)   : 250=0.01%, 500=0.03%, 750=0.04%, 1000=0.04%
  lat (msec)   : 2=76.29%, 4=3.28%, 10=9.44%, 20=2.88%, 50=7.24%
  lat (msec)   : 100=0.78%
  cpu          : usr=1.66%, sys=5.17%, ctx=187048, majf=0, minf=170
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,187307,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
RND4K_Write_Q32T16: (groupid=3, jobs=1): err= 0: pid=2894465: Sat Sep 26 07:56:23 2026
  write: IOPS=6080, BW=23.8MiB/s (24.9MB/s)(714MiB/30051msec); 0 zone resets
    slat (nsec): min=1980, max=350480, avg=3244.30, stdev=2320.28
    clat (usec): min=188, max=97901, avg=5257.74, stdev=10007.05
     lat (usec): min=200, max=97905, avg=5260.99, stdev=10007.37
    clat percentiles (usec):
     |  1.00th=[ 1434],  5.00th=[ 1614], 10.00th=[ 1647], 20.00th=[ 1680],
     | 30.00th=[ 1696], 40.00th=[ 1729], 50.00th=[ 1745], 60.00th=[ 1778],
     | 70.00th=[ 1811], 80.00th=[ 3654], 90.00th=[12518], 95.00th=[30016],
     | 99.00th=[48497], 99.50th=[53740], 99.90th=[69731], 99.95th=[73925],
     | 99.99th=[78119]
   bw (  KiB/s): min= 1784, max=76488, per=6.13%, avg=24369.83, stdev=26456.56, samples=60
   iops        : min=  446, max=19122, avg=6092.38, stdev=6614.16, samples=60
  lat (usec)   : 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
  lat (msec)   : 2=76.16%, 4=4.59%, 10=8.00%, 20=2.96%, 50=7.48%
  lat (msec)   : 100=0.81%
  cpu          : usr=1.48%, sys=5.28%, ctx=182496, majf=0, minf=111
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,182713,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
RND4K_Write_Q32T16: (groupid=3, jobs=1): err= 0: pid=2894466: Sat Sep 26 07:56:23 2026
  write: IOPS=6190, BW=24.2MiB/s (25.4MB/s)(727MiB/30049msec); 0 zone resets
    slat (nsec): min=1690, max=232510, avg=3250.67, stdev=1312.19
    clat (usec): min=136, max=96504, avg=5164.30, stdev=9948.35
     lat (usec): min=150, max=96508, avg=5167.55, stdev=9948.63
    clat percentiles (usec):
     |  1.00th=[ 1319],  5.00th=[ 1516], 10.00th=[ 1549], 20.00th=[ 1598],
     | 30.00th=[ 1631], 40.00th=[ 1663], 50.00th=[ 1745], 60.00th=[ 1795],
     | 70.00th=[ 1844], 80.00th=[ 3851], 90.00th=[11863], 95.00th=[29754],
     | 99.00th=[48497], 99.50th=[53740], 99.90th=[69731], 99.95th=[73925],
     | 99.99th=[74974]
   bw (  KiB/s): min= 1792, max=80840, per=6.24%, avg=24809.80, stdev=27504.85, samples=60
   iops        : min=  448, max=20210, avg=6202.37, stdev=6876.22, samples=60
  lat (usec)   : 250=0.01%, 500=0.02%, 750=0.02%, 1000=0.03%
  lat (msec)   : 2=77.09%, 4=3.10%, 10=8.74%, 20=3.01%, 50=7.20%
  lat (msec)   : 100=0.79%
  cpu          : usr=1.51%, sys=5.36%, ctx=185395, majf=0, minf=111
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,186011,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: bw=3158MiB/s (3312MB/s), 3158MiB/s-3158MiB/s (3312MB/s-3312MB/s), io=92.5GiB (99.4GB), run=30003-30003msec

Run status group 1 (all jobs):
  WRITE: bw=215MiB/s (225MB/s), 215MiB/s-215MiB/s (225MB/s-225MB/s), io=6460MiB (6774MB), run=30041-30041msec

Run status group 2 (all jobs):
   READ: bw=87.6MiB/s (91.9MB/s), 5566KiB/s-5667KiB/s (5700kB/s-5803kB/s), io=2632MiB (2760MB), run=30032-30035msec

Run status group 3 (all jobs):
  WRITE: bw=389MiB/s (407MB/s), 23.6MiB/s-25.3MiB/s (24.7MB/s-26.6MB/s), io=11.4GiB (12.2GB), run=30048-30051msec


root@pve:~# fio --time_based \
    --runtime=30 \
    --ramp_time=5 \
    --ioengine=io_uring \
    --direct=1 \
    --size=4G \
    --filename=/seq_test.fio \
    --name=PURE_SEQ \
    --bs=1m \
    --rw=write \
    --iodepth=8 \
    --numjobs=1
PURE_SEQ: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=io_uring, iodepth=8
fio-3.39
Starting 1 process
PURE_SEQ: Laying out IO file (1 file / 4096MiB)
Jobs: 1 (f=1): [W(1)][100.0%][w=213MiB/s][w=213 IOPS][eta 00m:00s]
PURE_SEQ: (groupid=0, jobs=1): err= 0: pid=2895120: Sat Sep 26 07:57:46 2026
  write: IOPS=211, BW=211MiB/s (222MB/s)(6349MiB/30035msec); 0 zone resets
    slat (usec): min=24, max=532, avg=40.04, stdev=17.44
    clat (usec): min=34490, max=64206, avg=37818.80, stdev=1206.60
     lat (usec): min=34531, max=64263, avg=37858.84, stdev=1206.73
    clat percentiles (usec):
     |  1.00th=[35914],  5.00th=[36439], 10.00th=[36963], 20.00th=[37487],
     | 30.00th=[37487], 40.00th=[37487], 50.00th=[37487], 60.00th=[37487],
     | 70.00th=[38011], 80.00th=[38011], 90.00th=[38536], 95.00th=[40109],
     | 99.00th=[42730], 99.50th=[42730], 99.90th=[45876], 99.95th=[50070],
     | 99.99th=[64226]
   bw (  KiB/s): min=204800, max=221184, per=100.00%, avg=216490.12, stdev=2869.06, samples=60
   iops        : min=  200, max=  216, avg=211.37, stdev= 2.79, samples=60
  lat (msec)   : 50=100.05%, 100=0.06%
  cpu          : usr=0.91%, sys=0.25%, ctx=6355, majf=0, minf=2099
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,6342,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
  WRITE: bw=211MiB/s (222MB/s), 211MiB/s-211MiB/s (222MB/s-222MB/s), io=6349MiB (6657MB), run=30035-30035msec

jhartbarger · Sep 25, 2026

Do you have other drives in this system that you have ran this test on and what did it look like? First looking at your second output screams to me you have a Record Size issue because S1M reads look great, R4K reads are terrible.

I think the issue is the combination of a suboptimal configuration of the zfs Record Size, and another part testing methodology. You typically structure a testing methodology and the datasets "use case" and in the Root Cause section #2 and #3 confirms my suspicion.

Performance Analysis:

Metric	Current Value	Expected (High-Perf NVMe)	Status / Observation
SEQ1M_Read	2,831 MiB/s	~3,200+ MiB/s	Healthy. Close to maximum real-world throughput for a PCIe Gen3 x4 connection.
SEQ1M_Write	185 MiB/s	~1,500+ MiB/s	Severe Throttling. Running at the speed of a mechanical HDD rather than an enterprise solid-state drive.
RND4K_Read	~3.6 MiB/s (per job)	~400+ MiB/s (total)	Queue Saturation. The median latency is great (701 µs), but the 95th percentile jumps off a cliff to 557 ms.

Root Cause Breakdown

Extreme Queue Concurrency Overload
The test parameters throw 512 concurrent asynchronous flight requests (16 jobs * 32 iodepth) simultaneously at the storage layer. This massively overflows the internal hardware queue structures, forcing requests to stall out in the ring buffers for over a second (99.99th percentile at 1.28 seconds).
Synchronous Write Bottleneck
Even though --direct=1 is passed, the filesystem or storage stack is treating the large sequential writes as blocking, synchronous operations. The high average completion latency (43.13 ms) shows the engine is spending its time waiting for hardware cache flushes rather than streaming data.
Block Allocation Disconnect
If this test is running on top of a filesystem like ZFS using a standard 128k record size, every minor 4k random read requires fetching an entire 128k block from disk, causing massive I/O amplification.

This is also limiting you but I don't think this is your root problem

Your Proxmox host (pve151) has a critical CPU frequency misconfiguration that is actively throttling your storage benchmark results. Your CPU cores are hard-capped at 2.10 GHz, artificial bottlenecks preventing your AMD EPYC processor from reaching its 3.32 GHz hardware limit.
Because your single-threaded FIO storage test (SEQ1M_Write) relies heavily on CPU processing for ZFS checksumming and compression, locking your clock speed at a lower threshold explains some of the performance issue.

luckylinux · Sep 25, 2026

jhartbarger said:
Do you have other drives in this system that you have ran this test on and what did it look like? First looking at your second output screams to me you have a Record Size issue because S1M reads look great, R4K reads are terrible.

No, I only have these 2 x Micron 7300 MAX in that System.

Code:

root@pve:~# zpool status
  pool: rpool
 state: ONLINE
config:

    NAME                                                        STATE     READ WRITE CKSUM
    rpool                                                       ONLINE       0     0     0
      mirror-0                                                  ONLINE       0     0     0
        nvme-Micron_7300_MTFDHBE3T2TDG_1946298769D8_root_crypt  ONLINE       0     0     0
        nvme-Micron_7300_MTFDHBE3T2TDG_194629C4947D_root_crypt  ONLINE       0     0     0

errors: No known data errors

jhartbarger said:
I think the issue is the combination of a suboptimal configuration of the zfs Record Size, and another part testing methodology. You typically structure a testing methodology and the datasets "use case" and in the Root Cause section #2 and #3 confirms my suspicion.

To be fair the fio Command was just something that I asked Gemini AI, I don't have much direct experience with fio at all (I only played with it previously trying to estimate Write Amplification on Proxmox VE Virtual Machines).

jhartbarger said:
Performance Analysis:

Metric Current Value Expected (High-Perf NVMe) Status / Observation
SEQ1M_Read 2,831 MiB/s ~3,200+ MiB/s Healthy. Close to maximum real-world throughput for a PCIe Gen3 x4 connection.
SEQ1M_Write 185 MiB/s ~1,500+ MiB/s Severe Throttling. Running at the speed of a mechanical HDD rather than an enterprise solid-state drive.
RND4K_Read ~3.6 MiB/s (per job) ~400+ MiB/s (total) Queue Saturation. The median latency is great (701 µs), but the 95th percentile jumps off a cliff to 557 ms.

Root Cause Breakdown

Extreme Queue Concurrency Overload
The test parameters throw 512 concurrent asynchronous flight requests (16 jobs * 32 iodepth) simultaneously at the storage layer. This massively overflows the internal hardware queue structures, forcing requests to stall out in the ring buffers for over a second (99.99th percentile at 1.28 seconds).

Synchronous Write Bottleneck
Even though --direct=1 is passed, the filesystem or storage stack is treating the large sequential writes as blocking, synchronous operations. The high average completion latency (43.13 ms) shows the engine is spending its time waiting for hardware cache flushes rather than streaming data.

Block Allocation Disconnect
If this test is running on top of a filesystem like ZFS using a standard 128k record size, every minor 4k random read requires fetching an entire 128k block from disk, causing massive I/O amplification.

Root Dataset is 128k yeah. But ZFS is kinda (?) smart, that is actually a maximum Value, ZFS can automatically use smaller Values of Block Size

.

So how do I go from there ? This seems to be reproducible across all NVMe Enterprise Drives I won (see my previous Post, Micron 9100 MAX 1.2TB seems to have the same Issue, on another one of these Hyve EPYC 7642 Server), not only these Drives.

jhartbarger said:
This is also limiting you but I don't think this is your root problem

Your Proxmox host (pve151) has a critical CPU frequency misconfiguration that is actively throttling your storage benchmark results. Your CPU cores are hard-capped at 2.10 GHz, artificial bottlenecks preventing your AMD EPYC processor from reaching its 3.32 GHz hardware limit.
Because your single-threaded FIO storage test (SEQ1M_Write) relies heavily on CPU processing for ZFS checksumming and compression, locking your clock speed at a lower threshold explains the performance issues.

I capped it at 2.1GHz in order to significantly limit Fan Noise and CPU Temperatures.

Switching from 2.3GHz to 2.1GHz transformed it from a Jet Engine to nearly tolerable

.

luckylinux · Sep 25, 2026

I just rebooted after removing powertop --auto-tune in /etc/rc.local to see if that was maybe doing some funky Stuff.

Performance seems to be the same (Bad) though with 2 x Micron 7300 MAX 3.2TB:

Code:

root@pve:~# fio --time_based \
    --runtime=30 \
    --ramp_time=5 \
    --ioengine=io_uring \
    --direct=1 \
    --size=2G \
    --stonewall \
    --name=SEQ1M_Read_Q8T1 --bs=1m --rw=read --iodepth=8 --numjobs=1 --filename=/testfile_seq.fio \
    --name=SEQ1M_Write_Q8T1 --bs=1m --rw=write --iodepth=8 --numjobs=1 --filename=/testfile_seq_w.fio \
    --name=RND4K_Read_Q32T16 --bs=4k --rw=randread --iodepth=32 --numjobs=16 --filename=/testfile_rnd.fio \
    --name=RND4K_Write_Q32T16 --bs=4k --rw=randwrite --iodepth=32 --numjobs=16 --filename=/testfile_rnd_w.fio

SEQ1M_Read_Q8T1: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=io_uring, iodepth=8
SEQ1M_Write_Q8T1: (g=1): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=io_uring, iodepth=8
RND4K_Read_Q32T16: (g=2): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=io_uring, iodepth=32
...
RND4K_Write_Q32T16: (g=3): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=io_uring, iodepth=32
...
fio-3.39
Starting 34 processes
Jobs: 16 (f=16): [_(18),w(16)][75.9%][w=516MiB/s][w=132k IOPS][eta 00m:45s]      
SEQ1M_Read_Q8T1: (groupid=0, jobs=1): err= 0: pid=41320: Sat Sep 26 08:13:15 2026
  read: IOPS=1043, BW=1044MiB/s (1094MB/s)(30.6GiB/30022msec)
    slat (nsec): min=1641, max=256314, avg=3642.80, stdev=2127.10
    clat (usec): min=793, max=147750, avg=7661.84, stdev=19010.74
     lat (usec): min=797, max=147753, avg=7665.49, stdev=19010.74
    clat percentiles (usec):
     |  1.00th=[   922],  5.00th=[   979], 10.00th=[  1012], 20.00th=[  1057],
     | 30.00th=[  1074], 40.00th=[  1106], 50.00th=[  1139], 60.00th=[  1172],
     | 70.00th=[  1221], 80.00th=[  1336], 90.00th=[ 23200], 95.00th=[ 55313],
     | 99.00th=[ 92799], 99.50th=[103285], 99.90th=[121111], 99.95th=[129500],
     | 99.99th=[145753]
   bw (  MiB/s): min=  751, max= 1530, per=100.00%, avg=1044.41, stdev=166.22, samples=60
   iops        : min=  751, max= 1530, avg=1044.30, stdev=166.23, samples=60
  lat (usec)   : 1000=7.75%
  lat (msec)   : 2=76.17%, 4=0.34%, 10=0.87%, 20=3.95%, 50=5.28%
  lat (msec)   : 100=5.03%, 250=0.63%
  cpu          : usr=0.37%, sys=1.31%, ctx=31321, majf=0, minf=77
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=31324,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8
SEQ1M_Write_Q8T1: (groupid=1, jobs=1): err= 0: pid=43077: Sat Sep 26 08:13:15 2026
  write: IOPS=188, BW=189MiB/s (198MB/s)(5677MiB/30042msec); 0 zone resets
    slat (usec): min=23, max=607, avg=44.31, stdev=20.37
    clat (usec): min=41794, max=73875, avg=42306.68, stdev=703.39
     lat (usec): min=41869, max=73913, avg=42350.99, stdev=703.27
    clat percentiles (usec):
     |  1.00th=[41681],  5.00th=[42206], 10.00th=[42206], 20.00th=[42206],
     | 30.00th=[42206], 40.00th=[42206], 50.00th=[42206], 60.00th=[42206],
     | 70.00th=[42206], 80.00th=[42206], 90.00th=[42730], 95.00th=[42730],
     | 99.00th=[43254], 99.50th=[43254], 99.90th=[47449], 99.95th=[63177],
     | 99.99th=[73925]
   bw (  KiB/s): min=192512, max=194949, per=100.00%, avg=193579.20, stdev=1033.84, samples=60
   iops        : min=  188, max=  190, avg=188.97, stdev= 1.01, samples=60
  lat (msec)   : 50=100.04%, 100=0.09%
  cpu          : usr=1.00%, sys=0.22%, ctx=5689, majf=0, minf=2102
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,5670,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8
RND4K_Read_Q32T16: (groupid=2, jobs=1): err= 0: pid=44519: Sat Sep 26 08:13:15 2026
  read: IOPS=1032, BW=4132KiB/s (4232kB/s)(126MiB/31218msec)
    slat (nsec): min=1640, max=92302, avg=3251.04, stdev=1688.43
    clat (usec): min=372, max=2720.2k, avg=31337.00, stdev=191292.91
     lat (usec): min=374, max=2720.2k, avg=31340.25, stdev=191292.93
    clat percentiles (usec):
     |  1.00th=[    433],  5.00th=[    469], 10.00th=[    494],
     | 20.00th=[    529], 30.00th=[    553], 40.00th=[    578],
     | 50.00th=[    611], 60.00th=[    644], 70.00th=[    685],
     | 80.00th=[    742], 90.00th=[    840], 95.00th=[    971],
     | 99.00th=[1266680], 99.50th=[1501561], 99.90th=[1702888],
     | 99.95th=[2105541], 99.99th=[2600469]
   bw (  KiB/s): min= 1616, max= 6968, per=6.69%, avg=4291.80, stdev=1244.54, samples=60
   iops        : min=  404, max= 1742, avg=1072.87, stdev=311.14, samples=60
  lat (usec)   : 500=12.22%, 750=69.15%, 1000=14.26%
  lat (msec)   : 2=1.55%, 4=0.01%, 750=0.97%, 1000=0.40%, 2000=1.47%
  lat (msec)   : >=2000=0.06%
  cpu          : usr=0.33%, sys=1.12%, ctx=32303, majf=0, minf=38
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=32221,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
RND4K_Read_Q32T16: (groupid=2, jobs=1): err= 0: pid=44520: Sat Sep 26 08:13:15 2026
  read: IOPS=995, BW=3986KiB/s (4082kB/s)(122MiB/31231msec)
    slat (nsec): min=1640, max=85601, avg=3318.54, stdev=1617.41
    clat (usec): min=366, max=2700.9k, avg=32425.68, stdev=194592.27
     lat (usec): min=368, max=2700.9k, avg=32429.00, stdev=194592.27
    clat percentiles (usec):
     |  1.00th=[    433],  5.00th=[    469], 10.00th=[    494],
     | 20.00th=[    529], 30.00th=[    553], 40.00th=[    578],
     | 50.00th=[    611], 60.00th=[    644], 70.00th=[    685],
     | 80.00th=[    742], 90.00th=[    848], 95.00th=[    979],
     | 99.00th=[1300235], 99.50th=[1501561], 99.90th=[1702888],
     | 99.95th=[2071987], 99.99th=[2566915]
   bw (  KiB/s): min=    8, max= 9840, per=6.36%, avg=4079.03, stdev=1955.16, samples=61
   iops        : min=    2, max= 2460, avg=1019.70, stdev=488.80, samples=61
  lat (usec)   : 500=11.88%, 750=69.15%, 1000=14.43%
  lat (msec)   : 2=1.62%, 4=0.03%, 10=0.01%, 750=0.99%, 1000=0.44%
  lat (msec)   : 2000=1.51%, >=2000=0.06%
  cpu          : usr=0.28%, sys=1.14%, ctx=31200, majf=0, minf=37
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=31090,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
RND4K_Read_Q32T16: (groupid=2, jobs=1): err= 0: pid=44521: Sat Sep 26 08:13:15 2026
  read: IOPS=1050, BW=4208KiB/s (4309kB/s)(129MiB/31324msec)
    slat (nsec): min=1610, max=72261, avg=3175.01, stdev=1226.53
    clat (usec): min=375, max=2770.0k, avg=30807.22, stdev=188983.69
     lat (usec): min=378, max=2770.0k, avg=30810.39, stdev=188983.72
    clat percentiles (usec):
     |  1.00th=[    433],  5.00th=[    465], 10.00th=[    490],
     | 20.00th=[    529], 30.00th=[    553], 40.00th=[    578],
     | 50.00th=[    611], 60.00th=[    644], 70.00th=[    685],
     | 80.00th=[    742], 90.00th=[    840], 95.00th=[    963],
     | 99.00th=[1266680], 99.50th=[1501561], 99.90th=[1702888],
     | 99.95th=[1887437], 99.99th=[2365588]
   bw (  KiB/s): min=  528, max= 9144, per=6.83%, avg=4384.62, stdev=1938.27, samples=60
   iops        : min=  132, max= 2286, avg=1096.07, stdev=484.57, samples=60
  lat (usec)   : 500=12.18%, 750=68.86%, 1000=14.63%
  lat (msec)   : 2=1.56%, 4=0.01%, 750=0.96%, 1000=0.41%, 2000=1.45%
  lat (msec)   : >=2000=0.04%
  cpu          : usr=0.30%, sys=1.16%, ctx=33064, majf=0, minf=37
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=32920,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
RND4K_Read_Q32T16: (groupid=2, jobs=1): err= 0: pid=44523: Sat Sep 26 08:13:15 2026
  read: IOPS=997, BW=3992KiB/s (4088kB/s)(122MiB/31271msec)
    slat (nsec): min=1640, max=77351, avg=3191.35, stdev=1402.18
    clat (usec): min=355, max=2706.7k, avg=32353.84, stdev=193941.44
     lat (usec): min=371, max=2706.7k, avg=32357.04, stdev=193941.46
    clat percentiles (usec):
     |  1.00th=[    433],  5.00th=[    469], 10.00th=[    490],
     | 20.00th=[    529], 30.00th=[    553], 40.00th=[    578],
     | 50.00th=[    611], 60.00th=[    644], 70.00th=[    685],
     | 80.00th=[    742], 90.00th=[    848], 95.00th=[    979],
     | 99.00th=[1283458], 99.50th=[1501561], 99.90th=[1702888],
     | 99.95th=[1904215], 99.99th=[2667578]
   bw (  KiB/s): min= 1680, max= 6701, per=6.47%, avg=4153.92, stdev=1184.39, samples=60
   iops        : min=  420, max= 1675, avg=1038.40, stdev=296.07, samples=60
  lat (usec)   : 500=12.56%, 750=68.70%, 1000=14.18%
  lat (msec)   : 2=1.65%, 4=0.01%, 750=0.96%, 1000=0.47%, 2000=1.53%
  lat (msec)   : >=2000=0.04%
  cpu          : usr=0.32%, sys=1.07%, ctx=31307, majf=0, minf=37
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=31179,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
RND4K_Read_Q32T16: (groupid=2, jobs=1): err= 0: pid=44524: Sat Sep 26 08:13:15 2026
  read: IOPS=994, BW=3980KiB/s (4076kB/s)(122MiB/31361msec)
    slat (nsec): min=1640, max=88241, avg=3255.16, stdev=1548.76
    clat (usec): min=337, max=2697.8k, avg=32545.85, stdev=194592.61
     lat (usec): min=340, max=2697.8k, avg=32549.11, stdev=194592.63
    clat percentiles (usec):
     |  1.00th=[    433],  5.00th=[    469], 10.00th=[    494],
     | 20.00th=[    529], 30.00th=[    553], 40.00th=[    578],
     | 50.00th=[    611], 60.00th=[    644], 70.00th=[    685],
     | 80.00th=[    742], 90.00th=[    848], 95.00th=[    979],
     | 99.00th=[1300235], 99.50th=[1518339], 99.90th=[1686111],
     | 99.95th=[2004878], 99.99th=[2533360]
   bw (  KiB/s): min=    8, max=10600, per=6.36%, avg=4081.95, stdev=1470.62, samples=61
   iops        : min=    2, max= 2650, avg=1020.41, stdev=367.65, samples=61
  lat (usec)   : 500=11.91%, 750=69.13%, 1000=14.48%
  lat (msec)   : 2=1.56%, 4=0.01%, 750=0.98%, 1000=0.46%, 2000=1.53%
  lat (msec)   : >=2000=0.05%
  cpu          : usr=0.30%, sys=1.10%, ctx=31307, majf=0, minf=38
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=31175,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
RND4K_Read_Q32T16: (groupid=2, jobs=1): err= 0: pid=44525: Sat Sep 26 08:13:15 2026
  read: IOPS=926, BW=3710KiB/s (3799kB/s)(114MiB/31369msec)
    slat (nsec): min=1640, max=94822, avg=3288.41, stdev=1884.71
    clat (usec): min=381, max=2835.6k, avg=34889.19, stdev=202040.49
     lat (usec): min=384, max=2835.6k, avg=34892.48, stdev=202040.51
    clat percentiles (usec):
     |  1.00th=[    433],  5.00th=[    469], 10.00th=[    494],
     | 20.00th=[    529], 30.00th=[    553], 40.00th=[    578],
     | 50.00th=[    611], 60.00th=[    644], 70.00th=[    685],
     | 80.00th=[    742], 90.00th=[    848], 95.00th=[    988],
     | 99.00th=[1317012], 99.50th=[1518339], 99.90th=[1736442],
     | 99.95th=[2298479], 99.99th=[2768241]
   bw (  KiB/s): min= 1440, max= 6968, per=6.03%, avg=3871.73, stdev=1169.91, samples=60
   iops        : min=  360, max= 1742, avg=967.87, stdev=292.49, samples=60
  lat (usec)   : 500=11.71%, 750=69.16%, 1000=14.37%
  lat (msec)   : 2=1.62%, 4=0.01%, 750=1.05%, 1000=0.48%, 2000=1.64%
  lat (msec)   : >=2000=0.06%
  cpu          : usr=0.31%, sys=1.00%, ctx=29204, majf=0, minf=38
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=29064,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
RND4K_Read_Q32T16: (groupid=2, jobs=1): err= 0: pid=44526: Sat Sep 26 08:13:15 2026
  read: IOPS=1047, BW=4196KiB/s (4296kB/s)(128MiB/31277msec)
    slat (nsec): min=1640, max=81081, avg=3239.97, stdev=1551.36
    clat (usec): min=366, max=2736.0k, avg=30861.91, stdev=188818.66
     lat (usec): min=370, max=2736.0k, avg=30865.15, stdev=188818.68
    clat percentiles (usec):
     |  1.00th=[    433],  5.00th=[    469], 10.00th=[    494],
     | 20.00th=[    529], 30.00th=[    553], 40.00th=[    578],
     | 50.00th=[    611], 60.00th=[    644], 70.00th=[    685],
     | 80.00th=[    742], 90.00th=[    840], 95.00th=[    971],
     | 99.00th=[1266680], 99.50th=[1501561], 99.90th=[1686111],
     | 99.95th=[1786774], 99.99th=[2466251]
   bw (  KiB/s): min= 1368, max= 7560, per=6.80%, avg=4365.98, stdev=1515.73, samples=60
   iops        : min=  342, max= 1890, avg=1091.45, stdev=378.94, samples=60
  lat (usec)   : 500=11.98%, 750=69.52%, 1000=14.18%
  lat (msec)   : 2=1.55%, 4=0.01%, 750=0.94%, 1000=0.42%, 2000=1.46%
  lat (msec)   : >=2000=0.04%
  cpu          : usr=0.37%, sys=1.10%, ctx=32917, majf=0, minf=38
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=32776,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
RND4K_Read_Q32T16: (groupid=2, jobs=1): err= 0: pid=44527: Sat Sep 26 08:13:15 2026
  read: IOPS=995, BW=3986KiB/s (4081kB/s)(122MiB/31333msec)
    slat (nsec): min=1640, max=73212, avg=3222.20, stdev=1307.51
    clat (usec): min=377, max=2850.8k, avg=32480.92, stdev=195513.38
     lat (usec): min=379, max=2850.8k, avg=32484.14, stdev=195513.39
    clat percentiles (usec):
     |  1.00th=[    433],  5.00th=[    469], 10.00th=[    494],
     | 20.00th=[    529], 30.00th=[    553], 40.00th=[    578],
     | 50.00th=[    611], 60.00th=[    644], 70.00th=[    685],
     | 80.00th=[    742], 90.00th=[    840], 95.00th=[    979],
     | 99.00th=[1283458], 99.50th=[1501561], 99.90th=[1686111],
     | 99.95th=[2399142], 99.99th=[2734687]
   bw (  KiB/s): min=  272, max= 7920, per=6.47%, avg=4154.57, stdev=1740.85, samples=60
   iops        : min=   68, max= 1980, avg=1038.57, stdev=435.17, samples=60
  lat (usec)   : 500=11.97%, 750=69.31%, 1000=14.27%
  lat (msec)   : 2=1.53%, 4=0.02%, 750=0.96%, 1000=0.48%, 2000=1.49%
  lat (msec)   : >=2000=0.06%
  cpu          : usr=0.31%, sys=1.08%, ctx=31340, majf=0, minf=38
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=31190,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
RND4K_Read_Q32T16: (groupid=2, jobs=1): err= 0: pid=44528: Sat Sep 26 08:13:15 2026
  read: IOPS=1001, BW=4009KiB/s (4105kB/s)(123MiB/31346msec)
    slat (nsec): min=1620, max=88841, avg=3289.72, stdev=1719.84
    clat (usec): min=378, max=2826.4k, avg=32254.89, stdev=194771.23
     lat (usec): min=381, max=2826.4k, avg=32258.18, stdev=194771.25
    clat percentiles (usec):
     |  1.00th=[    433],  5.00th=[    469], 10.00th=[    494],
     | 20.00th=[    529], 30.00th=[    553], 40.00th=[    578],
     | 50.00th=[    611], 60.00th=[    644], 70.00th=[    685],
     | 80.00th=[    742], 90.00th=[    848], 95.00th=[    971],
     | 99.00th=[1283458], 99.50th=[1501561], 99.90th=[1702888],
     | 99.95th=[2264925], 99.99th=[2801796]
   bw (  KiB/s): min=  376, max= 7190, per=6.51%, avg=4180.97, stdev=1429.79, samples=60
   iops        : min=   94, max= 1797, avg=1045.18, stdev=357.44, samples=60
  lat (usec)   : 500=11.88%, 750=69.33%, 1000=14.46%
  lat (msec)   : 2=1.44%, 4=0.02%, 750=0.97%, 1000=0.45%, 2000=1.50%
  lat (msec)   : >=2000=0.06%
  cpu          : usr=0.36%, sys=1.07%, ctx=31512, majf=0, minf=38
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=31383,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
RND4K_Read_Q32T16: (groupid=2, jobs=1): err= 0: pid=44529: Sat Sep 26 08:13:15 2026
  read: IOPS=971, BW=3892KiB/s (3985kB/s)(119MiB/31305msec)
    slat (nsec): min=1640, max=81881, avg=3288.25, stdev=1562.48
    clat (usec): min=374, max=2737.6k, avg=33221.09, stdev=196882.73
     lat (usec): min=376, max=2737.6k, avg=33224.38, stdev=196882.76
    clat percentiles (usec):
     |  1.00th=[    433],  5.00th=[    465], 10.00th=[    494],
     | 20.00th=[    529], 30.00th=[    553], 40.00th=[    578],
     | 50.00th=[    611], 60.00th=[    644], 70.00th=[    685],
     | 80.00th=[    742], 90.00th=[    840], 95.00th=[    971],
     | 99.00th=[1300235], 99.50th=[1501561], 99.90th=[1702888],
     | 99.95th=[2197816], 99.99th=[2667578]
   bw (  KiB/s): min=    8, max= 7704, per=6.21%, avg=3987.69, stdev=1342.96, samples=61
   iops        : min=    2, max= 1926, avg=996.85, stdev=335.77, samples=61
  lat (usec)   : 500=12.01%, 750=69.53%, 1000=14.07%
  lat (msec)   : 2=1.40%, 750=0.99%, 1000=0.48%, 2000=1.56%, >=2000=0.06%
  cpu          : usr=0.28%, sys=1.10%, ctx=30552, majf=0, minf=39
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=30428,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
RND4K_Read_Q32T16: (groupid=2, jobs=1): err= 0: pid=44530: Sat Sep 26 08:13:15 2026
  read: IOPS=1061, BW=4250KiB/s (4352kB/s)(130MiB/31308msec)
    slat (nsec): min=1640, max=92941, avg=3429.97, stdev=1747.74
    clat (usec): min=376, max=2706.4k, avg=30440.62, stdev=188841.72
     lat (usec): min=380, max=2706.4k, avg=30444.05, stdev=188841.73
    clat percentiles (usec):
     |  1.00th=[    433],  5.00th=[    469], 10.00th=[    494],
     | 20.00th=[    529], 30.00th=[    553], 40.00th=[    578],
     | 50.00th=[    611], 60.00th=[    644], 70.00th=[    685],
     | 80.00th=[    742], 90.00th=[    840], 95.00th=[    971],
     | 99.00th=[1266680], 99.50th=[1501561], 99.90th=[1702888],
     | 99.95th=[2105541], 99.99th=[2667578]
   bw (  KiB/s): min= 1624, max= 7040, per=6.90%, avg=4426.45, stdev=1222.50, samples=60
   iops        : min=  406, max= 1760, avg=1106.57, stdev=305.65, samples=60
  lat (usec)   : 500=11.83%, 750=69.20%, 1000=14.63%
  lat (msec)   : 2=1.61%, 4=0.01%, 750=0.94%, 1000=0.40%, 2000=1.41%
  lat (msec)   : >=2000=0.06%
  cpu          : usr=0.36%, sys=1.22%, ctx=33350, majf=0, minf=37
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=33232,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
RND4K_Read_Q32T16: (groupid=2, jobs=1): err= 0: pid=44531: Sat Sep 26 08:13:15 2026
  read: IOPS=1002, BW=4013KiB/s (4109kB/s)(123MiB/31344msec)
    slat (nsec): min=1630, max=99152, avg=3230.71, stdev=1493.16
    clat (usec): min=345, max=2813.7k, avg=32235.23, stdev=194315.45
     lat (usec): min=370, max=2813.7k, avg=32238.46, stdev=194315.46
    clat percentiles (usec):
     |  1.00th=[    429],  5.00th=[    465], 10.00th=[    490],
     | 20.00th=[    529], 30.00th=[    553], 40.00th=[    578],
     | 50.00th=[    611], 60.00th=[    644], 70.00th=[    685],
     | 80.00th=[    742], 90.00th=[    840], 95.00th=[    971],
     | 99.00th=[1283458], 99.50th=[1501561], 99.90th=[1702888],
     | 99.95th=[2231370], 99.99th=[2734687]
   bw (  KiB/s): min= 1752, max= 8640, per=6.52%, avg=4183.28, stdev=1387.08, samples=60
   iops        : min=  438, max= 2160, avg=1045.73, stdev=346.75, samples=60
  lat (usec)   : 500=12.33%, 750=68.94%, 1000=14.32%
  lat (msec)   : 2=1.50%, 4=0.02%, 10=0.01%, 750=0.97%, 1000=0.46%
  lat (msec)   : 2000=1.50%, >=2000=0.06%
  cpu          : usr=0.36%, sys=1.04%, ctx=31570, majf=0, minf=37
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=31415,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
RND4K_Read_Q32T16: (groupid=2, jobs=1): err= 0: pid=44533: Sat Sep 26 08:13:15 2026
  read: IOPS=1026, BW=4111KiB/s (4209kB/s)(126MiB/31281msec)
    slat (nsec): min=1620, max=77001, avg=3260.97, stdev=1492.10
    clat (usec): min=373, max=2733.3k, avg=31411.09, stdev=191025.92
     lat (usec): min=376, max=2733.3k, avg=31414.35, stdev=191025.94
    clat percentiles (usec):
     |  1.00th=[    433],  5.00th=[    469], 10.00th=[    494],
     | 20.00th=[    529], 30.00th=[    553], 40.00th=[    578],
     | 50.00th=[    611], 60.00th=[    644], 70.00th=[    685],
     | 80.00th=[    742], 90.00th=[    848], 95.00th=[    971],
     | 99.00th=[1283458], 99.50th=[1501561], 99.90th=[1702888],
     | 99.95th=[1920992], 99.99th=[2566915]
   bw (  KiB/s): min= 1498, max= 7880, per=6.67%, avg=4279.32, stdev=1291.46, samples=60
   iops        : min=  374, max= 1970, avg=1069.77, stdev=322.87, samples=60
  lat (usec)   : 500=11.90%, 750=69.16%, 1000=14.55%
  lat (msec)   : 2=1.58%, 4=0.01%, 10=0.01%, 750=0.95%, 1000=0.44%
  lat (msec)   : 2000=1.47%, >=2000=0.04%
  cpu          : usr=0.32%, sys=1.13%, ctx=32225, majf=0, minf=38
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=32116,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
RND4K_Read_Q32T16: (groupid=2, jobs=1): err= 0: pid=44534: Sat Sep 26 08:13:15 2026
  read: IOPS=960, BW=3847KiB/s (3940kB/s)(118MiB/31351msec)
    slat (nsec): min=1640, max=68711, avg=3190.74, stdev=1305.24
    clat (usec): min=373, max=2769.9k, avg=33762.53, stdev=198081.59
     lat (usec): min=376, max=2769.9k, avg=33765.72, stdev=198081.62
    clat percentiles (usec):
     |  1.00th=[    429],  5.00th=[    465], 10.00th=[    490],
     | 20.00th=[    529], 30.00th=[    553], 40.00th=[    578],
     | 50.00th=[    611], 60.00th=[    644], 70.00th=[    685],
     | 80.00th=[    742], 90.00th=[    848], 95.00th=[    979],
     | 99.00th=[1300235], 99.50th=[1518339], 99.90th=[1702888],
     | 99.95th=[1920992], 99.99th=[2734687]
   bw (  KiB/s): min=  769, max= 8864, per=6.25%, avg=4010.38, stdev=1879.01, samples=60
   iops        : min=  192, max= 2216, avg=1002.55, stdev=469.77, samples=60
  lat (usec)   : 500=12.14%, 750=68.86%, 1000=14.50%
  lat (msec)   : 2=1.47%, 4=0.01%, 750=0.98%, 1000=0.51%, 2000=1.59%
  lat (msec)   : >=2000=0.05%
  cpu          : usr=0.33%, sys=1.00%, ctx=30275, majf=0, minf=38
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=30124,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
RND4K_Read_Q32T16: (groupid=2, jobs=1): err= 0: pid=44535: Sat Sep 26 08:13:15 2026
  read: IOPS=1053, BW=4218KiB/s (4319kB/s)(129MiB/31240msec)
    slat (nsec): min=1630, max=84341, avg=3188.83, stdev=1471.43
    clat (usec): min=351, max=2752.5k, avg=30634.50, stdev=188909.02
     lat (usec): min=354, max=2752.5k, avg=30637.69, stdev=188909.03
    clat percentiles (usec):
     |  1.00th=[    429],  5.00th=[    469], 10.00th=[    490],
     | 20.00th=[    529], 30.00th=[    553], 40.00th=[    578],
     | 50.00th=[    611], 60.00th=[    644], 70.00th=[    685],
     | 80.00th=[    742], 90.00th=[    840], 95.00th=[    971],
     | 99.00th=[1249903], 99.50th=[1501561], 99.90th=[1702888],
     | 99.95th=[1971323], 99.99th=[2566915]
   bw (  KiB/s): min= 1712, max= 9400, per=6.84%, avg=4388.18, stdev=1495.09, samples=60
   iops        : min=  428, max= 2350, avg=1096.98, stdev=373.80, samples=60
  lat (usec)   : 500=12.29%, 750=69.30%, 1000=14.03%
  lat (msec)   : 2=1.61%, 4=0.02%, 750=0.93%, 1000=0.43%, 2000=1.43%
  lat (msec)   : >=2000=0.05%
  cpu          : usr=0.30%, sys=1.15%, ctx=33085, majf=0, minf=36
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=32913,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
RND4K_Read_Q32T16: (groupid=2, jobs=1): err= 0: pid=44536: Sat Sep 26 08:13:15 2026
  read: IOPS=944, BW=3783KiB/s (3874kB/s)(116MiB/31353msec)
    slat (nsec): min=1640, max=83092, avg=3307.84, stdev=1647.87
    clat (usec): min=374, max=2742.2k, avg=34125.52, stdev=199281.67
     lat (usec): min=377, max=2742.2k, avg=34128.83, stdev=199281.68
    clat percentiles (usec):
     |  1.00th=[    433],  5.00th=[    469], 10.00th=[    494],
     | 20.00th=[    529], 30.00th=[    553], 40.00th=[    578],
     | 50.00th=[    611], 60.00th=[    644], 70.00th=[    685],
     | 80.00th=[    742], 90.00th=[    848], 95.00th=[    988],
     | 99.00th=[1317012], 99.50th=[1518339], 99.90th=[1702888],
     | 99.95th=[2088764], 99.99th=[2667578]
   bw (  KiB/s): min=    8, max= 8584, per=6.05%, avg=3880.28, stdev=1292.00, samples=61
   iops        : min=    2, max= 2146, avg=969.98, stdev=322.98, samples=61
  lat (usec)   : 500=11.73%, 750=69.52%, 1000=14.06%
  lat (msec)   : 2=1.62%, 4=0.01%, 750=1.05%, 1000=0.46%, 2000=1.60%
  lat (msec)   : >=2000=0.05%
  cpu          : usr=0.32%, sys=1.04%, ctx=29726, majf=0, minf=36
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=29624,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
RND4K_Write_Q32T16: (groupid=3, jobs=1): err= 0: pid=46489: Sat Sep 26 08:13:15 2026
  write: IOPS=7275, BW=28.4MiB/s (29.8MB/s)(853MiB/30003msec); 0 zone resets
    slat (nsec): min=1640, max=794702, avg=3262.66, stdev=3329.97
    clat (usec): min=129, max=28342, avg=4394.23, stdev=5363.71
     lat (usec): min=132, max=28347, avg=4397.50, stdev=5364.02
    clat percentiles (usec):
     |  1.00th=[ 1450],  5.00th=[ 1598], 10.00th=[ 1631], 20.00th=[ 1663],
     | 30.00th=[ 1696], 40.00th=[ 1729], 50.00th=[ 1745], 60.00th=[ 1811],
     | 70.00th=[ 2835], 80.00th=[ 5997], 90.00th=[12780], 95.00th=[18482],
     | 99.00th=[23462], 99.50th=[24511], 99.90th=[25822], 99.95th=[26346],
     | 99.99th=[27395]
   bw (  KiB/s): min= 7184, max=60112, per=6.26%, avg=29115.27, stdev=18262.36, samples=60
   iops        : min= 1796, max=15028, avg=7278.75, stdev=4565.66, samples=60
  lat (usec)   : 250=0.01%, 500=0.04%, 750=0.07%, 1000=0.08%
  lat (msec)   : 2=65.69%, 4=8.33%, 10=12.61%, 20=9.61%, 50=3.58%
  cpu          : usr=2.12%, sys=6.46%, ctx=216468, majf=0, minf=209
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,218296,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
RND4K_Write_Q32T16: (groupid=3, jobs=1): err= 0: pid=46490: Sat Sep 26 08:13:15 2026
  write: IOPS=7262, BW=28.4MiB/s (29.8MB/s)(851MiB/30004msec); 0 zone resets
    slat (nsec): min=1630, max=857262, avg=3447.23, stdev=3120.89
    clat (usec): min=101, max=27810, avg=4401.83, stdev=5337.52
     lat (usec): min=117, max=27814, avg=4405.28, stdev=5337.73
    clat percentiles (usec):
     |  1.00th=[ 1450],  5.00th=[ 1598], 10.00th=[ 1631], 20.00th=[ 1663],
     | 30.00th=[ 1696], 40.00th=[ 1729], 50.00th=[ 1762], 60.00th=[ 1827],
     | 70.00th=[ 2868], 80.00th=[ 6063], 90.00th=[12780], 95.00th=[18220],
     | 99.00th=[23462], 99.50th=[24511], 99.90th=[26084], 99.95th=[26608],
     | 99.99th=[27395]
   bw (  KiB/s): min= 7280, max=60352, per=6.25%, avg=29063.97, stdev=18104.87, samples=60
   iops        : min= 1820, max=15088, avg=7265.90, stdev=4526.27, samples=60
  lat (usec)   : 250=0.02%, 500=0.07%, 750=0.08%, 1000=0.13%
  lat (msec)   : 2=64.43%, 4=9.39%, 10=12.55%, 20=9.83%, 50=3.51%
  cpu          : usr=2.19%, sys=6.41%, ctx=216249, majf=0, minf=282
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,217907,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
RND4K_Write_Q32T16: (groupid=3, jobs=1): err= 0: pid=46491: Sat Sep 26 08:13:15 2026
  write: IOPS=7270, BW=28.4MiB/s (29.8MB/s)(852MiB/30002msec); 0 zone resets
    slat (nsec): min=1690, max=4001.8k, avg=3544.62, stdev=9169.56
    clat (usec): min=117, max=28044, avg=4396.91, stdev=5378.03
     lat (usec): min=121, max=28046, avg=4400.45, stdev=5378.16
    clat percentiles (usec):
     |  1.00th=[ 1188],  5.00th=[ 1598], 10.00th=[ 1631], 20.00th=[ 1680],
     | 30.00th=[ 1696], 40.00th=[ 1729], 50.00th=[ 1762], 60.00th=[ 1827],
     | 70.00th=[ 2868], 80.00th=[ 5997], 90.00th=[12780], 95.00th=[18482],
     | 99.00th=[23725], 99.50th=[24773], 99.90th=[26346], 99.95th=[27132],
     | 99.99th=[27657]
   bw (  KiB/s): min= 7160, max=59632, per=6.26%, avg=29095.62, stdev=18270.96, samples=60
   iops        : min= 1790, max=14908, avg=7273.80, stdev=4567.76, samples=60
  lat (usec)   : 250=0.06%, 500=0.16%, 750=0.20%, 1000=0.27%
  lat (msec)   : 2=63.68%, 4=10.21%, 10=12.24%, 20=9.48%, 50=3.71%
  cpu          : usr=2.18%, sys=6.55%, ctx=214735, majf=0, minf=222
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,218144,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
RND4K_Write_Q32T16: (groupid=3, jobs=1): err= 0: pid=46492: Sat Sep 26 08:13:15 2026
  write: IOPS=7282, BW=28.5MiB/s (29.8MB/s)(854MiB/30002msec); 0 zone resets
    slat (nsec): min=1650, max=332814, avg=3344.53, stdev=2157.00
    clat (usec): min=115, max=28644, avg=4390.27, stdev=5358.15
     lat (usec): min=129, max=28646, avg=4393.62, stdev=5358.51
    clat percentiles (usec):
     |  1.00th=[ 1467],  5.00th=[ 1598], 10.00th=[ 1631], 20.00th=[ 1663],
     | 30.00th=[ 1696], 40.00th=[ 1729], 50.00th=[ 1745], 60.00th=[ 1795],
     | 70.00th=[ 2933], 80.00th=[ 5997], 90.00th=[12780], 95.00th=[18482],
     | 99.00th=[23462], 99.50th=[24773], 99.90th=[26608], 99.95th=[27132],
     | 99.99th=[28443]
   bw (  KiB/s): min= 7032, max=59408, per=6.27%, avg=29140.57, stdev=18221.79, samples=60
   iops        : min= 1758, max=14852, avg=7285.05, stdev=4555.48, samples=60
  lat (usec)   : 250=0.02%, 500=0.05%, 750=0.07%, 1000=0.12%
  lat (msec)   : 2=65.01%, 4=9.20%, 10=12.43%, 20=9.47%, 50=3.64%
  cpu          : usr=2.11%, sys=6.43%, ctx=217598, majf=0, minf=113
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,218488,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
RND4K_Write_Q32T16: (groupid=3, jobs=1): err= 0: pid=46493: Sat Sep 26 08:13:15 2026
  write: IOPS=7319, BW=28.6MiB/s (30.0MB/s)(858MiB/30003msec); 0 zone resets
    slat (nsec): min=1710, max=2133.6k, avg=3379.07, stdev=5312.18
    clat (usec): min=129, max=27997, avg=4367.99, stdev=5336.25
     lat (usec): min=132, max=28002, avg=4371.37, stdev=5336.50
    clat percentiles (usec):
     |  1.00th=[ 1467],  5.00th=[ 1598], 10.00th=[ 1631], 20.00th=[ 1663],
     | 30.00th=[ 1680], 40.00th=[ 1713], 50.00th=[ 1745], 60.00th=[ 1795],
     | 70.00th=[ 2835], 80.00th=[ 5932], 90.00th=[12649], 95.00th=[18220],
     | 99.00th=[23462], 99.50th=[24511], 99.90th=[25822], 99.95th=[27132],
     | 99.99th=[27657]
   bw (  KiB/s): min= 7256, max=59104, per=6.30%, avg=29288.38, stdev=18384.91, samples=60
   iops        : min= 1814, max=14776, avg=7322.00, stdev=4596.27, samples=60
  lat (usec)   : 250=0.01%, 500=0.02%, 750=0.04%, 1000=0.06%
  lat (msec)   : 2=66.05%, 4=8.29%, 10=12.24%, 20=9.67%, 50=3.63%
  cpu          : usr=2.05%, sys=6.52%, ctx=219148, majf=0, minf=324
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,219600,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
RND4K_Write_Q32T16: (groupid=3, jobs=1): err= 0: pid=46494: Sat Sep 26 08:13:15 2026
  write: IOPS=7234, BW=28.3MiB/s (29.6MB/s)(848MiB/30003msec); 0 zone resets
    slat (nsec): min=1630, max=1081.4k, avg=3566.74, stdev=3970.91
    clat (usec): min=124, max=27493, avg=4418.67, stdev=5364.93
     lat (usec): min=134, max=27496, avg=4422.23, stdev=5365.05
    clat percentiles (usec):
     |  1.00th=[ 1319],  5.00th=[ 1598], 10.00th=[ 1647], 20.00th=[ 1680],
     | 30.00th=[ 1713], 40.00th=[ 1729], 50.00th=[ 1762], 60.00th=[ 1844],
     | 70.00th=[ 2868], 80.00th=[ 6063], 90.00th=[12780], 95.00th=[18744],
     | 99.00th=[23462], 99.50th=[24511], 99.90th=[25822], 99.95th=[26346],
     | 99.99th=[27395]
   bw (  KiB/s): min= 7168, max=59632, per=6.23%, avg=28951.08, stdev=18114.62, samples=60
   iops        : min= 1792, max=14908, avg=7237.68, stdev=4528.70, samples=60
  lat (usec)   : 250=0.03%, 500=0.09%, 750=0.12%, 1000=0.20%
  lat (msec)   : 2=64.15%, 4=9.27%, 10=12.87%, 20=9.60%, 50=3.68%
  cpu          : usr=2.43%, sys=6.43%, ctx=213055, majf=0, minf=164
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,217067,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
RND4K_Write_Q32T16: (groupid=3, jobs=1): err= 0: pid=46495: Sat Sep 26 08:13:15 2026
  write: IOPS=7265, BW=28.4MiB/s (29.8MB/s)(852MiB/30002msec); 0 zone resets
    slat (nsec): min=1660, max=5782.5k, avg=3510.36, stdev=12683.17
    clat (usec): min=136, max=28079, avg=4400.65, stdev=5347.74
     lat (usec): min=140, max=28098, avg=4404.17, stdev=5348.28
    clat percentiles (usec):
     |  1.00th=[ 1319],  5.00th=[ 1582], 10.00th=[ 1631], 20.00th=[ 1680],
     | 30.00th=[ 1713], 40.00th=[ 1729], 50.00th=[ 1762], 60.00th=[ 1827],
     | 70.00th=[ 2868], 80.00th=[ 6063], 90.00th=[12780], 95.00th=[18482],
     | 99.00th=[23462], 99.50th=[24511], 99.90th=[26084], 99.95th=[27132],
     | 99.99th=[27919]
   bw (  KiB/s): min= 7112, max=59088, per=6.25%, avg=29071.37, stdev=18197.78, samples=60
   iops        : min= 1778, max=14772, avg=7267.73, stdev=4549.50, samples=60
  lat (usec)   : 250=0.03%, 500=0.09%, 750=0.12%, 1000=0.15%
  lat (msec)   : 2=64.43%, 4=9.49%, 10=12.48%, 20=9.62%, 50=3.60%
  cpu          : usr=2.07%, sys=6.56%, ctx=215282, majf=0, minf=154
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,217965,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
RND4K_Write_Q32T16: (groupid=3, jobs=1): err= 0: pid=46496: Sat Sep 26 08:13:15 2026
  write: IOPS=7307, BW=28.5MiB/s (29.9MB/s)(857MiB/30003msec); 0 zone resets
    slat (nsec): min=1740, max=2119.5k, avg=3355.07, stdev=8124.28
    clat (usec): min=128, max=28060, avg=4374.92, stdev=5334.06
     lat (usec): min=131, max=28064, avg=4378.28, stdev=5334.60
    clat percentiles (usec):
     |  1.00th=[ 1450],  5.00th=[ 1582], 10.00th=[ 1614], 20.00th=[ 1663],
     | 30.00th=[ 1680], 40.00th=[ 1713], 50.00th=[ 1745], 60.00th=[ 1795],
     | 70.00th=[ 2900], 80.00th=[ 5997], 90.00th=[12649], 95.00th=[18220],
     | 99.00th=[23200], 99.50th=[24773], 99.90th=[26084], 99.95th=[26870],
     | 99.99th=[27657]
   bw (  KiB/s): min= 7136, max=60240, per=6.29%, avg=29243.07, stdev=18411.30, samples=60
   iops        : min= 1784, max=15060, avg=7310.70, stdev=4602.87, samples=60
  lat (usec)   : 250=0.01%, 500=0.03%, 750=0.06%, 1000=0.08%
  lat (msec)   : 2=65.38%, 4=8.83%, 10=12.45%, 20=9.71%, 50=3.48%
  cpu          : usr=1.97%, sys=6.42%, ctx=219500, majf=0, minf=124
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,219256,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
RND4K_Write_Q32T16: (groupid=3, jobs=1): err= 0: pid=46497: Sat Sep 26 08:13:15 2026
  write: IOPS=7249, BW=28.3MiB/s (29.7MB/s)(850MiB/30003msec); 0 zone resets
    slat (nsec): min=1680, max=715250, avg=3758.92, stdev=3487.93
    clat (usec): min=100, max=28229, avg=4409.39, stdev=5372.95
     lat (usec): min=112, max=28234, avg=4413.15, stdev=5372.99
    clat percentiles (usec):
     |  1.00th=[ 1172],  5.00th=[ 1582], 10.00th=[ 1631], 20.00th=[ 1680],
     | 30.00th=[ 1713], 40.00th=[ 1729], 50.00th=[ 1762], 60.00th=[ 1844],
     | 70.00th=[ 2835], 80.00th=[ 6063], 90.00th=[12911], 95.00th=[18482],
     | 99.00th=[23462], 99.50th=[24773], 99.90th=[25822], 99.95th=[26608],
     | 99.99th=[27919]
   bw (  KiB/s): min= 7008, max=59936, per=6.24%, avg=29011.13, stdev=18169.35, samples=60
   iops        : min= 1752, max=14984, avg=7252.67, stdev=4542.37, samples=60
  lat (usec)   : 250=0.05%, 500=0.13%, 750=0.17%, 1000=0.29%
  lat (msec)   : 2=63.68%, 4=9.89%, 10=12.58%, 20=9.45%, 50=3.77%
  cpu          : usr=2.31%, sys=6.49%, ctx=215167, majf=0, minf=232
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,217515,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
RND4K_Write_Q32T16: (groupid=3, jobs=1): err= 0: pid=46498: Sat Sep 26 08:13:15 2026
  write: IOPS=7179, BW=28.0MiB/s (29.4MB/s)(841MiB/30003msec); 0 zone resets
    slat (nsec): min=1700, max=438086, avg=3934.48, stdev=3169.31
    clat (usec): min=100, max=27896, avg=4452.68, stdev=5387.26
     lat (usec): min=115, max=27900, avg=4456.61, stdev=5387.22
    clat percentiles (usec):
     |  1.00th=[ 1172],  5.00th=[ 1631], 10.00th=[ 1663], 20.00th=[ 1696],
     | 30.00th=[ 1729], 40.00th=[ 1762], 50.00th=[ 1795], 60.00th=[ 1860],
     | 70.00th=[ 2933], 80.00th=[ 6128], 90.00th=[12649], 95.00th=[18744],
     | 99.00th=[23462], 99.50th=[24511], 99.90th=[26084], 99.95th=[27132],
     | 99.99th=[27657]
   bw (  KiB/s): min= 7152, max=59096, per=6.18%, avg=28728.23, stdev=17888.52, samples=60
   iops        : min= 1788, max=14774, avg=7181.95, stdev=4472.15, samples=60
  lat (usec)   : 250=0.05%, 500=0.15%, 750=0.18%, 1000=0.29%
  lat (msec)   : 2=63.32%, 4=9.61%, 10=13.03%, 20=9.58%, 50=3.81%
  cpu          : usr=2.33%, sys=6.72%, ctx=211547, majf=0, minf=115
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,215392,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
RND4K_Write_Q32T16: (groupid=3, jobs=1): err= 0: pid=46499: Sat Sep 26 08:13:15 2026
  write: IOPS=7257, BW=28.4MiB/s (29.7MB/s)(851MiB/30003msec); 0 zone resets
    slat (nsec): min=1660, max=2420.3k, avg=3573.89, stdev=6020.43
    clat (usec): min=122, max=27816, avg=4405.04, stdev=5347.97
     lat (usec): min=125, max=27822, avg=4408.61, stdev=5348.12
    clat percentiles (usec):
     |  1.00th=[ 1450],  5.00th=[ 1614], 10.00th=[ 1647], 20.00th=[ 1680],
     | 30.00th=[ 1696], 40.00th=[ 1729], 50.00th=[ 1762], 60.00th=[ 1811],
     | 70.00th=[ 2933], 80.00th=[ 6063], 90.00th=[12780], 95.00th=[18482],
     | 99.00th=[23462], 99.50th=[24511], 99.90th=[25822], 99.95th=[26084],
     | 99.99th=[27132]
   bw (  KiB/s): min= 7200, max=59280, per=6.25%, avg=29041.08, stdev=18191.57, samples=60
   iops        : min= 1800, max=14820, avg=7260.18, stdev=4547.95, samples=60
  lat (usec)   : 250=0.02%, 500=0.05%, 750=0.07%, 1000=0.11%
  lat (msec)   : 2=65.10%, 4=8.86%, 10=12.52%, 20=9.71%, 50=3.57%
  cpu          : usr=2.19%, sys=6.50%, ctx=217792, majf=0, minf=198
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,217739,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
RND4K_Write_Q32T16: (groupid=3, jobs=1): err= 0: pid=46500: Sat Sep 26 08:13:15 2026
  write: IOPS=7267, BW=28.4MiB/s (29.8MB/s)(852MiB/30003msec); 0 zone resets
    slat (nsec): min=1660, max=441496, avg=3638.72, stdev=2838.50
    clat (usec): min=102, max=27980, avg=4398.99, stdev=5350.81
     lat (usec): min=117, max=27983, avg=4402.63, stdev=5350.99
    clat percentiles (usec):
     |  1.00th=[ 1319],  5.00th=[ 1598], 10.00th=[ 1631], 20.00th=[ 1680],
     | 30.00th=[ 1696], 40.00th=[ 1729], 50.00th=[ 1762], 60.00th=[ 1811],
     | 70.00th=[ 2900], 80.00th=[ 6063], 90.00th=[12649], 95.00th=[18482],
     | 99.00th=[23725], 99.50th=[24773], 99.90th=[26346], 99.95th=[26870],
     | 99.99th=[27657]
   bw (  KiB/s): min= 7128, max=59912, per=6.25%, avg=29079.88, stdev=18258.76, samples=60
   iops        : min= 1782, max=14978, avg=7269.88, stdev=4564.76, samples=60
  lat (usec)   : 250=0.04%, 500=0.09%, 750=0.14%, 1000=0.21%
  lat (msec)   : 2=64.59%, 4=9.04%, 10=12.69%, 20=9.73%, 50=3.49%
  cpu          : usr=2.26%, sys=6.59%, ctx=215086, majf=0, minf=119
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,218035,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
RND4K_Write_Q32T16: (groupid=3, jobs=1): err= 0: pid=46501: Sat Sep 26 08:13:15 2026
  write: IOPS=7293, BW=28.5MiB/s (29.9MB/s)(855MiB/30002msec); 0 zone resets
    slat (nsec): min=1690, max=196983, avg=3357.30, stdev=1983.02
    clat (usec): min=118, max=28069, avg=4383.41, stdev=5344.65
     lat (usec): min=127, max=28072, avg=4386.77, stdev=5344.81
    clat percentiles (usec):
     |  1.00th=[ 1287],  5.00th=[ 1582], 10.00th=[ 1631], 20.00th=[ 1663],
     | 30.00th=[ 1696], 40.00th=[ 1729], 50.00th=[ 1762], 60.00th=[ 1811],
     | 70.00th=[ 2835], 80.00th=[ 5997], 90.00th=[12649], 95.00th=[18482],
     | 99.00th=[23462], 99.50th=[24511], 99.90th=[26346], 99.95th=[26870],
     | 99.99th=[27919]
   bw (  KiB/s): min= 7160, max=61728, per=6.28%, avg=29186.53, stdev=18229.22, samples=60
   iops        : min= 1790, max=15432, avg=7296.53, stdev=4557.35, samples=60
  lat (usec)   : 250=0.03%, 500=0.06%, 750=0.10%, 1000=0.19%
  lat (msec)   : 2=64.94%, 4=9.05%, 10=12.37%, 20=9.72%, 50=3.56%
  cpu          : usr=2.19%, sys=6.49%, ctx=216213, majf=0, minf=226
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,218824,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
RND4K_Write_Q32T16: (groupid=3, jobs=1): err= 0: pid=46502: Sat Sep 26 08:13:15 2026
  write: IOPS=7256, BW=28.4MiB/s (29.7MB/s)(851MiB/30004msec); 0 zone resets
    slat (nsec): min=1690, max=409616, avg=3696.92, stdev=3008.03
    clat (usec): min=119, max=27965, avg=4405.08, stdev=5353.82
     lat (usec): min=128, max=27969, avg=4408.78, stdev=5354.17
    clat percentiles (usec):
     |  1.00th=[ 1303],  5.00th=[ 1614], 10.00th=[ 1647], 20.00th=[ 1680],
     | 30.00th=[ 1713], 40.00th=[ 1729], 50.00th=[ 1762], 60.00th=[ 1844],
     | 70.00th=[ 2868], 80.00th=[ 6063], 90.00th=[12780], 95.00th=[18482],
     | 99.00th=[23462], 99.50th=[24511], 99.90th=[26084], 99.95th=[27132],
     | 99.99th=[27657]
   bw (  KiB/s): min= 7176, max=60072, per=6.25%, avg=29040.48, stdev=18139.79, samples=60
   iops        : min= 1794, max=15018, avg=7260.02, stdev=4535.00, samples=60
  lat (usec)   : 250=0.04%, 500=0.10%, 750=0.14%, 1000=0.21%
  lat (msec)   : 2=64.34%, 4=9.46%, 10=12.44%, 20=9.75%, 50=3.53%
  cpu          : usr=2.30%, sys=6.69%, ctx=214547, majf=0, minf=289
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,217731,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
RND4K_Write_Q32T16: (groupid=3, jobs=1): err= 0: pid=46503: Sat Sep 26 08:13:15 2026
  write: IOPS=7260, BW=28.4MiB/s (29.7MB/s)(851MiB/30002msec); 0 zone resets
    slat (nsec): min=1690, max=5083.1k, avg=3665.07, stdev=12569.67
    clat (usec): min=122, max=71904, avg=4403.32, stdev=5412.55
     lat (usec): min=132, max=71919, avg=4406.98, stdev=5412.90
    clat percentiles (usec):
     |  1.00th=[ 1385],  5.00th=[ 1598], 10.00th=[ 1647], 20.00th=[ 1680],
     | 30.00th=[ 1696], 40.00th=[ 1729], 50.00th=[ 1762], 60.00th=[ 1811],
     | 70.00th=[ 2868], 80.00th=[ 5997], 90.00th=[12780], 95.00th=[18482],
     | 99.00th=[23725], 99.50th=[24511], 99.90th=[26084], 99.95th=[26608],
     | 99.99th=[69731]
   bw (  KiB/s): min= 7152, max=59320, per=6.25%, avg=29051.70, stdev=18276.20, samples=60
   iops        : min= 1788, max=14830, avg=7262.87, stdev=4569.09, samples=60
  lat (usec)   : 250=0.02%, 500=0.05%, 750=0.08%, 1000=0.13%
  lat (msec)   : 2=65.60%, 4=8.21%, 10=12.75%, 20=9.47%, 50=3.68%
  lat (msec)   : 100=0.01%
  cpu          : usr=2.06%, sys=6.78%, ctx=217557, majf=0, minf=220
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,217823,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
RND4K_Write_Q32T16: (groupid=3, jobs=1): err= 0: pid=46504: Sat Sep 26 08:13:15 2026
  write: IOPS=7231, BW=28.2MiB/s (29.6MB/s)(848MiB/30003msec); 0 zone resets
    slat (nsec): min=1680, max=787411, avg=3613.30, stdev=3884.61
    clat (usec): min=124, max=27809, avg=4420.91, stdev=5363.09
     lat (usec): min=134, max=27812, avg=4424.52, stdev=5363.39
    clat percentiles (usec):
     |  1.00th=[ 1270],  5.00th=[ 1598], 10.00th=[ 1647], 20.00th=[ 1680],
     | 30.00th=[ 1713], 40.00th=[ 1729], 50.00th=[ 1762], 60.00th=[ 1844],
     | 70.00th=[ 2900], 80.00th=[ 6128], 90.00th=[12780], 95.00th=[18482],
     | 99.00th=[23462], 99.50th=[24511], 99.90th=[25822], 99.95th=[26870],
     | 99.99th=[27657]
   bw (  KiB/s): min= 7320, max=60256, per=6.22%, avg=28936.48, stdev=18202.52, samples=60
   iops        : min= 1830, max=15064, avg=7234.02, stdev=4550.68, samples=60
  lat (usec)   : 250=0.04%, 500=0.11%, 750=0.14%, 1000=0.23%
  lat (msec)   : 2=63.89%, 4=9.86%, 10=12.52%, 20=9.61%, 50=3.62%
  cpu          : usr=2.32%, sys=6.51%, ctx=211659, majf=0, minf=204
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,216954,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: bw=1044MiB/s (1094MB/s), 1044MiB/s-1044MiB/s (1094MB/s-1094MB/s), io=30.6GiB (32.9GB), run=30022-30022msec

Run status group 1 (all jobs):
  WRITE: bw=189MiB/s (198MB/s), 189MiB/s-189MiB/s (198MB/s-198MB/s), io=5677MiB (5953MB), run=30042-30042msec

Run status group 2 (all jobs):
   READ: bw=62.7MiB/s (65.7MB/s), 3710KiB/s-4250KiB/s (3799kB/s-4352kB/s), io=1966MiB (2062MB), run=31218-31369msec

Run status group 3 (all jobs):
  WRITE: bw=454MiB/s (476MB/s), 28.0MiB/s-28.6MiB/s (29.4MB/s-30.0MB/s), io=13.3GiB (14.3GB), run=30002-30004msec

luckylinux · Sep 25, 2026

TrevorH said:
I'd run a more basic check at the drive level first and see if there is a problem underlying everything. Try dd if=/dev/nvme0n1 of=/dev/null iflag=direct bs=16M status=progress (nvme0n1 may vary) and see if it gets decent throughput.

At least after disabling powertop --auto-tune in /etc/rc.local I am now getting 3.1GB/s, which seems reasonable for an NVMe Drive:

Code:

root@pve:~# dd if=/dev/nvme0n1 of=/dev/null iflag=direct bs=16M status=progress
107927830528 bytes (108 GB, 101 GiB) copied, 34 s, 3.2 GB/s^C
6508+0 records in
6507+0 records out
109169344512 bytes (109 GB, 102 GiB) copied, 34.3601 s, 3.2 GB/s

jhartbarger · Sep 25, 2026

luckylinux said:
Root Dataset is 128k yeah. But ZFS is kinda (?) smart, that is actually a maximum Value, ZFS can automatically use smaller Values of Block Size

So how do I go from there ? This seems to be reproducible across all NVMe Enterprise Drives I won (see my previous Post, Micron 9100 MAX 1.2TB seems to have the same Issue, on another one of these Hyve EPYC 7642 Server), not only these Drives.

Ussing a 128k record size is just too small for this test, If you changed the record size to 1M your results would vastly improve for this test. You tune the record size for each dataset for the specific use case. The record size will be wildly different for a dataset used for mysql vs one for vm, or one for large video files.

The dd test confirms that your physical hardware layer is perfectly and performing well (3.2 GB/s) and by targeting the raw block device (/dev/nvme0n1) and using iflag=direct, you completely bypassed any ZFS, LUKS, and the kernel page cache thus proving that your drive and the massive speed drop you are seeing in fio is entirely a software-level limitation.

Native 4KB Firmware Configuration vs. Emulation: Enterprise Micron drives are built to handle pure native 4KB blocks. However, many enterprise NVMe controllers ship from the factory presenting a 512-byte logical sector size (512e) to the operating system for legacy server compatibility. If Proxmox auto-detected the drive as 512e, it likely built your pool with ashift=9. and this mismatch causes severe write amplification and internal garbage collection loops on the controller, completely destroying write performance under stress
PLP and Sync Overhead: Because the Micron 7300 MAX has full Power-Loss Protection (PLP), it can safely acknowledge synchronous cache flushes instantly out of its onboard controller RAM without waiting to write to physical flash. The fact that your writes are still stalling at 215 MiB/s indicates that the bottleneck is likely happening at the kernel virtualization layer or inside a LUKS dm-crypt worker thread pipeline rather than the drive itself.

luckylinux · Sep 25, 2026

jhartbarger said:
The test you are running using a 128k record size is just too small for this test, If you changed the record size to 1M your results would vastly improve for this test. You tune the record size for each dataset for the specific use case. The record size will be wildly different for a dataset used for mysql vs one for vm, or one for large video files.

Yeah, but for "standard" LXC Templates I think 128k is a good Default, or ? Or should I set that down to 16k ? Google Gemini seemed to suggest that high Latency and write Amplification are caused by this high 128k Value.

The Goal is to have something that is good for Production, but at the same Time, if "cloning" an LXC Templates on a 3000 MiB/s NVMe Drive is done at < 50 MiB/s Speeds, something feels very wrong.

I don't have huge PostgreSQL Databases in LXC Containers (none in the Templates), most LXC Containers are just a Linux Filesystem and yeah some have a PostgreSQL Databases, but nowhere near 1GB.

I also tried running a test on /boot (ext4 on top of mdadm), much better Results, so probably something between LUKS and/or ZFS is to blame:

Code:

root@pve:~# fio --time_based \
    --runtime=30 \
    --ramp_time=5 \
    --ioengine=io_uring \
    --direct=1 \
    --size=256M \
    --stonewall \
    --name=SEQ1M_Read_Q8T1 --bs=1m --rw=read --iodepth=8 --numjobs=1 --filename=/boot/test_seq_r.fio \
    --name=SEQ1M_Write_Q8T1 --bs=1m --rw=write --iodepth=8 --numjobs=1 --filename=/boot/test_seq_w.fio \
    --name=RND4K_Read_Q32T16 --bs=4k --rw=randread --iodepth=32 --numjobs=16 --filename=/boot/test_rnd_r.fio \
    --name=RND4K_Write_Q32T16 --bs=4k --rw=randwrite --iodepth=32 --numjobs=16 --filename=/boot/test_rnd_r.fio


SEQ1M_Read_Q8T1: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=io_uring, iodepth=8
SEQ1M_Write_Q8T1: (g=1): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=io_uring, iodepth=8
RND4K_Read_Q32T16: (g=2): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=io_uring, iodepth=32
...
RND4K_Write_Q32T16: (g=3): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=io_uring, iodepth=32
...
fio-3.39
Starting 34 processes
SEQ1M_Read_Q8T1: Laying out IO file (1 file / 256MiB)
SEQ1M_Write_Q8T1: Laying out IO file (1 file / 256MiB)
RND4K_Read_Q32T16: Laying out IO file (1 file / 256MiB)
Jobs: 16 (f=16): [_(18),w(16)][100.0%][w=712MiB/s][w=182k IOPS][eta 00m:00s]    
SEQ1M_Read_Q8T1: (groupid=0, jobs=1): err= 0: pid=103062: Sat Sep 26 08:37:34 2026
  read: IOPS=1285, BW=1285MiB/s (1348MB/s)(37.7GiB/30044msec)
    slat (usec): min=150, max=2217, avg=185.28, stdev=24.68
    clat (usec): min=304, max=180639, avg=6047.34, stdev=14565.23
     lat (usec): min=561, max=180806, avg=6232.62, stdev=14564.01
    clat percentiles (usec):
     |  1.00th=[   545],  5.00th=[   701], 10.00th=[   857], 20.00th=[  1090],
     | 30.00th=[  1270], 40.00th=[  1450], 50.00th=[  1647], 60.00th=[  1860],
     | 70.00th=[  2147], 80.00th=[  2540], 90.00th=[  3884], 95.00th=[ 48497],
     | 99.00th=[ 59507], 99.50th=[ 65274], 99.90th=[116917], 99.95th=[122160],
     | 99.99th=[168821]
   bw (  MiB/s): min=  573, max= 2108, per=100.00%, avg=1288.01, stdev=229.95, samples=60
   iops        : min=  573, max= 2108, avg=1287.85, stdev=229.98, samples=60
  lat (usec)   : 500=0.85%, 750=5.83%, 1000=9.09%
  lat (msec)   : 2=49.57%, 4=24.84%, 10=0.77%, 20=0.37%, 50=4.48%
  lat (msec)   : 100=4.06%, 250=0.17%
  cpu          : usr=0.22%, sys=24.78%, ctx=28140, majf=0, minf=1594
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=38612,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8
SEQ1M_Write_Q8T1: (groupid=1, jobs=1): err= 0: pid=104751: Sat Sep 26 08:37:34 2026
  write: IOPS=1807, BW=1808MiB/s (1895MB/s)(53.0GiB/30006msec); 0 zone resets
    slat (usec): min=232, max=2228, avg=298.13, stdev=31.75
    clat (usec): min=1027, max=12212, avg=4126.56, stdev=1126.18
     lat (usec): min=1365, max=12524, avg=4424.69, stdev=1123.68
    clat percentiles (usec):
     |  1.00th=[ 2606],  5.00th=[ 3064], 10.00th=[ 3261], 20.00th=[ 3425],
     | 30.00th=[ 3556], 40.00th=[ 3621], 50.00th=[ 3720], 60.00th=[ 3884],
     | 70.00th=[ 4146], 80.00th=[ 4686], 90.00th=[ 5735], 95.00th=[ 6652],
     | 99.00th=[ 8225], 99.50th=[ 8717], 99.90th=[ 9896], 99.95th=[10290],
     | 99.99th=[11338]
   bw (  MiB/s): min= 1598, max= 1890, per=100.00%, avg=1809.34, stdev=56.48, samples=60
   iops        : min= 1598, max= 1890, avg=1809.07, stdev=56.40, samples=60
  lat (msec)   : 2=0.05%, 4=65.24%, 10=34.64%, 20=0.08%
  cpu          : usr=2.36%, sys=53.21%, ctx=37247, majf=0, minf=38
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,54233,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8
RND4K_Read_Q32T16: (groupid=2, jobs=1): err= 0: pid=106320: Sat Sep 26 08:37:34 2026
  read: IOPS=35.9k, BW=140MiB/s (147MB/s)(4219MiB/30050msec)
    slat (usec): min=7, max=911, avg=11.71, stdev= 5.04
    clat (usec): min=42, max=152538, avg=877.69, stdev=3888.95
     lat (usec): min=50, max=152548, avg=889.40, stdev=3889.06
    clat percentiles (usec):
     |  1.00th=[   127],  5.00th=[   182], 10.00th=[   223], 20.00th=[   293],
     | 30.00th=[   351], 40.00th=[   408], 50.00th=[   461], 60.00th=[   515],
     | 70.00th=[   570], 80.00th=[   635], 90.00th=[   750], 95.00th=[   898],
     | 99.00th=[ 15533], 99.50th=[ 33817], 99.90th=[ 51643], 99.95th=[ 64226],
     | 99.99th=[105382]
   bw (  KiB/s): min=122797, max=162220, per=6.23%, avg=144142.43, stdev=9355.29, samples=60
   iops        : min=30699, max=40555, avg=36035.43, stdev=2338.78, samples=60
  lat (usec)   : 50=0.01%, 100=0.18%, 250=13.53%, 500=43.41%, 750=32.71%
  lat (usec)   : 1000=6.59%
  lat (msec)   : 2=1.79%, 4=0.21%, 10=0.34%, 20=0.35%, 50=0.77%
  lat (msec)   : 100=0.09%, 250=0.01%
  cpu          : usr=6.61%, sys=52.40%, ctx=378557, majf=0, minf=39
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=1080154,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
RND4K_Read_Q32T16: (groupid=2, jobs=1): err= 0: pid=106321: Sat Sep 26 08:37:34 2026
  read: IOPS=35.9k, BW=140MiB/s (147MB/s)(4219MiB/30042msec)
    slat (usec): min=7, max=534, avg=11.50, stdev= 4.54
    clat (usec): min=44, max=149730, avg=877.89, stdev=3900.54
     lat (usec): min=53, max=149740, avg=889.39, stdev=3900.66
    clat percentiles (usec):
     |  1.00th=[   126],  5.00th=[   180], 10.00th=[   223], 20.00th=[   289],
     | 30.00th=[   347], 40.00th=[   404], 50.00th=[   457], 60.00th=[   515],
     | 70.00th=[   570], 80.00th=[   635], 90.00th=[   750], 95.00th=[   889],
     | 99.00th=[ 15926], 99.50th=[ 34341], 99.90th=[ 51643], 99.95th=[ 63177],
     | 99.99th=[108528]
   bw (  KiB/s): min=123848, max=164120, per=6.22%, avg=144113.93, stdev=9212.61, samples=60
   iops        : min=30962, max=41030, avg=36028.27, stdev=2303.11, samples=60
  lat (usec)   : 50=0.01%, 100=0.19%, 250=13.80%, 500=43.70%, 750=32.36%
  lat (usec)   : 1000=6.42%
  lat (msec)   : 2=1.76%, 4=0.22%, 10=0.32%, 20=0.35%, 50=0.78%
  lat (msec)   : 100=0.09%, 250=0.01%
  cpu          : usr=5.97%, sys=52.04%, ctx=388056, majf=0, minf=136
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=1079949,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
RND4K_Read_Q32T16: (groupid=2, jobs=1): err= 0: pid=106322: Sat Sep 26 08:37:34 2026
  read: IOPS=36.0k, BW=141MiB/s (147MB/s)(4221MiB/30041msec)
    slat (usec): min=7, max=536, avg=11.35, stdev= 4.93
    clat (usec): min=44, max=161631, avg=877.44, stdev=3912.32
     lat (usec): min=57, max=161640, avg=888.79, stdev=3912.42
    clat percentiles (usec):
     |  1.00th=[   126],  5.00th=[   180], 10.00th=[   221], 20.00th=[   289],
     | 30.00th=[   347], 40.00th=[   404], 50.00th=[   457], 60.00th=[   515],
     | 70.00th=[   570], 80.00th=[   635], 90.00th=[   750], 95.00th=[   898],
     | 99.00th=[ 15401], 99.50th=[ 33817], 99.90th=[ 52167], 99.95th=[ 64226],
     | 99.99th=[108528]
   bw (  KiB/s): min=127568, max=166733, per=6.23%, avg=144200.53, stdev=7569.78, samples=60
   iops        : min=31892, max=41683, avg=36049.97, stdev=1892.50, samples=60
  lat (usec)   : 50=0.01%, 100=0.20%, 250=13.86%, 500=43.58%, 750=32.32%
  lat (usec)   : 1000=6.48%
  lat (msec)   : 2=1.76%, 4=0.22%, 10=0.34%, 20=0.37%, 50=0.76%
  lat (msec)   : 100=0.09%, 250=0.01%
  cpu          : usr=6.18%, sys=51.48%, ctx=393585, majf=0, minf=128
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=1080617,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
RND4K_Read_Q32T16: (groupid=2, jobs=1): err= 0: pid=106323: Sat Sep 26 08:37:34 2026
  read: IOPS=35.7k, BW=140MiB/s (146MB/s)(4194MiB/30050msec)
    slat (usec): min=7, max=484, avg=11.33, stdev= 4.52
    clat (usec): min=43, max=176360, avg=883.40, stdev=3947.89
     lat (usec): min=52, max=176370, avg=894.73, stdev=3948.00
    clat percentiles (usec):
     |  1.00th=[   126],  5.00th=[   180], 10.00th=[   223], 20.00th=[   289],
     | 30.00th=[   347], 40.00th=[   404], 50.00th=[   457], 60.00th=[   515],
     | 70.00th=[   570], 80.00th=[   635], 90.00th=[   750], 95.00th=[   889],
     | 99.00th=[ 16188], 99.50th=[ 34341], 99.90th=[ 52691], 99.95th=[ 64226],
     | 99.99th=[107480]
   bw (  KiB/s): min=105032, max=176769, per=6.19%, avg=143288.52, stdev=12103.49, samples=60
   iops        : min=26258, max=44192, avg=35821.92, stdev=3025.82, samples=60
  lat (usec)   : 50=0.01%, 100=0.20%, 250=13.81%, 500=43.66%, 750=32.33%
  lat (usec)   : 1000=6.46%
  lat (msec)   : 2=1.76%, 4=0.21%, 10=0.31%, 20=0.38%, 50=0.77%
  lat (msec)   : 100=0.10%, 250=0.01%
  cpu          : usr=5.85%, sys=51.34%, ctx=394325, majf=0, minf=61
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=1073756,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
RND4K_Read_Q32T16: (groupid=2, jobs=1): err= 0: pid=106324: Sat Sep 26 08:37:34 2026
  read: IOPS=35.8k, BW=140MiB/s (146MB/s)(4197MiB/30047msec)
    slat (usec): min=7, max=470, avg=11.30, stdev= 4.42
    clat (usec): min=41, max=162522, avg=882.86, stdev=3962.49
     lat (usec): min=54, max=162531, avg=894.16, stdev=3962.60
    clat percentiles (usec):
     |  1.00th=[   125],  5.00th=[   180], 10.00th=[   221], 20.00th=[   289],
     | 30.00th=[   347], 40.00th=[   404], 50.00th=[   461], 60.00th=[   515],
     | 70.00th=[   570], 80.00th=[   635], 90.00th=[   750], 95.00th=[   889],
     | 99.00th=[ 16188], 99.50th=[ 34341], 99.90th=[ 52167], 99.95th=[ 64226],
     | 99.99th=[113771]
   bw (  KiB/s): min=107432, max=165555, per=6.19%, avg=143372.13, stdev=10655.22, samples=60
   iops        : min=26858, max=41388, avg=35842.82, stdev=2663.76, samples=60
  lat (usec)   : 50=0.01%, 100=0.20%, 250=13.89%, 500=43.41%, 750=32.50%
  lat (usec)   : 1000=6.48%
  lat (msec)   : 2=1.75%, 4=0.21%, 10=0.32%, 20=0.36%, 50=0.78%
  lat (msec)   : 100=0.09%, 250=0.01%
  cpu          : usr=6.32%, sys=50.72%, ctx=395689, majf=0, minf=62
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=1074380,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
RND4K_Read_Q32T16: (groupid=2, jobs=1): err= 0: pid=106325: Sat Sep 26 08:37:34 2026
  read: IOPS=35.8k, BW=140MiB/s (147MB/s)(4205MiB/30049msec)
    slat (usec): min=7, max=482, avg=11.33, stdev= 4.83
    clat (usec): min=14, max=164676, avg=881.21, stdev=3922.13
     lat (usec): min=49, max=164686, avg=892.54, stdev=3922.24
    clat percentiles (usec):
     |  1.00th=[   126],  5.00th=[   180], 10.00th=[   221], 20.00th=[   289],
     | 30.00th=[   347], 40.00th=[   404], 50.00th=[   457], 60.00th=[   515],
     | 70.00th=[   570], 80.00th=[   635], 90.00th=[   750], 95.00th=[   898],
     | 99.00th=[ 16319], 99.50th=[ 34341], 99.90th=[ 51643], 99.95th=[ 64226],
     | 99.99th=[109577]
   bw (  KiB/s): min=112152, max=171703, per=6.20%, avg=143638.85, stdev=10199.20, samples=60
   iops        : min=28038, max=42925, avg=35909.47, stdev=2549.74, samples=60
  lat (usec)   : 20=0.01%, 50=0.01%, 100=0.20%, 250=13.86%, 500=43.58%
  lat (usec)   : 750=32.39%, 1000=6.43%
  lat (msec)   : 2=1.76%, 4=0.21%, 10=0.34%, 20=0.36%, 50=0.78%
  lat (msec)   : 100=0.09%, 250=0.01%
  cpu          : usr=6.27%, sys=51.13%, ctx=394721, majf=0, minf=37
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=1076375,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
RND4K_Read_Q32T16: (groupid=2, jobs=1): err= 0: pid=106326: Sat Sep 26 08:37:34 2026
  read: IOPS=36.3k, BW=142MiB/s (149MB/s)(4256MiB/30045msec)
    slat (usec): min=7, max=716, avg=11.73, stdev= 4.89
    clat (usec): min=44, max=151174, avg=870.00, stdev=3871.18
     lat (usec): min=53, max=151184, avg=881.73, stdev=3871.30
    clat percentiles (usec):
     |  1.00th=[   126],  5.00th=[   180], 10.00th=[   223], 20.00th=[   293],
     | 30.00th=[   351], 40.00th=[   408], 50.00th=[   461], 60.00th=[   515],
     | 70.00th=[   570], 80.00th=[   644], 90.00th=[   750], 95.00th=[   898],
     | 99.00th=[ 14877], 99.50th=[ 33817], 99.90th=[ 50594], 99.95th=[ 64226],
     | 99.99th=[109577]
   bw (  KiB/s): min=120272, max=169931, per=6.28%, avg=145374.22, stdev=9964.54, samples=60
   iops        : min=30068, max=42482, avg=36343.37, stdev=2491.08, samples=60
  lat (usec)   : 50=0.01%, 100=0.19%, 250=13.57%, 500=43.32%, 750=32.80%
  lat (usec)   : 1000=6.60%
  lat (msec)   : 2=1.78%, 4=0.22%, 10=0.33%, 20=0.35%, 50=0.75%
  lat (msec)   : 100=0.09%, 250=0.01%
  cpu          : usr=6.31%, sys=53.07%, ctx=378990, majf=0, minf=146
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=1089401,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
RND4K_Read_Q32T16: (groupid=2, jobs=1): err= 0: pid=106327: Sat Sep 26 08:37:34 2026
  read: IOPS=36.3k, BW=142MiB/s (149MB/s)(4265MiB/30035msec)
    slat (usec): min=7, max=696, avg=11.69, stdev= 5.04
    clat (usec): min=41, max=139198, avg=867.82, stdev=3830.47
     lat (usec): min=51, max=139228, avg=879.51, stdev=3830.58
    clat percentiles (usec):
     |  1.00th=[   127],  5.00th=[   182], 10.00th=[   225], 20.00th=[   293],
     | 30.00th=[   351], 40.00th=[   408], 50.00th=[   461], 60.00th=[   515],
     | 70.00th=[   570], 80.00th=[   644], 90.00th=[   750], 95.00th=[   898],
     | 99.00th=[ 14484], 99.50th=[ 33817], 99.90th=[ 50594], 99.95th=[ 63701],
     | 99.99th=[107480]
   bw (  KiB/s): min=129424, max=177515, per=6.29%, avg=145691.97, stdev=8387.33, samples=60
   iops        : min=32356, max=44378, avg=36422.80, stdev=2096.77, samples=60
  lat (usec)   : 50=0.01%, 100=0.19%, 250=13.46%, 500=43.38%, 750=32.78%
  lat (usec)   : 1000=6.59%
  lat (msec)   : 2=1.81%, 4=0.23%, 10=0.36%, 20=0.36%, 50=0.73%
  lat (msec)   : 100=0.09%, 250=0.01%
  cpu          : usr=6.39%, sys=53.16%, ctx=379073, majf=0, minf=67
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=1091763,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
RND4K_Read_Q32T16: (groupid=2, jobs=1): err= 0: pid=106328: Sat Sep 26 08:37:34 2026
  read: IOPS=36.5k, BW=143MiB/s (150MB/s)(4285MiB/30026msec)
    slat (usec): min=7, max=470, avg=11.49, stdev= 4.60
    clat (usec): min=41, max=142557, avg=863.71, stdev=3810.29
     lat (usec): min=50, max=142567, avg=875.19, stdev=3810.39
    clat percentiles (usec):
     |  1.00th=[   126],  5.00th=[   180], 10.00th=[   223], 20.00th=[   289],
     | 30.00th=[   351], 40.00th=[   404], 50.00th=[   461], 60.00th=[   515],
     | 70.00th=[   570], 80.00th=[   635], 90.00th=[   750], 95.00th=[   889],
     | 99.00th=[ 14484], 99.50th=[ 33817], 99.90th=[ 50594], 99.95th=[ 63177],
     | 99.99th=[101188]
   bw (  KiB/s): min=133434, max=168488, per=6.32%, avg=146369.17, stdev=7805.17, samples=60
   iops        : min=33358, max=42122, avg=36592.13, stdev=1951.26, samples=60
  lat (usec)   : 50=0.01%, 100=0.20%, 250=13.72%, 500=43.64%, 750=32.46%
  lat (usec)   : 1000=6.49%
  lat (msec)   : 2=1.75%, 4=0.22%, 10=0.33%, 20=0.36%, 50=0.73%
  lat (msec)   : 100=0.09%, 250=0.01%
  cpu          : usr=6.52%, sys=52.33%, ctx=387502, majf=0, minf=126
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=1096849,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
RND4K_Read_Q32T16: (groupid=2, jobs=1): err= 0: pid=106329: Sat Sep 26 08:37:34 2026
  read: IOPS=35.6k, BW=139MiB/s (146MB/s)(4181MiB/30046msec)
    slat (usec): min=7, max=496, avg=11.30, stdev= 4.86
    clat (usec): min=46, max=160160, avg=886.12, stdev=3932.00
     lat (usec): min=55, max=160169, avg=897.42, stdev=3932.11
    clat percentiles (usec):
     |  1.00th=[   126],  5.00th=[   180], 10.00th=[   223], 20.00th=[   289],
     | 30.00th=[   351], 40.00th=[   404], 50.00th=[   461], 60.00th=[   515],
     | 70.00th=[   570], 80.00th=[   635], 90.00th=[   750], 95.00th=[   898],
     | 99.00th=[ 16581], 99.50th=[ 34341], 99.90th=[ 51643], 99.95th=[ 64226],
     | 99.99th=[103285]
   bw (  KiB/s): min=125384, max=163438, per=6.17%, avg=142837.70, stdev=8786.42, samples=60
   iops        : min=31346, max=40859, avg=35709.22, stdev=2196.54, samples=60
  lat (usec)   : 50=0.01%, 100=0.21%, 250=13.79%, 500=43.64%, 750=32.40%
  lat (usec)   : 1000=6.41%
  lat (msec)   : 2=1.75%, 4=0.21%, 10=0.33%, 20=0.37%, 50=0.80%
  lat (msec)   : 100=0.09%, 250=0.01%
  cpu          : usr=6.34%, sys=50.87%, ctx=394275, majf=0, minf=38
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=1070364,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
RND4K_Read_Q32T16: (groupid=2, jobs=1): err= 0: pid=106330: Sat Sep 26 08:37:34 2026
  read: IOPS=36.3k, BW=142MiB/s (149MB/s)(4255MiB/30047msec)
    slat (usec): min=7, max=456, avg=11.61, stdev= 4.62
    clat (usec): min=41, max=173518, avg=870.22, stdev=3866.14
     lat (usec): min=50, max=173528, avg=881.83, stdev=3866.25
    clat percentiles (usec):
     |  1.00th=[   126],  5.00th=[   180], 10.00th=[   223], 20.00th=[   293],
     | 30.00th=[   351], 40.00th=[   408], 50.00th=[   461], 60.00th=[   515],
     | 70.00th=[   570], 80.00th=[   635], 90.00th=[   750], 95.00th=[   898],
     | 99.00th=[ 14746], 99.50th=[ 33817], 99.90th=[ 52167], 99.95th=[ 63177],
     | 99.99th=[109577]
   bw (  KiB/s): min=126717, max=167519, per=6.28%, avg=145372.45, stdev=8479.86, samples=60
   iops        : min=31679, max=41879, avg=36342.92, stdev=2119.95, samples=60
  lat (usec)   : 50=0.01%, 100=0.20%, 250=13.54%, 500=43.38%, 750=32.78%
  lat (usec)   : 1000=6.55%
  lat (msec)   : 2=1.79%, 4=0.22%, 10=0.35%, 20=0.36%, 50=0.74%
  lat (msec)   : 100=0.09%, 250=0.01%
  cpu          : usr=6.34%, sys=52.48%, ctx=384708, majf=0, minf=180
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=1089373,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
RND4K_Read_Q32T16: (groupid=2, jobs=1): err= 0: pid=106331: Sat Sep 26 08:37:34 2026
  read: IOPS=38.0k, BW=148MiB/s (155MB/s)(4456MiB/30049msec)
    slat (usec): min=7, max=3044, avg=12.22, stdev= 8.60
    clat (usec): min=45, max=148077, avg=829.93, stdev=3631.73
     lat (usec): min=55, max=148095, avg=842.16, stdev=3631.86
    clat percentiles (usec):
     |  1.00th=[   128],  5.00th=[   182], 10.00th=[   225], 20.00th=[   293],
     | 30.00th=[   355], 40.00th=[   412], 50.00th=[   465], 60.00th=[   519],
     | 70.00th=[   578], 80.00th=[   644], 90.00th=[   758], 95.00th=[   898],
     | 99.00th=[ 10814], 99.50th=[ 32375], 99.90th=[ 49546], 99.95th=[ 61604],
     | 99.99th=[101188]
   bw (  KiB/s): min=139520, max=175302, per=6.57%, avg=152209.10, stdev=7301.87, samples=60
   iops        : min=34880, max=43825, avg=38052.08, stdev=1825.43, samples=60
  lat (usec)   : 50=0.01%, 100=0.18%, 250=13.34%, 500=42.96%, 750=33.13%
  lat (usec)   : 1000=6.85%
  lat (msec)   : 2=1.88%, 4=0.26%, 10=0.37%, 20=0.29%, 50=0.64%
  lat (msec)   : 100=0.09%, 250=0.01%
  cpu          : usr=6.76%, sys=56.54%, ctx=357651, majf=0, minf=102
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=1140608,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
RND4K_Read_Q32T16: (groupid=2, jobs=1): err= 0: pid=106332: Sat Sep 26 08:37:34 2026
  read: IOPS=36.2k, BW=141MiB/s (148MB/s)(4250MiB/30046msec)
    slat (usec): min=7, max=483, avg=11.48, stdev= 4.50
    clat (usec): min=42, max=159060, avg=871.59, stdev=3890.62
     lat (usec): min=50, max=159068, avg=883.07, stdev=3890.74
    clat percentiles (usec):
     |  1.00th=[   126],  5.00th=[   180], 10.00th=[   223], 20.00th=[   289],
     | 30.00th=[   351], 40.00th=[   408], 50.00th=[   461], 60.00th=[   515],
     | 70.00th=[   570], 80.00th=[   635], 90.00th=[   750], 95.00th=[   889],
     | 99.00th=[ 15139], 99.50th=[ 33817], 99.90th=[ 50594], 99.95th=[ 64226],
     | 99.99th=[110625]
   bw (  KiB/s): min=112192, max=173410, per=6.27%, avg=145186.43, stdev=9121.57, samples=60
   iops        : min=28048, max=43352, avg=36296.38, stdev=2280.35, samples=60
  lat (usec)   : 50=0.01%, 100=0.19%, 250=13.68%, 500=43.55%, 750=32.64%
  lat (usec)   : 1000=6.43%
  lat (msec)   : 2=1.76%, 4=0.22%, 10=0.33%, 20=0.33%, 50=0.77%
  lat (msec)   : 100=0.09%, 250=0.01%
  cpu          : usr=6.30%, sys=52.10%, ctx=388939, majf=0, minf=112
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=1087878,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
RND4K_Read_Q32T16: (groupid=2, jobs=1): err= 0: pid=106333: Sat Sep 26 08:37:34 2026
  read: IOPS=35.8k, BW=140MiB/s (147MB/s)(4208MiB/30044msec)
    slat (usec): min=7, max=381, avg=11.32, stdev= 4.46
    clat (usec): min=35, max=143938, avg=880.55, stdev=3902.51
     lat (usec): min=49, max=143947, avg=891.87, stdev=3902.62
    clat percentiles (usec):
     |  1.00th=[   126],  5.00th=[   180], 10.00th=[   223], 20.00th=[   289],
     | 30.00th=[   351], 40.00th=[   408], 50.00th=[   461], 60.00th=[   515],
     | 70.00th=[   570], 80.00th=[   635], 90.00th=[   750], 95.00th=[   898],
     | 99.00th=[ 15926], 99.50th=[ 33817], 99.90th=[ 51119], 99.95th=[ 64226],
     | 99.99th=[104334]
   bw (  KiB/s): min=118364, max=170076, per=6.21%, avg=143758.37, stdev=10250.30, samples=60
   iops        : min=29591, max=42519, avg=35939.38, stdev=2562.59, samples=60
  lat (usec)   : 50=0.01%, 100=0.20%, 250=13.66%, 500=43.58%, 750=32.55%
  lat (usec)   : 1000=6.45%
  lat (msec)   : 2=1.76%, 4=0.21%, 10=0.33%, 20=0.37%, 50=0.77%
  lat (msec)   : 100=0.09%, 250=0.01%
  cpu          : usr=6.01%, sys=51.29%, ctx=395560, majf=0, minf=49
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=1077107,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
RND4K_Read_Q32T16: (groupid=2, jobs=1): err= 0: pid=106334: Sat Sep 26 08:37:34 2026
  read: IOPS=36.3k, BW=142MiB/s (149MB/s)(4265MiB/30047msec)
    slat (usec): min=7, max=3208, avg=11.63, stdev= 5.84
    clat (usec): min=37, max=167318, avg=868.30, stdev=3848.70
     lat (usec): min=52, max=167330, avg=879.93, stdev=3848.81
    clat percentiles (usec):
     |  1.00th=[   126],  5.00th=[   180], 10.00th=[   225], 20.00th=[   293],
     | 30.00th=[   351], 40.00th=[   408], 50.00th=[   461], 60.00th=[   519],
     | 70.00th=[   570], 80.00th=[   644], 90.00th=[   750], 95.00th=[   898],
     | 99.00th=[ 14353], 99.50th=[ 33817], 99.90th=[ 51643], 99.95th=[ 64226],
     | 99.99th=[103285]
   bw (  KiB/s): min=127272, max=172833, per=6.29%, avg=145680.57, stdev=8648.01, samples=60
   iops        : min=31818, max=43208, avg=36419.90, stdev=2162.00, samples=60
  lat (usec)   : 50=0.01%, 100=0.20%, 250=13.44%, 500=43.32%, 750=32.89%
  lat (usec)   : 1000=6.60%
  lat (msec)   : 2=1.80%, 4=0.22%, 10=0.35%, 20=0.35%, 50=0.73%
  lat (msec)   : 100=0.09%, 250=0.01%
  cpu          : usr=6.30%, sys=52.91%, ctx=382517, majf=0, minf=69
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=1091692,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
RND4K_Read_Q32T16: (groupid=2, jobs=1): err= 0: pid=106335: Sat Sep 26 08:37:34 2026
  read: IOPS=36.4k, BW=142MiB/s (149MB/s)(4269MiB/30035msec)
    slat (usec): min=7, max=555, avg=11.45, stdev= 4.56
    clat (usec): min=42, max=156502, avg=867.29, stdev=3859.35
     lat (usec): min=51, max=156511, avg=878.74, stdev=3859.46
    clat percentiles (usec):
     |  1.00th=[   126],  5.00th=[   180], 10.00th=[   221], 20.00th=[   289],
     | 30.00th=[   351], 40.00th=[   404], 50.00th=[   461], 60.00th=[   515],
     | 70.00th=[   570], 80.00th=[   635], 90.00th=[   750], 95.00th=[   889],
     | 99.00th=[ 15008], 99.50th=[ 33817], 99.90th=[ 51119], 99.95th=[ 63701],
     | 99.99th=[104334]
   bw (  KiB/s): min=129827, max=173995, per=6.30%, avg=145878.15, stdev=9766.83, samples=60
   iops        : min=32456, max=43498, avg=36469.33, stdev=2441.71, samples=60
  lat (usec)   : 50=0.01%, 100=0.19%, 250=13.89%, 500=43.46%, 750=32.47%
  lat (usec)   : 1000=6.50%
  lat (msec)   : 2=1.75%, 4=0.22%, 10=0.32%, 20=0.35%, 50=0.75%
  lat (msec)   : 100=0.09%, 250=0.01%
  cpu          : usr=5.97%, sys=52.52%, ctx=389242, majf=0, minf=141
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=1092918,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
RND4K_Write_Q32T16: (groupid=3, jobs=1): err= 0: pid=107798: Sat Sep 26 08:37:34 2026
  write: IOPS=11.6k, BW=45.2MiB/s (47.4MB/s)(1355MiB/30004msec); 0 zone resets
    slat (usec): min=10, max=384, avg=16.57, stdev= 5.31
    clat (usec): min=151, max=42622, avg=2749.48, stdev=1067.87
     lat (usec): min=176, max=42635, avg=2766.05, stdev=1067.82
    clat percentiles (usec):
     |  1.00th=[ 1516],  5.00th=[ 1762], 10.00th=[ 1926], 20.00th=[ 2114],
     | 30.00th=[ 2311], 40.00th=[ 2507], 50.00th=[ 2704], 60.00th=[ 2933],
     | 70.00th=[ 3130], 80.00th=[ 3326], 90.00th=[ 3556], 95.00th=[ 3752],
     | 99.00th=[ 4080], 99.50th=[ 4228], 99.90th=[ 5604], 99.95th=[31851],
     | 99.99th=[36439]
   bw (  KiB/s): min=41872, max=48712, per=6.25%, avg=46303.93, stdev=1406.07, samples=60
   iops        : min=10468, max=12178, avg=11575.80, stdev=351.58, samples=60
  lat (usec)   : 250=0.01%, 500=0.01%, 750=0.03%, 1000=0.06%
  lat (msec)   : 2=13.46%, 4=84.99%, 10=1.36%, 20=0.02%, 50=0.07%
  cpu          : usr=3.18%, sys=26.80%, ctx=246247, majf=0, minf=39
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,346934,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
RND4K_Write_Q32T16: (groupid=3, jobs=1): err= 0: pid=107799: Sat Sep 26 08:37:34 2026
  write: IOPS=11.6k, BW=45.3MiB/s (47.5MB/s)(1359MiB/30002msec); 0 zone resets
    slat (usec): min=10, max=435, avg=14.57, stdev= 4.70
    clat (usec): min=193, max=42605, avg=2744.65, stdev=1072.11
     lat (usec): min=223, max=42617, avg=2759.22, stdev=1072.05
    clat percentiles (usec):
     |  1.00th=[ 1483],  5.00th=[ 1762], 10.00th=[ 1909], 20.00th=[ 2114],
     | 30.00th=[ 2311], 40.00th=[ 2507], 50.00th=[ 2704], 60.00th=[ 2933],
     | 70.00th=[ 3130], 80.00th=[ 3326], 90.00th=[ 3556], 95.00th=[ 3720],
     | 99.00th=[ 4047], 99.50th=[ 4178], 99.90th=[ 5407], 99.95th=[31851],
     | 99.99th=[36439]
   bw (  KiB/s): min=41979, max=48736, per=6.27%, avg=46424.27, stdev=1350.30, samples=60
   iops        : min=10494, max=12184, avg=11605.90, stdev=337.64, samples=60
  lat (usec)   : 250=0.01%, 500=0.02%, 750=0.05%, 1000=0.09%
  lat (msec)   : 2=13.65%, 4=84.84%, 10=1.26%, 20=0.02%, 50=0.07%
  cpu          : usr=2.51%, sys=23.92%, ctx=260452, majf=0, minf=45
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,347825,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
RND4K_Write_Q32T16: (groupid=3, jobs=1): err= 0: pid=107800: Sat Sep 26 08:37:34 2026
  write: IOPS=11.6k, BW=45.2MiB/s (47.4MB/s)(1357MiB/30004msec); 0 zone resets
    slat (usec): min=10, max=193, avg=16.02, stdev= 4.74
    clat (usec): min=143, max=42626, avg=2745.76, stdev=1072.40
     lat (usec): min=170, max=42642, avg=2761.77, stdev=1072.33
    clat percentiles (usec):
     |  1.00th=[ 1500],  5.00th=[ 1762], 10.00th=[ 1909], 20.00th=[ 2114],
     | 30.00th=[ 2311], 40.00th=[ 2507], 50.00th=[ 2704], 60.00th=[ 2933],
     | 70.00th=[ 3097], 80.00th=[ 3294], 90.00th=[ 3556], 95.00th=[ 3720],
     | 99.00th=[ 4080], 99.50th=[ 4228], 99.90th=[ 5735], 99.95th=[32113],
     | 99.99th=[36439]
   bw (  KiB/s): min=41912, max=48632, per=6.26%, avg=46377.43, stdev=1401.72, samples=60
   iops        : min=10478, max=12158, avg=11594.20, stdev=350.48, samples=60
  lat (usec)   : 250=0.01%, 500=0.02%, 750=0.04%, 1000=0.08%
  lat (msec)   : 2=13.56%, 4=84.84%, 10=1.38%, 20=0.02%, 50=0.07%
  cpu          : usr=2.89%, sys=26.05%, ctx=250416, majf=0, minf=38
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,347485,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
RND4K_Write_Q32T16: (groupid=3, jobs=1): err= 0: pid=107801: Sat Sep 26 08:37:34 2026
  write: IOPS=11.6k, BW=45.2MiB/s (47.4MB/s)(1356MiB/30004msec); 0 zone resets
    slat (usec): min=10, max=1184, avg=16.65, stdev= 5.92
    clat (usec): min=216, max=42486, avg=2747.83, stdev=1069.78
     lat (usec): min=239, max=42500, avg=2764.48, stdev=1069.69
    clat percentiles (usec):
     |  1.00th=[ 1500],  5.00th=[ 1762], 10.00th=[ 1909], 20.00th=[ 2114],
     | 30.00th=[ 2311], 40.00th=[ 2507], 50.00th=[ 2704], 60.00th=[ 2933],
     | 70.00th=[ 3130], 80.00th=[ 3326], 90.00th=[ 3556], 95.00th=[ 3720],
     | 99.00th=[ 4080], 99.50th=[ 4228], 99.90th=[ 5473], 99.95th=[32113],
     | 99.99th=[36439]
   bw (  KiB/s): min=41776, max=49000, per=6.25%, avg=46328.95, stdev=1441.67, samples=60
   iops        : min=10444, max=12250, avg=11582.03, stdev=360.48, samples=60
  lat (usec)   : 250=0.01%, 500=0.02%, 750=0.06%, 1000=0.08%
  lat (msec)   : 2=13.47%, 4=84.95%, 10=1.35%, 20=0.02%, 50=0.07%
  cpu          : usr=3.22%, sys=26.88%, ctx=245328, majf=0, minf=38
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,347133,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
RND4K_Write_Q32T16: (groupid=3, jobs=1): err= 0: pid=107802: Sat Sep 26 08:37:34 2026
  write: IOPS=11.6k, BW=45.2MiB/s (47.4MB/s)(1356MiB/30003msec); 0 zone resets
    slat (usec): min=10, max=401, avg=16.74, stdev= 4.94
    clat (usec): min=123, max=42549, avg=2748.21, stdev=1070.75
     lat (usec): min=139, max=42565, avg=2764.95, stdev=1070.69
    clat percentiles (usec):
     |  1.00th=[ 1500],  5.00th=[ 1762], 10.00th=[ 1909], 20.00th=[ 2114],
     | 30.00th=[ 2311], 40.00th=[ 2507], 50.00th=[ 2704], 60.00th=[ 2933],
     | 70.00th=[ 3130], 80.00th=[ 3294], 90.00th=[ 3556], 95.00th=[ 3752],
     | 99.00th=[ 4080], 99.50th=[ 4228], 99.90th=[ 5342], 99.95th=[31851],
     | 99.99th=[36439]
   bw (  KiB/s): min=41747, max=48888, per=6.25%, avg=46321.15, stdev=1416.31, samples=60
   iops        : min=10436, max=12222, avg=11580.10, stdev=354.11, samples=60
  lat (usec)   : 250=0.01%, 500=0.01%, 750=0.04%, 1000=0.06%
  lat (msec)   : 2=13.60%, 4=84.83%, 10=1.37%, 20=0.02%, 50=0.07%
  cpu          : usr=3.23%, sys=27.01%, ctx=245013, majf=0, minf=39
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,347062,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
RND4K_Write_Q32T16: (groupid=3, jobs=1): err= 0: pid=107803: Sat Sep 26 08:37:34 2026
  write: IOPS=11.6k, BW=45.2MiB/s (47.4MB/s)(1357MiB/30003msec); 0 zone resets
    slat (usec): min=10, max=514, avg=16.44, stdev= 5.28
    clat (usec): min=214, max=42498, avg=2746.76, stdev=1070.55
     lat (usec): min=226, max=42510, avg=2763.21, stdev=1070.46
    clat percentiles (usec):
     |  1.00th=[ 1500],  5.00th=[ 1762], 10.00th=[ 1909], 20.00th=[ 2114],
     | 30.00th=[ 2311], 40.00th=[ 2507], 50.00th=[ 2704], 60.00th=[ 2933],
     | 70.00th=[ 3130], 80.00th=[ 3326], 90.00th=[ 3556], 95.00th=[ 3720],
     | 99.00th=[ 4080], 99.50th=[ 4228], 99.90th=[ 5407], 99.95th=[31851],
     | 99.99th=[36439]
   bw (  KiB/s): min=41720, max=48849, per=6.26%, avg=46351.72, stdev=1443.27, samples=60
   iops        : min=10430, max=12212, avg=11587.75, stdev=360.87, samples=60
  lat (usec)   : 250=0.01%, 500=0.02%, 750=0.05%, 1000=0.09%
  lat (msec)   : 2=13.69%, 4=84.79%, 10=1.28%, 20=0.02%, 50=0.07%
  cpu          : usr=3.28%, sys=26.49%, ctx=247414, majf=0, minf=51
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,347292,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
RND4K_Write_Q32T16: (groupid=3, jobs=1): err= 0: pid=107804: Sat Sep 26 08:37:34 2026
  write: IOPS=11.6k, BW=45.3MiB/s (47.5MB/s)(1359MiB/30004msec); 0 zone resets
    slat (usec): min=10, max=370, avg=16.14, stdev= 4.88
    clat (usec): min=160, max=42443, avg=2741.95, stdev=1071.22
     lat (usec): min=176, max=42454, avg=2758.09, stdev=1071.16
    clat percentiles (usec):
     |  1.00th=[ 1467],  5.00th=[ 1762], 10.00th=[ 1909], 20.00th=[ 2114],
     | 30.00th=[ 2311], 40.00th=[ 2507], 50.00th=[ 2704], 60.00th=[ 2933],
     | 70.00th=[ 3097], 80.00th=[ 3294], 90.00th=[ 3556], 95.00th=[ 3720],
     | 99.00th=[ 4080], 99.50th=[ 4228], 99.90th=[ 5473], 99.95th=[31851],
     | 99.99th=[36439]
   bw (  KiB/s): min=41835, max=49160, per=6.27%, avg=46438.78, stdev=1418.89, samples=60
   iops        : min=10458, max=12290, avg=11609.50, stdev=354.77, samples=60
  lat (usec)   : 250=0.01%, 500=0.03%, 750=0.06%, 1000=0.10%
  lat (msec)   : 2=13.75%, 4=84.61%, 10=1.36%, 20=0.02%, 50=0.07%
  cpu          : usr=2.97%, sys=26.25%, ctx=249918, majf=0, minf=47
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,347945,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
RND4K_Write_Q32T16: (groupid=3, jobs=1): err= 0: pid=107805: Sat Sep 26 08:37:34 2026
  write: IOPS=11.6k, BW=45.3MiB/s (47.5MB/s)(1358MiB/30003msec); 0 zone resets
    slat (usec): min=10, max=535, avg=16.08, stdev= 5.00
    clat (usec): min=194, max=42558, avg=2745.10, stdev=1070.42
     lat (usec): min=215, max=42573, avg=2761.18, stdev=1070.33
    clat percentiles (usec):
     |  1.00th=[ 1483],  5.00th=[ 1762], 10.00th=[ 1909], 20.00th=[ 2114],
     | 30.00th=[ 2311], 40.00th=[ 2507], 50.00th=[ 2704], 60.00th=[ 2933],
     | 70.00th=[ 3130], 80.00th=[ 3326], 90.00th=[ 3556], 95.00th=[ 3720],
     | 99.00th=[ 4080], 99.50th=[ 4228], 99.90th=[ 5407], 99.95th=[31851],
     | 99.99th=[36439]
   bw (  KiB/s): min=41851, max=48696, per=6.26%, avg=46386.68, stdev=1413.31, samples=60
   iops        : min=10462, max=12174, avg=11596.52, stdev=353.37, samples=60
  lat (usec)   : 250=0.01%, 500=0.02%, 750=0.05%, 1000=0.10%
  lat (msec)   : 2=13.59%, 4=84.82%, 10=1.33%, 20=0.02%, 50=0.07%
  cpu          : usr=2.95%, sys=26.14%, ctx=250005, majf=0, minf=38
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,347554,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
RND4K_Write_Q32T16: (groupid=3, jobs=1): err= 0: pid=107806: Sat Sep 26 08:37:34 2026
  write: IOPS=11.6k, BW=45.2MiB/s (47.4MB/s)(1357MiB/30003msec); 0 zone resets
    slat (usec): min=10, max=418, avg=14.41, stdev= 3.88
    clat (usec): min=208, max=42706, avg=2749.38, stdev=1069.12
     lat (usec): min=231, max=42718, avg=2763.79, stdev=1069.07
    clat percentiles (usec):
     |  1.00th=[ 1516],  5.00th=[ 1778], 10.00th=[ 1926], 20.00th=[ 2147],
     | 30.00th=[ 2311], 40.00th=[ 2507], 50.00th=[ 2704], 60.00th=[ 2933],
     | 70.00th=[ 3130], 80.00th=[ 3326], 90.00th=[ 3556], 95.00th=[ 3720],
     | 99.00th=[ 4080], 99.50th=[ 4228], 99.90th=[ 5342], 99.95th=[31851],
     | 99.99th=[36439]
   bw (  KiB/s): min=41816, max=48720, per=6.26%, avg=46347.67, stdev=1378.47, samples=60
   iops        : min=10454, max=12180, avg=11586.72, stdev=344.64, samples=60
  lat (usec)   : 250=0.01%, 500=0.02%, 750=0.05%, 1000=0.06%
  lat (msec)   : 2=13.37%, 4=85.09%, 10=1.34%, 20=0.02%, 50=0.07%
  cpu          : usr=2.20%, sys=23.90%, ctx=261562, majf=0, minf=39
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,347262,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
RND4K_Write_Q32T16: (groupid=3, jobs=1): err= 0: pid=107807: Sat Sep 26 08:37:34 2026
  write: IOPS=11.6k, BW=45.2MiB/s (47.4MB/s)(1357MiB/30004msec); 0 zone resets
    slat (usec): min=10, max=534, avg=16.47, stdev= 5.73
    clat (usec): min=165, max=42576, avg=2746.42, stdev=1069.99
     lat (usec): min=196, max=42591, avg=2762.89, stdev=1069.87
    clat percentiles (usec):
     |  1.00th=[ 1500],  5.00th=[ 1762], 10.00th=[ 1909], 20.00th=[ 2114],
     | 30.00th=[ 2311], 40.00th=[ 2507], 50.00th=[ 2704], 60.00th=[ 2933],
     | 70.00th=[ 3097], 80.00th=[ 3326], 90.00th=[ 3556], 95.00th=[ 3720],
     | 99.00th=[ 4080], 99.50th=[ 4228], 99.90th=[ 5276], 99.95th=[31851],
     | 99.99th=[36439]
   bw (  KiB/s): min=41896, max=48768, per=6.26%, avg=46357.30, stdev=1402.59, samples=60
   iops        : min=10474, max=12192, avg=11589.13, stdev=350.69, samples=60
  lat (usec)   : 250=0.01%, 500=0.02%, 750=0.05%, 1000=0.09%
  lat (msec)   : 2=13.46%, 4=84.94%, 10=1.35%, 20=0.02%, 50=0.07%
  cpu          : usr=3.12%, sys=26.70%, ctx=246526, majf=0, minf=60
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,347334,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
RND4K_Write_Q32T16: (groupid=3, jobs=1): err= 0: pid=107808: Sat Sep 26 08:37:34 2026
  write: IOPS=11.6k, BW=45.3MiB/s (47.5MB/s)(1358MiB/30004msec); 0 zone resets
    slat (usec): min=10, max=947, avg=16.02, stdev= 5.13
    clat (usec): min=202, max=42762, avg=2744.26, stdev=1070.17
     lat (usec): min=220, max=42777, avg=2760.27, stdev=1070.09
    clat percentiles (usec):
     |  1.00th=[ 1483],  5.00th=[ 1762], 10.00th=[ 1909], 20.00th=[ 2114],
     | 30.00th=[ 2311], 40.00th=[ 2507], 50.00th=[ 2704], 60.00th=[ 2933],
     | 70.00th=[ 3097], 80.00th=[ 3294], 90.00th=[ 3556], 95.00th=[ 3720],
     | 99.00th=[ 4080], 99.50th=[ 4228], 99.90th=[ 5669], 99.95th=[31851],
     | 99.99th=[36439]
   bw (  KiB/s): min=41800, max=48856, per=6.26%, avg=46402.45, stdev=1426.07, samples=60
   iops        : min=10450, max=12214, avg=11600.45, stdev=356.54, samples=60
  lat (usec)   : 250=0.01%, 500=0.03%, 750=0.06%, 1000=0.10%
  lat (msec)   : 2=13.60%, 4=84.81%, 10=1.32%, 20=0.02%, 50=0.07%
  cpu          : usr=2.94%, sys=26.05%, ctx=250590, majf=0, minf=38
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,347671,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
RND4K_Write_Q32T16: (groupid=3, jobs=1): err= 0: pid=107809: Sat Sep 26 08:37:34 2026
  write: IOPS=11.6k, BW=45.2MiB/s (47.4MB/s)(1355MiB/30003msec); 0 zone resets
    slat (usec): min=10, max=422, avg=16.53, stdev= 5.01
    clat (usec): min=192, max=42580, avg=2749.28, stdev=1070.59
     lat (usec): min=211, max=42597, avg=2765.81, stdev=1070.53
    clat percentiles (usec):
     |  1.00th=[ 1516],  5.00th=[ 1762], 10.00th=[ 1926], 20.00th=[ 2147],
     | 30.00th=[ 2311], 40.00th=[ 2507], 50.00th=[ 2704], 60.00th=[ 2933],
     | 70.00th=[ 3130], 80.00th=[ 3326], 90.00th=[ 3556], 95.00th=[ 3720],
     | 99.00th=[ 4080], 99.50th=[ 4228], 99.90th=[ 5800], 99.95th=[31851],
     | 99.99th=[36439]
   bw (  KiB/s): min=41851, max=48624, per=6.25%, avg=46307.82, stdev=1410.50, samples=60
   iops        : min=10462, max=12156, avg=11576.77, stdev=352.64, samples=60
  lat (usec)   : 250=0.01%, 500=0.01%, 750=0.04%, 1000=0.06%
  lat (msec)   : 2=13.34%, 4=85.11%, 10=1.35%, 20=0.02%, 50=0.07%
  cpu          : usr=3.41%, sys=26.45%, ctx=246673, majf=0, minf=38
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,346963,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
RND4K_Write_Q32T16: (groupid=3, jobs=1): err= 0: pid=107810: Sat Sep 26 08:37:34 2026
  write: IOPS=11.6k, BW=45.2MiB/s (47.4MB/s)(1355MiB/30004msec); 0 zone resets
    slat (usec): min=10, max=263, avg=16.51, stdev= 4.89
    clat (usec): min=200, max=42753, avg=2749.30, stdev=1068.77
     lat (usec): min=222, max=42765, avg=2765.80, stdev=1068.67
    clat percentiles (usec):
     |  1.00th=[ 1516],  5.00th=[ 1762], 10.00th=[ 1926], 20.00th=[ 2114],
     | 30.00th=[ 2311], 40.00th=[ 2507], 50.00th=[ 2704], 60.00th=[ 2933],
     | 70.00th=[ 3130], 80.00th=[ 3326], 90.00th=[ 3556], 95.00th=[ 3720],
     | 99.00th=[ 4047], 99.50th=[ 4228], 99.90th=[ 5669], 99.95th=[31851],
     | 99.99th=[36439]
   bw (  KiB/s): min=42004, max=48680, per=6.25%, avg=46308.03, stdev=1384.81, samples=60
   iops        : min=10501, max=12170, avg=11576.80, stdev=346.21, samples=60
  lat (usec)   : 250=0.01%, 500=0.02%, 750=0.04%, 1000=0.07%
  lat (msec)   : 2=13.40%, 4=85.10%, 10=1.30%, 20=0.02%, 50=0.07%
  cpu          : usr=3.13%, sys=26.69%, ctx=246725, majf=0, minf=39
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,346965,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
RND4K_Write_Q32T16: (groupid=3, jobs=1): err= 0: pid=107811: Sat Sep 26 08:37:34 2026
  write: IOPS=11.6k, BW=45.2MiB/s (47.4MB/s)(1355MiB/30004msec); 0 zone resets
    slat (usec): min=10, max=761, avg=16.71, stdev= 5.17
    clat (usec): min=190, max=42463, avg=2749.38, stdev=1067.51
     lat (usec): min=204, max=42477, avg=2766.08, stdev=1067.41
    clat percentiles (usec):
     |  1.00th=[ 1516],  5.00th=[ 1762], 10.00th=[ 1926], 20.00th=[ 2147],
     | 30.00th=[ 2311], 40.00th=[ 2507], 50.00th=[ 2704], 60.00th=[ 2933],
     | 70.00th=[ 3130], 80.00th=[ 3326], 90.00th=[ 3556], 95.00th=[ 3720],
     | 99.00th=[ 4080], 99.50th=[ 4228], 99.90th=[ 5604], 99.95th=[31851],
     | 99.99th=[36439]
   bw (  KiB/s): min=41795, max=48640, per=6.25%, avg=46302.82, stdev=1381.67, samples=60
   iops        : min=10448, max=12160, avg=11575.50, stdev=345.47, samples=60
  lat (usec)   : 250=0.01%, 500=0.01%, 750=0.04%, 1000=0.05%
  lat (msec)   : 2=13.38%, 4=85.08%, 10=1.35%, 20=0.02%, 50=0.07%
  cpu          : usr=3.10%, sys=27.05%, ctx=245469, majf=0, minf=38
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,346925,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
RND4K_Write_Q32T16: (groupid=3, jobs=1): err= 0: pid=107812: Sat Sep 26 08:37:34 2026
  write: IOPS=11.6k, BW=45.2MiB/s (47.4MB/s)(1357MiB/30004msec); 0 zone resets
    slat (usec): min=10, max=354, avg=16.50, stdev= 5.20
    clat (usec): min=175, max=42588, avg=2747.11, stdev=1069.50
     lat (usec): min=198, max=42603, avg=2763.61, stdev=1069.45
    clat percentiles (usec):
     |  1.00th=[ 1500],  5.00th=[ 1762], 10.00th=[ 1909], 20.00th=[ 2114],
     | 30.00th=[ 2311], 40.00th=[ 2507], 50.00th=[ 2704], 60.00th=[ 2933],
     | 70.00th=[ 3130], 80.00th=[ 3294], 90.00th=[ 3556], 95.00th=[ 3720],
     | 99.00th=[ 4080], 99.50th=[ 4228], 99.90th=[ 5473], 99.95th=[31851],
     | 99.99th=[36439]
   bw (  KiB/s): min=41928, max=48704, per=6.26%, avg=46344.78, stdev=1407.10, samples=60
   iops        : min=10482, max=12176, avg=11586.00, stdev=351.81, samples=60
  lat (usec)   : 250=0.01%, 500=0.02%, 750=0.05%, 1000=0.08%
  lat (msec)   : 2=13.50%, 4=84.94%, 10=1.32%, 20=0.02%, 50=0.07%
  cpu          : usr=3.27%, sys=26.59%, ctx=246409, majf=0, minf=59
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,347240,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
RND4K_Write_Q32T16: (groupid=3, jobs=1): err= 0: pid=107813: Sat Sep 26 08:37:34 2026
  write: IOPS=11.6k, BW=45.2MiB/s (47.4MB/s)(1356MiB/30003msec); 0 zone resets
    slat (usec): min=10, max=548, avg=16.73, stdev= 5.04
    clat (usec): min=205, max=42701, avg=2747.68, stdev=1069.86
     lat (usec): min=218, max=42716, avg=2764.41, stdev=1069.74
    clat percentiles (usec):
     |  1.00th=[ 1516],  5.00th=[ 1762], 10.00th=[ 1926], 20.00th=[ 2114],
     | 30.00th=[ 2311], 40.00th=[ 2507], 50.00th=[ 2704], 60.00th=[ 2933],
     | 70.00th=[ 3130], 80.00th=[ 3326], 90.00th=[ 3556], 95.00th=[ 3720],
     | 99.00th=[ 4080], 99.50th=[ 4228], 99.90th=[ 5342], 99.95th=[31851],
     | 99.99th=[36439]
   bw (  KiB/s): min=41699, max=48728, per=6.25%, avg=46330.78, stdev=1366.29, samples=60
   iops        : min=10424, max=12182, avg=11582.48, stdev=341.63, samples=60
  lat (usec)   : 250=0.01%, 500=0.01%, 750=0.04%, 1000=0.07%
  lat (msec)   : 2=13.48%, 4=85.01%, 10=1.30%, 20=0.02%, 50=0.07%
  cpu          : usr=3.33%, sys=26.89%, ctx=244971, majf=0, minf=38
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,347135,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: bw=1285MiB/s (1348MB/s), 1285MiB/s-1285MiB/s (1348MB/s-1348MB/s), io=37.7GiB (40.5GB), run=30044-30044msec

Run status group 1 (all jobs):
  WRITE: bw=1808MiB/s (1895MB/s), 1808MiB/s-1808MiB/s (1895MB/s-1895MB/s), io=53.0GiB (56.9GB), run=30006-30006msec

Run status group 2 (all jobs):
   READ: bw=2261MiB/s (2371MB/s), 139MiB/s-148MiB/s (146MB/s-155MB/s), io=66.4GiB (71.2GB), run=30026-30050msec

Run status group 3 (all jobs):
  WRITE: bw=724MiB/s (759MB/s), 45.2MiB/s-45.3MiB/s (47.4MB/s-47.5MB/s), io=21.2GiB (22.8GB), run=30002-30004msec

Disk stats (read/write):
    md3: ios=20633316/6830544, sectors=249223840/179868480, merge=0/0, ticks=17841143/19426283, in_queue=37267426, util=99.81%, aggrios=10316853/6838872, aggsectors=124614448/180568616, aggrmerge=0/17, aggrticks=8902897/10705748, aggrin_queue=19608646, aggrutil=78.44%
  nvme0n1: ios=19164741/6838993, sectors=184936128/180568616, merge=0/17, ticks=8546566/1947867, in_queue=10494433, util=58.67%
  nvme1n1: ios=1468965/6838752, sectors=64292768/180568616, merge=0/17, ticks=9259229/19463630, in_queue=28722859, util=78.44%

luckylinux · Sep 26, 2026

Not sure what just happened, now it sends about 2 GiB in 3s (zfs send | zfs receive)

.

TrevorH · Sep 26, 2026

luckylinux said:
At least after disabling powertop --auto-tune in /etc/rc.local I am now getting 3.1GB/s, which seems reasonable for an NVMe Drive:

Code:

root@pve:~# dd if=/dev/nvme0n1 of=/dev/null iflag=direct bs=16M status=progress 107927830528 bytes (108 GB, 101 GiB) copied, 34 s, 3.2 GB/s^C 6508+0 records in 6507+0 records out 109169344512 bytes (109 GB, 102 GiB) copied, 34.3601 s, 3.2 GB/s

Your original results were appalling and almost certainly the cause of everything else you were complaining about. Since it's PCIe 3.0 3.2GB/s is much better and depending on SSD model, might be as good as you'll get. I had a weird issue like this a few years ago and I tried lots of stuff but the eventual fix was discovering that X2APIC was disabled in the BIOS. Re-enabling it fixed it for me then.

luckylinux · Sep 26, 2026

TrevorH said:
Your original results were appalling and almost certainly the cause of everything else you were complaining about. Since it's PCIe 3.0 3.2GB/s is much better and depending on SSD model, might be as good as you'll get. I had a weird issue like this a few years ago and I tried lots of stuff but the eventual fix was discovering that X2APIC was disabled in the BIOS. Re-enabling it fixed it for me then.

But it's not consistent. I just tried another zfs send | zfs receive and it's slow as hell again

.

TrevorH · Sep 26, 2026

What do you get if you run dmesg -T | grep -i x2a ? What about lspci -vvv -s bb:cc.x | grep Lnk where bb:cc.x is the number from the lefthand side of the line in lspci -nn that is for the nvme drive? Does that change when it's quick vs slow?

[Promox VE] LUKS + ZFS - Extremely slow NVMe Root Pool

Well-Known Member

Active Member

Well-Known Member

Active Member

Active Member

Well-Known Member

Well-Known Member

Active Member

Well-Known Member

Well-Known Member

Active Member

Well-Known Member

Well-Known Member

Well-Known Member

Active Member

Well-Known Member

Well-Known Member

Active Member

Well-Known Member

Active Member