CWWK/Topton/... Nxxx quad NIC router

ohm · Jul 25, 2023

athurdent said:
Odd, my tech specs say
View attachment 30501
As for NVME x1, and I am no expert so might be completely wrong, I was under the impression with PCIe 3.x there's 1G throughput. This should be sufficient for a firewall VM, even with heavy logging / database activity from Zenarmor.

You can verify your RAM specs by checking if you have DDR4 or DDR5 ram installed. DDR5 is not pin compatible with DDR4.

As for non-volatile storage performance requirements for a firewall VM, it is really subjective depending on the individual's environment and needs. If you are also planning for other RAM intensive containers and plugins, and disk swap space becomes frequently used, then you'd want disk access to be optimized as much as possible.

Also, previous-gen PCIe 3.0 NVMEs are amazingly-priced now, and the most value-for-money ones are of the x4 variants with speeds in excess of 3000 MB/s. As a performance freak, I simply couldn't bear running a x4-capable disk on a x1 interface.

swixter · Jul 25, 2023

I think most of us lose sight of what we wanna do with the device. Different hardware for different purposes. Honestly, USB2 or USB3 doesn' t make a difference with the use case of this device. DDR4 or DDR5, PCIe x1 or better. It's not a gaming machine. By the way, don' t forget "more performance means generally more heat to dissipate". If we go further for the better, well better buy a real PC in stead

athurdent · Jul 25, 2023

ohm said:
You can verify your RAM specs by checking if you have DDR4 or DDR5 ram installed. DDR5 is not pin compatible with DDR4.

As for non-volatile storage performance requirements for a firewall VM, it is really subjective depending on the individual's environment and needs. If you are also planning for other RAM intensive containers and plugins, and disk swap space becomes frequently used, then you'd want disk access to be optimized as much as possible.

Also, previous-gen PCIe 3.0 NVMEs are amazingly-priced now, and the most value-for-money ones are of the x4 variants with speeds in excess of 3000 MB/s. As a performance freak, I simply couldn't bear running a x4-capable disk on a x1 interface.

Thank you for the info!
I bought a Crucial DDR5 module along with the HUNSN. Works fine so far:

Code:

root@n100:~# dmidecode --type 17
# dmidecode 3.4
Getting SMBIOS data from sysfs.
SMBIOS 3.5.0 present.

Handle 0x0028, DMI type 17, 92 bytes
Memory Device
    Array Handle: 0x0027
    Error Information Handle: Not Provided
    Total Width: 64 bits
    Data Width: 64 bits
    Size: 16 GB
    Form Factor: SODIMM
    Set: None
    Locator: Controller0-ChannelA-DIMM0
    Bank Locator: BANK 0
    Type: DDR5
    Type Detail: Synchronous
    Speed: 4800 MT/s
    Manufacturer: Crucial Technology
    Serial Number: ***
    Asset Tag: 9876543210
    Part Number: CT16G48C40S5.M8A1
    Rank: 1
    Configured Memory Speed: 4800 MT/s
    Minimum Voltage: 1.1 V
    Maximum Voltage: 1.1 V
    Configured Voltage: 1.1 V
    Memory Technology: DRAM
    Memory Operating Mode Capability: Volatile memory
    Firmware Version: Not Specified
    Module Manufacturer ID: Bank 6, Hex 0x9B
    Module Product ID: Unknown
    Memory Subsystem Controller Manufacturer ID: Unknown
    Memory Subsystem Controller Product ID: Unknown
    Non-Volatile Size: None
    Volatile Size: 16 GB
    Cache Size: None
    Logical Size: None

Handle 0x0029, DMI type 17, 92 bytes
Memory Device
    Array Handle: 0x0027
    Error Information Handle: Not Provided
    Total Width: Unknown
    Data Width: Unknown
    Size: No Module Installed
    Form Factor: Unknown
    Set: None
    Locator: Controller1-ChannelA-DIMM0
    Bank Locator: BANK 0
    Type: Unknown
    Type Detail: None

athurdent · Jul 25, 2023

swixter said:
I think most of us lose sight of what we wanna do with the device. Different hardware for different purposes. Honestly, USB2 or USB3 doesn' t make a difference with the use case of this device. DDR4 or DDR5, PCIe x1 or better. It's not a gaming machine. By the way, don' t forget "more performance means generally more heat to dissipate". If we go further for the better, well better buy a real PC in stead

Yeah, the only thing that really bugs me so far is the low 2.5G performance with Debian.

The PS they delivered with the device suffers from coil whine, replaced it with a Leicke PS for now and contacted the seller for a solution. If I buy a fanless device that is supposed to be silent, no fun when the PS starts to compete with my tinnitus...

WRT temperatures, while it's not really warm I guess, you can see when I put one of those HiFi USB fans on there for now. Have ordered a 80mm x 10,8mm PWM fan, let's see if I can get that mounted in the case and tuned for silent operation.

athurdent · Jul 26, 2023

athurdent said:
I installed Proxmox on mine, iperf3 tests never go above 1.6 Gbps with Linux. Usually with Intel NICs, I could fix this compiling in drivers from their website, no luck finding any for the 226 NICs though.

I found the problem:

[Tue Jul 25 18:53:38 2023] igc 0000:01:00.0: 2.000 Gb/s available PCIe bandwidth, limited by 2.5 GT/s PCIe x1 link at 0000:00:1c.0 (capable of 4.000 Gb/s with 5.0 GT/s PCIe x1 link)

using this one

[Tue Jul 25 18:53:38 2023] igc 0000:04:00.0: 4.000 Gb/s available PCIe bandwidth (5.0 GT/s PCIe x1 link)

all is well.

So there's one 2.5G interface in my box that does not actually do 2.5G.

ohm · Jul 26, 2023

athurdent said:
I found the problem:

[Tue Jul 25 18:53:38 2023] igc 0000:01:00.0: 2.000 Gb/s available PCIe bandwidth, limited by 2.5 GT/s PCIe x1 link at 0000:00:1c.0 (capable of 4.000 Gb/s with 5.0 GT/s PCIe x1 link)

using this one

[Tue Jul 25 18:53:38 2023] igc 0000:04:00.0: 4.000 Gb/s available PCIe bandwidth (5.0 GT/s PCIe x1 link)

all is well.

So there's one 2.5G interface in my box that does not actually do 2.5G.

Interesting...

Here's mine(4-port with NVME x4). Looks like all ports are legit 2.5Gbe.

01:00.0 Non-Volatile memory controller: Sandisk Corp WD Black 2018/SN750 / PC SN720 NVMe SSD (prog-if 02 [NVM Express])

                LnkCap: Port #0, Speed 8GT/s, Width x4, ASPM L1, Exit Latency L1 <8us

                LnkSta: Speed 8GT/s, Width x4

02:00.0 Ethernet controller: Intel Corporation Ethernet Controller I226-V (rev 04)

                LnkCap: Port #0, Speed 5GT/s, Width x1, ASPM L1, Exit Latency L1 <4us

                LnkSta: Speed 5GT/s, Width x1

03:00.0 Ethernet controller: Intel Corporation Ethernet Controller I226-V (rev 04)

                LnkCap: Port #0, Speed 5GT/s, Width x1, ASPM L1, Exit Latency L1 <4us

                LnkSta: Speed 5GT/s, Width x1

04:00.0 Ethernet controller: Intel Corporation Ethernet Controller I226-V (rev 04)

                LnkCap: Port #0, Speed 5GT/s, Width x1, ASPM L1, Exit Latency L1 <4us

                LnkSta: Speed 5GT/s, Width x1

05:00.0 Ethernet controller: Intel Corporation Ethernet Controller I226-V (rev 04)

                LnkCap: Port #0, Speed 5GT/s, Width x1, ASPM L1, Exit Latency L1 <4us

                LnkSta: Speed 5GT/s, Width x1

Zer0_C00L · Jul 26, 2023

limerx said:
I hope this isn't the wrong place to post.

I've been looking at one of these n100 boxes but the main gripe for me is the lack of usb3.0, no practical reason why, I just can't swallow the idea of being stuck with 2.0 in 2023.

So in my search I found this box from BeiKong/BKHD (G30M): 241.36US $ |BKHD 2023 Mini PC Soft Router Firewall VPN G30M 4 LAN 2.5G Intel Alder Lake N N100 Quad Cores Pfsense OPNsense DP HD MI NVMe| | - AliExpress
Edit: Google translate of the product specs from TaoBao. Source: https://m.tb.cn/h.5c4x19J?tk=STzdduGtCKA
View attachment 30491

I've attempted to search for this model on this forum and Google but found nothing on this box.

My initial thoughts is that the case doesn't look like the best heat sink, but does anyone else have an opinion on the specs of this box vs say the variation C from CWWK (which seems to be the most popular)?

Why would you NEED USB3?! Like if I had to choose between 2x USB3 ports and one extra PICe lane, or a 2.5G port **** USB3!

Like sure one or two USB3 ports but honestly USB2 is fine, by and large USB3 is important for like set up maybe from time to time for copying large files not over a network but it'd not be that much harder to just copy over via network.

dantkt0 · Jul 26, 2023

Has anyone gotten the mini PCIE slot to work? Tried adding a PCIE coral TPU card, and its not showing up at all in lspci, all the slots seem to be enabled in the BIOS

athurdent · Jul 26, 2023

ohm said:
Interesting...

Here's mine(4-port with NVME x4). Looks like all ports are legit 2.5Gbe.

Thanks. I asked HUNSN if there's a BIOS setting to get full bandwidth for the first interface. I have no idea about PCI stuff, so most likely that is a noobish question and the only solution would be a soldering iron.

But, on the other hand, it would be odd to sell a firewall appliance that has one slower interface. Nothing in the tech specs I could find about that either.

lucker · Jul 26, 2023

goh said:
Did you perhaps have time to explore this yet? This 4xNVMe board could potentially be very useful, pity that they seem to have abandoned developing it themselves.

Not yet, alas, but eager to test. Lack of spare time and a great fear not be able to roll back to the "standard"BIOS keeps me out of it for now. I hope very much to do this in a couple of weeks whlie awaiting the new OPNsense release based on FreeBSD 13.2 to finally strating setup the rig and build the system. I will definitely post about my endeavor. Stay tuned

lucker · Jul 26, 2023

Here is my second post (the first one) with the test outcomes.

I installed the second NVMe (the same 1TB Apacer PP3480 AP1TPP3480-R) and attached to each cooper "slabs".

On Ubuntu with default BIOS (factory optimized) PL1=20000, PL2=0 I finally get good performance test results:
CPU Mark: 10878
Memory Mark: 2914

Code:

                          PassMark PerformanceTest Linux


Intel Core i3-N305 (x86_64)
8 cores @ 3800 MHz  |  31.1 GiB RAM
Number of Processes: 8  |  Test Iterations: 1  |  Test Duration: Medium
--------------------------------------------------------------------------
CPU Mark:                          10878
  Integer Math                     35646 Million Operations/s
  Floating Point Math              20852 Million Operations/s
  Prime Numbers                    30.5 Million Primes/s
  Sorting                          19659 Thousand Strings/s
  Encryption                       9855 MB/s
  Compression                      120836 KB/s
  CPU Single Threaded              2310 Million Operations/s
  Physics                          774 Frames/s
  Extended Instructions (SSE)      5342 Million Matrices/s

Memory Mark:                       2914
  Database Operations              6461 Thousand Operations/s
  Memory Read Cached               21268 MB/s
  Memory Read Uncached             12694 MB/s
  Memory Write                     11008 MB/s
  Available RAM                    29885 Megabytes
  Memory Latency                   31 Nanoseconds
  Memory Threaded                  32505 MB/s
--------------------------------------------------------------------------

With the same default BIOS settings I tested both NVMe slots: native (on MB, x4) and on H-type PCB (x1). This test was performed on the open upended box with the ambient t=25C
in Windows x4
View attachment 30507 View attachment 30508
in Windows x1
View attachment 30509

in Ubuntu x4 then x1

FInally I made a ZFS mirror in Ubuntu (didn't tune it, just ran defaults); this one was performed on the closed horizontal box; ambient t=25C

Ubuntu ZFS mirror plain 2023-07-17 19-19-21.png

disk t were near 60C (forgot to take a sreenshot)

Pretty much satisfied with the outcomes I moved to FreeBSD 13.2. Alas, there is no support (yet) for Adler Lake in drm-kmod, so, no graphics, just console output.

Partitioned disks to have a root (ZFS mirror), a data (geli encrypted ZFS mirror), and a swap (not mirrored) partitions. I even made the EFI partitions mirrored with gmirror. Total smoke & mirror

Code:

gpart show
=>        40  2000409184  nda0  GPT  (954G)
          40     1048576     1  efi  (512M)
     1048616    67108864     2  freebsd-zfs  (32G)
    68157480  1927282688     3  freebsd-zfs  (919G)
  1995440168     4968448     4  freebsd-swap  (2.4G)
  2000408616         608        - free -  (304K)

=>        40  2000409184  nda1  GPT  (954G)
          40     1048576     1  efi  (512M)
     1048616    67108864     2  freebsd-zfs  (32G)
    68157480  1927282688     3  freebsd-zfs  (919G)
  1995440168     4968448     4  freebsd-swap  (2.4G)
2000408616         608        - free -  (304K)

16:17 /root # zpool list
NAME    SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
zdata   912G  7.76G   904G        -         -     0%     0%  1.00x    ONLINE  -
zroot  31.5G  26.6G  4.89G        -         -    45%    84%  1.00x    ONLINE  -

16:17 /root # zpool status
  pool: zdata
state: ONLINE
config:

        NAME               STATE     READ WRITE CKSUM
        zdata              ONLINE       0     0     0
          mirror-0         ONLINE       0     0     0
            gpt/data0.eli  ONLINE       0     0     0
            gpt/data1.eli  ONLINE       0     0     0

errors: No known data errors

  pool: zroot
state: ONLINE
config:

        NAME          STATE     READ WRITE CKSUM
        zroot         ONLINE       0     0     0
          mirror-0    ONLINE       0     0     0
            gpt/sys0  ONLINE       0     0     0
            gpt/sys1  ONLINE       0     0     0

errors: No known data errors

All ZFS properties by their defaults, didn't have time to play with it, will do later.

The fio outcomes make me scratch my head.

I tested both zpools (zroot & zdata on the same two mirrored NVMes) with two sets:

Code:

# Single 4KiB random write process
fio --name=random-write --ioengine=posixaio --rw=randwrite --bs=4k --size=4g --numjobs=1 --iodepth=1 --runtime=60 --time_based --end_fsync=1
# 16 parallel 64KiB random write processes
fio --name=random-write --ioengine=posixaio --rw=randwrite --bs=64k --size=256m --numjobs=16 --iodepth=16 --runtime=60 --time_based --end_fsync=1

... kindly suggested by Jim Salter at arstechnica.com; there are interpretations as well.

The tests ran on closed horizontal box with default BIOS and the ambient t=28C
fio Single 4KiB random write process zroot
At end
nvme0 Temperature: 52 Celsius
nvme1 Temperature: 49 Celsius
Results: WRITE: bw=22.7MiB/s (23.8MB/s), 22.7MiB/s-22.7MiB/s (23.8MB/s-23.8MB/s), io=1379MiB (1446MB), run=60639-60639msec

Code:

12:16 / # fio --name=random-write --ioengine=posixaio --rw=randwrite --bs=4k --size=4g --numjobs=1 --iodepth=1 --runtime=60 --time_based --end_fsync=1
random-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=posixaio, iodepth=1
fio-3.35
Starting 1 process
random-write: Laying out IO file (1 file / 4096MiB)
Jobs: 1 (f=1): [w(1)][33.7%][w=21.6MiB/s][w=5521 IOPS][eta 02m:00s]
random-write: (groupid=0, jobs=1): err= 0: pid=1847: Wed Jul 26 12:17:59 2023
  write: IOPS=5822, BW=22.7MiB/s (23.8MB/s)(1379MiB/60639msec); 0 zone resets
    slat (nsec): min=838, max=127709, avg=1538.21, stdev=3268.30
    clat (nsec): min=632, max=826775k, avg=169869.16, stdev=10714149.50
     lat (usec): min=6, max=826775, avg=171.41, stdev=10714.15
    clat percentiles (usec):
     |  1.00th=[     7],  5.00th=[     7], 10.00th=[     8], 20.00th=[    26],
     | 30.00th=[    28], 40.00th=[    28], 50.00th=[    28], 60.00th=[    29],
     | 70.00th=[    29], 80.00th=[    29], 90.00th=[    30], 95.00th=[    42],
     | 99.00th=[    60], 99.50th=[    70], 99.90th=[    94], 99.95th=[   102],
     | 99.99th=[801113]
   bw (  KiB/s): min=  360, max=69011, per=100.00%, avg=34241.29, stdev=15177.02, samples=82
   iops        : min=   90, max=17252, avg=8559.93, stdev=3794.17, samples=82
  lat (nsec)   : 750=0.05%, 1000=0.14%
  lat (usec)   : 2=0.15%, 4=0.02%, 10=13.38%, 20=0.65%, 50=83.75%
  lat (usec)   : 100=1.81%, 250=0.04%, 500=0.01%
  lat (msec)   : 250=0.01%, 750=0.01%, 1000=0.02%
  cpu          : usr=0.74%, sys=1.67%, ctx=353886, majf=0, minf=1
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,353098,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=22.7MiB/s (23.8MB/s), 22.7MiB/s-22.7MiB/s (23.8MB/s-23.8MB/s), io=1379MiB (1446MB), run=60639-60639msec

fio Single 4KiB random write process zdata
At end
nvme0 Temperature: 57 Celsius
nvme1 Temperature: 55 Celsius
Results: WRITE: bw=24.8MiB/s (26.0MB/s), 24.8MiB/s-24.8MiB/s (26.0MB/s-26.0MB/s), io=1611MiB (1690MB), run=65029-65029msec

Code:

12:22 /zdata # fio --name=random-write --ioengine=posixaio --rw=randwrite --bs=4k --size=4g --numjobs=1 --iodepth=1 --runtime=60 --time_based --end_fsync=1
random-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=posixaio, iodepth=1
fio-3.35
Starting 1 process
random-write: Laying out IO file (1 file / 4096MiB)
Jobs: 1 (f=1): [F(1)][100.0%][eta 00m:00s]
random-write: (groupid=0, jobs=1): err= 0: pid=1866: Wed Jul 26 12:24:39 2023
  write: IOPS=6343, BW=24.8MiB/s (26.0MB/s)(1611MiB/65029msec); 0 zone resets
    slat (nsec): min=831, max=20204k, avg=2126.78, stdev=45197.55
    clat (nsec): min=645, max=101480k, avg=142286.82, stdev=254894.62
     lat (usec): min=6, max=101480, avg=144.41, stdev=258.64
    clat percentiles (usec):
     |  1.00th=[    8],  5.00th=[   26], 10.00th=[   30], 20.00th=[   40],
     | 30.00th=[  115], 40.00th=[  131], 50.00th=[  151], 60.00th=[  163],
     | 70.00th=[  178], 80.00th=[  188], 90.00th=[  196], 95.00th=[  200],
     | 99.00th=[ 1139], 99.50th=[ 1156], 99.90th=[ 1172], 99.95th=[ 1598],
     | 99.99th=[ 3064]
   bw (  KiB/s): min=18988, max=107107, per=100.00%, avg=27545.43, stdev=12161.83, samples=119
   iops        : min= 4747, max=26776, avg=6886.03, stdev=3040.44, samples=119
  lat (nsec)   : 750=0.02%, 1000=0.05%
  lat (usec)   : 2=0.07%, 4=0.04%, 10=1.37%, 20=0.50%, 50=21.43%
  lat (usec)   : 100=5.11%, 250=70.05%, 500=0.10%, 750=0.05%, 1000=0.02%
  lat (msec)   : 2=1.15%, 4=0.03%, 10=0.01%, 20=0.01%, 50=0.01%
  lat (msec)   : 100=0.01%, 250=0.01%
  cpu          : usr=1.94%, sys=1.98%, ctx=418405, majf=0, minf=1
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,412496,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=24.8MiB/s (26.0MB/s), 24.8MiB/s-24.8MiB/s (26.0MB/s-26.0MB/s), io=1611MiB (1690MB), run=65029-65029msec

fio 16 parallel 64KiB random write processes zroot
At end
nvme0 Temperature: 57 Celsius
nvme1 Temperature: 54 Celsius
Results: WRITE: bw=349MiB/s (366MB/s), 20.2MiB/s-24.0MiB/s (21.2MB/s-25.2MB/s), io=20.5GiB (22.0GB), run=60145-60155msec

fio 16 parallel 64KiB random write processes zdata
At end
nvme0 Temperature: 61 Celsius
nvme1 Temperature: 59 Celsius
CPU: 48.0
Results: WRITE: bw=381MiB/s (400MB/s), 23.7MiB/s-24.1MiB/s (24.8MB/s-25.2MB/s), io=24.6GiB (26.5GB), run=66189-66199msec

Code:

12:24 /zdata # fio --name=random-write --ioengine=posixaio --rw=randwrite --bs=64k --size=256m --numjobs=16 --iodepth=16 --runtime=60 --time_based --end_fsync=1
random-write: (g=0): rw=randwrite, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=posixaio, iodepth=16
...
fio-3.35
Starting 16 processes
random-write: Laying out IO file (1 file / 256MiB)
random-write: Laying out IO file (1 file / 256MiB)
random-write: Laying out IO file (1 file / 256MiB)
random-write: Laying out IO file (1 file / 256MiB)
random-write: Laying out IO file (1 file / 256MiB)
random-write: Laying out IO file (1 file / 256MiB)
random-write: Laying out IO file (1 file / 256MiB)
random-write: Laying out IO file (1 file / 256MiB)
random-write: Laying out IO file (1 file / 256MiB)
random-write: Laying out IO file (1 file / 256MiB)
random-write: Laying out IO file (1 file / 256MiB)
random-write: Laying out IO file (1 file / 256MiB)
random-write: Laying out IO file (1 file / 256MiB)
random-write: Laying out IO file (1 file / 256MiB)
random-write: Laying out IO file (1 file / 256MiB)
Jobs: 16 (f=16): [F(16)][100.0%][eta 00m:00s]
random-write: (groupid=0, jobs=1): err= 0: pid=1895: Wed Jul 26 12:31:58 2023
  write: IOPS=379, BW=23.7MiB/s (24.8MB/s)(1569MiB/66198msec); 0 zone resets
    slat (nsec): min=905, max=32722k, avg=16218.16, stdev=216016.98
    clat (usec): min=10, max=169218, avg=38148.50, stdev=13295.86
     lat (usec): min=63, max=169221, avg=38164.72, stdev=13264.77
    clat percentiles (usec):
     |  1.00th=[   660],  5.00th=[ 14353], 10.00th=[ 25035], 20.00th=[ 28705],
     | 30.00th=[ 32637], 40.00th=[ 35914], 50.00th=[ 39584], 60.00th=[ 43254],
     | 70.00th=[ 46400], 80.00th=[ 49021], 90.00th=[ 51643], 95.00th=[ 52691],
     | 99.00th=[ 55313], 99.50th=[ 78119], 99.90th=[127402], 99.95th=[145753],
     | 99.99th=[147850]
   bw (  KiB/s): min=18267, max=170624, per=6.85%, avg=26748.86, stdev=14628.55, samples=119
   iops        : min=  285, max= 2666, avg=417.57, stdev=228.56, samples=119
  lat (usec)   : 20=0.18%, 50=0.04%, 100=0.01%, 250=0.20%, 500=0.33%
  lat (usec)   : 750=0.39%, 1000=0.37%
  lat (msec)   : 2=1.30%, 4=1.54%, 10=0.39%, 20=0.65%, 50=78.99%
  lat (msec)   : 100=15.30%, 250=0.31%
  cpu          : usr=0.26%, sys=0.26%, ctx=22470, majf=0, minf=1
  IO depths    : 1=0.3%, 2=0.7%, 4=1.7%, 8=53.7%, 16=43.6%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=96.1%, 8=2.3%, 16=1.7%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,25098,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
random-write: (groupid=0, jobs=1): err= 0: pid=1896: Wed Jul 26 12:31:58 2023
  write: IOPS=383, BW=24.0MiB/s (25.2MB/s)(1588MiB/66196msec); 0 zone resets
    slat (nsec): min=900, max=32682k, avg=15795.26, stdev=213950.23
    clat (usec): min=10, max=149757, avg=37666.82, stdev=13905.14
     lat (usec): min=97, max=149767, avg=37682.61, stdev=13876.54
    clat percentiles (usec):
     |  1.00th=[   355],  5.00th=[  2638], 10.00th=[ 24511], 20.00th=[ 28181],
     | 30.00th=[ 32375], 40.00th=[ 35390], 50.00th=[ 39584], 60.00th=[ 43254],
     | 70.00th=[ 45876], 80.00th=[ 49021], 90.00th=[ 51643], 95.00th=[ 52691],
     | 99.00th=[ 55313], 99.50th=[ 61604], 99.90th=[133694], 99.95th=[135267],
     | 99.99th=[143655]
   bw (  KiB/s): min=18340, max=213675, per=6.94%, avg=27099.60, stdev=18289.38, samples=119
   iops        : min=  286, max= 3338, avg=423.06, stdev=285.72, samples=119
  lat (usec)   : 20=0.25%, 50=0.04%, 100=0.02%, 250=0.40%, 500=0.67%
  lat (usec)   : 750=0.65%, 1000=0.63%
  lat (msec)   : 2=1.90%, 4=1.14%, 10=0.33%, 20=0.56%, 50=78.06%
  lat (msec)   : 100=15.01%, 250=0.35%
  cpu          : usr=0.25%, sys=0.27%, ctx=23064, majf=0, minf=1
  IO depths    : 1=0.4%, 2=0.8%, 4=2.0%, 8=53.3%, 16=43.5%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=96.1%, 8=2.4%, 16=1.4%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,25410,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
random-write: (groupid=0, jobs=1): err= 0: pid=1897: Wed Jul 26 12:31:58 2023
  write: IOPS=380, BW=23.8MiB/s (24.9MB/s)(1573MiB/66199msec); 0 zone resets
    slat (nsec): min=932, max=32854k, avg=16029.64, stdev=217102.85
    clat (usec): min=10, max=123456, avg=38017.95, stdev=12342.31
     lat (usec): min=97, max=123464, avg=38033.98, stdev=12309.29
    clat percentiles (usec):
     |  1.00th=[   824],  5.00th=[ 19006], 10.00th=[ 25035], 20.00th=[ 28705],
     | 30.00th=[ 32637], 40.00th=[ 35914], 50.00th=[ 39584], 60.00th=[ 43254],
     | 70.00th=[ 45876], 80.00th=[ 49021], 90.00th=[ 51643], 95.00th=[ 52691],
     | 99.00th=[ 55313], 99.50th=[ 55837], 99.90th=[106431], 99.95th=[109577],
     | 99.99th=[112722]
   bw (  KiB/s): min=18358, max=157253, per=6.87%, avg=26823.98, stdev=13559.55, samples=119
   iops        : min=  286, max= 2457, avg=418.76, stdev=211.85, samples=119
  lat (usec)   : 20=0.21%, 50=0.02%, 250=0.13%, 500=0.27%, 750=0.29%
  lat (usec)   : 1000=0.25%
  lat (msec)   : 2=1.01%, 4=1.80%, 10=0.44%, 20=0.71%, 50=79.42%
  lat (msec)   : 100=15.32%, 250=0.12%
  cpu          : usr=0.33%, sys=0.18%, ctx=22341, majf=0, minf=1
  IO depths    : 1=0.3%, 2=0.6%, 4=1.7%, 8=53.9%, 16=43.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=95.8%, 8=2.3%, 16=1.8%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,25169,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
random-write: (groupid=0, jobs=1): err= 0: pid=1898: Wed Jul 26 12:31:58 2023
  write: IOPS=380, BW=23.8MiB/s (25.0MB/s)(1576MiB/66199msec); 0 zone resets
    slat (nsec): min=926, max=32678k, avg=16177.31, stdev=215562.62
    clat (usec): min=10, max=164345, avg=37964.11, stdev=13238.52
     lat (usec): min=64, max=164349, avg=37980.29, stdev=13207.43
    clat percentiles (usec):
     |  1.00th=[   685],  5.00th=[ 13698], 10.00th=[ 24773], 20.00th=[ 28443],
     | 30.00th=[ 32637], 40.00th=[ 35390], 50.00th=[ 39584], 60.00th=[ 43254],
     | 70.00th=[ 45876], 80.00th=[ 49021], 90.00th=[ 51643], 95.00th=[ 52691],
     | 99.00th=[ 55313], 99.50th=[ 60031], 99.90th=[143655], 99.95th=[152044],
     | 99.99th=[164627]
   bw (  KiB/s): min=16956, max=165556, per=6.88%, avg=26861.86, stdev=14284.45, samples=119
   iops        : min=  264, max= 2586, avg=419.34, stdev=223.13, samples=119
  lat (usec)   : 20=0.21%, 50=0.04%, 100=0.02%, 250=0.19%, 500=0.32%
  lat (usec)   : 750=0.30%, 1000=0.35%
  lat (msec)   : 2=1.20%, 4=1.51%, 10=0.56%, 20=1.16%, 50=78.64%
  lat (msec)   : 100=15.32%, 250=0.18%
  cpu          : usr=0.23%, sys=0.30%, ctx=22762, majf=0, minf=1
  IO depths    : 1=0.3%, 2=0.6%, 4=1.6%, 8=53.1%, 16=44.3%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=96.2%, 8=2.5%, 16=1.3%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,25216,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
random-write: (groupid=0, jobs=1): err= 0: pid=1899: Wed Jul 26 12:31:58 2023
  write: IOPS=380, BW=23.8MiB/s (24.9MB/s)(1573MiB/66198msec); 0 zone resets
    slat (nsec): min=921, max=32777k, avg=16183.16, stdev=216624.03
    clat (usec): min=10, max=147175, avg=38049.61, stdev=12714.52
     lat (usec): min=56, max=147178, avg=38065.79, stdev=12682.09
    clat percentiles (usec):
     |  1.00th=[   816],  5.00th=[ 15401], 10.00th=[ 25035], 20.00th=[ 28705],
     | 30.00th=[ 32637], 40.00th=[ 35390], 50.00th=[ 39584], 60.00th=[ 43254],
     | 70.00th=[ 45876], 80.00th=[ 49021], 90.00th=[ 51643], 95.00th=[ 52691],
     | 99.00th=[ 55313], 99.50th=[ 57410], 99.90th=[122160], 99.95th=[129500],
     | 99.99th=[129500]
   bw (  KiB/s): min=18358, max=161536, per=6.87%, avg=26802.72, stdev=13832.40, samples=119
   iops        : min=  286, max= 2524, avg=418.42, stdev=216.12, samples=119
  lat (usec)   : 20=0.19%, 50=0.04%, 100=0.02%, 250=0.13%, 500=0.27%
  lat (usec)   : 750=0.30%, 1000=0.27%
  lat (msec)   : 2=1.17%, 4=1.63%, 10=0.45%, 20=1.06%, 50=79.06%
  lat (msec)   : 100=15.22%, 250=0.20%
  cpu          : usr=0.33%, sys=0.20%, ctx=22628, majf=0, minf=1
  IO depths    : 1=0.3%, 2=0.6%, 4=1.7%, 8=53.0%, 16=44.5%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=96.3%, 8=2.3%, 16=1.4%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,25165,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
random-write: (groupid=0, jobs=1): err= 0: pid=1900: Wed Jul 26 12:31:58 2023
  write: IOPS=384, BW=24.0MiB/s (25.2MB/s)(1589MiB/66198msec); 0 zone resets
    slat (nsec): min=896, max=32721k, avg=17229.79, stdev=282389.46
    clat (usec): min=9, max=174240, avg=37643.93, stdev=12792.40
     lat (usec): min=61, max=174242, avg=37661.16, stdev=12759.84
    clat percentiles (usec):
     |  1.00th=[   545],  5.00th=[  5145], 10.00th=[ 24773], 20.00th=[ 28181],
     | 30.00th=[ 32375], 40.00th=[ 35390], 50.00th=[ 39584], 60.00th=[ 42730],
     | 70.00th=[ 45876], 80.00th=[ 49021], 90.00th=[ 51643], 95.00th=[ 52691],
     | 99.00th=[ 55313], 99.50th=[ 55837], 99.90th=[ 86508], 99.95th=[141558],
     | 99.99th=[141558]
   bw (  KiB/s): min=18468, max=181632, per=6.95%, avg=27108.26, stdev=15564.52, samples=119
   iops        : min=  288, max= 2838, avg=423.18, stdev=243.18, samples=119
  lat (usec)   : 10=0.01%, 20=0.21%, 50=0.04%, 100=0.02%, 250=0.26%
  lat (usec)   : 500=0.42%, 750=0.38%, 1000=0.41%
  lat (msec)   : 2=1.42%, 4=1.58%, 10=0.41%, 20=0.95%, 50=78.71%
  lat (msec)   : 100=15.12%, 250=0.08%
  cpu          : usr=0.27%, sys=0.29%, ctx=23103, majf=0, minf=1
  IO depths    : 1=0.3%, 2=0.7%, 4=1.6%, 8=52.9%, 16=44.5%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=96.1%, 8=2.7%, 16=1.2%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,25422,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
random-write: (groupid=0, jobs=1): err= 0: pid=1901: Wed Jul 26 12:31:58 2023
  write: IOPS=379, BW=23.7MiB/s (24.9MB/s)(1569MiB/66199msec); 0 zone resets
    slat (nsec): min=912, max=38168k, avg=17900.39, stdev=323755.58
    clat (usec): min=9, max=155984, avg=38114.46, stdev=13025.89
     lat (usec): min=108, max=155986, avg=38132.36, stdev=12994.65
    clat percentiles (usec):
     |  1.00th=[   725],  5.00th=[ 15795], 10.00th=[ 25035], 20.00th=[ 28705],
     | 30.00th=[ 32900], 40.00th=[ 35914], 50.00th=[ 39584], 60.00th=[ 43254],
     | 70.00th=[ 46400], 80.00th=[ 49021], 90.00th=[ 51643], 95.00th=[ 52691],
     | 99.00th=[ 55313], 99.50th=[ 56361], 99.90th=[135267], 99.95th=[145753],
     | 99.99th=[147850]
   bw (  KiB/s): min=18395, max=166578, per=6.85%, avg=26748.00, stdev=14317.90, samples=119
   iops        : min=  287, max= 2602, avg=417.58, stdev=223.65, samples=119
  lat (usec)   : 10=0.01%, 20=0.21%, 50=0.04%, 100=0.02%, 250=0.14%
  lat (usec)   : 500=0.31%, 750=0.30%, 1000=0.31%
  lat (msec)   : 2=1.31%, 4=1.64%, 10=0.41%, 20=0.61%, 50=79.26%
  lat (msec)   : 100=15.17%, 250=0.27%
  cpu          : usr=0.37%, sys=0.14%, ctx=22407, majf=0, minf=1
  IO depths    : 1=0.3%, 2=0.6%, 4=1.8%, 8=53.7%, 16=43.6%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=95.8%, 8=2.4%, 16=1.7%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,25105,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
random-write: (groupid=0, jobs=1): err= 0: pid=1902: Wed Jul 26 12:31:58 2023
  write: IOPS=382, BW=23.9MiB/s (25.1MB/s)(1582MiB/66199msec); 0 zone resets
    slat (nsec): min=906, max=3593.1k, avg=16061.10, stdev=71119.77
    clat (usec): min=11, max=172560, avg=37825.27, stdev=13167.51
     lat (usec): min=87, max=172565, avg=37841.33, stdev=13135.21
    clat percentiles (usec):
     |  1.00th=[   635],  5.00th=[  5735], 10.00th=[ 24773], 20.00th=[ 28443],
     | 30.00th=[ 32637], 40.00th=[ 35390], 50.00th=[ 39584], 60.00th=[ 43254],
     | 70.00th=[ 45876], 80.00th=[ 49021], 90.00th=[ 51643], 95.00th=[ 52691],
     | 99.00th=[ 55313], 99.50th=[ 57934], 99.90th=[137364], 99.95th=[139461],
     | 99.99th=[139461]
   bw (  KiB/s): min=18432, max=179608, per=6.91%, avg=26979.46, stdev=15355.52, samples=119
   iops        : min=  288, max= 2806, avg=421.18, stdev=239.89, samples=119
  lat (usec)   : 20=0.21%, 50=0.05%, 100=0.02%, 250=0.16%, 500=0.33%
  lat (usec)   : 750=0.36%, 1000=0.34%
  lat (msec)   : 2=1.24%, 4=1.87%, 10=0.64%, 20=0.66%, 50=78.73%
  lat (msec)   : 100=15.19%, 250=0.21%
  cpu          : usr=0.36%, sys=0.18%, ctx=23037, majf=0, minf=1
  IO depths    : 1=0.3%, 2=0.7%, 4=1.7%, 8=52.8%, 16=44.5%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=96.2%, 8=2.4%, 16=1.4%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,25305,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
random-write: (groupid=0, jobs=1): err= 0: pid=1903: Wed Jul 26 12:31:58 2023
  write: IOPS=385, BW=24.1MiB/s (25.2MB/s)(1593MiB/66198msec); 0 zone resets
    slat (nsec): min=914, max=4741.1k, avg=16094.13, stdev=72947.41
    clat (usec): min=9, max=154508, avg=37545.98, stdev=12911.57
     lat (usec): min=73, max=154512, avg=37562.07, stdev=12878.97
    clat percentiles (usec):
     |  1.00th=[   457],  5.00th=[  4178], 10.00th=[ 24511], 20.00th=[ 28181],
     | 30.00th=[ 32113], 40.00th=[ 35390], 50.00th=[ 39060], 60.00th=[ 42730],
     | 70.00th=[ 45876], 80.00th=[ 49021], 90.00th=[ 51643], 95.00th=[ 52691],
     | 99.00th=[ 54789], 99.50th=[ 55313], 99.90th=[110625], 99.95th=[111674],
     | 99.99th=[147850]
   bw (  KiB/s): min=18358, max=194560, per=6.96%, avg=27163.65, stdev=16656.74, samples=119
   iops        : min=  286, max= 3040, avg=424.08, stdev=260.27, samples=119
  lat (usec)   : 10=0.01%, 20=0.21%, 50=0.04%, 100=0.02%, 250=0.34%
  lat (usec)   : 500=0.41%, 750=0.41%, 1000=0.41%
  lat (msec)   : 2=1.48%, 4=1.60%, 10=0.47%, 20=0.86%, 50=78.77%
  lat (msec)   : 100=14.86%, 250=0.10%
  cpu          : usr=0.36%, sys=0.19%, ctx=23047, majf=0, minf=1
  IO depths    : 1=0.3%, 2=0.7%, 4=1.9%, 8=52.9%, 16=44.2%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=95.9%, 8=2.6%, 16=1.5%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,25493,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
random-write: (groupid=0, jobs=1): err= 0: pid=1904: Wed Jul 26 12:31:58 2023
  write: IOPS=378, BW=23.7MiB/s (24.8MB/s)(1567MiB/66196msec); 0 zone resets
    slat (nsec): min=906, max=32761k, avg=15979.06, stdev=216582.08
    clat (usec): min=11, max=174239, avg=38182.77, stdev=12929.11
     lat (usec): min=99, max=174240, avg=38198.75, stdev=12897.40
    clat percentiles (usec):
     |  1.00th=[   734],  5.00th=[ 19006], 10.00th=[ 25035], 20.00th=[ 28705],
     | 30.00th=[ 32900], 40.00th=[ 35914], 50.00th=[ 39584], 60.00th=[ 43254],
     | 70.00th=[ 45876], 80.00th=[ 49021], 90.00th=[ 51643], 95.00th=[ 52691],
     | 99.00th=[ 55313], 99.50th=[ 65274], 99.90th=[114820], 99.95th=[145753],
     | 99.99th=[164627]
   bw (  KiB/s): min=18358, max=162874, per=6.85%, avg=26720.88, stdev=13929.60, samples=119
   iops        : min=  286, max= 2544, avg=417.16, stdev=217.58, samples=119
  lat (usec)   : 20=0.19%, 50=0.05%, 250=0.20%, 500=0.28%, 750=0.30%
  lat (usec)   : 1000=0.30%
  lat (msec)   : 2=1.22%, 4=1.51%, 10=0.47%, 20=0.56%, 50=79.31%
  lat (msec)   : 100=15.38%, 250=0.23%
  cpu          : usr=0.36%, sys=0.16%, ctx=22492, majf=0, minf=1
  IO depths    : 1=0.3%, 2=0.6%, 4=1.6%, 8=53.4%, 16=44.1%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=95.8%, 8=2.6%, 16=1.5%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,25071,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
random-write: (groupid=0, jobs=1): err= 0: pid=1905: Wed Jul 26 12:31:58 2023
  write: IOPS=383, BW=23.9MiB/s (25.1MB/s)(1585MiB/66195msec); 0 zone resets
    slat (nsec): min=930, max=32549k, avg=15956.50, stdev=213756.09
    clat (usec): min=9, max=154155, avg=37753.27, stdev=13252.65
     lat (usec): min=48, max=154160, avg=37769.23, stdev=13222.69
    clat percentiles (usec):
     |  1.00th=[   437],  5.00th=[  4293], 10.00th=[ 24773], 20.00th=[ 28443],
     | 30.00th=[ 32375], 40.00th=[ 35390], 50.00th=[ 39584], 60.00th=[ 42730],
     | 70.00th=[ 45876], 80.00th=[ 49021], 90.00th=[ 51643], 95.00th=[ 52691],
     | 99.00th=[ 55313], 99.50th=[ 55837], 99.90th=[131597], 99.95th=[133694],
     | 99.99th=[154141]
   bw (  KiB/s): min=18395, max=191744, per=6.92%, avg=27003.11, stdev=16433.72, samples=119
   iops        : min=  287, max= 2996, avg=421.58, stdev=256.78, samples=119
  lat (usec)   : 10=0.01%, 20=0.20%, 50=0.07%, 100=0.02%, 250=0.33%
  lat (usec)   : 500=0.52%, 750=0.52%, 1000=0.51%
  lat (msec)   : 2=1.55%, 4=1.16%, 10=0.50%, 20=0.63%, 50=78.67%
  lat (msec)   : 100=15.10%, 250=0.21%
  cpu          : usr=0.26%, sys=0.28%, ctx=22844, majf=0, minf=1
  IO depths    : 1=0.3%, 2=0.7%, 4=2.0%, 8=53.3%, 16=43.7%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=96.1%, 8=2.0%, 16=1.9%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,25355,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
random-write: (groupid=0, jobs=1): err= 0: pid=1906: Wed Jul 26 12:31:58 2023
  write: IOPS=381, BW=23.9MiB/s (25.0MB/s)(1579MiB/66194msec); 0 zone resets
    slat (nsec): min=941, max=32682k, avg=15898.77, stdev=214435.40
    clat (usec): min=9, max=150478, avg=37892.24, stdev=12680.73
     lat (usec): min=57, max=150483, avg=37908.14, stdev=12649.04
    clat percentiles (usec):
     |  1.00th=[   652],  5.00th=[ 15401], 10.00th=[ 24773], 20.00th=[ 28443],
     | 30.00th=[ 32637], 40.00th=[ 35390], 50.00th=[ 39584], 60.00th=[ 43254],
     | 70.00th=[ 45876], 80.00th=[ 49021], 90.00th=[ 51643], 95.00th=[ 52691],
     | 99.00th=[ 55313], 99.50th=[ 55837], 99.90th=[115868], 99.95th=[120062],
     | 99.99th=[149947]
   bw (  KiB/s): min=18358, max=172710, per=6.90%, avg=26928.76, stdev=14837.50, samples=119
   iops        : min=  286, max= 2698, avg=420.39, stdev=231.79, samples=119
  lat (usec)   : 10=0.01%, 20=0.20%, 50=0.04%, 100=0.02%, 250=0.19%
  lat (usec)   : 500=0.33%, 750=0.34%, 1000=0.32%
  lat (msec)   : 2=1.29%, 4=1.66%, 10=0.33%, 20=0.78%, 50=79.04%
  lat (msec)   : 100=15.32%, 250=0.13%
  cpu          : usr=0.32%, sys=0.21%, ctx=22695, majf=0, minf=1
  IO depths    : 1=0.3%, 2=0.7%, 4=1.7%, 8=53.1%, 16=44.2%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=96.0%, 8=2.4%, 16=1.6%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,25267,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
random-write: (groupid=0, jobs=1): err= 0: pid=1907: Wed Jul 26 12:31:58 2023
  write: IOPS=378, BW=23.7MiB/s (24.8MB/s)(1566MiB/66189msec); 0 zone resets
    slat (nsec): min=911, max=3682.9k, avg=15906.91, stdev=70743.41
    clat (usec): min=11, max=175675, avg=38195.12, stdev=13799.71
     lat (usec): min=70, max=175676, avg=38211.03, stdev=13768.71
    clat percentiles (usec):
     |  1.00th=[   611],  5.00th=[  5997], 10.00th=[ 25035], 20.00th=[ 28705],
     | 30.00th=[ 32900], 40.00th=[ 35914], 50.00th=[ 39584], 60.00th=[ 43254],
     | 70.00th=[ 46400], 80.00th=[ 49021], 90.00th=[ 51643], 95.00th=[ 52691],
     | 99.00th=[ 55313], 99.50th=[ 86508], 99.90th=[141558], 99.95th=[147850],
     | 99.99th=[152044]
   bw (  KiB/s): min=18468, max=178917, per=6.85%, avg=26719.06, stdev=15286.60, samples=119
   iops        : min=  288, max= 2795, avg=417.13, stdev=238.81, samples=119
  lat (usec)   : 20=0.22%, 50=0.05%, 100=0.02%, 250=0.19%, 500=0.37%
  lat (usec)   : 750=0.32%, 1000=0.36%
  lat (msec)   : 2=1.29%, 4=1.84%, 10=0.44%, 20=0.57%, 50=78.67%
  lat (msec)   : 100=15.23%, 250=0.44%
  cpu          : usr=0.22%, sys=0.30%, ctx=22430, majf=0, minf=1
  IO depths    : 1=0.3%, 2=0.7%, 4=1.9%, 8=53.4%, 16=43.7%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=95.9%, 8=2.2%, 16=1.8%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,25055,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
random-write: (groupid=0, jobs=1): err= 0: pid=1908: Wed Jul 26 12:31:58 2023
  write: IOPS=380, BW=23.8MiB/s (25.0MB/s)(1576MiB/66192msec); 0 zone resets
    slat (nsec): min=918, max=2963.1k, avg=15897.21, stdev=66714.14
    clat (usec): min=11, max=150516, avg=37971.62, stdev=13677.73
     lat (usec): min=77, max=150519, avg=37987.52, stdev=13646.68
    clat percentiles (usec):
     |  1.00th=[   506],  5.00th=[  3884], 10.00th=[ 25035], 20.00th=[ 28443],
     | 30.00th=[ 32637], 40.00th=[ 35390], 50.00th=[ 39584], 60.00th=[ 43254],
     | 70.00th=[ 45876], 80.00th=[ 49021], 90.00th=[ 51643], 95.00th=[ 52691],
     | 99.00th=[ 55313], 99.50th=[ 59507], 99.90th=[143655], 99.95th=[145753],
     | 99.99th=[149947]
   bw (  KiB/s): min=18358, max=193410, per=6.89%, avg=26879.81, stdev=16424.91, samples=119
   iops        : min=  286, max= 3022, avg=419.65, stdev=256.65, samples=119
  lat (usec)   : 20=0.27%, 50=0.03%, 100=0.01%, 250=0.30%, 500=0.39%
  lat (usec)   : 750=0.46%, 1000=0.41%
  lat (msec)   : 2=1.55%, 4=1.62%, 10=0.44%, 20=0.39%, 50=78.69%
  lat (msec)   : 100=15.12%, 250=0.33%
  cpu          : usr=0.29%, sys=0.24%, ctx=22713, majf=0, minf=1
  IO depths    : 1=0.3%, 2=0.7%, 4=1.9%, 8=53.2%, 16=43.8%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=96.0%, 8=2.4%, 16=1.6%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,25212,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
random-write: (groupid=0, jobs=1): err= 0: pid=1909: Wed Jul 26 12:31:58 2023
  write: IOPS=381, BW=23.8MiB/s (25.0MB/s)(1577MiB/66191msec); 0 zone resets
    slat (nsec): min=927, max=3670.3k, avg=15999.83, stdev=74216.85
    clat (usec): min=10, max=172589, avg=37929.29, stdev=13187.42
     lat (usec): min=120, max=172595, avg=37945.29, stdev=13155.39
    clat percentiles (usec):
     |  1.00th=[   766],  5.00th=[ 12256], 10.00th=[ 24773], 20.00th=[ 28443],
     | 30.00th=[ 32375], 40.00th=[ 35390], 50.00th=[ 39584], 60.00th=[ 43254],
     | 70.00th=[ 45876], 80.00th=[ 49021], 90.00th=[ 51643], 95.00th=[ 52691],
     | 99.00th=[ 55313], 99.50th=[ 67634], 99.90th=[119014], 99.95th=[145753],
     | 99.99th=[160433]
   bw (  KiB/s): min=18103, max=168704, per=6.89%, avg=26878.84, stdev=14591.38, samples=119
   iops        : min=  282, max= 2636, avg=419.62, stdev=227.99, samples=119
  lat (usec)   : 20=0.21%, 50=0.04%, 100=0.01%, 250=0.13%, 500=0.32%
  lat (usec)   : 750=0.29%, 1000=0.30%
  lat (msec)   : 2=1.21%, 4=1.73%, 10=0.59%, 20=1.13%, 50=78.58%
  lat (msec)   : 100=15.18%, 250=0.28%
  cpu          : usr=0.26%, sys=0.25%, ctx=22327, majf=0, minf=1
  IO depths    : 1=0.3%, 2=0.7%, 4=1.8%, 8=53.8%, 16=43.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=95.8%, 8=2.1%, 16=2.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,25239,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
random-write: (groupid=0, jobs=1): err= 0: pid=1910: Wed Jul 26 12:31:58 2023
  write: IOPS=378, BW=23.7MiB/s (24.8MB/s)(1568MiB/66192msec); 0 zone resets
    slat (nsec): min=929, max=32766k, avg=15881.65, stdev=216544.50
    clat (usec): min=11, max=161012, avg=38151.05, stdev=12990.80
     lat (usec): min=65, max=161014, avg=38166.93, stdev=12959.76
    clat percentiles (usec):
     |  1.00th=[   832],  5.00th=[ 16909], 10.00th=[ 25035], 20.00th=[ 28705],
     | 30.00th=[ 32900], 40.00th=[ 35914], 50.00th=[ 39584], 60.00th=[ 43254],
     | 70.00th=[ 45876], 80.00th=[ 49021], 90.00th=[ 51643], 95.00th=[ 52691],
     | 99.00th=[ 55313], 99.50th=[ 60031], 99.90th=[143655], 99.95th=[152044],
     | 99.99th=[160433]
   bw (  KiB/s): min=17083, max=158012, per=6.85%, avg=26725.08, stdev=13605.58, samples=119
   iops        : min=  266, max= 2468, avg=417.20, stdev=212.52, samples=119
  lat (usec)   : 20=0.20%, 50=0.04%, 100=0.02%, 250=0.12%, 500=0.30%
  lat (usec)   : 750=0.26%, 1000=0.26%
  lat (msec)   : 2=1.08%, 4=1.67%, 10=0.47%, 20=1.00%, 50=79.01%
  lat (msec)   : 100=15.35%, 250=0.22%
  cpu          : usr=0.29%, sys=0.23%, ctx=22398, majf=0, minf=1
  IO depths    : 1=0.3%, 2=0.6%, 4=1.4%, 8=53.4%, 16=44.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=95.9%, 8=3.1%, 16=1.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,25080,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=381MiB/s (400MB/s), 23.7MiB/s-24.1MiB/s (24.8MB/s-25.2MB/s), io=24.6GiB (26.5GB), run=66189-66199msec

The SSD temperature rose steeply, but decreasing to ca. 55-57C also in several seconds. Idle system results in SSD temperature ca. 50C, and it takes couple of dozen minutes to settle given no load on SSDs.

Since my jerry-rigged SSD heatsinks deny me the use of the suggested by CWWK 80x80x10 fan (see my previous post), I bought another 4pin fan 80x80x15 and attached it on the outside of the box.

Alas, I couldn't figure out how to control the fan properly in the BIOS. I set it to start at 52C, to stop at 50C, with starting PWM = 30 and steep 2pwm. Still the BIOS shows me the fan speed actual is ca. 1500PWM and nothing I did changed this behavior. I'm still not sure that actual PWM in changing with the temperature rising at all (the fan is rather silent).

But with this monstrosity on the back the temperature has droped significantly! For both SSDs and the CPU!
In all SSD tests the CPU temp never went up more than 37C! Though it might be as well due to vertically placed box (I hanged it on some fabric jerry-rigged loops), but still, very impressive.

fio Single 4KiB random write process zroot
At start
nvme0 Temperature: 34 Celsius
nvme1 Temperature: 30 Celsius
At end
nvme0 Temperature: 39 Celsius
nvme1 Temperature: 35 Celsius
CPU 35-37
Results: WRITE: bw=21.8MiB/s (22.8MB/s), 21.8MiB/s-21.8MiB/s (22.8MB/s-22.8MB/s), io=1307MiB (1371MB), run=60026-60026msec

Code:

7, max=44455, per=100.00%, avg=27420.79, stdev=11030.01, samples=96
   iops        : min=  101, max=11113, avg=6854.91, stdev=2757.52, samples=96
  lat (usec)   : 10=8.86%, 20=0.27%, 50=89.72%, 100=1.00%, 250=0.12%
  lat (usec)   : 500=0.01%
  lat (msec)   : 750=0.02%, 1000=0.01%
  cpu          : usr=0.70%, sys=1.66%, ctx=334626, majf=1, minf=1
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,334623,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=21.8MiB/s (22.8MB/s), 21.8MiB/s-21.8MiB/s (22.8MB/s-22.8MB/s), io=1307MiB (1371MB), run=60026-60026msec

fio Single 4KiB random write process zdata
At start
nvme0 Temperature: 36 Celsius
nvme1 Temperature: 31 Celsius
At end
nvme0 Temperature: 39 Celsius
nvme1 Temperature: 35 Celsius
Results: WRITE: bw=20.4MiB/s (21.4MB/s), 20.4MiB/s-20.4MiB/s (21.4MB/s-21.4MB/s), io=1233MiB (1293MB), run=60303-60303msec

Code:

14:57 /zdata # fio --name=random-write --ioengine=posixaio --rw=randwrite --bs=4k --size=4g --numjobs=1 --iodepth=1 --runtime=60 --time_based --end_fsync=1
random-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=posixaio, iodepth=1
fio-3.35
Starting 1 process
random-write: Laying out IO file (1 file / 4096MiB)
Jobs: 1 (f=1): [w(1)][30.2%][w=21.7MiB/s][w=5554 IOPS][eta 02m:21s]
random-write: (groupid=0, jobs=1): err= 0: pid=1742: Wed Jul 26 15:07:21 2023
  write: IOPS=5233, BW=20.4MiB/s (21.4MB/s)(1233MiB/60303msec); 0 zone resets
    slat (nsec): min=916, max=59956, avg=1400.89, stdev=569.24
    clat (usec): min=5, max=651495, avg=189.33, stdev=8754.56
     lat (usec): min=6, max=651496, avg=190.73, stdev=8754.55
    clat percentiles (usec):
     |  1.00th=[     7],  5.00th=[     8], 10.00th=[    20], 20.00th=[    26],
     | 30.00th=[    28], 40.00th=[    28], 50.00th=[    29], 60.00th=[    29],
     | 70.00th=[    29], 80.00th=[    30], 90.00th=[    34], 95.00th=[    47],
     | 99.00th=[    59], 99.50th=[    67], 99.90th=[    86], 99.95th=[    97],
     | 99.99th=[471860]
   bw (  KiB/s): min= 1593, max=36470, per=100.00%, avg=21301.46, stdev=4893.47, samples=118
   iops        : min=  398, max= 9117, avg=5324.91, stdev=1223.35, samples=118
  lat (usec)   : 10=9.34%, 20=0.69%, 50=87.10%, 100=2.83%, 250=0.01%
  lat (usec)   : 500=0.01%
  lat (msec)   : 500=0.03%, 750=0.01%
  cpu          : usr=0.65%, sys=1.71%, ctx=315585, majf=0, minf=1
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,315584,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=20.4MiB/s (21.4MB/s), 20.4MiB/s-20.4MiB/s (21.4MB/s-21.4MB/s), io=1233MiB (1293MB), run=60303-60303msec

fio 16 parallel 64KiB random write processes zroot
At start
nvme0 Temperature: 33 Celsius
nvme1 Temperature: 36 Celsius
At end
nvme0 Temperature: 42 Celsius
nvme1 Temperature: 36 Celsius
CPU 37
Results: WRITE: bw=249MiB/s (261MB/s), 15.3MiB/s-16.2MiB/s (16.0MB/s-17.0MB/s), io=14.6GiB (15.7GB), run=60213-60221msec

fio 16 parallel 64KiB random write processes zdata
At start
nvme0 Temperature: 32 Celsius
nvme1 Temperature: 36 Celsius
At end
nvme0 Temperature: 41 Celsius
nvme1 Temperature: 36 Celsius
CPU 37C
Results: WRITE: bw=268MiB/s (281MB/s), 15.4MiB/s-17.9MiB/s (16.2MB/s-18.7MB/s), io=15.8GiB (16.9GB), run=60219-60243msec

Code:

15:13 /zdata # fio --name=random-write --ioengine=posixaio --rw=randwrite --bs=64k --size=256m --numjobs=16 --iodepth=16 --runtime=60 --time_based --end_fsync=1
random-write: (g=0): rw=randwrite, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=posixaio, iodepth=16
...
fio-3.35
Starting 16 processes
random-write: Laying out IO file (1 file / 256MiB)
random-write: Laying out IO file (1 file / 256MiB)
random-write: Laying out IO file (1 file / 256MiB)
random-write: Laying out IO file (1 file / 256MiB)
random-write: Laying out IO file (1 file / 256MiB)
random-write: Laying out IO file (1 file / 256MiB)
random-write: Laying out IO file (1 file / 256MiB)
random-write: Laying out IO file (1 file / 256MiB)
random-write: Laying out IO file (1 file / 256MiB)
random-write: Laying out IO file (1 file / 256MiB)
random-write: Laying out IO file (1 file / 256MiB)
random-write: Laying out IO file (1 file / 256MiB)
random-write: Laying out IO file (1 file / 256MiB)
random-write: Laying out IO file (1 file / 256MiB)
random-write: Laying out IO file (1 file / 256MiB)
Jobs: 16 (f=16): [w(5),F(1),w(1),F(4),w(1),F(1),w(1),F(2)][100.0%][w=210MiB/s][w=3364 IOPS][eta 00m:00s]
random-write: (groupid=0, jobs=1): err= 0: pid=1788: Wed Jul 26 15:14:54 2023
  write: IOPS=266, BW=16.7MiB/s (17.5MB/s)(1004MiB/60243msec); 0 zone resets
    slat (usec): min=3, max=13224, avg=87.68, stdev=374.11
    clat (usec): min=11, max=494234, avg=59072.60, stdev=118470.50
     lat (usec): min=63, max=494240, avg=59160.28, stdev=118440.12
    clat percentiles (usec):
     |  1.00th=[   127],  5.00th=[   848], 10.00th=[  1909], 20.00th=[  3523],
     | 30.00th=[  4948], 40.00th=[  6390], 50.00th=[  8160], 60.00th=[ 10683],
     | 70.00th=[ 14091], 80.00th=[ 21103], 90.00th=[320865], 95.00th=[329253],
     | 99.00th=[379585], 99.50th=[438305], 99.90th=[492831], 99.95th=[492831],
     | 99.99th=[492831]
   bw (  KiB/s): min= 2039, max=48975, per=6.23%, avg=17107.69, stdev=8231.93, samples=119
   iops        : min=   31, max=  765, avg=266.97, stdev=128.65, samples=119
  lat (usec)   : 20=0.21%, 50=0.55%, 100=0.21%, 250=0.54%, 500=2.02%
  lat (usec)   : 750=1.03%, 1000=1.15%
  lat (msec)   : 2=4.86%, 4=12.81%, 10=34.27%, 20=21.40%, 50=5.31%
  lat (msec)   : 500=15.64%
  cpu          : usr=0.47%, sys=0.31%, ctx=3330, majf=0, minf=1
  IO depths    : 1=5.4%, 2=11.7%, 4=24.5%, 8=51.0%, 16=7.3%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=94.1%, 8=0.1%, 16=5.8%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,16065,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
random-write: (groupid=0, jobs=1): err= 0: pid=1789: Wed Jul 26 15:14:54 2023
  write: IOPS=268, BW=16.8MiB/s (17.6MB/s)(1009MiB/60242msec); 0 zone resets
    slat (nsec): min=1534, max=19496k, avg=90937.10, stdev=408111.89
    clat (usec): min=9, max=503735, avg=58598.32, stdev=117595.68
     lat (usec): min=58, max=503747, avg=58689.25, stdev=117564.93
    clat percentiles (usec):
     |  1.00th=[   113],  5.00th=[   750], 10.00th=[  1680], 20.00th=[  3654],
     | 30.00th=[  5080], 40.00th=[  6521], 50.00th=[  8455], 60.00th=[ 11076],
     | 70.00th=[ 14615], 80.00th=[ 21103], 90.00th=[316670], 95.00th=[329253],
     | 99.00th=[375391], 99.50th=[429917], 99.90th=[497026], 99.95th=[505414],
     | 99.99th=[505414]
   bw (  KiB/s): min= 2035, max=67927, per=6.24%, avg=17153.34, stdev=9093.36, samples=119
   iops        : min=   31, max= 1061, avg=267.70, stdev=142.11, samples=119
  lat (usec)   : 10=0.01%, 20=0.14%, 50=0.57%, 100=0.19%, 250=0.76%
  lat (usec)   : 500=1.93%, 750=1.41%, 1000=1.46%
  lat (msec)   : 2=5.21%, 4=11.15%, 10=33.37%, 20=22.66%, 50=5.54%
  lat (msec)   : 100=0.10%, 500=15.41%, 750=0.10%
  cpu          : usr=0.49%, sys=0.29%, ctx=3297, majf=0, minf=1
  IO depths    : 1=5.4%, 2=11.8%, 4=24.4%, 8=51.0%, 16=7.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=94.1%, 8=0.1%, 16=5.8%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,16148,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
random-write: (groupid=0, jobs=1): err= 0: pid=1790: Wed Jul 26 15:14:54 2023
  write: IOPS=286, BW=17.9MiB/s (18.7MB/s)(1077MiB/60240msec); 0 zone resets
    slat (nsec): min=1256, max=11326k, avg=90110.82, stdev=328752.16
    clat (usec): min=11, max=490502, avg=55013.07, stdev=114559.68
     lat (usec): min=55, max=490509, avg=55103.18, stdev=114527.38
    clat percentiles (usec):
     |  1.00th=[    63],  5.00th=[   537], 10.00th=[  1287], 20.00th=[  2868],
     | 30.00th=[  4490], 40.00th=[  6259], 50.00th=[  7767], 60.00th=[ 10028],
     | 70.00th=[ 12911], 80.00th=[ 19006], 90.00th=[316670], 95.00th=[329253],
     | 99.00th=[375391], 99.50th=[425722], 99.90th=[467665], 99.95th=[488637],
     | 99.99th=[488637]
   bw (  KiB/s): min= 4589, max=57638, per=6.68%, avg=18349.87, stdev=9351.09, samples=119
   iops        : min=   71, max=  900, avg=286.37, stdev=146.08, samples=119
  lat (usec)   : 20=0.20%, 50=0.67%, 100=0.32%, 250=0.79%, 500=2.76%
  lat (usec)   : 750=1.86%, 1000=1.69%
  lat (msec)   : 2=6.42%, 4=11.78%, 10=33.12%, 20=21.50%, 50=4.20%
  lat (msec)   : 100=0.09%, 500=14.58%
  cpu          : usr=0.57%, sys=0.26%, ctx=4146, majf=0, minf=1
  IO depths    : 1=5.3%, 2=11.7%, 4=24.5%, 8=51.1%, 16=7.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=94.1%, 8=0.1%, 16=5.8%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,17234,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
random-write: (groupid=0, jobs=1): err= 0: pid=1791: Wed Jul 26 15:14:54 2023
  write: IOPS=278, BW=17.4MiB/s (18.3MB/s)(1050MiB/60239msec); 0 zone resets
    slat (nsec): min=1177, max=24783k, avg=92785.82, stdev=404000.74
    clat (usec): min=12, max=528090, avg=56425.89, stdev=115659.26
     lat (usec): min=66, max=528147, avg=56518.68, stdev=115627.67
    clat percentiles (usec):
     |  1.00th=[   127],  5.00th=[   766], 10.00th=[  1729], 20.00th=[  3392],
     | 30.00th=[  5080], 40.00th=[  6456], 50.00th=[  8356], 60.00th=[ 10290],
     | 70.00th=[ 13435], 80.00th=[ 19792], 90.00th=[316670], 95.00th=[329253],
     | 99.00th=[379585], 99.50th=[425722], 99.90th=[467665], 99.95th=[526386],
     | 99.99th=[526386]
   bw (  KiB/s): min= 2039, max=44361, per=6.52%, avg=17914.75, stdev=7848.88, samples=119
   iops        : min=   31, max=  693, avg=279.57, stdev=122.68, samples=119
  lat (usec)   : 20=0.17%, 50=0.47%, 100=0.28%, 250=0.72%, 500=2.03%
  lat (usec)   : 750=1.26%, 1000=1.27%
  lat (msec)   : 2=5.33%, 4=11.73%, 10=35.20%, 20=21.84%, 50=4.75%
  lat (msec)   : 500=14.86%, 750=0.10%
  cpu          : usr=0.57%, sys=0.26%, ctx=3452, majf=0, minf=1
  IO depths    : 1=5.3%, 2=11.6%, 4=24.3%, 8=51.3%, 16=7.5%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=94.1%, 8=0.2%, 16=5.8%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,16799,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
random-write: (groupid=0, jobs=1): err= 0: pid=1792: Wed Jul 26 15:14:54 2023
  write: IOPS=265, BW=16.6MiB/s (17.4MB/s)(998MiB/60237msec); 0 zone resets
    slat (nsec): min=1471, max=69144k, avg=102783.96, stdev=702543.30
    clat (usec): min=10, max=469409, avg=59221.11, stdev=117804.83
     lat (usec): min=81, max=471686, avg=59323.89, stdev=117771.15
    clat percentiles (usec):
     |  1.00th=[   151],  5.00th=[   848], 10.00th=[  1942], 20.00th=[  3720],
     | 30.00th=[  5407], 40.00th=[  7046], 50.00th=[  8848], 60.00th=[ 11076],
     | 70.00th=[ 14222], 80.00th=[ 20841], 90.00th=[316670], 95.00th=[333448],
     | 99.00th=[379585], 99.50th=[429917], 99.90th=[467665], 99.95th=[467665],
     | 99.99th=[467665]
   bw (  KiB/s): min= 2039, max=47911, per=6.19%, avg=16999.68, stdev=8324.99, samples=119
   iops        : min=   31, max=  748, avg=265.27, stdev=130.09, samples=119
  lat (usec)   : 20=0.17%, 50=0.39%, 100=0.27%, 250=0.51%, 500=2.02%
  lat (usec)   : 750=1.12%, 1000=1.36%
  lat (msec)   : 2=4.41%, 4=11.44%, 10=34.43%, 20=23.03%, 50=5.02%
  lat (msec)   : 100=0.10%, 500=15.71%
  cpu          : usr=0.50%, sys=0.28%, ctx=3125, majf=0, minf=1
  IO depths    : 1=5.4%, 2=11.7%, 4=24.4%, 8=51.0%, 16=7.5%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=94.1%, 8=0.1%, 16=5.8%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,15973,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
random-write: (groupid=0, jobs=1): err= 0: pid=1793: Wed Jul 26 15:14:54 2023
  write: IOPS=273, BW=17.1MiB/s (17.9MB/s)(1028MiB/60236msec); 0 zone resets
    slat (usec): min=2, max=40757, avg=88.78, stdev=480.97
    clat (usec): min=11, max=527176, avg=57549.64, stdev=116817.67
     lat (usec): min=69, max=529540, avg=57638.42, stdev=116787.30
    clat percentiles (usec):
     |  1.00th=[   155],  5.00th=[   881], 10.00th=[  1958], 20.00th=[  3884],
     | 30.00th=[  5407], 40.00th=[  6521], 50.00th=[  8356], 60.00th=[ 10159],
     | 70.00th=[ 13173], 80.00th=[ 20055], 90.00th=[316670], 95.00th=[333448],
     | 99.00th=[379585], 99.50th=[429917], 99.90th=[467665], 99.95th=[526386],
     | 99.99th=[526386]
   bw (  KiB/s): min= 2039, max=47426, per=6.38%, avg=17540.04, stdev=8023.88, samples=119
   iops        : min=   31, max=  741, avg=273.72, stdev=125.39, samples=119
  lat (usec)   : 20=0.15%, 50=0.44%, 100=0.24%, 250=0.55%, 500=2.07%
  lat (usec)   : 750=0.97%, 1000=1.08%
  lat (msec)   : 2=4.69%, 4=10.59%, 10=38.22%, 20=21.04%, 50=4.72%
  lat (msec)   : 500=15.13%, 750=0.10%
  cpu          : usr=0.60%, sys=0.20%, ctx=3101, majf=0, minf=1
  IO depths    : 1=5.5%, 2=11.7%, 4=24.4%, 8=50.9%, 16=7.5%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=94.1%, 8=0.1%, 16=5.8%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,16446,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
random-write: (groupid=0, jobs=1): err= 0: pid=1794: Wed Jul 26 15:14:54 2023
  write: IOPS=270, BW=16.9MiB/s (17.7MB/s)(1019MiB/60240msec); 0 zone resets
    slat (usec): min=2, max=63824, avg=88.60, stdev=625.35
    clat (usec): min=9, max=528691, avg=58212.78, stdev=117442.06
     lat (usec): min=98, max=528697, avg=58301.39, stdev=117412.05
    clat percentiles (usec):
     |  1.00th=[   169],  5.00th=[   898], 10.00th=[  1942], 20.00th=[  3621],
     | 30.00th=[  5276], 40.00th=[  6587], 50.00th=[  8356], 60.00th=[ 10290],
     | 70.00th=[ 13304], 80.00th=[ 20055], 90.00th=[320865], 95.00th=[333448],
     | 99.00th=[379585], 99.50th=[425722], 99.90th=[467665], 99.95th=[530580],
     | 99.99th=[530580]
   bw (  KiB/s): min=  118, max=54330, per=6.33%, avg=17399.29, stdev=7556.67, samples=119
   iops        : min=    1, max=  848, avg=271.47, stdev=118.06, samples=119
  lat (usec)   : 10=0.01%, 20=0.13%, 50=0.39%, 100=0.20%, 250=0.60%
  lat (usec)   : 500=2.01%, 750=1.06%, 1000=1.05%
  lat (msec)   : 2=4.95%, 4=12.22%, 10=36.49%, 20=20.81%, 50=4.57%
  lat (msec)   : 100=0.10%, 500=15.31%, 750=0.10%
  cpu          : usr=0.50%, sys=0.29%, ctx=2969, majf=0, minf=1
  IO depths    : 1=5.4%, 2=11.7%, 4=24.3%, 8=51.0%, 16=7.6%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=94.1%, 8=0.1%, 16=5.8%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,16298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
random-write: (groupid=0, jobs=1): err= 0: pid=1795: Wed Jul 26 15:14:54 2023
  write: IOPS=270, BW=16.9MiB/s (17.8MB/s)(1020MiB/60240msec); 0 zone resets
    slat (usec): min=2, max=56943, avg=92.00, stdev=573.43
    clat (usec): min=10, max=515450, avg=57970.02, stdev=117207.69
     lat (usec): min=90, max=516931, avg=58062.02, stdev=117178.10
    clat percentiles (usec):
     |  1.00th=[   129],  5.00th=[   766], 10.00th=[  1811], 20.00th=[  3687],
     | 30.00th=[  5276], 40.00th=[  6849], 50.00th=[  8356], 60.00th=[ 10552],
     | 70.00th=[ 13435], 80.00th=[ 20055], 90.00th=[320865], 95.00th=[329253],
     | 99.00th=[379585], 99.50th=[425722], 99.90th=[505414], 99.95th=[513803],
     | 99.99th=[513803]
   bw (  KiB/s): min= 2043, max=62580, per=6.32%, avg=17376.38, stdev=8582.29, samples=119
   iops        : min=   31, max=  977, avg=271.17, stdev=134.09, samples=119
  lat (usec)   : 20=0.17%, 50=0.51%, 100=0.20%, 250=0.61%, 500=2.15%
  lat (usec)   : 750=1.29%, 1000=1.38%
  lat (msec)   : 2=4.73%, 4=11.13%, 10=35.49%, 20=22.29%, 50=4.68%
  lat (msec)   : 100=0.02%, 500=15.15%, 750=0.20%
  cpu          : usr=0.52%, sys=0.26%, ctx=3143, majf=0, minf=1
  IO depths    : 1=5.5%, 2=11.8%, 4=24.6%, 8=50.9%, 16=7.3%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=94.1%, 8=0.1%, 16=5.8%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,16318,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
random-write: (groupid=0, jobs=1): err= 0: pid=1796: Wed Jul 26 15:14:54 2023
  write: IOPS=278, BW=17.4MiB/s (18.2MB/s)(1047MiB/60221msec); 0 zone resets
    slat (usec): min=2, max=66209, avg=87.31, stdev=643.62
    clat (usec): min=13, max=525983, avg=56608.22, stdev=116220.57
     lat (usec): min=112, max=525994, avg=56695.54, stdev=116192.32
    clat percentiles (usec):
     |  1.00th=[   258],  5.00th=[  1106], 10.00th=[  2245], 20.00th=[  3687],
     | 30.00th=[  5145], 40.00th=[  6521], 50.00th=[  8029], 60.00th=[ 10028],
     | 70.00th=[ 13042], 80.00th=[ 19006], 90.00th=[316670], 95.00th=[333448],
     | 99.00th=[375391], 99.50th=[425722], 99.90th=[501220], 99.95th=[526386],
     | 99.99th=[526386]
   bw (  KiB/s): min= 2011, max=37632, per=6.51%, avg=17879.18, stdev=7286.28, samples=119
   iops        : min=   31, max=  588, avg=279.03, stdev=113.90, samples=119
  lat (usec)   : 20=0.08%, 50=0.46%, 100=0.10%, 250=0.33%, 500=1.89%
  lat (usec)   : 750=0.91%, 1000=0.84%
  lat (msec)   : 2=4.32%, 4=12.60%, 10=38.55%, 20=20.88%, 50=4.04%
  lat (msec)   : 500=14.81%, 750=0.19%
  cpu          : usr=0.54%, sys=0.27%, ctx=2780, majf=0, minf=1
  IO depths    : 1=5.4%, 2=11.7%, 4=24.2%, 8=51.1%, 16=7.7%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=94.1%, 8=0.1%, 16=5.7%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,16750,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
random-write: (groupid=0, jobs=1): err= 0: pid=1797: Wed Jul 26 15:14:54 2023
  write: IOPS=274, BW=17.2MiB/s (18.0MB/s)(1034MiB/60237msec); 0 zone resets
    slat (usec): min=2, max=23433, avg=92.28, stdev=427.66
    clat (usec): min=11, max=528438, avg=57320.11, stdev=116573.05
     lat (usec): min=111, max=528500, avg=57412.39, stdev=116540.99
    clat percentiles (usec):
     |  1.00th=[   159],  5.00th=[   930], 10.00th=[  1942], 20.00th=[  3752],
     | 30.00th=[  5276], 40.00th=[  6849], 50.00th=[  8356], 60.00th=[ 10421],
     | 70.00th=[ 13304], 80.00th=[ 20055], 90.00th=[316670], 95.00th=[333448],
     | 99.00th=[375391], 99.50th=[429917], 99.90th=[505414], 99.95th=[526386],
     | 99.99th=[526386]
   bw (  KiB/s): min= 8192, max=45988, per=6.48%, avg=17797.44, stdev=6499.65, samples=118
   iops        : min=  128, max=  718, avg=277.74, stdev=101.54, samples=118
  lat (usec)   : 20=0.14%, 50=0.44%, 100=0.23%, 250=0.57%, 500=1.83%
  lat (usec)   : 750=0.99%, 1000=1.03%
  lat (msec)   : 2=5.12%, 4=10.91%, 10=36.91%, 20=21.74%, 50=4.90%
  lat (msec)   : 500=15.00%, 750=0.19%
  cpu          : usr=0.55%, sys=0.27%, ctx=3077, majf=0, minf=1
  IO depths    : 1=5.3%, 2=11.6%, 4=24.3%, 8=51.2%, 16=7.5%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=94.1%, 8=0.1%, 16=5.8%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,16538,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
random-write: (groupid=0, jobs=1): err= 0: pid=1798: Wed Jul 26 15:14:54 2023
  write: IOPS=265, BW=16.6MiB/s (17.4MB/s)(998MiB/60236msec); 0 zone resets
    slat (usec): min=2, max=15253, avg=96.18, stdev=423.66
    clat (usec): min=12, max=526637, avg=59389.86, stdev=118369.89
     lat (usec): min=69, max=529144, avg=59486.04, stdev=118335.92
    clat percentiles (usec):
     |  1.00th=[   215],  5.00th=[  1221], 10.00th=[  2409], 20.00th=[  4146],
     | 30.00th=[  5604], 40.00th=[  6915], 50.00th=[  8455], 60.00th=[ 10683],
     | 70.00th=[ 13829], 80.00th=[ 20055], 90.00th=[320865], 95.00th=[329253],
     | 99.00th=[379585], 99.50th=[425722], 99.90th=[526386], 99.95th=[526386],
     | 99.99th=[526386]
   bw (  KiB/s): min= 4071, max=37401, per=6.20%, avg=17023.50, stdev=6842.57, samples=119
   iops        : min=   63, max=  584, avg=265.66, stdev=106.94, samples=119
  lat (usec)   : 20=0.13%, 50=0.35%, 100=0.19%, 250=0.41%, 500=1.52%
  lat (usec)   : 750=0.75%, 1000=0.90%
  lat (msec)   : 2=3.63%, 4=11.07%, 10=37.60%, 20=23.49%, 50=4.22%
  lat (msec)   : 500=15.53%, 750=0.20%
  cpu          : usr=0.62%, sys=0.17%, ctx=2858, majf=0, minf=1
  IO depths    : 1=5.4%, 2=11.7%, 4=24.5%, 8=50.9%, 16=7.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=94.1%, 8=0.1%, 16=5.8%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,15965,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
random-write: (groupid=0, jobs=1): err= 0: pid=1799: Wed Jul 26 15:14:54 2023
  write: IOPS=267, BW=16.7MiB/s (17.5MB/s)(1007MiB/60236msec); 0 zone resets
    slat (nsec): min=1647, max=18765k, avg=91876.28, stdev=419277.15
    clat (usec): min=10, max=527975, avg=58909.07, stdev=118071.19
     lat (usec): min=117, max=528068, avg=59000.94, stdev=118039.41
    clat percentiles (usec):
     |  1.00th=[   190],  5.00th=[  1090], 10.00th=[  2147], 20.00th=[  3884],
     | 30.00th=[  5473], 40.00th=[  6849], 50.00th=[  8356], 60.00th=[ 10683],
     | 70.00th=[ 13829], 80.00th=[ 20841], 90.00th=[320865], 95.00th=[329253],
     | 99.00th=[379585], 99.50th=[429917], 99.90th=[492831], 99.95th=[526386],
     | 99.99th=[526386]
   bw (  KiB/s): min= 2039, max=43816, per=6.26%, avg=17187.39, stdev=7226.37, samples=119
   iops        : min=   31, max=  684, avg=268.18, stdev=112.93, samples=119
  lat (usec)   : 20=0.16%, 50=0.32%, 100=0.18%, 250=0.58%, 500=1.85%
  lat (usec)   : 750=0.84%, 1000=0.89%
  lat (msec)   : 2=4.42%, 4=11.51%, 10=36.58%, 20=22.03%, 50=5.06%
  lat (msec)   : 500=15.49%, 750=0.10%
  cpu          : usr=0.55%, sys=0.24%, ctx=2765, majf=0, minf=1
  IO depths    : 1=5.4%, 2=11.6%, 4=24.4%, 8=51.1%, 16=7.5%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=94.1%, 8=0.2%, 16=5.8%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,16109,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
random-write: (groupid=0, jobs=1): err= 0: pid=1800: Wed Jul 26 15:14:54 2023
  write: IOPS=265, BW=16.6MiB/s (17.4MB/s)(1000MiB/60219msec); 0 zone resets
    slat (usec): min=2, max=52864, avg=97.78, stdev=573.40
    clat (usec): min=10, max=503765, avg=59232.60, stdev=117990.65
     lat (usec): min=62, max=503785, avg=59330.38, stdev=117955.13
    clat percentiles (usec):
     |  1.00th=[   206],  5.00th=[  1123], 10.00th=[  2114], 20.00th=[  3884],
     | 30.00th=[  5473], 40.00th=[  6980], 50.00th=[  8586], 60.00th=[ 10945],
     | 70.00th=[ 14091], 80.00th=[ 21103], 90.00th=[320865], 95.00th=[333448],
     | 99.00th=[379585], 99.50th=[429917], 99.90th=[467665], 99.95th=[505414],
     | 99.99th=[505414]
   bw (  KiB/s): min= 2035, max=37146, per=6.21%, avg=17060.85, stdev=6609.13, samples=119
   iops        : min=   31, max=  580, avg=266.24, stdev=103.30, samples=119
  lat (usec)   : 20=0.13%, 50=0.35%, 100=0.28%, 250=0.37%, 500=1.64%
  lat (usec)   : 750=0.71%, 1000=0.99%
  lat (msec)   : 2=4.81%, 4=11.53%, 10=35.50%, 20=23.08%, 50=4.85%
  lat (msec)   : 100=0.07%, 500=15.60%, 750=0.10%
  cpu          : usr=0.54%, sys=0.25%, ctx=2921, majf=0, minf=1
  IO depths    : 1=5.5%, 2=11.8%, 4=24.4%, 8=51.0%, 16=7.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=94.1%, 8=0.2%, 16=5.7%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,16004,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
random-write: (groupid=0, jobs=1): err= 0: pid=1801: Wed Jul 26 15:14:54 2023
  write: IOPS=246, BW=15.4MiB/s (16.2MB/s)(928MiB/60235msec); 0 zone resets
    slat (usec): min=2, max=27052, avg=90.44, stdev=471.55
    clat (usec): min=12, max=527206, avg=63941.46, stdev=121992.34
     lat (usec): min=135, max=529699, avg=64031.91, stdev=121959.35
    clat percentiles (usec):
     |  1.00th=[   227],  5.00th=[  1090], 10.00th=[  2507], 20.00th=[  4359],
     | 30.00th=[  5866], 40.00th=[  7504], 50.00th=[  9765], 60.00th=[ 12125],
     | 70.00th=[ 15401], 80.00th=[ 24773], 90.00th=[320865], 95.00th=[333448],
     | 99.00th=[379585], 99.50th=[463471], 99.90th=[526386], 99.95th=[526386],
     | 99.99th=[526386]
   bw (  KiB/s): min= 2039, max=35089, per=5.76%, avg=15834.72, stdev=5966.46, samples=119
   iops        : min=   31, max=  548, avg=247.08, stdev=93.23, samples=119
  lat (usec)   : 20=0.10%, 50=0.32%, 100=0.18%, 250=0.53%, 500=2.09%
  lat (usec)   : 750=0.76%, 1000=0.81%
  lat (msec)   : 2=3.18%, 4=9.72%, 10=33.06%, 20=25.75%, 50=6.48%
  lat (msec)   : 100=0.11%, 500=16.70%, 750=0.22%
  cpu          : usr=0.53%, sys=0.20%, ctx=2417, majf=0, minf=1
  IO depths    : 1=5.4%, 2=11.7%, 4=24.2%, 8=50.8%, 16=7.8%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=94.2%, 8=0.1%, 16=5.7%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,14853,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
random-write: (groupid=0, jobs=1): err= 0: pid=1802: Wed Jul 26 15:14:54 2023
  write: IOPS=261, BW=16.3MiB/s (17.1MB/s)(984MiB/60219msec); 0 zone resets
    slat (nsec): min=1644, max=13969k, avg=94335.77, stdev=407603.33
    clat (usec): min=10, max=530058, avg=60270.86, stdev=119198.69
     lat (usec): min=67, max=532215, avg=60365.19, stdev=119165.12
    clat percentiles (usec):
     |  1.00th=[   212],  5.00th=[   971], 10.00th=[  2442], 20.00th=[  4113],
     | 30.00th=[  5735], 40.00th=[  6980], 50.00th=[  8455], 60.00th=[ 10552],
     | 70.00th=[ 13829], 80.00th=[ 21890], 90.00th=[320865], 95.00th=[333448],
     | 99.00th=[379585], 99.50th=[438305], 99.90th=[530580], 99.95th=[530580],
     | 99.99th=[530580]
   bw (  KiB/s): min= 2039, max=36771, per=6.11%, avg=16783.76, stdev=6898.44, samples=119
   iops        : min=   31, max=  574, avg=261.92, stdev=107.83, samples=119
  lat (usec)   : 20=0.15%, 50=0.33%, 100=0.22%, 250=0.47%, 500=1.84%
  lat (usec)   : 750=1.11%, 1000=1.00%
  lat (msec)   : 2=3.32%, 4=10.91%, 10=38.91%, 20=20.75%, 50=4.92%
  lat (msec)   : 100=0.10%, 500=15.86%, 750=0.10%
  cpu          : usr=0.58%, sys=0.19%, ctx=2694, majf=0, minf=1
  IO depths    : 1=5.6%, 2=11.8%, 4=24.3%, 8=50.7%, 16=7.6%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=94.2%, 8=0.1%, 16=5.8%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,15740,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
random-write: (groupid=0, jobs=1): err= 0: pid=1803: Wed Jul 26 15:14:54 2023
  write: IOPS=255, BW=16.0MiB/s (16.7MB/s)(961MiB/60219msec); 0 zone resets
    slat (nsec): min=1661, max=65749k, avg=96158.72, stdev=687869.49
    clat (usec): min=11, max=502100, avg=61651.42, stdev=120522.78
     lat (usec): min=79, max=503684, avg=61747.58, stdev=120488.95
    clat percentiles (usec):
     |  1.00th=[   217],  5.00th=[  1090], 10.00th=[  2278], 20.00th=[  4080],
     | 30.00th=[  5669], 40.00th=[  7046], 50.00th=[  8717], 60.00th=[ 11076],
     | 70.00th=[ 15270], 80.00th=[ 22676], 90.00th=[320865], 95.00th=[333448],
     | 99.00th=[379585], 99.50th=[459277], 99.90th=[501220], 99.95th=[501220],
     | 99.99th=[501220]
   bw (  KiB/s): min= 2035, max=44835, per=5.97%, avg=16390.27, stdev=7472.30, samples=119
   iops        : min=   31, max=  700, avg=255.75, stdev=116.78, samples=119
  lat (usec)   : 20=0.14%, 50=0.42%, 100=0.14%, 250=0.44%, 500=2.09%
  lat (usec)   : 750=0.64%, 1000=0.77%
  lat (msec)   : 2=4.03%, 4=10.85%, 10=36.60%, 20=22.00%, 50=5.54%
  lat (msec)   : 500=16.23%, 750=0.10%
  cpu          : usr=0.41%, sys=0.34%, ctx=2660, majf=0, minf=1
  IO depths    : 1=5.4%, 2=11.6%, 4=24.2%, 8=51.0%, 16=7.8%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=94.1%, 8=0.1%, 16=5.7%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,15378,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=268MiB/s (281MB/s), 15.4MiB/s-17.9MiB/s (16.2MB/s-18.7MB/s), io=15.8GiB (16.9GB), run=60219-60243msec

While temp is dropped to the point it doesn't bother me anymore, the SSD performence looks like kind of strange. I'm scratching my head whether this is test specifics (fio) or the FreeBSD in general and probably can be tuned out. Cannot make the sense out of it.

So, if anyone wonders whether one should order the internal fan along with the box I would highly recommend to do that. Maybe I will order the 80x80x10 4pin fan later from Ali as well and remove the cooper slugs to test the temp on SSD. But maybe no. Will see.

lucker · Jul 26, 2023

It seems like tor some reason the forum limiting the number of attached files.
Here the screenshots for testing
With the same default BIOS settings I tested both NVMe slots: native (on MB, x4) and on H-type PCB (x1). This test performed on the open upended box with the ambient t=25C
in Windows x4

in Windows x1

Mike9474593 · Jul 26, 2023

@athurdent What type of board do you have? is there any indication on the board, like CW-ADLN-something? Or do you have a photo of the board? Would be very interesting what kind of board you got there.

athurdent · Jul 26, 2023

Mike9474593 said:
@athurdent What type of board do you have? is there any indication on the board, like CW-ADLN-something? Or do you have a photo of the board? Would be very interesting what kind of board you got there.

Looks like this, I'll return it and order a N305 one without all those USB / HDMI extras.

Edit, sorry, here's the sticker, accidentally redacted it

Mike9474593 · Jul 26, 2023

athurdent said:
Looks like this, I'll return it and order a N305 one without all those USB / HDMI extras.

View attachment 30529

Cool, thanks. Can you remove the RAM and the SSD maybe? I really don't get what version you got here...

athurdent · Jul 26, 2023

Mike9474593 said:
Cool, thanks. Can you remove the RAM and the SSD maybe? I really don't get what version you got here...

Sorry, see my edit

Mike9474593 · Jul 26, 2023

Thanks. The CW-ADLN-4L version, explains why DDR5 works

Probably CWWKs table is wrong, because it says your version has only 1 USB 3.0 Port. I'm asking myself if they run out of PCI lanes becasue of the additional USB 3.0 ports and do some PCI switching or sharing somewhere. This would explain your performance problem...

athurdent · Jul 26, 2023

Mike9474593 said:
Thanks. The CW-ADLN-4L version, explains why DDR5 works
Probably CWWKs table is wrong, because it says your version has only 1 USB 3.0 Port. I'm asking myself if they run out of PCI lanes becasue of the additional USB 3.0 ports and do some PCI switching or sharing somewhere. This would explain your performance problem...

Well, interesting development. They came back to me with a new BIOS, there was a parameter mismatch in PCIE LAN1 according to their engineering. BIOS file is around 20 days old, so this was probably already a known issue.

And after a long journey with the Ventoy tool they are leveraging and me working with macOS, plus figuring out what to type to actually flash the BIOS as their autostart did not autostart (I am getting too old for this, it's most likely just me

)

Voila:

Code:

[    2.700099] igc 0000:01:00.0: enabling device (0000 -> 0002)
[    2.704288] igc 0000:01:00.0: PTM enabled, 4ns granularity
[    2.752344] igc 0000:01:00.0 (unnamed net_device) (uninitialized): PHC added
[    2.777427] igc 0000:01:00.0: 4.000 Gb/s available PCIe bandwidth (5.0 GT/s PCIe x1 link)
[    2.777435] igc 0000:01:00.0 eth0: MAC: ***
[    2.777480] igc 0000:02:00.0: enabling device (0000 -> 0002)
[    2.777833] igc 0000:02:00.0: PTM enabled, 4ns granularity
[    2.825595] igc 0000:02:00.0 (unnamed net_device) (uninitialized): PHC added
[    2.851034] igc 0000:02:00.0: 4.000 Gb/s available PCIe bandwidth (5.0 GT/s PCIe x1 link)
[    2.851044] igc 0000:02:00.0 eth1: MAC: ***
[    2.851094] igc 0000:03:00.0: enabling device (0000 -> 0002)
[    2.851239] igc 0000:03:00.0: PTM enabled, 4ns granularity
[    2.898874] igc 0000:03:00.0 (unnamed net_device) (uninitialized): PHC added
[    2.924024] igc 0000:03:00.0: 4.000 Gb/s available PCIe bandwidth (5.0 GT/s PCIe x1 link)
[    2.924033] igc 0000:03:00.0 eth2: MAC: ***
[    2.924072] igc 0000:04:00.0: enabling device (0000 -> 0002)
[    2.924214] igc 0000:04:00.0: PTM enabled, 4ns granularity
[    2.971769] igc 0000:04:00.0 (unnamed net_device) (uninitialized): PHC added
[    2.996011] igc 0000:04:00.0: 4.000 Gb/s available PCIe bandwidth (5.0 GT/s PCIe x1 link)
[    2.996020] igc 0000:04:00.0 eth3: MAC: ***

iperf3 results are fine now, full 2.5Gbps.

I can really recommend HUNSN support at this point, quick and friendly!

Now for some stability testing, hope that change did not break anything. Running with Optimized BIOS defaults for now.

athurdent · Jul 27, 2023

So, while I have probably found a fan that would fit inside the HUNSN, I ordered an adapter cable with a plug that is far too big. Does anybody know how these mini mini four pin connectors are called? (It's smaller than it looks).
Also, as this connector is mounted far too close to the edge of the board, ultra thin & flexible cables would be needed.

alexw1982 · Jul 27, 2023

lucker said:
With the same default BIOS settings I tested both NVMe slots: native (on MB, x4) and on H-type PCB (x1). This test was performed on the open upended box with the ambient t=25C
in Windows x4
View attachment 30507 View attachment 30508
in Windows x1
View attachment 30509 View attachment 30510
in Ubuntu x4 then x1
View attachment 30512 View attachment 30511 View attachment 30513

I might be wrong but shouldnt the x4 NVME disk be much faster in the individual benchmark than the x1 connected? Looking at the numbers they both look like x1 connected drives (1Gb/s max) the mirrored speed could be so high due to the caching of zfs (I could be wrong though)

CWWK/Topton/... Nxxx quad NIC router

Member

New Member

Member

Member

Member

Member

Member

New Member

Member

Member

Member

Member

Member

Member

Member

Member

Member

Member

Member

Member