[Promox VE] LUKS + ZFS - Extremely slow NVMe Root Pool

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.
L

luckylinux

Well-Known Member
Mar 18, 2012
1,840
612
113
TrevorH said:
What do you get if you run dmesg -T | grep -i x2a ? What about lspci -vvv -s bb:cc.x | grep Lnk where bb:cc.x is the number from the lefthand side of the line in lspci -nn that is for the nvme drive? Does that change when it's quick vs slow?
Click to expand...
Now with 2 x Micron 7300 MAX 3.2TB it's FAST (> 300 MiB/s) zfs send | zfs receive whereas yesterday it was < 50 MiB/s:

Code: 
root@pve:/# dmesg -T | grep -i x2a
[Sat Sep 26 08:02:33 2026] AMD-Vi: Extended features (0x58f77ef22294ade, 0x0): PPR X2APIC NX GT IA GA PC GA_vAPIC
[Sat Sep 26 08:02:33 2026] AMD-Vi: X2APIC enabled

root@pve:/ lspci | grep -i NVMe
81:00.0 Non-Volatile memory controller: Micron Technology Inc 7300 MAX NVMe SSD (rev 01)
c2:00.0 Non-Volatile memory controller: Micron Technology Inc 7300 MAX NVMe SSD (rev 01)

root@pve:/# lspci -vvv -s 81:00.0 | grep Lnk
        LnkCap:    Port #0, Speed 8GT/s, Width x4, ASPM not supported
        LnkCtl:    ASPM Disabled; RCB 64 bytes, LnkDisable- CommClk+
        LnkSta:    Speed 8GT/s, Width x4
        LnkCap2: Supported Link Speeds: 2.5-8GT/s, Crosslink- Retimer- 2Retimers- DRS-
        LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
        LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete+ EqualizationPhase1+
        LnkCtl3: LnkEquIntrruptEn- PerformEqu-

root@pve:/# lspci -vvv -s c2:00.0 | grep Lnk
        LnkCap:    Port #0, Speed 8GT/s, Width x4, ASPM not supported
        LnkCtl:    ASPM Disabled; RCB 64 bytes, LnkDisable- CommClk+
        LnkSta:    Speed 8GT/s, Width x4
        LnkCap2: Supported Link Speeds: 2.5-8GT/s, Crosslink- Retimer- 2Retimers- DRS-
        LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
        LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete+ EqualizationPhase1+
        LnkCtl3: LnkEquIntrruptEn- PerformEqu-
 
jhartbarger

jhartbarger

Active Member
Jan 5, 2022
127
168
43
luckylinux said:
I also tried running a test on /boot (ext4 on top of mdadm), much better Results, so probably something between LUKS and/or ZFS is to blame:
Click to expand...
This is your likely culprit that something is going on in the virtualization layer that's causing your performance issues
 
jhartbarger

jhartbarger

Active Member
Jan 5, 2022
127
168
43
luckylinux said:
Yeah, but for "standard" LXC Templates I think 128k is a good Default, or ? Or should I set that down to 16k ? Google Gemini seemed to suggest that high Latency and write Amplification are caused by this high 128k Value.
Click to expand...
Based on what you have said I would agree that you should and what I would do is create a minimum of two different datasets, one that holds your containers, and another for userdata that is configured optimally for its use case.

For example my Plex Server container contains mounts to 4 different datasets, (config files, database, logs and media) all with different record sizes and or dataset configs and that's not even to mention the mount to the /tempfs ram drive for temp transcoding.
 
Last edited:
M

mattventura

Well-Known Member
Nov 9, 2022
799
446
63
Why LUKS instead of native ZFS encryption? ZFS always wants to be as close to bare metal disks as possible.

I also notice your two drives have very different PCIe addresses. Is one of them maybe going through the chipset while the other is connected directly to the CPU? Unlikely but could be bottlenecking somewhere.
 
L

luckylinux

Well-Known Member
Mar 18, 2012
1,840
612
113
jhartbarger said:
Based on what you have said I would agree that you should and what I would do is create a minimum of two different datasets, one that holds your containers, and another for userdata that is configured optimally for its use case.

For example my Plex Server container contains mounts to 4 different datasets, (config files, database, logs and media) all with different record sizes and or dataset configs and that's not even to mention the mount to the /tempfs ram drive for temp transcoding.
Click to expand...
Funnily enough a couple of Years ago I had a Virtual Disk formatted as ZFS Pool inside the KVM Guest (inside a VM) so I could do exactly that.

Turns out that Performance an Write Amplification was an absolute Nightmare :(.

You could do that with LXC Containers and Mount-Points though, just not worth, if you have lots of small-ish Podman Containers (say < 1 GB Data).
 
L

luckylinux

Well-Known Member
Mar 18, 2012
1,840
612
113
mattventura said:
Why LUKS instead of native ZFS encryption? ZFS always wants to be as close to bare metal disks as possible.
Click to expand...
Where should I begin :) ?
  • Better standardized
  • I always use this
  • Supported since a longer Time (older)
  • Compatible with openzfs-2.0-linux (yes, I don't want shiny new Features, please give me something Reliable *)
  • Unattended unlocking with clevis + tang

(*): minus a LUKS <-> ZFS Alignment Bug around 1-2 Years ago which was giving some intermittent Issues when ZFS Pool was nearly full, causing me to write this troubleshooting Tool and the well-known workaround at the Time of using a Loop Device for "standardizing" and aligning Writes

github.com

GitHub - luckylinux/workaround-zfs-on-luks-bug: workaround-zfs-on-luks-bug

workaround-zfs-on-luks-bug. Contribute to luckylinux/workaround-zfs-on-luks-bug development by creating an account on GitHub.
github.com github.com

(I really missed BASH back then ;))

ZFS wants to be as close as possible, though I also use Partitions, because unless you using an unencrypted ZFS Boot bpool Pool, which had yet another Bug with GRUB2 and I didn't want to learn ZFS boot menu Bootloader and don't want/can't use a separate USB Thumb Drive for Boot, then you really have no other Options.

The only Case where I would consider, though not sure if supported, an encrypted ZFS Pool, would be if I need to do a Remote Backup to a non-encrypted Location (Cloud Backup).

mattventura said:
I also notice your two drives have very different PCIe addresses. Is one of them maybe going through the chipset while the other is connected directly to the CPU? Unlikely but could be bottlenecking somewhere.
Click to expand...
Good Call, though I doubt it.

Code: 
root@pve:/# lspci -tv
-+-[0000:00]-+-00.0  Advanced Micro Devices, Inc. [AMD] Starship/Matisse Root Complex
|           +-00.2  Advanced Micro Devices, Inc. [AMD] Starship/Matisse IOMMU
|           +-01.0  Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
|           +-02.0  Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
|           +-03.0  Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
|           +-03.2-[01]----00.0  Marvell Technology Group Ltd. 88SE9235 PCIe 2.0 x2 4-port SATA 6 Gb/s Controller
|           +-03.5-[02-03]----00.0-[03]----00.0  ASPEED Technology, Inc. ASPEED Graphics Family
|           +-03.6-[04]--+-00.0  Broadcom Inc. and subsidiaries NetXtreme BCM5720 Gigabit Ethernet PCIe
|           |            \-00.1  Broadcom Inc. and subsidiaries NetXtreme BCM5720 Gigabit Ethernet PCIe
|           +-04.0  Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
|           +-05.0  Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
|           +-07.0  Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
|           +-07.1-[05]--+-00.0  Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Function
|           |            \-00.2  Advanced Micro Devices, Inc. [AMD] Starship/Matisse PTDMA
|           +-08.0  Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
|           +-08.1-[06]--+-00.0  Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP
|           |            +-00.2  Advanced Micro Devices, Inc. [AMD] Starship/Matisse PTDMA
|           |            \-00.3  Advanced Micro Devices, Inc. [AMD] Starship USB 3.0 Host Controller
|           +-14.0  Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller
|           +-14.3  Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge
|           +-18.0  Advanced Micro Devices, Inc. [AMD] Starship Device 24; Function 0
|           +-18.1  Advanced Micro Devices, Inc. [AMD] Starship Device 24; Function 1
|           +-18.2  Advanced Micro Devices, Inc. [AMD] Starship Device 24; Function 2
|           +-18.3  Advanced Micro Devices, Inc. [AMD] Starship Device 24; Function 3
|           +-18.4  Advanced Micro Devices, Inc. [AMD] Starship Device 24; Function 4
|           +-18.5  Advanced Micro Devices, Inc. [AMD] Starship Device 24; Function 5
|           +-18.6  Advanced Micro Devices, Inc. [AMD] Starship Device 24; Function 6
|           \-18.7  Advanced Micro Devices, Inc. [AMD] Starship Device 24; Function 7
+-[0000:40]-+-00.0  Advanced Micro Devices, Inc. [AMD] Starship/Matisse Root Complex
|           +-00.2  Advanced Micro Devices, Inc. [AMD] Starship/Matisse IOMMU
|           +-01.0  Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
|           +-02.0  Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
|           +-03.0  Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
|           +-04.0  Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
|           +-05.0  Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
|           +-07.0  Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
|           +-07.1-[41]--+-00.0  Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Function
|           |            \-00.2  Advanced Micro Devices, Inc. [AMD] Starship/Matisse PTDMA
|           +-08.0  Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
|           \-08.1-[42]--+-00.0  Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP
|                        +-00.1  Advanced Micro Devices, Inc. [AMD] Starship/Matisse Cryptographic Coprocessor PSPCPP
|                        +-00.2  Advanced Micro Devices, Inc. [AMD] Starship/Matisse PTDMA
|                        \-00.3  Advanced Micro Devices, Inc. [AMD] Starship USB 3.0 Host Controller
+-[0000:80]-+-00.0  Advanced Micro Devices, Inc. [AMD] Starship/Matisse Root Complex
|           +-00.2  Advanced Micro Devices, Inc. [AMD] Starship/Matisse IOMMU
|           +-01.0  Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
|           +-01.1-[81]----00.0  Micron Technology Inc 7300 MAX NVMe SSD
|           +-02.0  Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
|           +-03.0  Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
|           +-03.1-[82]----00.0  Mellanox Technologies MT27520 Family [ConnectX-3 Pro]
|           +-04.0  Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
|           +-05.0  Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
|           +-07.0  Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
|           +-07.1-[83]--+-00.0  Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Function
|           |            \-00.2  Advanced Micro Devices, Inc. [AMD] Starship/Matisse PTDMA
|           +-08.0  Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
|           +-08.1-[84]--+-00.0  Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP
|           |            \-00.2  Advanced Micro Devices, Inc. [AMD] Starship/Matisse PTDMA
|           +-08.2-[85]----00.0  Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode]
|           \-08.3-[86]----00.0  Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode]
\-[0000:c0]-+-00.0  Advanced Micro Devices, Inc. [AMD] Starship/Matisse Root Complex
             +-00.2  Advanced Micro Devices, Inc. [AMD] Starship/Matisse IOMMU
             +-01.0  Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
             +-01.2-[c1]--+-00.0  Mellanox Technologies MT27710 Family [ConnectX-4 Lx]
             |            \-00.1  Mellanox Technologies MT27710 Family [ConnectX-4 Lx]
             +-02.0  Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
             +-03.0  Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
             +-03.1-[c2]----00.0  Micron Technology Inc 7300 MAX NVMe SSD
             +-04.0  Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
             +-05.0  Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
             +-07.0  Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
             +-07.1-[c3]--+-00.0  Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Function
             |            \-00.2  Advanced Micro Devices, Inc. [AMD] Starship/Matisse PTDMA
             +-08.0  Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
             \-08.1-[c4]--+-00.0  Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP
                          \-00.2  Advanced Micro Devices, Inc. [AMD] Starship/Matisse PTDMA
For Reference: the Mellanox ConnectX-4 LX is built-in into the Hyve G10 Server (via OCP Mezzanine), the Mellanox ConnectX-3 Pro is a NIC I installed in the PCIe Slot (via PCIe Riser).

While this is quite confusing to me
Code: 
 |           +-08.2-[85]----00.0  Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode]

|           \-08.3-[86]----00.0  Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode]
The Chipset is IMHO the very first Tree (USB Controller, ASPEED BMC/IPMI, Gigabit Ethernet, SATA):
Code: 
-+-[0000:00]-+-00.0  Advanced Micro Devices, Inc. [AMD] Starship/Matisse Root Complex
 
Last edited:
L

luckylinux

Well-Known Member
Mar 18, 2012
1,840
612
113
jhartbarger said:
This is your likely culprit that something is going on in the virtualization layer that's causing your performance issues
Click to expand...
What do you mean by Virtualization Layer ? The Server is not really doing any hard Work right now.

It's highly inconsistent. I might do 5 Times a zfs send | zfs receive right now and they all achieve > 1000 MiB/s.

I try that in a few Minutes/Hours and it's 50 < MiB/s.
 
TrevorH

TrevorH

Active Member
Oct 25, 2024
311
135
43
If your drive is still dropping to 200 or 300MB/s when tested with a simple tool then you have a hardware problem.
 
TrevorH

TrevorH

Active Member
Oct 25, 2024
311
135
43
I think you need to gather data about the drives when they are not performing correctly.
 
You must log in or register to reply here.
Top Bottom