Problems making SR-IOV work in Proxmox for Mellanox CX3

[kevindd992002]

I'm reading in some threads here that I can enable SR-IOV with the built-in mlx driver (as confirmed by @fohdeesha too) of the CX3 in Proxmox/Debian. So no need to install the aggressive MLNX_OFED driver from Nvidia and here's what I tried doing:

wget https://www.mellanox.com/downloads/MFT/mft-4.26.1-3-x86_64-deb.tgz
apt-get install gcc make dkms pve-headers
tar -xvzf mft-4.22.1-11-x86_64-deb.tgz
cd mft-4.22.1-11-x86_64-deb/
./install.sh
mst start

As soon as I try doing mst start, I get this:

Starting MST (Mellanox Software Tools) driver set
Loading MST PCI modulemodprobe: ERROR: could not insert 'mst_pci': Key was rejected by service
- Failure: 1
Loading MST PCI configuration modulemodprobe: ERROR: could not insert 'mst_pciconf': Key was rejected by service
- Failure: 1
Create devices

mst_pci driver not found
Unloading MST PCI module (unused) - Success
Unloading MST PCI configuration module (unused) - Success

Why can it not start? Probably because of secure boot being enabled on the host? If so, can I temporarily disable it while I try enabling SR-IOV and then re-enable it back afterwards?

Here's the output of lspci for reference:

02:00.0 Ethernet controller: Mellanox Technologies MT27500 Family [ConnectX-3]
Subsystem: Mellanox Technologies ConnectX-3 10 GbE Single Port SFP+ Adapter
Flags: bus master, fast devsel, latency 0, IRQ 16, IOMMU group 8
Memory at f7100000 (64-bit, non-prefetchable) [size=1M]
Memory at f2800000 (64-bit, prefetchable) [size=8M]
Expansion ROM at f7000000 [disabled] [size=1M]
Capabilities: [40] Power Management version 3
Capabilities: [48] Vital Product Data
Capabilities: [9c] MSI-X: Enable+ Count=128 Masked-
Capabilities: [60] Express Endpoint, MSI 00
Capabilities: [c0] Vendor Specific Information: Len=18 <?>
Capabilities: [100] Alternative Routing-ID Interpretation (ARI)
Capabilities: [148] Device Serial Number 24-8a-07-03-00-79-4b-e0
Capabilities: [154] Advanced Error Reporting
Capabilities: [18c] Secondary PCI Express
Kernel driver in use: mlx4_core
Kernel modules: mlx4_core

 

[heromode]

From what i understand you need to disable secureboot in BIOS. And if you want it enabled you must sign the modules. And you must sign all modules after every kernel upgrade. And you must sign any future modules you install. There are lots of guides how to do that, but in the realworld, secureboot is a hot mess, and most people just disable it in linux. Unless you have a very tightly maintained server that only runs specific software, you never try new things, and you maintain a very strict upgrade routine, that you are able to test somewhere else before each upgrade of your production server, it's a nightmare. Just my notion..

I remember trying to read up on how to run proxmox with secureboot, and after a few hours of reading my brain melted, and it became clear that it's aking to masochism in a home use setting.

 

[kevindd992002]

That's what I thought. So Proxmox just recently supported secure boot. It didn't before, which is probably the time when you were reading up on it. When I installed 8.1 now, it "just worked", until I realized how unsigned modules can cause this. Oh well, I hope temporarily disabling it doesn't do any bad effect on my system (I'm assuming not).

 

[mattventura]

The settings you change with the MST tools should mostly persist on the card itself, so you can disable secure boot, make your changes, then re-enable. You can also put the card in another system to configure it, then put it back in the original system.

 

[heromode]

Thing with secureboot is, whenever something goes wrong (and it will, unless every installer and .deb package etc you run is compatible with latest debian secureboot specs), and you reboot your server, it will fail. That means you need to reboot into bios, disable secureboot, save and reboot, and try to fix the issue. Once you have made some change, it's reboot into bios, enable secureboot, save and reboot, and try again. If it works, nice, if not, then reboot into bios, disable secureboot, save and reboot, and try again. Then reboot into bios, enable secureboot, save and reboot, and try again. And on and on and on

Prepare to be spending ALOT of time, hours upon hours looking at your POST screen. If remote, your only option is doing all this rebooting using IPMI or something. And if that fails through these numerous boots, you're screwed.

Everytime you do apt-get dist-upgrade, you will know you might be in for hours of work, and 10 or 20 reboots. Over and over and over again.
that's why secureboot is a mess. It's useless in actual real life scenarios, unless you are a masochist.

edit: Every fix requires atleast FOUR reboots: reboot #1 into BIOS, disable secureboot, save and reboot #2 into OS, fix the issue. Reboot #3 into bios, re-enable secureboot, save and reboot#4 into OS.

If it doesn't work you're looking at atleast 4 more reboots. Now you're at 8 reboots, and you forgot something. That's 4 more reboots. Now you're at 12 reboots. You get the point

Anything not tested by the proxmox team is a potential risk. Every .DEB file, every ./install script. Most of them will know NOTHING about needing to sign the module during update-initramfs etc.. not to mention proxmox has it's own custom scripts now for UEFI / root on ZFS systems.

It's just LOL.

 

[kevindd992002]

Ok, so I disabled secure boot and then loaded and got this:

root@pve:~# mst start
Starting MST (Mellanox Software Tools) driver set
Loading MST PCI module - Success
Loading MST PCI configuration module - Success
Create devices
Unloading MST PCI module (unused) - Success
Unloading MST PCI configuration module (unused) - Success
root@pve:~# mst status
MST modules:
------------
MST PCI module is not loaded
MST PCI configuration module is not loaded

PCI Devices:
------------

No devices were found.


root@pve:~#

Why aren't the modules stay being loaded?

 

[kevindd992002]

Ok. I was able to load the mst module now and the status of the NIC. The solution was to use the older version of MFT (4.22.1-307-LTS) because 4.26.1-LTS no longer supports CX3. But then it fails to query the current configuration of the NIC.

root@pve:~# mst start
Starting MST (Mellanox Software Tools) driver set
Loading MST PCI module - Success
Loading MST PCI configuration module - Success
Create devices
root@pve:~# mst status -v
MST modules:
------------
MST PCI module loaded
MST PCI configuration module loaded
PCI devices:
------------
DEVICE_TYPE MST PCI RDMA NET NUMA
ConnectX3(rev:1) /dev/mst/mt4099_pciconf0 02:00.0 mlx4_0 net-enp2s0 -1

ConnectX3(rev:1) /dev/mst/mt4099_pci_cr0 02:00.0 mlx4_0 net-enp2s0 -1

root@pve:~# mlxconfig -d /dev/mst/mt4099_pciconf0 q

Device #1:
----------

Device type: ConnectX3
Device: /dev/mst/mt4099_pciconf0

Configurations: Next Boot
-E- Failed to query device current configuration

 

[klui]

When I installed Proxmox 8 I used MFT 4.25.1 and it works fine with a CX3 after a dist-upgrade. But I don't use secure boot.

 

[kevindd992002]

Are you saying you were able to use mxlconfig to set the SRIOV parameters on the card? I have secure boot disabled now so I was able to load the mst modules successfully as you see. But I can't use mxlconfig when I try to set the SRIOV parameters. Similar to the problem explained here:

Issue with Connectx-3 - Failed to query device current configuration

Hi All, I’ve a connectx-3 with FW 2.42.5000, I’m trying to modify the config of the card however everytime I’m checking the configuration I got an Error to check the configuration on the card even after reseting the config too. Any idea how to solve this problem? flint -d...

forums.developer.nvidia.com

I already tried resetting the NIC using the flint command there but same issue. Same result even with the mstconfig command (from the mstflint package from the debian sources).

 

[klui]

I can query the card's parameters w/out any problem. I have SR-IOV working on a system that doesn't have any SR-IOV setting in the BIOS.

 

[kevindd992002]

Do you know where to get 4.25.1? It'a no longer in nvidia's website. What did you mean by "after a dist-upgrade"? Did you have to update Proxmox first before it worked?

 

[mach3.2]

It's under archived version.

NVIDIA Firmware Tools (MFT)

The NVIDIA Firmware Tools (MFT) package is a set of firmware management tools

network.nvidia.com

Direct link for x64 debian .deb:

https://www.mellanox.com/downloads/MFT/mft-4.25.1-11-x86_64-deb.tgz

 

[kevindd992002]

Same issue with 4.25.1-11. I have two of the same card (MCX311A-XCAT) that has the same issue. This says that it is a known issue with this card:

https://forums.servethehome.com/index.php?threads/connectx-3-custom-firmware-files.39457/

 

[klui]

No, it worked before. I just performed a dist-upgrade before I tried again.

EDIT: I don't recall changing the SRIOV settings using mlxconfig. At the most only link type setting to ETH.

My card is an MCX354A-FCBT (A2-A5)

Maybe it's due to Proxmox 8.1's kernel? This system is still on 8. BTW newer kernels like Proxmox 8's prevents ethtool from obtaining CX3s' module information.

root@pve8:~# uname -a
Linux pve8 6.5.11-7-pve #1 SMP PREEMPT_DYNAMIC PMX 6.5.11-7 (2023-12-05T09:44Z) x86_64 GNU/Linux
root@pve8:~/mft-4.25.1-11-x86_64-deb# mst version
mst, mft 4.25.1-11, built on Sep 18 2023, 11:50:10. Git SHA Hash: N/A
root@pve8:~# mlxconfig -d /dev/mst/mt4099_pci_cr0 q

Device #1:
----------

Device type:    ConnectX3
Device:         /dev/mst/mt4099_pci_cr0

Configurations:                                      Next Boot
        SRIOV_EN                                    True(1)
        NUM_OF_VFS                                  8
        LINK_TYPE_P1                                ETH(2)
        LINK_TYPE_P2                                ETH(2)
        LOG_BAR_SIZE                                3
        BOOT_PKEY_P1                                0
        BOOT_PKEY_P2                                0
        BOOT_OPTION_ROM_EN_P1                       True(1)
        BOOT_VLAN_EN_P1                             False(0)
        BOOT_RETRY_CNT_P1                           0
        LEGACY_BOOT_PROTOCOL_P1                     PXE(1)
        BOOT_VLAN_P1                                1
        BOOT_OPTION_ROM_EN_P2                       True(1)
        BOOT_VLAN_EN_P2                             False(0)
        BOOT_RETRY_CNT_P2                           0
        LEGACY_BOOT_PROTOCOL_P2                     PXE(1)
        BOOT_VLAN_P2                                1
        IP_VER_P1                                   IPv4(0)
        IP_VER_P2                                   IPv4(0)
        CQ_TIMESTAMP                                True(1)
root@pve8:~# mlxfwmanager
Querying Mellanox devices firmware ...

Device #1:
----------

  Device Type:      ConnectX3
  Part Number:      MCX354A-FCB_A2-A5
  Description:      ConnectX-3 VPI adapter card; dual-port QSFP; FDR IB (56Gb/s) and 40GigE; PCIe3.0 x8 8GT/s; RoHS R6
  PSID:             MT_1090120019

 

[kevindd992002]

Ahh, that's probably why. We have a different CX3. It's not because of Proxmox 8.1's new kernel because I tried this on a windows 11 machine too (with another NIC of the same model) and same issue. There's a handful of threads having the same issue with this single port version of the CX3 and it looks like the solution is to dump the ini from the card, edit it with the sriov settings, and compile a custom firmware image with the edited ini, then burn that fw image over the card. The problem is I don't know where to get the latest custom fw image that I can use.

 

[kevindd992002]

I just found why this is happening. It seems to be an issues with the mft tools and the single port CX3 (as discussed here). The solution is to generate an image with a modified ini (to enable sriov) but the mlx fw file you can use with that is older (2.40.5030) than the latest available bin (2.42.5000). I did just that and it worked. I can see the vf's in proxmox now.

However, is it worth running an older fw to make sriov work or maybe just replace this card with a CX4 or even a dual port CX3 that don't exhibit this issue? With these cards, you can simply use mlxconfig to edit the fw settings directly without needing to generate a modified image and reflashing.

@fohdeesha , thoughts?