Proxmox - Mellanox ConnectX-3 VFs stuck in same SR-IOV group

[Psycho_Robotico]

Hi,

I'm trying to get SR-IOV working on Proxmox (followed the guide for enabling SR-IOV: PCI(e) Passthrough - Proxmox VE), but always end up with all virtual functions stuck in the same IOMMU group. Switching PCIe slots didn't help either:

IOMMU Group 48:
        40:03.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
        40:03.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge [1022:1483]
        41:00.0 Ethernet controller [0200]: Mellanox Technologies MT27500 Family [ConnectX-3] [15b3:1003]
        41:00.1 Ethernet controller [0200]: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function] [15b3:1004]
        41:00.2 Ethernet controller [0200]: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function] [15b3:1004]
        41:00.3 Ethernet controller [0200]: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function] [15b3:1004]
        41:00.4 Ethernet controller [0200]: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function] [15b3:1004]
        41:00.5 Ethernet controller [0200]: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function] [15b3:1004]
        41:00.6 Ethernet controller [0200]: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function] [15b3:1004]
        41:00.7 Ethernet controller [0200]: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function] [15b3:1004]
        41:01.0 Ethernet controller [0200]: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function] [15b3:1004]
        41:01.1 Ethernet controller [0200]: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function] [15b3:1004]
        41:01.2 Ethernet controller [0200]: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function] [15b3:1004]
        41:01.3 Ethernet controller [0200]: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function] [15b3:1004]
        41:01.4 Ethernet controller [0200]: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function] [15b3:1004]
        41:01.5 Ethernet controller [0200]: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function] [15b3:1004]
        41:01.6 Ethernet controller [0200]: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function] [15b3:1004]
        41:01.7 Ethernet controller [0200]: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function] [15b3:1004]
        41:02.0 Ethernet controller [0200]: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function] [15b3:1004]
        41:02.1 Ethernet controller [0200]: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function] [15b3:1004]
        41:02.2 Ethernet controller [0200]: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function] [15b3:1004]
        41:02.3 Ethernet controller [0200]: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function] [15b3:1004]
        41:02.4 Ethernet controller [0200]: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function] [15b3:1004]
        41:02.5 Ethernet controller [0200]: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function] [15b3:1004]
        41:02.6 Ethernet controller [0200]: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function] [15b3:1004]
        41:02.7 Ethernet controller [0200]: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function] [15b3:1004]
        41:03.0 Ethernet controller [0200]: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function] [15b3:1004]

My setup

• Proxmox 6.1-5 (Kernel: 5.3.13-2)
  
• Supermicro H11SSL
• AMD Epyc 7502
• Mellanox ConnectX-3

Any ideas?

 

[vangoose]

ConnectX-3 doesn't support SRV-IO last time I checked when I tried to enable it in VMware.
ConnectX-3 VPI supports it.

 

[Psycho_Robotico]

ConnectX-3 doesn't support SRV-IO last time I checked when I tried to enable it in VMware.
ConnectX-3 VPI supports it.

The card should be fine. I bought a batch of HP branded ones and flashed them to stock mellanox firmware plus set up a number of VFs. Actually, a card of the same batch and identical configuration is doing its job in another Proxmox machine.

 

[Psycho_Robotico]

I setup another system (Athlon 200GE) with the same version of proxmox and identical mellanox NIC. There all VFs end up in different IOMMU groups. Next, I took out the SSD and put it in the Supermicro/Epyc box. There VFs are stuck in the same group - hence it really shouldn't be an issue with my configuration. I've already switched bios settings around several times, nothing helped.

Is anyone seeing a similar issue with Supermicro's H11SSL and can post their IOMMU groupings?

 

[Psycho_Robotico]

After A LOT of experimenting, this seems to come down to the CPU/microcode, as it's an eng. sample. This thread can be closed!