Passthrough fail for Mellanox ConnectX3

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.
C

Chura

New Member
Apr 16, 2023
14
0
1
I can't make the Mellanox be used on a Proxmox VM (Actually, I think its not accessible to the host as well)
1685707484035.png

When I try to attach it:
kvm: -device vfio-pci,host=0000:01:00.0,id=hostpci0,bus=pci.0,addr=0x10: vfio 0000:01:00.0: Failed to set up TRIGGER eventfd signaling for interrupt INTX-0: VFIO_DEVICE_SET_IRQS failure: Device or resource busy
TASK ERROR: start failed: QEMU exited with code 1

I have IOMMU enabled, I used to have a M2 2.5G working just fine
Tried to blacklist it from the PVE, same results.
 
T

TheRac25

New Member
May 30, 2023
12
0
1
I haven't been able to get this to work with CX2 or CX3.
Blacklist mlx4_core no effect.
"Blacklist with fake install" no effect.
"allow_unsafe_interrupts" guest starts, sees card then host freezes or kernel fault
VFs passthrough fine, except not windows guest on CX3 unless you patch the sabotaged mlx4_core and no rdma on VFs
very disappointed that such mediocrity is rewarded with such status
 
T

TheRac25

New Member
May 30, 2023
12
0
1
1. enable sr-iov options on the host motherboard, sr-iov vt-d etc etc
2. enable sriov in the card, which probably involves reflashing the firmware, not sure when they went to dynamic firmware variables (CX3 pro?)
3. configure mlx4_core module parameters, num_vfs and optionally port_type_array/probe_vf using grub command line or initramfs parameters
4. reboot and make sure you can see the VFs in lspci, ip link, etc. also make sure each VF is its own iommu group
5. set VF mac and optionally vlan(s), (hardware vlan is not working or broken on my CX2 sample)
I increment the PF mac address by 1 for each VF and set the locally administered bit to 1 (second bit first octet)

script to do this automagically "nano /etc/systemd/system/mlnx_sriov.service" and "systemctl enable mlnx_sriov"

Code: 
[Unit]
Description=Script to set VF parameters on boot

[Service]
Type=oneshot
ExecStart=/usr/bin/bash -c '/usr/bin/ip link set enpXs0 vf 0 mac XX:XX:XX:XX:XX:XX'
ExecStart=/usr/bin/bash -c '/usr/bin/ip link set enpXs0 vf 0 vlan X'
ExecStart=/usr/bin/bash -c '/usr/bin/ip link set enpXs0 vf 1 mac XX:XX:XX:XX:XX:XX'
ExecStart=/usr/bin/bash -c '/usr/bin/ip link set enpXs0 vf 1 vlan X'

[Install]
WantedBy=multi-user.target

6. attach VF to the guest as you would any other pcie device
 
C

Chura

New Member
Apr 16, 2023
14
0
1
zer0sum said:
You should be able to get it going, but it does depend on all of your hardware.
I had it working on a tiny Lenovo m920q :)


I could only ever pass through both ports, instead of just a single one, so I switched to 10G Intel based cards
Click to expand...
I gave up on SR-IOV, seems like M720q do not support it :/
but I can't even get IOMMU passthrough, keep getting device is busy.
 
I

i386

Well-Known Member
Mar 18, 2016
4,245
1,546
113
34
Germany
I don't know if it helps but under windows connect-x 3 present them self as two devices: one as a nic and one device appears under system devices. I don't know if you this is the same behavior in linux or bsd.
 
C

Chura

New Member
Apr 16, 2023
14
0
1
Not sure how to proceed :(

Code: 
root@pxx:~# lspci -v -s 01:00.0
01:00.0 Ethernet controller: Mellanox Technologies MT27500 Family [ConnectX-3]
        Subsystem: Mellanox Technologies MT27500 Family [ConnectX-3]
        Flags: fast devsel, IRQ 16, IOMMU group 9
        Memory at b2200000 (64-bit, non-prefetchable) [disabled] [size=1M]
        Memory at b2800000 (64-bit, prefetchable) [disabled] [size=8M]
        Expansion ROM at b2100000 [disabled] [size=1M]
        Capabilities: [40] Power Management version 3
        Capabilities: [48] Vital Product Data
        Capabilities: [9c] MSI-X: Enable- Count=128 Masked-
        Capabilities: [60] Express Endpoint, MSI 00
        Capabilities: [c0] Vendor Specific Information: Len=18 <?>
        Capabilities: [100] Alternative Routing-ID Interpretation (ARI)
        Capabilities: [148] Device Serial Number ec-0d-9a-03-00-05-13-30
        Capabilities: [154] Advanced Error Reporting
        Capabilities: [18c] Secondary PCI Express
        Capabilities: [108] Single Root I/O Virtualization (SR-IOV)
        Kernel driver in use: vfio-pci
        Kernel modules: mlx4_core
I've managed to blacklist mlx4_core modules
Code: 
root@pxx:~# lsmod | grep mlx
root@pxx:~#
 
C

Chura

New Member
Apr 16, 2023
14
0
1
If I use those ports on proxmox, add them to the bridge, fully usable.
1685742632690.png

Just can't make them passthrough
 
zer0sum

zer0sum

Well-Known Member
Mar 8, 2013
849
474
63
Do you get any output from echo 4 > /sys/class/net/enp39s0f0/device/sriov_numvfs
 
C

Chura

New Member
Apr 16, 2023
14
0
1
zer0sum said:
Do you get any output from echo 4 > /sys/class/net/enp39s0f0/device/sriov_numvfs
Click to expand...
They exist, but can't write them or something

Code: 
root@pxx:~# ls -al /sys/class/net/enp1s0*/device/sriov_numvfs
-rw-r--r-- 1 root root 4096 Jun  5 18:19 /sys/class/net/enp1s0d1/device/sriov_numvfs
-rw-r--r-- 1 root root 4096 Jun  5 18:19 /sys/class/net/enp1s0/device/sriov_numvfs


root@pxx:~# echo 4 > /sys/class/net/enp1s0/device/sriov_numvfs
-bash: echo: write error: No such file or directory
root@pxx:~# echo 4 > /sys/class/net/enp1s0d1/device/sriov_numvfs
-bash: echo: write error: No such file or directory
 
G

gb00s

Well-Known Member
Jul 25, 2018
1,190
602
113
Poland
May I ask why not using the Mellanox card with a second and manually set bridge and then share it for all VM's instead of passing it through?
 
C

Chura

New Member
Apr 16, 2023
14
0
1
gb00s said:
May I ask why not using the Mellanox card with a second and manually set bridge and then share it for all VM's instead of passing it through?
Click to expand...
This is what I'm doing now because I can't support it out. I think it makes more sense to have it in passthrough, for both security and possibly performance at high speed.
 
zer0sum

zer0sum

Well-Known Member
Mar 8, 2013
849
474
63
Chura said:
They exist, but can't write them or something

Code: 
root@pxx:~# ls -al /sys/class/net/enp1s0*/device/sriov_numvfs
-rw-r--r-- 1 root root 4096 Jun  5 18:19 /sys/class/net/enp1s0d1/device/sriov_numvfs
-rw-r--r-- 1 root root 4096 Jun  5 18:19 /sys/class/net/enp1s0/device/sriov_numvfs


root@pxx:~# echo 4 > /sys/class/net/enp1s0/device/sriov_numvfs
-bash: echo: write error: No such file or directory
root@pxx:~# echo 4 > /sys/class/net/enp1s0d1/device/sriov_numvfs
-bash: echo: write error: No such file or directory
Click to expand...
I feel like this may be an issue with your bios not supporting SRIOV?
 
P

piranha32

Active Member
Mar 4, 2023
240
176
43
Chura said:
Can you elaborate? You mean all is not lost ? (Yet)
Click to expand...
I don't have mellanox card to test it, but the module supports num_vfs param:
Code: 
$ modinfo mlx4_core
[...]
parm:           num_vfs:enable #num_vfs functions if num_vfs > 0
num_vfs=port1,port2,port1+2 (array of byte)
[...]
If you need an example, I used this guide to configure VFs with ixgbe intel driver: Setting up SR-IOV in Proxmox VE
 
  • Like
Reactions: Chura
C

Chura

New Member
Apr 16, 2023
14
0
1
piranha32 said:
I don't have mellanox card to test it, but the module supports num_vfs param:
Code: 
$ modinfo mlx4_core
[...]
parm:           num_vfs:enable #num_vfs functions if num_vfs > 0
num_vfs=port1,port2,port1+2 (array of byte)
[...]
If you need an example, I used this guide to configure VFs with ixgbe intel driver: Setting up SR-IOV in Proxmox VE
Click to expand...
This was really helpful! while not the direct answer since if aimed at intel NIC. but it gave me more direction what I should look for.
and.....


root@pxx:~# lspci | grep ^01
01:00.0 Ethernet controller: Mellanox Technologies MT27500 Family [ConnectX-3]
01:00.1 Ethernet controller: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]
01:00.2 Ethernet controller: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]
01:00.3 Ethernet controller: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]
01:00.4 Ethernet controller: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]
 
You must log in or register to reply here.
Top Bottom