Hi guys - paging
@Basriram - Apologies for bringing up an old thread, but I'm in the middle of playing with PVE (planning to move off Hyper-V since I need native container support) and I'm running into issue with the CX3 cards and SR-IOV.
I've gone through the steps above to patch the mlx driver in PVE, but...I'm using the latest PVE 8.4.1 and..when I do
modinfo mlx4_core, I'm on
6.8.12-11-pve vs what was just above:
name: mlx4_core
vermagic: 6.8.12-5-pve SMP preempt mod_unload modversions
Does this patch still work with the latest PVE 8.4.1?
Here's what I'm facing.
Background Info:
- CX3 dual port 40gb cards in the PVE host, all flashed to the latest firmware (2.42.5000), connected to an SX6036 switch.
- PVE 8.4.1 setup and running successfully.
- A test Windows 11 VM with Mellanox drivers installed (5.50.54000).
- PVE hookscript for this VM configured to use vlan 20
The VM starts fine, no yellow bangs in Device Manager, both Mellanox devices show up correctly and use the right driver versions. The ethernet adapter pulls a correct IP address based on vlan 20 from my DHCP server, and...then nothing. Stays at "Unidentified Network" and no traffic seems to pass. I can't even ping the gateway for that VLAN.
If I simply pass a virtio Nic from the same bond/bridge, everything works fine, so I know there's no issues with the VLAN routing/LACP/Switch etc.
What am I doing wrong/missing?
Edit: I'm seeing this in dmesg (The device vfio-pci 0000:81:00.1 is the VF adapter passed to the VM)
Code:
[ 828.231590] mlx4_core 0000:81:00.0: default mac on vf 0 port 1 to 248A07DA1753 will take effect only after vf restart
[ 828.233364] mlx4_core 0000:81:00.0: updating vf 0 port 1 config will take effect on next VF restart
[ 829.172536] tap100i0: entered promiscuous mode
[ 829.208563] fwbr100i0: port 1(tap100i0) entered blocking state
[ 829.208567] fwbr100i0: port 1(tap100i0) entered disabled state
[ 829.208586] tap100i0: entered allmulticast mode
[ 829.208730] fwbr100i0: port 1(tap100i0) entered blocking state
[ 829.208733] fwbr100i0: port 1(tap100i0) entered forwarding state
[ 829.225425] fwln100o0: entered promiscuous mode
[ 829.241318] fwbr100i0: port 2(fwln100o0) entered blocking state
[ 829.241322] fwbr100i0: port 2(fwln100o0) entered disabled state
[ 829.241339] fwln100o0: entered allmulticast mode
[ 829.241426] fwbr100i0: port 2(fwln100o0) entered blocking state
[ 829.241428] fwbr100i0: port 2(fwln100o0) entered forwarding state
[ 831.564475] vfio-pci 0000:81:00.1: enabling device (0000 -> 0002)
[ 831.898305] mlx4_core 0000:81:00.0: default mac on vf 0 port 1 to 248A07DA1753 will take effect only after vf restart
[ 831.899954] mlx4_core 0000:81:00.0: updating vf 0 port 1 config will take effect on next VF restart
[ 840.834036] mlx4_core 0000:81:00.0: Received reset from slave:1
[ 841.046605] mlx4_core 0000:81:00.0: denying Global Pause change for slave:1
[ 1531.841656] vmbr0: the hash_elasticity option has been deprecated and is always 16
[ 1543.321475] mlx4_core 0000:81:00.0: denying Global Pause change for slave:1