Error enabling switchdev for ConnectX-5 Ex in Proxmox 8.4.1

[eric_h]

I got a ConnectX-5 Ex to play with switchdev mode offloading and I'm following the steps in this guide, but the command to enable switchdev mode returns an invalid argument error:

root@pve:~# /usr/sbin/devlink dev eswitch set pci/0000:41:00.0 mode switchdev
Error: mlx5_core: Failed setting eswitch to offloads.
kernel answers: Invalid argument

I found other guides on using switchdev, but none mentioned needing to make any changes before running the devlink command to enable switchdev mode, and I didn't see anything in mstconfig that looked related to enabling it.

Is there something I've missed, or could this be a difference between the ConnectX-5 and the ConnectX-5 Ex?

Thanks in advance for any help.

 

[necr]

I used switchdev on regular CX5, it was working, there were additional parameters to be changed, something similar to these:

switchdev-utils/templates/firstboot/mellanox_fw_update.yaml at master · Mellanox/switchdev-utils

Contribute to Mellanox/switchdev-utils development by creating an account on GitHub.

github.com

Are the basics covered? Firmware up to date? Can you manually create a SR-IOV VF successfully?

 

[NablaSquaredG]

Is there something I've missed, or could this be a difference between the ConnectX-5 and the ConnectX-5 Ex?

Did you unbind the driver from all VFs before changing the mode to eswitch?

My standard script basically works like this

1. Set number of VFs (/usr/bin/echo 8 > /sys/class/net/ens21f0np0/device/sriov_numvfs)
2. Set Static MACs for VFs (/usr/bin/ip link set ens21f0np0 vf 0 mac d2:77:a4:7c:0d:02)
3. Unbind VFs (echo 0000:b3:00.2 > /sys/bus/pci/drivers/mlx5_core/unbind)
4. Change eSwitch mode from Legacy to Switchdev (/usr/sbin/devlink dev eswitch set pci/0000:b3:00.0 mode switchdev)

(Code snippets are just examples)

 

[eric_h]

Thank you both for your help with this.

The VFs are being created successfully, and I was getting the same error when unbinding all the VFs before changing the switch mode.

I found this thread about updating the firmware on the same Dell branded card I'm using, and after updating the firmware from 16.26.6000 to 16.26.6000 (from Mellanox ConnectX-5 Ethernet Adapter Firmware | Driver Details | Dell US), and rebooting it's showing up in switchdev mode:

root@pve:~# devlink dev eswitch show pci/0000:41:00.0
pci/0000:41:00.0: mode switchdev inline-mode none encap-mode basic