Help with ConnectX 3 SR-IOV with Linux host and windows guest via kvm

[naptastic]

Okay, I changed my parameters to: num_vfs=3,3,0 probe_vf=3,3,0. On boot, this gets me a total of 4 Infiniband and 4 Ethernet interfaces with random MAC/GUID assignments. I can use `driverctl set-override $dev none` and it will unbind the driver. Then `ip link set $dev mac 00:11:22:33:44:55 vlan 3` puts the right values in `ip link`. When I re-bind mlx4_core to that device, it comes back as a usable network interface with the right properties.

At least, they look right. I haven't actually tested them yet.

IB GUIDs are weird and probably not right. The `ip link set` for node_guid and port_guid do not work (RTNETLINK answers: Operation not supported) but I can set a MAC address and then re-bind the driver and the GUID will be constructed from that MAC address. The output in `ip link`, in this situation, is still wrong; it still shows the MAC as 00:00:00:00:00:00.

So, here's a functioning workflow for my Ethernet VFs:

1. Use mlxconfig to set NUM_OF_VFS to x
2. Use num_vfs=0,x,0 probe_vf=0,x,0
3. Reboot.
4. Unbind mlx4_core from the vfs using driverctl. (I didn't try any other methods.)
5. Use `ip link set dev vf num mac stuff etc` to set the VF's properties
6. Either re-bind mlx4_core to the VF if you want the host to use that interface, or attach it to a virtual machine.

I don't know how to change the GUIDs on IB VFs. Random GUIDs get assigned, which makes them useless for SRP. (Time to learn NVMe-oF!)

This experience is making me appreciate the mlx5 driver model and adapters that use it. Shame they cost so much and run so hot.

Thanks so much for the help. I could not have gotten this far without it.

 

[dsrhdev]

hello,
1) regarding eth
- you are not obliged to bind/rebind driver just for vlan/link state update (vlan and link state attributes can be changed online)
- you are not obliged to set mac address on VF and use random if it is acceptable for your software
- you can assign required mac for VF via networkmanager/networkd daemons
anyway if you want to change some VF's attributes, like mac, vlan and apply changes there is more unified way - use devlink command, like

devlink dev reload pci/0000:11:22:33.4

2) regarding IB GUIDS
- there is no known method (to me) to set port/node guid for CX-3 cards, is works for mlx5_core driver (i.e. CX-4 and above)
- if you need to hardcode GUIDS for SRP you can use networkmanager/networkd daemons or custom scripts or try to use mlnx_ofed drivers instead of in-tree driver
- nvmeof is many much newer and effective protocol against to srp, and you can configure it with nvmetcli, but by default the nmeof client creates (as for now) appropriate count of kernel threads (one per core count), it can be reduced by --nr-io-queues=<x> parameter, also you can try iSER protocol to have access to iSCSI target w/wo RDMA, and configure it via targetcli as well

 

[BackupProphet]

Im am updating the mlx4 driver to fix some of the SR-IOV issues, with Claude assisting me. I have for example fixed so you can load for example 4 -> 0 -> 8 virtual functions in runtime. Am also backporting a few things from mlx5. Stay tuned for a release.

Another thing, NVMe-oF works great on these cards when using IB. You can configure it using nvmetcli.

 

[naptastic]

I'm going to halt my own experimentation for now. I can't get traffic to pass through VFs no matter what I try, and if VFs are present, my system resumes from suspend immediately, consistently, and I need suspend/resume to work. I will be happy to test any patches you come up with.

 

[BackupProphet]

That may be a firmware feature in newer mellanox cards. I will do a check, but my guess is that will not work.