It's been a learning curve, but I have managed to enable SR-IOV on my Proxmox Host using a dual port Mellanox Connectx-3 Pro.
When you enable SR-IOV on these cards, you end up having a PF for each port enp1s0, enp1s0d1 respectively and then the mlx4_core driver also creates the VFs as Port1 only, Port2 only, and Port1+2 together as expected. I understand that "probing" (probe_vf= ) the VFs on driver initialization allows the Host to use the interface, and will be seen on the host as an additional nic.
In this case, I have probed one dual port VF on the Proxmox Host, which causes enp1s0v4 and enp1s0d1v4 interfaces to show up as seen in the output of $ ip link below:
PF Devices: enp1s0, enp1s0d1
(Probed) VF Devices: enp1s0v4, enp1s0d1v4
I want to create a bridge on the Proxmox Host to each port, but what I would like to know is it better to create a bridge on the PF, or is it better to bridge the VF?
Some sr-iov documentation says the PFs will disappear once you initialize a VF, but I do not seem to experience that, even when I passthrough the unprobed VFs to guest VMs, all the interfaces listed above remain, and seem to function perfectly...
Should I just bridge the PF devices (enp1s0, and enp1s0d1), and do not probe any VFs within the Host, leaving all VFs for passthrough to VMs?
Or should I bridge the VF devices (enp1s0v4, enp1s0d1v4) instead?
When you enable SR-IOV on these cards, you end up having a PF for each port enp1s0, enp1s0d1 respectively and then the mlx4_core driver also creates the VFs as Port1 only, Port2 only, and Port1+2 together as expected. I understand that "probing" (probe_vf= ) the VFs on driver initialization allows the Host to use the interface, and will be seen on the host as an additional nic.
In this case, I have probed one dual port VF on the Proxmox Host, which causes enp1s0v4 and enp1s0d1v4 interfaces to show up as seen in the output of $ ip link below:
PF Devices: enp1s0, enp1s0d1
(Probed) VF Devices: enp1s0v4, enp1s0d1v4
Code:
3: enp1s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
link/ether 24:8a:07:bc:d5:a3 brd ff:ff:ff:ff:ff:ff
vf 0 link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, vlan 4095, spoof checking off, link-state auto
vf 1 link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, vlan 4095, spoof checking off, link-state auto
vf 2 link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, vlan 4095, spoof checking off, link-state auto
vf 3 link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, vlan 4095, spoof checking off, link-state auto
vf 4 link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, vlan 4095, spoof checking off, link-state auto
vf 5 link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, vlan 4095, spoof checking off, link-state auto
vf 6 link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, vlan 4095, spoof checking off, link-state auto
vf 7 link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, vlan 4095, spoof checking off, link-state auto
4: enp1s0d1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
link/ether 24:8a:07:bc:d5:a4 brd ff:ff:ff:ff:ff:ff
vf 0 link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, vlan 4095, spoof checking off, link-state auto
vf 1 link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, vlan 4095, spoof checking off, link-state auto
vf 2 link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, vlan 4095, spoof checking off, link-state auto
vf 3 link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, vlan 4095, spoof checking off, link-state auto
vf 4 link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, vlan 4095, spoof checking off, link-state auto
vf 5 link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, vlan 4095, spoof checking off, link-state auto
vf 6 link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, vlan 4095, spoof checking off, link-state auto
vf 7 link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, vlan 4095, spoof checking off, link-state auto
5: enp1s0v4: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
link/ether 6a:4f:10:17:f8:18 brd ff:ff:ff:ff:ff:ff
6: enp1s0d1v4: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
link/ether 02:58:41:3a:23:53 brd ff:ff:ff:ff:ff:ff
Some sr-iov documentation says the PFs will disappear once you initialize a VF, but I do not seem to experience that, even when I passthrough the unprobed VFs to guest VMs, all the interfaces listed above remain, and seem to function perfectly...
Should I just bridge the PF devices (enp1s0, and enp1s0d1), and do not probe any VFs within the Host, leaving all VFs for passthrough to VMs?
Or should I bridge the VF devices (enp1s0v4, enp1s0d1v4) instead?