EDIT: The original script i posted only works when partitioning interfaces for use with pcie passthrough in VM's. But i quickly ran into issues when i wanted to use interfaces for the proxmox host as well, because the script was being run after proxmox configures network interfaces, and starts the firewall service, which resulted in the interfaces being non-active, and the firewall was unaware of them resulting in any rules created did nothing.
I have updated the script so it works both for the proxmox host as bridgeable interfaces, raw interfaces, as well as using passthrough for VM's.
I have moved the original script to the end for reference. The original script is still valid for a simple setup where you only want to partition and detach interfaces from the host, and use them as native interfaces in VM's.
I discuss the changes i made later down in the thread. Comments and suggestions are highly welcome. As i write this everything seems to work, but issues might arise later, or in different use cases.
---
So as i didn't find any resources for this i ended up creating a systemd script for proxmox to partition, set static MAC's, and detaching NIC Virtual Functions to use for my SFN7022F dual port SFP+ adapter in Proxmox. I posted this earlier in the proxmox forums, but thought i'd share it here as well in case anyone needs it as a template.
It's been working great so far, and the start, stop and reload functions have been really handy.
To set the static MAC's, i first partitioned the ports manually, and then used the generated random MAC's.
This example creates 8 VF's per port, sets static MAC addresses for the VF's, and then detaches 7 interfaces per port from the host to use in pcie passthrough mode in a VM.
I wanted to keep 2 interfaces per port for use in proxmox: One interface for use as a bridge host interface, so i can use that for accessing the host with SSH and WebUI over 10Gbit, for use in containers, and also VM's that uses virtio networking.
The second one for things like a iscsi target serving incoming iscsi initiators, which i don't want to run over a bridge.
1. Enable 'Full Feature / virtualization mode' Firmware, and 'SRIOV' Switch mode, and set the number of virtual functions per port:
2. Create the systemd script, use a filename that suits you:
3. My example. To get the correct interface names and pcie addresses for your system, you need to first do this manually step by step. The double \\ in the script is to fix an annoying but non-fatal error message that a single \ would produce.
4. Register, enable and start the script:
5. Check that the VF's have been created (this also prints in the logs), and check the status of the service:
6: Use the script with these commands:
------
7: (for reference) This original example creates 4 VF's per port, sets the MAC addresses for the VF's, and then detaches them from the host to use in pcie passthrough mode in a VM:
I have updated the script so it works both for the proxmox host as bridgeable interfaces, raw interfaces, as well as using passthrough for VM's.
I have moved the original script to the end for reference. The original script is still valid for a simple setup where you only want to partition and detach interfaces from the host, and use them as native interfaces in VM's.
I discuss the changes i made later down in the thread. Comments and suggestions are highly welcome. As i write this everything seems to work, but issues might arise later, or in different use cases.
---
So as i didn't find any resources for this i ended up creating a systemd script for proxmox to partition, set static MAC's, and detaching NIC Virtual Functions to use for my SFN7022F dual port SFP+ adapter in Proxmox. I posted this earlier in the proxmox forums, but thought i'd share it here as well in case anyone needs it as a template.
It's been working great so far, and the start, stop and reload functions have been really handy.
To set the static MAC's, i first partitioned the ports manually, and then used the generated random MAC's.
This example creates 8 VF's per port, sets static MAC addresses for the VF's, and then detaches 7 interfaces per port from the host to use in pcie passthrough mode in a VM.
I wanted to keep 2 interfaces per port for use in proxmox: One interface for use as a bridge host interface, so i can use that for accessing the host with SSH and WebUI over 10Gbit, for use in containers, and also VM's that uses virtio networking.
The second one for things like a iscsi target serving incoming iscsi initiators, which i don't want to run over a bridge.
1. Enable 'Full Feature / virtualization mode' Firmware, and 'SRIOV' Switch mode, and set the number of virtual functions per port:
Code:
sfboot firmware-variant=full-feature
sfboot switch-mode=sriov
sfboot vf-count=8
2. Create the systemd script, use a filename that suits you:
Code:
nano /etc/systemd/system/sriov-vfs.service
3. My example. To get the correct interface names and pcie addresses for your system, you need to first do this manually step by step. The double \\ in the script is to fix an annoying but non-fatal error message that a single \ would produce.
Code:
[Unit]
Description=Enable SR-IOV and detach guest VFs from host
Before=network-online.target network-pre.target
Wants=network-pre.target
[Service]
Type=oneshot
RemainAfterExit=yes
# Create NIC VFs
ExecStart=/usr/bin/bash -c 'echo 8 > /sys/class/net/ens6f0np0/device/sriov_numvfs'
ExecStart=/usr/bin/bash -c 'echo 8 > /sys/class/net/ens6f1np1/device/sriov_numvfs'
# Set static MACs for VFs
ExecStart=/usr/bin/bash -c '/usr/bin/ip link set ens6f0np0 vf 0 mac 76:9e:17:83:39:e5'
ExecStart=/usr/bin/bash -c '/usr/bin/ip link set ens6f0np0 vf 1 mac 46:2c:6d:24:6b:1b'
ExecStart=/usr/bin/bash -c '/usr/bin/ip link set ens6f0np0 vf 2 mac 3e:47:48:12:ed:94'
ExecStart=/usr/bin/bash -c '/usr/bin/ip link set ens6f0np0 vf 3 mac be:e3:6a:f3:8f:ac'
ExecStart=/usr/bin/bash -c '/usr/bin/ip link set ens6f0np0 vf 4 mac 62:8f:3d:bb:02:08'
ExecStart=/usr/bin/bash -c '/usr/bin/ip link set ens6f0np0 vf 5 mac ae:91:57:b9:14:7f'
ExecStart=/usr/bin/bash -c '/usr/bin/ip link set ens6f0np0 vf 6 mac 5a:c2:08:a9:68:a7'
ExecStart=/usr/bin/bash -c '/usr/bin/ip link set ens6f0np0 vf 7 mac b2:f0:18:af:cb:c5'
ExecStart=/usr/bin/bash -c '/usr/bin/ip link set ens6f1np1 vf 0 mac 16:47:7c:a8:95:98'
ExecStart=/usr/bin/bash -c '/usr/bin/ip link set ens6f1np1 vf 1 mac a6:c7:c5:7f:9c:22'
ExecStart=/usr/bin/bash -c '/usr/bin/ip link set ens6f1np1 vf 2 mac b6:0f:45:34:5e:19'
ExecStart=/usr/bin/bash -c '/usr/bin/ip link set ens6f1np1 vf 3 mac 2a:f7:37:84:31:30'
ExecStart=/usr/bin/bash -c '/usr/bin/ip link set ens6f1np1 vf 4 mac 8a:fa:f8:c5:0b:93'
ExecStart=/usr/bin/bash -c '/usr/bin/ip link set ens6f1np1 vf 5 mac b2:f5:d5:2f:79:06'
ExecStart=/usr/bin/bash -c '/usr/bin/ip link set ens6f1np1 vf 6 mac c2:92:f5:fa:32:20'
ExecStart=/usr/bin/bash -c '/usr/bin/ip link set ens6f1np1 vf 7 mac 2e:fb:29:1e:48:31'
# Detach VFs from host
ExecStart=/usr/bin/bash -c 'echo 0000:03:00.3 > /sys/bus/pci/devices/0000\\:03\\:00.3/driver/unbind'
ExecStart=/usr/bin/bash -c 'echo 0000:03:00.4 > /sys/bus/pci/devices/0000\\:03\\:00.4/driver/unbind'
ExecStart=/usr/bin/bash -c 'echo 0000:03:00.5 > /sys/bus/pci/devices/0000\\:03\\:00.5/driver/unbind'
ExecStart=/usr/bin/bash -c 'echo 0000:03:00.6 > /sys/bus/pci/devices/0000\\:03\\:00.6/driver/unbind'
ExecStart=/usr/bin/bash -c 'echo 0000:03:00.7 > /sys/bus/pci/devices/0000\\:03\\:00.7/driver/unbind'
ExecStart=/usr/bin/bash -c 'echo 0000:03:01.0 > /sys/bus/pci/devices/0000\\:03\\:01.0/driver/unbind'
ExecStart=/usr/bin/bash -c 'echo 0000:03:01.1 > /sys/bus/pci/devices/0000\\:03\\:01.1/driver/unbind'
ExecStart=/usr/bin/bash -c 'echo 0000:03:01.3 > /sys/bus/pci/devices/0000\\:03\\:01.3/driver/unbind'
ExecStart=/usr/bin/bash -c 'echo 0000:03:01.4 > /sys/bus/pci/devices/0000\\:03\\:01.4/driver/unbind'
ExecStart=/usr/bin/bash -c 'echo 0000:03:01.5 > /sys/bus/pci/devices/0000\\:03\\:01.5/driver/unbind'
ExecStart=/usr/bin/bash -c 'echo 0000:03:01.6 > /sys/bus/pci/devices/0000\\:03\\:01.6/driver/unbind'
ExecStart=/usr/bin/bash -c 'echo 0000:03:01.7 > /sys/bus/pci/devices/0000\\:03\\:01.7/driver/unbind'
ExecStart=/usr/bin/bash -c 'echo 0000:03:02.0 > /sys/bus/pci/devices/0000\\:03\\:02.0/driver/unbind'
ExecStart=/usr/bin/bash -c 'echo 0000:03:02.1 > /sys/bus/pci/devices/0000\\:03\\:02.1/driver/unbind'
# List new VFs
ExecStart=/usr/bin/lspci -D -d1924:
# Destroy VFs
ExecStop=/usr/bin/bash -c 'echo 0 > /sys/class/net/ens6f0np0/device/sriov_numvfs'
ExecStop=/usr/bin/bash -c 'echo 0 > /sys/class/net/ens6f1np1/device/sriov_numvfs'
# Reload NIC VFs
ExecReload=/usr/bin/bash -c 'echo 0 > /sys/class/net/ens6f0np0/device/sriov_numvfs'
ExecReload=/usr/bin/bash -c 'echo 0 > /sys/class/net/ens6f1np1/device/sriov_numvfs'
ExecReload=/usr/bin/bash -c 'echo 8 > /sys/class/net/ens6f0np0/device/sriov_numvfs'
ExecReload=/usr/bin/bash -c 'echo 8 > /sys/class/net/ens6f1np1/device/sriov_numvfs'
ExecReload=/usr/bin/bash -c '/usr/bin/ip link set ens6f0np0 vf 0 mac 76:9e:17:83:39:e5'
ExecReload=/usr/bin/bash -c '/usr/bin/ip link set ens6f0np0 vf 1 mac 46:2c:6d:24:6b:1b'
ExecReload=/usr/bin/bash -c '/usr/bin/ip link set ens6f0np0 vf 2 mac 3e:47:48:12:ed:94'
ExecReload=/usr/bin/bash -c '/usr/bin/ip link set ens6f0np0 vf 3 mac be:e3:6a:f3:8f:ac'
ExecReload=/usr/bin/bash -c '/usr/bin/ip link set ens6f0np0 vf 4 mac 62:8f:3d:bb:02:08'
ExecReload=/usr/bin/bash -c '/usr/bin/ip link set ens6f0np0 vf 5 mac ae:91:57:b9:14:7f'
ExecReload=/usr/bin/bash -c '/usr/bin/ip link set ens6f0np0 vf 6 mac 5a:c2:08:a9:68:a7'
ExecReload=/usr/bin/bash -c '/usr/bin/ip link set ens6f0np0 vf 7 mac b2:f0:18:af:cb:c5'
ExecReload=/usr/bin/bash -c '/usr/bin/ip link set ens6f1np1 vf 0 mac 16:47:7c:a8:95:98'
ExecReload=/usr/bin/bash -c '/usr/bin/ip link set ens6f1np1 vf 1 mac a6:c7:c5:7f:9c:22'
ExecReload=/usr/bin/bash -c '/usr/bin/ip link set ens6f1np1 vf 2 mac b6:0f:45:34:5e:19'
ExecReload=/usr/bin/bash -c '/usr/bin/ip link set ens6f1np1 vf 3 mac 2a:f7:37:84:31:30'
ExecReload=/usr/bin/bash -c '/usr/bin/ip link set ens6f1np1 vf 4 mac 8a:fa:f8:c5:0b:93'
ExecReload=/usr/bin/bash -c '/usr/bin/ip link set ens6f1np1 vf 5 mac b2:f5:d5:2f:79:06'
ExecReload=/usr/bin/bash -c '/usr/bin/ip link set ens6f1np1 vf 6 mac c2:92:f5:fa:32:20'
ExecReload=/usr/bin/bash -c '/usr/bin/ip link set ens6f1np1 vf 7 mac 2e:fb:29:1e:48:31'
ExecReload=/usr/bin/bash -c 'echo 0000:03:00.3 > /sys/bus/pci/devices/0000\\:03\\:00.3/driver/unbind'
ExecReload=/usr/bin/bash -c 'echo 0000:03:00.4 > /sys/bus/pci/devices/0000\\:03\\:00.4/driver/unbind'
ExecReload=/usr/bin/bash -c 'echo 0000:03:00.5 > /sys/bus/pci/devices/0000\\:03\\:00.5/driver/unbind'
ExecReload=/usr/bin/bash -c 'echo 0000:03:00.6 > /sys/bus/pci/devices/0000\\:03\\:00.6/driver/unbind'
ExecReload=/usr/bin/bash -c 'echo 0000:03:00.7 > /sys/bus/pci/devices/0000\\:03\\:00.7/driver/unbind'
ExecReload=/usr/bin/bash -c 'echo 0000:03:01.0 > /sys/bus/pci/devices/0000\\:03\\:01.0/driver/unbind'
ExecReload=/usr/bin/bash -c 'echo 0000:03:01.1 > /sys/bus/pci/devices/0000\\:03\\:01.1/driver/unbind'
ExecReload=/usr/bin/bash -c 'echo 0000:03:01.3 > /sys/bus/pci/devices/0000\\:03\\:01.3/driver/unbind'
ExecReload=/usr/bin/bash -c 'echo 0000:03:01.4 > /sys/bus/pci/devices/0000\\:03\\:01.4/driver/unbind'
ExecReload=/usr/bin/bash -c 'echo 0000:03:01.5 > /sys/bus/pci/devices/0000\\:03\\:01.5/driver/unbind'
ExecReload=/usr/bin/bash -c 'echo 0000:03:01.6 > /sys/bus/pci/devices/0000\\:03\\:01.6/driver/unbind'
ExecReload=/usr/bin/bash -c 'echo 0000:03:01.7 > /sys/bus/pci/devices/0000\\:03\\:01.7/driver/unbind'
ExecReload=/usr/bin/bash -c 'echo 0000:03:02.0 > /sys/bus/pci/devices/0000\\:03\\:02.0/driver/unbind'
ExecReload=/usr/bin/bash -c 'echo 0000:03:02.1 > /sys/bus/pci/devices/0000\\:03\\:02.1/driver/unbind'
ExecReload=/usr/bin/lspci -D -d1924:
[Install]
WantedBy=multi-user.target
4. Register, enable and start the script:
Code:
systemctl daemon-reload
systemctl enable sriov-vfs.service
systemctl start sriov-vfs.service
5. Check that the VF's have been created (this also prints in the logs), and check the status of the service:
Code:
# lspci -D -d1924:
0000:03:00.0 Ethernet controller: Solarflare Communications SFC9120 10G Ethernet Controller (rev 01)
0000:03:00.1 Ethernet controller: Solarflare Communications SFC9120 10G Ethernet Controller (rev 01)
0000:03:00.2 Ethernet controller: Solarflare Communications SFC9120 10G Ethernet Controller (Virtual Function) (rev 01)
0000:03:00.3 Ethernet controller: Solarflare Communications SFC9120 10G Ethernet Controller (Virtual Function) (rev 01)
0000:03:00.4 Ethernet controller: Solarflare Communications SFC9120 10G Ethernet Controller (Virtual Function) (rev 01)
0000:03:00.5 Ethernet controller: Solarflare Communications SFC9120 10G Ethernet Controller (Virtual Function) (rev 01)
0000:03:00.6 Ethernet controller: Solarflare Communications SFC9120 10G Ethernet Controller (Virtual Function) (rev 01)
0000:03:00.7 Ethernet controller: Solarflare Communications SFC9120 10G Ethernet Controller (Virtual Function) (rev 01)
0000:03:01.0 Ethernet controller: Solarflare Communications SFC9120 10G Ethernet Controller (Virtual Function) (rev 01)
0000:03:01.1 Ethernet controller: Solarflare Communications SFC9120 10G Ethernet Controller (Virtual Function) (rev 01)
0000:03:01.2 Ethernet controller: Solarflare Communications SFC9120 10G Ethernet Controller (Virtual Function) (rev 01)
0000:03:01.3 Ethernet controller: Solarflare Communications SFC9120 10G Ethernet Controller (Virtual Function) (rev 01)
0000:03:01.4 Ethernet controller: Solarflare Communications SFC9120 10G Ethernet Controller (Virtual Function) (rev 01)
0000:03:01.5 Ethernet controller: Solarflare Communications SFC9120 10G Ethernet Controller (Virtual Function) (rev 01)
0000:03:01.6 Ethernet controller: Solarflare Communications SFC9120 10G Ethernet Controller (Virtual Function) (rev 01)
0000:03:01.7 Ethernet controller: Solarflare Communications SFC9120 10G Ethernet Controller (Virtual Function) (rev 01)
0000:03:02.0 Ethernet controller: Solarflare Communications SFC9120 10G Ethernet Controller (Virtual Function) (rev 01)
0000:03:02.1 Ethernet controller: Solarflare Communications SFC9120 10G Ethernet Controller (Virtual Function) (rev 01)
# systemctl status sriov-vfs.service
● sriov-vfs.service - Enable SR-IOV and detach guest VFs from host
Loaded: loaded (/etc/systemd/system/sriov-vfs.service; enabled; vendor preset: enabled)
Active: active (exited) since Sat 2023-05-06 15:43:36 EEST; 52min ago
Process: 3002 ExecStart=/usr/bin/bash -c echo 8 > /sys/class/net/ens6f0np0/device/sriov_numvfs (code=exited, status=0/SUCCESS)
Process: 3226 ExecStart=/usr/bin/bash -c echo 8 > /sys/class/net/ens6f1np1/device/sriov_numvfs (code=exited, status=0/SUCCESS)
Process: 3243 ExecStart=/usr/bin/bash -c /usr/bin/ip link set ens6f0np0 vf 0 mac 76:9e:17:83:39:e5 (code=exited, status=0/SUCCESS)
Process: 3245 ExecStart=/usr/bin/bash -c /usr/bin/ip link set ens6f0np0 vf 1 mac 46:2c:6d:24:6b:1b (code=exited, status=0/SUCCESS)
Process: 3246 ExecStart=/usr/bin/bash -c /usr/bin/ip link set ens6f0np0 vf 2 mac 3e:47:48:12:ed:94 (code=exited, status=0/SUCCESS)
Process: 3247 ExecStart=/usr/bin/bash -c /usr/bin/ip link set ens6f0np0 vf 3 mac be:e3:6a:f3:8f:ac (code=exited, status=0/SUCCESS)
Process: 3248 ExecStart=/usr/bin/bash -c /usr/bin/ip link set ens6f0np0 vf 4 mac 62:8f:3d:bb:02:08 (code=exited, status=0/SUCCESS)
Process: 3249 ExecStart=/usr/bin/bash -c /usr/bin/ip link set ens6f0np0 vf 5 mac ae:91:57:b9:14:7f (code=exited, status=0/SUCCESS)
Process: 3250 ExecStart=/usr/bin/bash -c /usr/bin/ip link set ens6f0np0 vf 6 mac 5a:c2:08:a9:68:a7 (code=exited, status=0/SUCCESS)
Process: 3251 ExecStart=/usr/bin/bash -c /usr/bin/ip link set ens6f0np0 vf 7 mac b2:f0:18:af:cb:c5 (code=exited, status=0/SUCCESS)
Process: 3252 ExecStart=/usr/bin/bash -c /usr/bin/ip link set ens6f1np1 vf 0 mac 16:47:7c:a8:95:98 (code=exited, status=0/SUCCESS)
Process: 3253 ExecStart=/usr/bin/bash -c /usr/bin/ip link set ens6f1np1 vf 1 mac a6:c7:c5:7f:9c:22 (code=exited, status=0/SUCCESS)
Process: 3254 ExecStart=/usr/bin/bash -c /usr/bin/ip link set ens6f1np1 vf 2 mac b6:0f:45:34:5e:19 (code=exited, status=0/SUCCESS)
Process: 3255 ExecStart=/usr/bin/bash -c /usr/bin/ip link set ens6f1np1 vf 3 mac 2a:f7:37:84:31:30 (code=exited, status=0/SUCCESS)
Process: 3256 ExecStart=/usr/bin/bash -c /usr/bin/ip link set ens6f1np1 vf 4 mac 8a:fa:f8:c5:0b:93 (code=exited, status=0/SUCCESS)
Process: 3258 ExecStart=/usr/bin/bash -c /usr/bin/ip link set ens6f1np1 vf 5 mac b2:f5:d5:2f:79:06 (code=exited, status=0/SUCCESS)
Process: 3259 ExecStart=/usr/bin/bash -c /usr/bin/ip link set ens6f1np1 vf 6 mac c2:92:f5:fa:32:20 (code=exited, status=0/SUCCESS)
Process: 3260 ExecStart=/usr/bin/bash -c /usr/bin/ip link set ens6f1np1 vf 7 mac 2e:fb:29:1e:48:31 (code=exited, status=0/SUCCESS)
Process: 3261 ExecStart=/usr/bin/bash -c echo 0000:03:00.3 > /sys/bus/pci/devices/0000\:03\:00.3/driver/unbind (code=exited, status=0/SUCCESS)
Process: 3263 ExecStart=/usr/bin/bash -c echo 0000:03:00.4 > /sys/bus/pci/devices/0000\:03\:00.4/driver/unbind (code=exited, status=0/SUCCESS)
Process: 3265 ExecStart=/usr/bin/bash -c echo 0000:03:00.5 > /sys/bus/pci/devices/0000\:03\:00.5/driver/unbind (code=exited, status=0/SUCCESS)
Process: 3266 ExecStart=/usr/bin/bash -c echo 0000:03:00.6 > /sys/bus/pci/devices/0000\:03\:00.6/driver/unbind (code=exited, status=0/SUCCESS)
Process: 3268 ExecStart=/usr/bin/bash -c echo 0000:03:00.7 > /sys/bus/pci/devices/0000\:03\:00.7/driver/unbind (code=exited, status=0/SUCCESS)
Process: 3269 ExecStart=/usr/bin/bash -c echo 0000:03:01.0 > /sys/bus/pci/devices/0000\:03\:01.0/driver/unbind (code=exited, status=0/SUCCESS)
Process: 3270 ExecStart=/usr/bin/bash -c echo 0000:03:01.1 > /sys/bus/pci/devices/0000\:03\:01.1/driver/unbind (code=exited, status=0/SUCCESS)
Process: 3272 ExecStart=/usr/bin/bash -c echo 0000:03:01.3 > /sys/bus/pci/devices/0000\:03\:01.3/driver/unbind (code=exited, status=0/SUCCESS)
Process: 3273 ExecStart=/usr/bin/bash -c echo 0000:03:01.4 > /sys/bus/pci/devices/0000\:03\:01.4/driver/unbind (code=exited, status=0/SUCCESS)
Process: 3274 ExecStart=/usr/bin/bash -c echo 0000:03:01.5 > /sys/bus/pci/devices/0000\:03\:01.5/driver/unbind (code=exited, status=0/SUCCESS)
Process: 3379 ExecStart=/usr/bin/bash -c echo 0000:03:01.6 > /sys/bus/pci/devices/0000\:03\:01.6/driver/unbind (code=exited, status=0/SUCCESS)
Process: 3380 ExecStart=/usr/bin/bash -c echo 0000:03:01.7 > /sys/bus/pci/devices/0000\:03\:01.7/driver/unbind (code=exited, status=0/SUCCESS)
Process: 3383 ExecStart=/usr/bin/bash -c echo 0000:03:02.0 > /sys/bus/pci/devices/0000\:03\:02.0/driver/unbind (code=exited, status=0/SUCCESS)
Process: 3384 ExecStart=/usr/bin/bash -c echo 0000:03:02.1 > /sys/bus/pci/devices/0000\:03\:02.1/driver/unbind (code=exited, status=0/SUCCESS)
Process: 3385 ExecStart=/usr/bin/lspci -D -d1924: (code=exited, status=0/SUCCESS)
Main PID: 3385 (code=exited, status=0/SUCCESS)
CPU: 185ms
May 06 15:43:36 pve lspci[3385]: 0000:03:01.1 Ethernet controller: Solarflare Communications SFC9120 10G Ethernet Controller (Virtual Function) (rev 01)
May 06 15:43:36 pve lspci[3385]: 0000:03:01.2 Ethernet controller: Solarflare Communications SFC9120 10G Ethernet Controller (Virtual Function) (rev 01)
May 06 15:43:36 pve lspci[3385]: 0000:03:01.3 Ethernet controller: Solarflare Communications SFC9120 10G Ethernet Controller (Virtual Function) (rev 01)
May 06 15:43:36 pve lspci[3385]: 0000:03:01.4 Ethernet controller: Solarflare Communications SFC9120 10G Ethernet Controller (Virtual Function) (rev 01)
May 06 15:43:36 pve lspci[3385]: 0000:03:01.5 Ethernet controller: Solarflare Communications SFC9120 10G Ethernet Controller (Virtual Function) (rev 01)
May 06 15:43:36 pve lspci[3385]: 0000:03:01.6 Ethernet controller: Solarflare Communications SFC9120 10G Ethernet Controller (Virtual Function) (rev 01)
May 06 15:43:36 pve lspci[3385]: 0000:03:01.7 Ethernet controller: Solarflare Communications SFC9120 10G Ethernet Controller (Virtual Function) (rev 01)
May 06 15:43:36 pve lspci[3385]: 0000:03:02.0 Ethernet controller: Solarflare Communications SFC9120 10G Ethernet Controller (Virtual Function) (rev 01)
May 06 15:43:36 pve lspci[3385]: 0000:03:02.1 Ethernet controller: Solarflare Communications SFC9120 10G Ethernet Controller (Virtual Function) (rev 01)
May 06 15:43:36 pve systemd[1]: Finished Enable SR-IOV and detach guest VFs from host.
6: Use the script with these commands:
Code:
systemctl start sriov-vfs.service
systemctl stop sriov-vfs.service
systemctl reload sriov-vfs.service
systemctl status sriov-vfs.service
systemctl enable sriov-vfs.service
systemctl disable sriov-vfs.service
------
7: (for reference) This original example creates 4 VF's per port, sets the MAC addresses for the VF's, and then detaches them from the host to use in pcie passthrough mode in a VM:
Code:
[Unit]
Description=Enable SR-IOV and detach guest VFs from host
Requires=network.target
After=network.target
[Service]
Type=oneshot
RemainAfterExit=yes
# Create NIC VFs
ExecStart=/usr/bin/bash -c 'echo 4 > /sys/class/net/ens6f0np0/device/sriov_numvfs'
ExecStart=/usr/bin/bash -c 'echo 4 > /sys/class/net/ens6f1np1/device/sriov_numvfs'
# Set static MACs for VFs
ExecStart=/usr/bin/bash -c '/usr/bin/ip link set ens6f0np0 vf 0 mac xx:xx:xx:xx:xx:xx'
ExecStart=/usr/bin/bash -c '/usr/bin/ip link set ens6f0np0 vf 1 mac xx:xx:xx:xx:xx:xx'
ExecStart=/usr/bin/bash -c '/usr/bin/ip link set ens6f0np0 vf 2 mac xx:xx:xx:xx:xx:xx'
ExecStart=/usr/bin/bash -c '/usr/bin/ip link set ens6f0np0 vf 3 mac xx:xx:xx:xx:xx:xx'
ExecStart=/usr/bin/bash -c '/usr/bin/ip link set ens6f1np1 vf 0 mac xx:xx:xx:xx:xx:xx'
ExecStart=/usr/bin/bash -c '/usr/bin/ip link set ens6f1np1 vf 1 mac xx:xx:xx:xx:xx:xx'
ExecStart=/usr/bin/bash -c '/usr/bin/ip link set ens6f1np1 vf 2 mac xx:xx:xx:xx:xx:xx'
ExecStart=/usr/bin/bash -c '/usr/bin/ip link set ens6f1np1 vf 3 mac xx:xx:xx:xx:xx:xx'
# Detach VFs from host
ExecStart=/usr/bin/bash -c 'echo 0000:03:00.2 > /sys/bus/pci/devices/0000\\:03\\:00.2/driver/unbind'
ExecStart=/usr/bin/bash -c 'echo 0000:03:00.3 > /sys/bus/pci/devices/0000\\:03\\:00.3/driver/unbind'
ExecStart=/usr/bin/bash -c 'echo 0000:03:00.4 > /sys/bus/pci/devices/0000\\:03\\:00.4/driver/unbind'
ExecStart=/usr/bin/bash -c 'echo 0000:03:00.5 > /sys/bus/pci/devices/0000\\:03\\:00.5/driver/unbind'
ExecStart=/usr/bin/bash -c 'echo 0000:03:00.6 > /sys/bus/pci/devices/0000\\:03\\:00.6/driver/unbind'
ExecStart=/usr/bin/bash -c 'echo 0000:03:00.7 > /sys/bus/pci/devices/0000\\:03\\:00.7/driver/unbind'
ExecStart=/usr/bin/bash -c 'echo 0000:03:01.0 > /sys/bus/pci/devices/0000\\:03\\:01.0/driver/unbind'
ExecStart=/usr/bin/bash -c 'echo 0000:03:01.1 > /sys/bus/pci/devices/0000\\:03\\:01.1/driver/unbind'
# List new VFs
ExecStart=/usr/bin/lspci -D -d1924:
# Destroy VFs
ExecStop=/usr/bin/bash -c 'echo 0 > /sys/class/net/ens6f0np0/device/sriov_numvfs'
ExecStop=/usr/bin/bash -c 'echo 0 > /sys/class/net/ens6f1np1/device/sriov_numvfs'
# Reload NIC VFs
ExecReload=/usr/bin/bash -c 'echo 0 > /sys/class/net/ens6f0np0/device/sriov_numvfs'
ExecReload=/usr/bin/bash -c 'echo 0 > /sys/class/net/ens6f1np1/device/sriov_numvfs'
ExecReload=/usr/bin/bash -c 'echo 4 > /sys/class/net/ens6f0np0/device/sriov_numvfs'
ExecReload=/usr/bin/bash -c 'echo 4 > /sys/class/net/ens6f1np1/device/sriov_numvfs'
ExecReload=/usr/bin/bash -c '/usr/bin/ip link set ens6f0np0 vf 0 mac xx:xx:xx:xx:xx:xx'
ExecReload=/usr/bin/bash -c '/usr/bin/ip link set ens6f0np0 vf 1 mac xx:xx:xx:xx:xx:xx'
ExecReload=/usr/bin/bash -c '/usr/bin/ip link set ens6f0np0 vf 2 mac xx:xx:xx:xx:xx:xx'
ExecReload=/usr/bin/bash -c '/usr/bin/ip link set ens6f0np0 vf 3 mac xx:xx:xx:xx:xx:xx'
ExecReload=/usr/bin/bash -c '/usr/bin/ip link set ens6f1np1 vf 0 mac xx:xx:xx:xx:xx:xx'
ExecReload=/usr/bin/bash -c '/usr/bin/ip link set ens6f1np1 vf 1 mac xx:xx:xx:xx:xx:xx'
ExecReload=/usr/bin/bash -c '/usr/bin/ip link set ens6f1np1 vf 2 mac xx:xx:xx:xx:xx:xx'
ExecReload=/usr/bin/bash -c '/usr/bin/ip link set ens6f1np1 vf 3 mac xx:xx:xx:xx:xx:xx'
ExecReload=/usr/bin/bash -c 'echo 0000:03:00.2 > /sys/bus/pci/devices/0000\\:03\\:00.2/driver/unbind'
ExecReload=/usr/bin/bash -c 'echo 0000:03:00.3 > /sys/bus/pci/devices/0000\\:03\\:00.3/driver/unbind'
ExecReload=/usr/bin/bash -c 'echo 0000:03:00.4 > /sys/bus/pci/devices/0000\\:03\\:00.4/driver/unbind'
ExecReload=/usr/bin/bash -c 'echo 0000:03:00.5 > /sys/bus/pci/devices/0000\\:03\\:00.5/driver/unbind'
ExecReload=/usr/bin/bash -c 'echo 0000:03:00.6 > /sys/bus/pci/devices/0000\\:03\\:00.6/driver/unbind'
ExecReload=/usr/bin/bash -c 'echo 0000:03:00.7 > /sys/bus/pci/devices/0000\\:03\\:00.7/driver/unbind'
ExecReload=/usr/bin/bash -c 'echo 0000:03:01.0 > /sys/bus/pci/devices/0000\\:03\\:01.0/driver/unbind'
ExecReload=/usr/bin/bash -c 'echo 0000:03:01.1 > /sys/bus/pci/devices/0000\\:03\\:01.1/driver/unbind'
ExecReload=/usr/bin/lspci -D -d1924:
[Install]
WantedBy=multi-user.target
Last edited: