My Proxmox/Debian systemd script for partitioning and detaching Solarflare SFN7x22F VF's

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

heromode

Active Member
May 25, 2020
379
199
43
EDIT: The original script i posted only works when partitioning interfaces for use with pcie passthrough in VM's. But i quickly ran into issues when i wanted to use interfaces for the proxmox host as well, because the script was being run after proxmox configures network interfaces, and starts the firewall service, which resulted in the interfaces being non-active, and the firewall was unaware of them resulting in any rules created did nothing.

I have updated the script so it works both for the proxmox host as bridgeable interfaces, raw interfaces, as well as using passthrough for VM's.

I have moved the original script to the end for reference. The original script is still valid for a simple setup where you only want to partition and detach interfaces from the host, and use them as native interfaces in VM's.

I discuss the changes i made later down in the thread. Comments and suggestions are highly welcome. As i write this everything seems to work, but issues might arise later, or in different use cases.


---

So as i didn't find any resources for this i ended up creating a systemd script for proxmox to partition, set static MAC's, and detaching NIC Virtual Functions to use for my SFN7022F dual port SFP+ adapter in Proxmox. I posted this earlier in the proxmox forums, but thought i'd share it here as well in case anyone needs it as a template.

It's been working great so far, and the start, stop and reload functions have been really handy.

To set the static MAC's, i first partitioned the ports manually, and then used the generated random MAC's.

This example creates 8 VF's per port, sets static MAC addresses for the VF's, and then detaches 7 interfaces per port from the host to use in pcie passthrough mode in a VM.

I wanted to keep 2 interfaces per port for use in proxmox: One interface for use as a bridge host interface, so i can use that for accessing the host with SSH and WebUI over 10Gbit, for use in containers, and also VM's that uses virtio networking.

The second one for things like a iscsi target serving incoming iscsi initiators, which i don't want to run over a bridge.


1. Enable 'Full Feature / virtualization mode' Firmware, and 'SRIOV' Switch mode, and set the number of virtual functions per port:

Code:
sfboot firmware-variant=full-feature
sfboot switch-mode=sriov
sfboot vf-count=8

2. Create the systemd script, use a filename that suits you:

Code:
nano /etc/systemd/system/sriov-vfs.service

3. My example. To get the correct interface names and pcie addresses for your system, you need to first do this manually step by step. The double \\ in the script is to fix an annoying but non-fatal error message that a single \ would produce.

Code:
[Unit]
Description=Enable SR-IOV and detach guest VFs from host
Before=network-online.target network-pre.target
Wants=network-pre.target
[Service]
Type=oneshot
RemainAfterExit=yes
# Create NIC VFs
ExecStart=/usr/bin/bash -c 'echo 8 > /sys/class/net/ens6f0np0/device/sriov_numvfs'
ExecStart=/usr/bin/bash -c 'echo 8 > /sys/class/net/ens6f1np1/device/sriov_numvfs'
# Set static MACs for VFs
ExecStart=/usr/bin/bash -c '/usr/bin/ip link set ens6f0np0 vf 0 mac 76:9e:17:83:39:e5'
ExecStart=/usr/bin/bash -c '/usr/bin/ip link set ens6f0np0 vf 1 mac 46:2c:6d:24:6b:1b'
ExecStart=/usr/bin/bash -c '/usr/bin/ip link set ens6f0np0 vf 2 mac 3e:47:48:12:ed:94'
ExecStart=/usr/bin/bash -c '/usr/bin/ip link set ens6f0np0 vf 3 mac be:e3:6a:f3:8f:ac'
ExecStart=/usr/bin/bash -c '/usr/bin/ip link set ens6f0np0 vf 4 mac 62:8f:3d:bb:02:08'
ExecStart=/usr/bin/bash -c '/usr/bin/ip link set ens6f0np0 vf 5 mac ae:91:57:b9:14:7f'
ExecStart=/usr/bin/bash -c '/usr/bin/ip link set ens6f0np0 vf 6 mac 5a:c2:08:a9:68:a7'
ExecStart=/usr/bin/bash -c '/usr/bin/ip link set ens6f0np0 vf 7 mac b2:f0:18:af:cb:c5'
ExecStart=/usr/bin/bash -c '/usr/bin/ip link set ens6f1np1 vf 0 mac 16:47:7c:a8:95:98'
ExecStart=/usr/bin/bash -c '/usr/bin/ip link set ens6f1np1 vf 1 mac a6:c7:c5:7f:9c:22'
ExecStart=/usr/bin/bash -c '/usr/bin/ip link set ens6f1np1 vf 2 mac b6:0f:45:34:5e:19'
ExecStart=/usr/bin/bash -c '/usr/bin/ip link set ens6f1np1 vf 3 mac 2a:f7:37:84:31:30'
ExecStart=/usr/bin/bash -c '/usr/bin/ip link set ens6f1np1 vf 4 mac 8a:fa:f8:c5:0b:93'
ExecStart=/usr/bin/bash -c '/usr/bin/ip link set ens6f1np1 vf 5 mac b2:f5:d5:2f:79:06'
ExecStart=/usr/bin/bash -c '/usr/bin/ip link set ens6f1np1 vf 6 mac c2:92:f5:fa:32:20'
ExecStart=/usr/bin/bash -c '/usr/bin/ip link set ens6f1np1 vf 7 mac 2e:fb:29:1e:48:31'
# Detach VFs from host
ExecStart=/usr/bin/bash -c 'echo 0000:03:00.3 > /sys/bus/pci/devices/0000\\:03\\:00.3/driver/unbind'
ExecStart=/usr/bin/bash -c 'echo 0000:03:00.4 > /sys/bus/pci/devices/0000\\:03\\:00.4/driver/unbind'
ExecStart=/usr/bin/bash -c 'echo 0000:03:00.5 > /sys/bus/pci/devices/0000\\:03\\:00.5/driver/unbind'
ExecStart=/usr/bin/bash -c 'echo 0000:03:00.6 > /sys/bus/pci/devices/0000\\:03\\:00.6/driver/unbind'
ExecStart=/usr/bin/bash -c 'echo 0000:03:00.7 > /sys/bus/pci/devices/0000\\:03\\:00.7/driver/unbind'
ExecStart=/usr/bin/bash -c 'echo 0000:03:01.0 > /sys/bus/pci/devices/0000\\:03\\:01.0/driver/unbind'
ExecStart=/usr/bin/bash -c 'echo 0000:03:01.1 > /sys/bus/pci/devices/0000\\:03\\:01.1/driver/unbind'
ExecStart=/usr/bin/bash -c 'echo 0000:03:01.3 > /sys/bus/pci/devices/0000\\:03\\:01.3/driver/unbind'
ExecStart=/usr/bin/bash -c 'echo 0000:03:01.4 > /sys/bus/pci/devices/0000\\:03\\:01.4/driver/unbind'
ExecStart=/usr/bin/bash -c 'echo 0000:03:01.5 > /sys/bus/pci/devices/0000\\:03\\:01.5/driver/unbind'
ExecStart=/usr/bin/bash -c 'echo 0000:03:01.6 > /sys/bus/pci/devices/0000\\:03\\:01.6/driver/unbind'
ExecStart=/usr/bin/bash -c 'echo 0000:03:01.7 > /sys/bus/pci/devices/0000\\:03\\:01.7/driver/unbind'
ExecStart=/usr/bin/bash -c 'echo 0000:03:02.0 > /sys/bus/pci/devices/0000\\:03\\:02.0/driver/unbind'
ExecStart=/usr/bin/bash -c 'echo 0000:03:02.1 > /sys/bus/pci/devices/0000\\:03\\:02.1/driver/unbind'
# List new VFs
ExecStart=/usr/bin/lspci -D -d1924:
# Destroy VFs
ExecStop=/usr/bin/bash -c 'echo 0 > /sys/class/net/ens6f0np0/device/sriov_numvfs'
ExecStop=/usr/bin/bash -c 'echo 0 > /sys/class/net/ens6f1np1/device/sriov_numvfs'
# Reload NIC VFs
ExecReload=/usr/bin/bash -c 'echo 0 > /sys/class/net/ens6f0np0/device/sriov_numvfs'
ExecReload=/usr/bin/bash -c 'echo 0 > /sys/class/net/ens6f1np1/device/sriov_numvfs'
ExecReload=/usr/bin/bash -c 'echo 8 > /sys/class/net/ens6f0np0/device/sriov_numvfs'
ExecReload=/usr/bin/bash -c 'echo 8 > /sys/class/net/ens6f1np1/device/sriov_numvfs'
ExecReload=/usr/bin/bash -c '/usr/bin/ip link set ens6f0np0 vf 0 mac 76:9e:17:83:39:e5'
ExecReload=/usr/bin/bash -c '/usr/bin/ip link set ens6f0np0 vf 1 mac 46:2c:6d:24:6b:1b'
ExecReload=/usr/bin/bash -c '/usr/bin/ip link set ens6f0np0 vf 2 mac 3e:47:48:12:ed:94'
ExecReload=/usr/bin/bash -c '/usr/bin/ip link set ens6f0np0 vf 3 mac be:e3:6a:f3:8f:ac'
ExecReload=/usr/bin/bash -c '/usr/bin/ip link set ens6f0np0 vf 4 mac 62:8f:3d:bb:02:08'
ExecReload=/usr/bin/bash -c '/usr/bin/ip link set ens6f0np0 vf 5 mac ae:91:57:b9:14:7f'
ExecReload=/usr/bin/bash -c '/usr/bin/ip link set ens6f0np0 vf 6 mac 5a:c2:08:a9:68:a7'
ExecReload=/usr/bin/bash -c '/usr/bin/ip link set ens6f0np0 vf 7 mac b2:f0:18:af:cb:c5'
ExecReload=/usr/bin/bash -c '/usr/bin/ip link set ens6f1np1 vf 0 mac 16:47:7c:a8:95:98'
ExecReload=/usr/bin/bash -c '/usr/bin/ip link set ens6f1np1 vf 1 mac a6:c7:c5:7f:9c:22'
ExecReload=/usr/bin/bash -c '/usr/bin/ip link set ens6f1np1 vf 2 mac b6:0f:45:34:5e:19'
ExecReload=/usr/bin/bash -c '/usr/bin/ip link set ens6f1np1 vf 3 mac 2a:f7:37:84:31:30'
ExecReload=/usr/bin/bash -c '/usr/bin/ip link set ens6f1np1 vf 4 mac 8a:fa:f8:c5:0b:93'
ExecReload=/usr/bin/bash -c '/usr/bin/ip link set ens6f1np1 vf 5 mac b2:f5:d5:2f:79:06'
ExecReload=/usr/bin/bash -c '/usr/bin/ip link set ens6f1np1 vf 6 mac c2:92:f5:fa:32:20'
ExecReload=/usr/bin/bash -c '/usr/bin/ip link set ens6f1np1 vf 7 mac 2e:fb:29:1e:48:31'
ExecReload=/usr/bin/bash -c 'echo 0000:03:00.3 > /sys/bus/pci/devices/0000\\:03\\:00.3/driver/unbind'
ExecReload=/usr/bin/bash -c 'echo 0000:03:00.4 > /sys/bus/pci/devices/0000\\:03\\:00.4/driver/unbind'
ExecReload=/usr/bin/bash -c 'echo 0000:03:00.5 > /sys/bus/pci/devices/0000\\:03\\:00.5/driver/unbind'
ExecReload=/usr/bin/bash -c 'echo 0000:03:00.6 > /sys/bus/pci/devices/0000\\:03\\:00.6/driver/unbind'
ExecReload=/usr/bin/bash -c 'echo 0000:03:00.7 > /sys/bus/pci/devices/0000\\:03\\:00.7/driver/unbind'
ExecReload=/usr/bin/bash -c 'echo 0000:03:01.0 > /sys/bus/pci/devices/0000\\:03\\:01.0/driver/unbind'
ExecReload=/usr/bin/bash -c 'echo 0000:03:01.1 > /sys/bus/pci/devices/0000\\:03\\:01.1/driver/unbind'
ExecReload=/usr/bin/bash -c 'echo 0000:03:01.3 > /sys/bus/pci/devices/0000\\:03\\:01.3/driver/unbind'
ExecReload=/usr/bin/bash -c 'echo 0000:03:01.4 > /sys/bus/pci/devices/0000\\:03\\:01.4/driver/unbind'
ExecReload=/usr/bin/bash -c 'echo 0000:03:01.5 > /sys/bus/pci/devices/0000\\:03\\:01.5/driver/unbind'
ExecReload=/usr/bin/bash -c 'echo 0000:03:01.6 > /sys/bus/pci/devices/0000\\:03\\:01.6/driver/unbind'
ExecReload=/usr/bin/bash -c 'echo 0000:03:01.7 > /sys/bus/pci/devices/0000\\:03\\:01.7/driver/unbind'
ExecReload=/usr/bin/bash -c 'echo 0000:03:02.0 > /sys/bus/pci/devices/0000\\:03\\:02.0/driver/unbind'
ExecReload=/usr/bin/bash -c 'echo 0000:03:02.1 > /sys/bus/pci/devices/0000\\:03\\:02.1/driver/unbind'
ExecReload=/usr/bin/lspci -D -d1924:
[Install]
WantedBy=multi-user.target

4. Register, enable and start the script:

Code:
systemctl daemon-reload
systemctl enable sriov-vfs.service
systemctl start sriov-vfs.service

5. Check that the VF's have been created (this also prints in the logs), and check the status of the service:

Code:
# lspci -D -d1924:
0000:03:00.0 Ethernet controller: Solarflare Communications SFC9120 10G Ethernet Controller (rev 01)
0000:03:00.1 Ethernet controller: Solarflare Communications SFC9120 10G Ethernet Controller (rev 01)
0000:03:00.2 Ethernet controller: Solarflare Communications SFC9120 10G Ethernet Controller (Virtual Function) (rev 01)
0000:03:00.3 Ethernet controller: Solarflare Communications SFC9120 10G Ethernet Controller (Virtual Function) (rev 01)
0000:03:00.4 Ethernet controller: Solarflare Communications SFC9120 10G Ethernet Controller (Virtual Function) (rev 01)
0000:03:00.5 Ethernet controller: Solarflare Communications SFC9120 10G Ethernet Controller (Virtual Function) (rev 01)
0000:03:00.6 Ethernet controller: Solarflare Communications SFC9120 10G Ethernet Controller (Virtual Function) (rev 01)
0000:03:00.7 Ethernet controller: Solarflare Communications SFC9120 10G Ethernet Controller (Virtual Function) (rev 01)
0000:03:01.0 Ethernet controller: Solarflare Communications SFC9120 10G Ethernet Controller (Virtual Function) (rev 01)
0000:03:01.1 Ethernet controller: Solarflare Communications SFC9120 10G Ethernet Controller (Virtual Function) (rev 01)
0000:03:01.2 Ethernet controller: Solarflare Communications SFC9120 10G Ethernet Controller (Virtual Function) (rev 01)
0000:03:01.3 Ethernet controller: Solarflare Communications SFC9120 10G Ethernet Controller (Virtual Function) (rev 01)
0000:03:01.4 Ethernet controller: Solarflare Communications SFC9120 10G Ethernet Controller (Virtual Function) (rev 01)
0000:03:01.5 Ethernet controller: Solarflare Communications SFC9120 10G Ethernet Controller (Virtual Function) (rev 01)
0000:03:01.6 Ethernet controller: Solarflare Communications SFC9120 10G Ethernet Controller (Virtual Function) (rev 01)
0000:03:01.7 Ethernet controller: Solarflare Communications SFC9120 10G Ethernet Controller (Virtual Function) (rev 01)
0000:03:02.0 Ethernet controller: Solarflare Communications SFC9120 10G Ethernet Controller (Virtual Function) (rev 01)
0000:03:02.1 Ethernet controller: Solarflare Communications SFC9120 10G Ethernet Controller (Virtual Function) (rev 01)

# systemctl status sriov-vfs.service
● sriov-vfs.service - Enable SR-IOV and detach guest VFs from host
Loaded: loaded (/etc/systemd/system/sriov-vfs.service; enabled; vendor preset: enabled)
Active: active (exited) since Sat 2023-05-06 15:43:36 EEST; 52min ago
Process: 3002 ExecStart=/usr/bin/bash -c echo 8 > /sys/class/net/ens6f0np0/device/sriov_numvfs (code=exited, status=0/SUCCESS)
Process: 3226 ExecStart=/usr/bin/bash -c echo 8 > /sys/class/net/ens6f1np1/device/sriov_numvfs (code=exited, status=0/SUCCESS)
Process: 3243 ExecStart=/usr/bin/bash -c /usr/bin/ip link set ens6f0np0 vf 0 mac 76:9e:17:83:39:e5 (code=exited, status=0/SUCCESS)
Process: 3245 ExecStart=/usr/bin/bash -c /usr/bin/ip link set ens6f0np0 vf 1 mac 46:2c:6d:24:6b:1b (code=exited, status=0/SUCCESS)
Process: 3246 ExecStart=/usr/bin/bash -c /usr/bin/ip link set ens6f0np0 vf 2 mac 3e:47:48:12:ed:94 (code=exited, status=0/SUCCESS)
Process: 3247 ExecStart=/usr/bin/bash -c /usr/bin/ip link set ens6f0np0 vf 3 mac be:e3:6a:f3:8f:ac (code=exited, status=0/SUCCESS)
Process: 3248 ExecStart=/usr/bin/bash -c /usr/bin/ip link set ens6f0np0 vf 4 mac 62:8f:3d:bb:02:08 (code=exited, status=0/SUCCESS)
Process: 3249 ExecStart=/usr/bin/bash -c /usr/bin/ip link set ens6f0np0 vf 5 mac ae:91:57:b9:14:7f (code=exited, status=0/SUCCESS)
Process: 3250 ExecStart=/usr/bin/bash -c /usr/bin/ip link set ens6f0np0 vf 6 mac 5a:c2:08:a9:68:a7 (code=exited, status=0/SUCCESS)
Process: 3251 ExecStart=/usr/bin/bash -c /usr/bin/ip link set ens6f0np0 vf 7 mac b2:f0:18:af:cb:c5 (code=exited, status=0/SUCCESS)
Process: 3252 ExecStart=/usr/bin/bash -c /usr/bin/ip link set ens6f1np1 vf 0 mac 16:47:7c:a8:95:98 (code=exited, status=0/SUCCESS)
Process: 3253 ExecStart=/usr/bin/bash -c /usr/bin/ip link set ens6f1np1 vf 1 mac a6:c7:c5:7f:9c:22 (code=exited, status=0/SUCCESS)
Process: 3254 ExecStart=/usr/bin/bash -c /usr/bin/ip link set ens6f1np1 vf 2 mac b6:0f:45:34:5e:19 (code=exited, status=0/SUCCESS)
Process: 3255 ExecStart=/usr/bin/bash -c /usr/bin/ip link set ens6f1np1 vf 3 mac 2a:f7:37:84:31:30 (code=exited, status=0/SUCCESS)
Process: 3256 ExecStart=/usr/bin/bash -c /usr/bin/ip link set ens6f1np1 vf 4 mac 8a:fa:f8:c5:0b:93 (code=exited, status=0/SUCCESS)
Process: 3258 ExecStart=/usr/bin/bash -c /usr/bin/ip link set ens6f1np1 vf 5 mac b2:f5:d5:2f:79:06 (code=exited, status=0/SUCCESS)
Process: 3259 ExecStart=/usr/bin/bash -c /usr/bin/ip link set ens6f1np1 vf 6 mac c2:92:f5:fa:32:20 (code=exited, status=0/SUCCESS)
Process: 3260 ExecStart=/usr/bin/bash -c /usr/bin/ip link set ens6f1np1 vf 7 mac 2e:fb:29:1e:48:31 (code=exited, status=0/SUCCESS)
Process: 3261 ExecStart=/usr/bin/bash -c echo 0000:03:00.3 > /sys/bus/pci/devices/0000\:03\:00.3/driver/unbind (code=exited, status=0/SUCCESS)
Process: 3263 ExecStart=/usr/bin/bash -c echo 0000:03:00.4 > /sys/bus/pci/devices/0000\:03\:00.4/driver/unbind (code=exited, status=0/SUCCESS)
Process: 3265 ExecStart=/usr/bin/bash -c echo 0000:03:00.5 > /sys/bus/pci/devices/0000\:03\:00.5/driver/unbind (code=exited, status=0/SUCCESS)
Process: 3266 ExecStart=/usr/bin/bash -c echo 0000:03:00.6 > /sys/bus/pci/devices/0000\:03\:00.6/driver/unbind (code=exited, status=0/SUCCESS)
Process: 3268 ExecStart=/usr/bin/bash -c echo 0000:03:00.7 > /sys/bus/pci/devices/0000\:03\:00.7/driver/unbind (code=exited, status=0/SUCCESS)
Process: 3269 ExecStart=/usr/bin/bash -c echo 0000:03:01.0 > /sys/bus/pci/devices/0000\:03\:01.0/driver/unbind (code=exited, status=0/SUCCESS)
Process: 3270 ExecStart=/usr/bin/bash -c echo 0000:03:01.1 > /sys/bus/pci/devices/0000\:03\:01.1/driver/unbind (code=exited, status=0/SUCCESS)
Process: 3272 ExecStart=/usr/bin/bash -c echo 0000:03:01.3 > /sys/bus/pci/devices/0000\:03\:01.3/driver/unbind (code=exited, status=0/SUCCESS)
Process: 3273 ExecStart=/usr/bin/bash -c echo 0000:03:01.4 > /sys/bus/pci/devices/0000\:03\:01.4/driver/unbind (code=exited, status=0/SUCCESS)
Process: 3274 ExecStart=/usr/bin/bash -c echo 0000:03:01.5 > /sys/bus/pci/devices/0000\:03\:01.5/driver/unbind (code=exited, status=0/SUCCESS)
Process: 3379 ExecStart=/usr/bin/bash -c echo 0000:03:01.6 > /sys/bus/pci/devices/0000\:03\:01.6/driver/unbind (code=exited, status=0/SUCCESS)
Process: 3380 ExecStart=/usr/bin/bash -c echo 0000:03:01.7 > /sys/bus/pci/devices/0000\:03\:01.7/driver/unbind (code=exited, status=0/SUCCESS)
Process: 3383 ExecStart=/usr/bin/bash -c echo 0000:03:02.0 > /sys/bus/pci/devices/0000\:03\:02.0/driver/unbind (code=exited, status=0/SUCCESS)
Process: 3384 ExecStart=/usr/bin/bash -c echo 0000:03:02.1 > /sys/bus/pci/devices/0000\:03\:02.1/driver/unbind (code=exited, status=0/SUCCESS)
Process: 3385 ExecStart=/usr/bin/lspci -D -d1924: (code=exited, status=0/SUCCESS)
Main PID: 3385 (code=exited, status=0/SUCCESS)
CPU: 185ms

May 06 15:43:36 pve lspci[3385]: 0000:03:01.1 Ethernet controller: Solarflare Communications SFC9120 10G Ethernet Controller (Virtual Function) (rev 01)
May 06 15:43:36 pve lspci[3385]: 0000:03:01.2 Ethernet controller: Solarflare Communications SFC9120 10G Ethernet Controller (Virtual Function) (rev 01)
May 06 15:43:36 pve lspci[3385]: 0000:03:01.3 Ethernet controller: Solarflare Communications SFC9120 10G Ethernet Controller (Virtual Function) (rev 01)
May 06 15:43:36 pve lspci[3385]: 0000:03:01.4 Ethernet controller: Solarflare Communications SFC9120 10G Ethernet Controller (Virtual Function) (rev 01)
May 06 15:43:36 pve lspci[3385]: 0000:03:01.5 Ethernet controller: Solarflare Communications SFC9120 10G Ethernet Controller (Virtual Function) (rev 01)
May 06 15:43:36 pve lspci[3385]: 0000:03:01.6 Ethernet controller: Solarflare Communications SFC9120 10G Ethernet Controller (Virtual Function) (rev 01)
May 06 15:43:36 pve lspci[3385]: 0000:03:01.7 Ethernet controller: Solarflare Communications SFC9120 10G Ethernet Controller (Virtual Function) (rev 01)
May 06 15:43:36 pve lspci[3385]: 0000:03:02.0 Ethernet controller: Solarflare Communications SFC9120 10G Ethernet Controller (Virtual Function) (rev 01)
May 06 15:43:36 pve lspci[3385]: 0000:03:02.1 Ethernet controller: Solarflare Communications SFC9120 10G Ethernet Controller (Virtual Function) (rev 01)
May 06 15:43:36 pve systemd[1]: Finished Enable SR-IOV and detach guest VFs from host.

6: Use the script with these commands:

Code:
systemctl start sriov-vfs.service
systemctl stop sriov-vfs.service
systemctl reload sriov-vfs.service
systemctl status sriov-vfs.service
systemctl enable sriov-vfs.service
systemctl disable sriov-vfs.service

------
7: (for reference) This original example creates 4 VF's per port, sets the MAC addresses for the VF's, and then detaches them from the host to use in pcie passthrough mode in a VM:

Code:
[Unit]
Description=Enable SR-IOV and detach guest VFs from host
Requires=network.target
After=network.target
[Service]
Type=oneshot
RemainAfterExit=yes
# Create NIC VFs
ExecStart=/usr/bin/bash -c 'echo 4 > /sys/class/net/ens6f0np0/device/sriov_numvfs'
ExecStart=/usr/bin/bash -c 'echo 4 > /sys/class/net/ens6f1np1/device/sriov_numvfs'
# Set static MACs for VFs
ExecStart=/usr/bin/bash -c '/usr/bin/ip link set ens6f0np0 vf 0 mac xx:xx:xx:xx:xx:xx'
ExecStart=/usr/bin/bash -c '/usr/bin/ip link set ens6f0np0 vf 1 mac xx:xx:xx:xx:xx:xx'
ExecStart=/usr/bin/bash -c '/usr/bin/ip link set ens6f0np0 vf 2 mac xx:xx:xx:xx:xx:xx'
ExecStart=/usr/bin/bash -c '/usr/bin/ip link set ens6f0np0 vf 3 mac xx:xx:xx:xx:xx:xx'
ExecStart=/usr/bin/bash -c '/usr/bin/ip link set ens6f1np1 vf 0 mac xx:xx:xx:xx:xx:xx'
ExecStart=/usr/bin/bash -c '/usr/bin/ip link set ens6f1np1 vf 1 mac xx:xx:xx:xx:xx:xx'
ExecStart=/usr/bin/bash -c '/usr/bin/ip link set ens6f1np1 vf 2 mac xx:xx:xx:xx:xx:xx'
ExecStart=/usr/bin/bash -c '/usr/bin/ip link set ens6f1np1 vf 3 mac xx:xx:xx:xx:xx:xx'
# Detach VFs from host
ExecStart=/usr/bin/bash -c 'echo 0000:03:00.2 > /sys/bus/pci/devices/0000\\:03\\:00.2/driver/unbind'
ExecStart=/usr/bin/bash -c 'echo 0000:03:00.3 > /sys/bus/pci/devices/0000\\:03\\:00.3/driver/unbind'
ExecStart=/usr/bin/bash -c 'echo 0000:03:00.4 > /sys/bus/pci/devices/0000\\:03\\:00.4/driver/unbind'
ExecStart=/usr/bin/bash -c 'echo 0000:03:00.5 > /sys/bus/pci/devices/0000\\:03\\:00.5/driver/unbind'
ExecStart=/usr/bin/bash -c 'echo 0000:03:00.6 > /sys/bus/pci/devices/0000\\:03\\:00.6/driver/unbind'
ExecStart=/usr/bin/bash -c 'echo 0000:03:00.7 > /sys/bus/pci/devices/0000\\:03\\:00.7/driver/unbind'
ExecStart=/usr/bin/bash -c 'echo 0000:03:01.0 > /sys/bus/pci/devices/0000\\:03\\:01.0/driver/unbind'
ExecStart=/usr/bin/bash -c 'echo 0000:03:01.1 > /sys/bus/pci/devices/0000\\:03\\:01.1/driver/unbind'
# List new VFs
ExecStart=/usr/bin/lspci -D -d1924:
# Destroy VFs
ExecStop=/usr/bin/bash -c 'echo 0 > /sys/class/net/ens6f0np0/device/sriov_numvfs'
ExecStop=/usr/bin/bash -c 'echo 0 > /sys/class/net/ens6f1np1/device/sriov_numvfs'
# Reload NIC VFs
ExecReload=/usr/bin/bash -c 'echo 0 > /sys/class/net/ens6f0np0/device/sriov_numvfs'
ExecReload=/usr/bin/bash -c 'echo 0 > /sys/class/net/ens6f1np1/device/sriov_numvfs'
ExecReload=/usr/bin/bash -c 'echo 4 > /sys/class/net/ens6f0np0/device/sriov_numvfs'
ExecReload=/usr/bin/bash -c 'echo 4 > /sys/class/net/ens6f1np1/device/sriov_numvfs'
ExecReload=/usr/bin/bash -c '/usr/bin/ip link set ens6f0np0 vf 0 mac xx:xx:xx:xx:xx:xx'
ExecReload=/usr/bin/bash -c '/usr/bin/ip link set ens6f0np0 vf 1 mac xx:xx:xx:xx:xx:xx'
ExecReload=/usr/bin/bash -c '/usr/bin/ip link set ens6f0np0 vf 2 mac xx:xx:xx:xx:xx:xx'
ExecReload=/usr/bin/bash -c '/usr/bin/ip link set ens6f0np0 vf 3 mac xx:xx:xx:xx:xx:xx'
ExecReload=/usr/bin/bash -c '/usr/bin/ip link set ens6f1np1 vf 0 mac xx:xx:xx:xx:xx:xx'
ExecReload=/usr/bin/bash -c '/usr/bin/ip link set ens6f1np1 vf 1 mac xx:xx:xx:xx:xx:xx'
ExecReload=/usr/bin/bash -c '/usr/bin/ip link set ens6f1np1 vf 2 mac xx:xx:xx:xx:xx:xx'
ExecReload=/usr/bin/bash -c '/usr/bin/ip link set ens6f1np1 vf 3 mac xx:xx:xx:xx:xx:xx'
ExecReload=/usr/bin/bash -c 'echo 0000:03:00.2 > /sys/bus/pci/devices/0000\\:03\\:00.2/driver/unbind'
ExecReload=/usr/bin/bash -c 'echo 0000:03:00.3 > /sys/bus/pci/devices/0000\\:03\\:00.3/driver/unbind'
ExecReload=/usr/bin/bash -c 'echo 0000:03:00.4 > /sys/bus/pci/devices/0000\\:03\\:00.4/driver/unbind'
ExecReload=/usr/bin/bash -c 'echo 0000:03:00.5 > /sys/bus/pci/devices/0000\\:03\\:00.5/driver/unbind'
ExecReload=/usr/bin/bash -c 'echo 0000:03:00.6 > /sys/bus/pci/devices/0000\\:03\\:00.6/driver/unbind'
ExecReload=/usr/bin/bash -c 'echo 0000:03:00.7 > /sys/bus/pci/devices/0000\\:03\\:00.7/driver/unbind'
ExecReload=/usr/bin/bash -c 'echo 0000:03:01.0 > /sys/bus/pci/devices/0000\\:03\\:01.0/driver/unbind'
ExecReload=/usr/bin/bash -c 'echo 0000:03:01.1 > /sys/bus/pci/devices/0000\\:03\\:01.1/driver/unbind'
ExecReload=/usr/bin/lspci -D -d1924:
[Install]
WantedBy=multi-user.target
 
Last edited:

Stephan

Well-Known Member
Apr 21, 2017
920
697
93
Germany
Handy. Why network-online.target instead of network.target?

Also to reduce duplicate code I like to write this more compact with nested for-loops and associative arrays which contain MACs as a /usr/local/sbin script in SysV rc.d style with brief parameter parsing at the top.
 
  • Like
Reactions: gb00s

heromode

Active Member
May 25, 2020
379
199
43
Handy. Why network-online.target instead of network.target?

Also to reduce duplicate code I like to write this more compact with nested for-loops and associative arrays which contain MACs as a /usr/local/sbin script in SysV rc.d style with brief parameter parsing at the top.
Exactly. I was hoping for knowledgeable feedback. I'm not an expert in these..

Any improvements are welcome. The only other place this example exists is i posted it in proxmox forums, but apart from that, there are examples, but none to my knowledge with the start, stop and reload functions. Let's create the perfect script..

For me editing and testing is a bit stressful since there are running VM's using the interfaces, but i should start by changing to network.target when i get the chance.. I'll be updating this thread when i get the chance of testing. Thx for feedback, keep it coming plz.
 
  • Like
Reactions: Patrick

heromode

Active Member
May 25, 2020
379
199
43
I'm not sure but i think that changing the vf count with sfboot requires a cold-boot each time. That is not something i'm willing to do on this system for testing, powering the system with multiple SSD's, GPU's and HDD's on and off repetitively tens of times.

If anyone has a test rig not used for anything else then i welcome any improvements. For now this example works perfectly for me, and it's running on a system with GPU pcie passthrough desktop VM's in active use, so in that sense any testing that requires the dreaded cold boot is basically not possible for me.
 

heromode

Active Member
May 25, 2020
379
199
43
Handy. Why network-online.target instead of network.target?

Also to reduce duplicate code I like to write this more compact with nested for-loops and associative arrays which contain MACs as a /usr/local/sbin script in SysV rc.d style with brief parameter parsing at the top.
OK. I changed to network.target in the original simple script. Seems to work fine, it makes no difference since the interfaces are never used by the host. But that script doesn't work when leaving some of the interfaces attached to the host for use in proxmox. I had to do some reading:

Running Services After the Network Is Up
https://linuxconfig.org/how-to-create-systemd-service-unit-in-linux

Based on the first article especially, i set [Unit] to:

Code:
[Unit]
Description=Enable SR-IOV and detach guest VFs from host
Before=network-online.target network-pre.target
Wants=network-pre.target
Now in journalctl it clearly initializes the interfaces before networking and firewall service:

Code:
May 06 15:43:36 pve systemd[1]: Finished Enable SR-IOV and detach guest VFs from host.
May 06 15:43:36 pve systemd[1]: Reached target Network (Pre).
May 06 15:43:36 pve systemd[1]: Starting Network initialization...
May 06 15:43:36 pve networking[3386]: networking: Configuring network interfaces

...

May 06 15:43:37 pve systemd[1]: Finished Network initialization.
May 06 15:43:37 pve systemd[1]: Reached target Network.
May 06 15:43:37 pve systemd[1]: Reached target Network is Online.
As i write this i can't remember why i included Before=network-online.target, I will have to test without it.

As for the more compact code and nested for-loops, generally when you set static MAC's, they are seldom changed, unless you want random MAC's, in which case you would just remove that code.

Changing the number of VF's using the sfboot tool also changes the order of the pcie addresses generated, so the script will break. That makes trying to create any kind of logic pretty cumbersome, as each time requires a cold reboot. You would basically have to start with 1 VF per port, and work upwards, with a cold reboot between each iteration, to know how your system organises the pcie addresses. And that would also not be portable, and could also change with the addition or removal of other pcie devices. So i see it as an excercise in futility really.

Please if anyone more knowledgeable than me spots some problems, or has insight, let me know so i can improve the OP.
 
Last edited: