Lenovo Tiny (M920q or M720q) with CX3 and SR-IOV

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

sophware

New Member
Nov 1, 2019
5
0
1
I haven't been able to get SR-IOV working on these Tinies. My thought is that the BIOS (motherboard, not just card card) needs to support it and that the Lenovos' BIOS doesn't.

Fairly confident about the former (that actual BIOS support is needed), since there is a plethora of search results saying so, and because my rack servers do have such options.

As to the latter, it seems likely people here would have experience (though maybe not in this forum, Networking). Anyone care to comment?

======

Less important:

Also, getting the FastBoot or the UEFI version isn't working for me to boot to SAN via iSCSI. It's probably a config thing on my part. If I'm doing things correctly, what I'm expecting is to boot to an install CD (like ESXi) and see a drive appear. I get no ctrl+B--the only way UEFI or FastBoot gets its config is from me putting it in the UEFI version when not booting Legacy. This is done by going into the motherboard BIOS and going to the ATA settings. I've had no success going directly to the card from some POST method. Anyway, no drive shows up. I think the target is set up correctly and suspect that the client/initiator side isn't.
 
Last edited:

sug7

New Member
Jul 11, 2022
2
0
1
Hi,

were you able to get this working? I have a M720q and looking to purchase a SR-IOV supported NIC card.
 

zer0sum

Well-Known Member
Mar 8, 2013
849
474
63
I have had it working without any issues on an m920q with an 8500T, using either a CX3 or an AOC-STGN-i2s (82599).
The 82599 based cards work a lot better as you can pass through each port individually, whereas on the CX3 you can only pass through both

Where are you getting stuck with it exactly?
 
  • Like
Reactions: sug7

sug7

New Member
Jul 11, 2022
2
0
1
I have had it working without any issues on an m920q with an 8500T, using either a CX3 or an AOC-STGN-i2s (82599).
The 82599 based cards work a lot better as you can pass through each port individually, whereas on the CX3 you can only pass through both

Where are you getting stuck with it exactly?
Thanks for the info!

I have 2 x M720q running i7-8700Ts, both currently have I340-T4s. I was looking to confirm it would support SR-IOV before purchasing upgraded network cards.
 

mach3.2

Active Member
Feb 7, 2022
130
87
28
I have had it working without any issues on an m920q with an 8500T, using either a CX3 or an AOC-STGN-i2s (82599).
The 82599 based cards work a lot better as you can pass through each port individually, whereas on the CX3 you can only pass through both

Where are you getting stuck with it exactly?
You seem to be describing hardware passthrough(VT-d) instead of SR-IOV though. Both are distinctly different technologies.

Are you sure you actually got SR-IOV working?
 

lexanam

New Member
Jun 20, 2022
3
3
3
I have an m920q and have been trying to get SR-IOV working but no luck so far. I have AOC-STGN-I2S card which I am trying to enable for this. Found your post and this gives me hope! Would you be able to share the steps on how to get this working.
 

zer0sum

Well-Known Member
Mar 8, 2013
849
474
63
Nothing fancy really...I think it was something like this :)

You want to enable IOMMU at boot time. I'm using Proxmox and zfs so I edit /etc/kernel/cmdline
The process is a little different for grub, but really easy to work out

nano /etc/kernel/cmdline
root=ZFS=rpool/ROOT/pve-1 boot=zfs quiet intel_iommu=on iommu=pt

You may or may not need these other switches: pci_pt_e820_access=on pci=assign-busses pcie_acs_override=downstream,multifunction
It just depends on the hypervisor or operating system and how your IOMMU groups look.

Next, add the vfio modules to /etc/modules
nano /etc/modules

vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd


Update boot
proxmox-boot-tool refresh

REBOOT

Run this command to just check that it is working before creating a script to automate the vf's at boot time
echo 4 > /sys/class/net/enp39s0f0/device/sriov_numvfs

It's not a bad idea to also blacklist the ixgbevf driver so there is no contention, but it's not necessary really.
nano /etc/modprobe.d/pve-blacklist.conf
# blacklist Intel VF driver
blacklist ixgbevf

Then create the script. This one creates 4 VF's on each port.
Probably a good idea to use a MAC address generator for the addresses :)

Code:
nano /etc/systemd/system/sriov-NIC.service

[Unit]
Description=Script to enable SR-IOV on boot

[Service]
Type=oneshot
ExecStart=/usr/bin/bash -c '/usr/bin/echo 4 > /sys/class/net/enp39s0f0/device/sriov_numvfs'
ExecStart=/usr/bin/bash -c '/usr/bin/echo 4 > /sys/class/net/enp39s0f1/device/sriov_numvfs'

# Setting static MAC for VFs
#ExecStart=/usr/bin/bash -c '/usr/bin/ip link set <name of your NIC1> vf M mac <mac addr of vf M of NIC1>'
ExecStart=/usr/bin/bash -c '/usr/bin/ip link set enp39s0f0 vf 0 mac 00:52:44:00:00:00'
ExecStart=/usr/bin/bash -c '/usr/bin/ip link set enp39s0f0 vf 1 mac 00:52:44:00:00:11'
ExecStart=/usr/bin/bash -c '/usr/bin/ip link set enp39s0f0 vf 2 mac 00:52:44:00:00:22'
ExecStart=/usr/bin/bash -c '/usr/bin/ip link set enp39s0f0 vf 3 mac 00:52:44:00:00:33'
ExecStart=/usr/bin/bash -c '/usr/bin/ip link set enp39s0f1 vf 0 mac 00:53:44:44:00:00'
ExecStart=/usr/bin/bash -c '/usr/bin/ip link set enp39s0f1 vf 1 mac 00:53:44:55:00:11'
ExecStart=/usr/bin/bash -c '/usr/bin/ip link set enp39s0f1 vf 2 mac 00:53:44:66:00:22'
ExecStart=/usr/bin/bash -c '/usr/bin/ip link set enp39s0f1 vf 3 mac 00:53:44:77:00:33'

# Setting trust for VF for promiscuous mode
ExecStart=/usr/bin/bash -c '/usr/bin/ip link set dev enp39s0f1 vf 0 trust on'

[Install]
WantedBy=multi-user.target
Then you want to start that script and test it
systemctl start sriov-NIC
systemctl status sriov-NIC


If it all looks good then enable it so it runs at boot time
systemctl enable sriov-NIC


You can check the status of the script by running systemctl daemon-reload


I could only ever pass through both ports, instead of just a single one, so I switched to 10G Intel based cards

To see the IOMMU groups someone created a really cool little script :)

#!/bin/bash
for d in /sys/kernel/iommu_groups/*/devices/*; do
n=${d#*/iommu_groups/*}; n=${n%%/*}
printf 'IOMMU Group %s ' "$n"
lspci -nns "${d##*/}"
done
 
Last edited:

lexanam

New Member
Jun 20, 2022
3
3
3
This has been super helpful. I finally have SRIOV working. Had some settings a bit different maybe but the service unit file is absolutely amazing. Thanks a ton.

For reference and maybe helping someone in the future, I have a Lenovo ThinkCentre m920q with a SuperMicro AOC-STGN-I2S card installed. Running on Proxmox VE with btrfs filesystem. Here are the steps I took to enable SRIOV. Note that this configuration works for me and I understand I may not have everything correct and can be improved. By no means I am an expert in this area. Let me know if something needs to change and I will give it a try.

Enabling PCI Passthrough and SRIOV

Update `/etc/default/grub`
Add the following flags to *GRUB_CMDLINE_LINUX_DEFAULT* variable

Code:
pci=assign-busses intel_iommu=on iommu=pt pcie_acs_override=downstream,multifunction
Update `/etc/modules` to enable PCI Passthrough

Add the following:
Code:
vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd
Update/Add `/etc/modprobe.d/ixgbe.conf`

Add the following:
Code:
options ixgbe allow_unsupported_sfp=1
Update grub and initramfs and reboot
Code:
update-grub
update-initramfs -u -k all
reboot
Test SRIOV and Virtual Functions
Code:
echo 4 > /sys/class/net/enp1s0f0/device/sriov_numvfs
echo 4 > /sys/class/net/enp1s0f1/device/sriov_numvfs
No error means good :) Check new interfaces added using the following, you should see interfaces with "Virtual Function" in their name.
Code:
lspci | grep Ethernet
Create service unit to enable virtual functions at boot

Create a service unit file
Code:
nano /etc/systemd/system/sriov-nic.service
Paste the following and save and exit
Code:
[Unit]
Description=Script to enable SR-IOV on boot
[Service]
Type=oneshot

ExecStart=/usr/bin/bash -c '/usr/bin/echo 4 > /sys/class/net/enp1s0f0/device/sriov_numvfs'
ExecStart=/usr/bin/bash -c '/usr/bin/echo 4 > /sys/class/net/enp1s0f1/device/sriov_numvfs'

# Setting static MAC for VFs
#ExecStart=/usr/bin/bash -c '/usr/bin/ip link set <name of your NIC1> vf M mac <mac addr of vf M of NIC1>'

## On interface enp1s0f0 mac aa:bb:cc:dd:ee:f1
ExecStart=/usr/bin/bash -c '/usr/bin/ip link set enp1s0f0 vf 0 mac aa:bb:cc:f1:00:00'
ExecStart=/usr/bin/bash -c '/usr/bin/ip link set enp1s0f0 vf 1 mac aa:bb:cc:f1:00:01'
ExecStart=/usr/bin/bash -c '/usr/bin/ip link set enp1s0f0 vf 2 mac aa:bb:cc:f1:00:02'
ExecStart=/usr/bin/bash -c '/usr/bin/ip link set enp1s0f0 vf 3 mac aa:bb:cc:f1:00:03'

## On interface enp1s0f1 mac aa:bb:cc:dd:ee:f2
ExecStart=/usr/bin/bash -c '/usr/bin/ip link set enp1s0f1 vf 0 mac aa:bb:cc:f2:00:00'
ExecStart=/usr/bin/bash -c '/usr/bin/ip link set enp1s0f1 vf 1 mac aa:bb:cc:f2:00:01'
ExecStart=/usr/bin/bash -c '/usr/bin/ip link set enp1s0f1 vf 2 mac aa:bb:cc:f2:00:02'
ExecStart=/usr/bin/bash -c '/usr/bin/ip link set enp1s0f1 vf 3 mac aa:bb:cc:f2:00:03'

# Setting trust for VF for promiscuous mode
#ExecStart=/usr/bin/bash -c '/usr/bin/ip link set <name of your NIC1> vf M trust on'
## interface enp1s0f0 mac aa:bb:cc:dd:ee:f1
ExecStart=/usr/bin/bash -c '/usr/bin/ip link set dev enp1s0f0 vf 0 trust on'
ExecStart=/usr/bin/bash -c '/usr/bin/ip link set dev enp1s0f0 vf 1 trust on'
ExecStart=/usr/bin/bash -c '/usr/bin/ip link set dev enp1s0f0 vf 2 trust on'
ExecStart=/usr/bin/bash -c '/usr/bin/ip link set dev enp1s0f0 vf 3 trust on'

## interface enp1s0f1 mac aa:bb:cc:dd:ee:f2
ExecStart=/usr/bin/bash -c '/usr/bin/ip link set dev enp1s0f1 vf 0 trust on'
ExecStart=/usr/bin/bash -c '/usr/bin/ip link set dev enp1s0f1 vf 1 trust on'
ExecStart=/usr/bin/bash -c '/usr/bin/ip link set dev enp1s0f1 vf 2 trust on'
ExecStart=/usr/bin/bash -c '/usr/bin/ip link set dev enp1s0f1 vf 3 trust on'
[Install]
WantedBy=multi-user.target
Start, verify and then enable
Start service
Code:
systemctl start sriov-nic
systemctl status sriov-nic
lspci | grep Ethernet
If everything looks good enable the service to start at boot
Code:
systemctl enable --now sriov-nic
Enable interfaces in Proxmox VE
In Proxmox VE > Nodes > Network, set each of the VF interface to "autostart" so they can connect to the network and can be brought up.

Edits and Notes
Although the original config did create virtual functions (VF) I was not able to use them in a VM. As soon as the VM started it reset the NIC (VF) and along with it all the devices in the same IOMMU group. Turns out Lenovo ThinkCentre m920q was keeping the physical function and all virtual functions in the same IOMMU group. Moreover, the VF's once created would not connect to network on activation. To resolve these errors the following had to done:

1. Add the flag to allow PCIe ACS override
Add "pcie_acs_override=downstream,multifunction" to /etc/default.grub. This helps create VFs in different IOMMU group

2. Do not blacklist VF driver
Remove the "blacklist ixgbevf" from ixgbe.conf
The driver actually helps Proxmox VE see the interfaces for each of the virtual function that can be then enabled as needed
 
Last edited:

TinyLenovo

New Member
Feb 1, 2023
14
8
3
Thanks to zer0sum and lexanam for pioneering this and capturing for the rest of us to follow.

Would either of you be able to provide some guidance for updates to this for an HBA controller vs a NIC? i.e. There are a number of steps that are nic specific which wouldn't apply, or should be done differently for an HBA.

I have an PCI HBA in IT mode (IBM 81Y4494 H1110) that I'd like to passthrough to a TrueNAS vm on Proxmox, but having trouble getting the passthrough setup. This is on a m920x i5-8500. I can see the drives in Proxmox, and I can find the PCI device to passthrough, but when I assign it, the VM goes into a bootloop instead of loading the from the drive it's stored on.
*My PVE boot drive is an SSD (ext) on the front SATA connector. I'm not sure if there is any conflict using a drive on this connector when trying to pass through PCI HBA through.

I have updated /etc/default/grub and /etc/modules, but haven't added any files or scripts as I'm not sure if they're necessary for the HBA. BIOS is the latest from Lenovo. I believe I have the right settings set there (?), but a refresher for confirmation wouldn't hurt.

Update: I've since been able to add the drives individually to the VM via command line so I can use TrueNAS now. I'd still like to understand how to pass the device successfully, as I I understand it's better for TrueNAS to have control over the HBA vs the drives individually.
 
Last edited:

kino0924

New Member
Aug 10, 2022
14
1
3
I know this thread is about M920q but Im trying to do same thing with M910x(same as P320 afaik)
So Im trying to enable VFs with X540-AT2


Code:
root@pve2:~# dmesg | grep -e i5-7500T
[    0.201065] smpboot: CPU0: Intel(R) Core(TM) i5-7500T CPU @ 2.70GHz (family: 0x6, model: 0x9e, stepping: 0x9)

root@pve2:~# lspci -vv | grep Eth
00:1f.6 Ethernet controller: Intel Corporation Ethernet Connection (2) I219-LM
        Subsystem: Lenovo Ethernet Connection (2) I219-LM
01:00.0 Ethernet controller: Intel Corporation Ethernet Controller 10-Gigabit X540-AT2 (rev 01)
        Subsystem: Super Micro Computer Inc Ethernet Controller 10-Gigabit X540-AT2
01:00.1 Ethernet controller: Intel Corporation Ethernet Controller 10-Gigabit X540-AT2 (rev 01)
        Subsystem: Super Micro Computer Inc Ethernet Controller 10-Gigabit X540-AT2

root@pve2:~# ./ls-iommu.sh
IOMMU Group 0 00:00.0 Host bridge [0600]: Intel Corporation Xeon E3-1200 v6/7th Gen Core Processor Host Bridge/DRAM Registers [8086:591f] (rev 05)
IOMMU Group 1 00:01.0 PCI bridge [0604]: Intel Corporation 6th-10th Gen Core Processor PCIe Controller (x16) [8086:1901] (rev 05)
IOMMU Group 1 01:00.0 Ethernet controller [0200]: Intel Corporation Ethernet Controller 10-Gigabit X540-AT2 [8086:1528] (rev 01)
IOMMU Group 1 01:00.1 Ethernet controller [0200]: Intel Corporation Ethernet Controller 10-Gigabit X540-AT2 [8086:1528] (rev 01)
IOMMU Group 2 00:02.0 VGA compatible controller [0300]: Intel Corporation HD Graphics 630 [8086:5912] (rev 04)
IOMMU Group 3 00:14.0 USB controller [0c03]: Intel Corporation 200 Series/Z370 Chipset Family USB 3.0 xHCI Controller [8086:a2af]
IOMMU Group 3 00:14.2 Signal processing controller [1180]: Intel Corporation 200 Series PCH Thermal Subsystem [8086:a2b1]
IOMMU Group 4 00:16.0 Communication controller [0780]: Intel Corporation 200 Series PCH CSME HECI #1 [8086:a2ba]
IOMMU Group 4 00:16.3 Serial controller [0700]: Intel Corporation Device [8086:a2bd]
IOMMU Group 5 00:17.0 SATA controller [0106]: Intel Corporation 200 Series PCH SATA controller [AHCI mode] [8086:a282]
IOMMU Group 6 00:1b.0 PCI bridge [0604]: Intel Corporation 200 Series PCH PCI Express Root Port #21 [8086:a2eb] (rev f0)
IOMMU Group 7 00:1f.0 ISA bridge [0601]: Intel Corporation 200 Series PCH LPC Controller (Q270) [8086:a2c6]
IOMMU Group 7 00:1f.2 Memory controller [0580]: Intel Corporation 200 Series/Z370 Chipset Family Power Management Controller [8086:a2a1]
IOMMU Group 7 00:1f.3 Audio device [0403]: Intel Corporation 200 Series PCH HD Audio [8086:a2f0]
IOMMU Group 7 00:1f.4 SMBus [0c05]: Intel Corporation 200 Series/Z370 Chipset Family SMBus Controller [8086:a2a3]
IOMMU Group 8 00:1f.6 Ethernet controller [0200]: Intel Corporation Ethernet Connection (2) I219-LM [8086:15b7]
IOMMU Group 9 02:00.0 Non-Volatile memory controller [0108]: KIOXIA Corporation Device [1e0f:0001]

I enabled VT-d from BIOS and I dont think I theres anything else I can touch from this BIOS.
I also saw IOMMU is enabled from dmesg


Code:
root@pve2:~# dmesg | grep IOMMU
[    0.047216] DMAR: IOMMU enabled
[    0.140893] DMAR-IR: IOAPIC id 2 under DRHD base  0xfed91000 IOMMU 1
[    0.406294] DMAR: IOMMU feature fl1gp_support inconsistent
[    0.406295] DMAR: IOMMU feature pgsel_inv inconsistent
[    0.406296] DMAR: IOMMU feature nwfs inconsistent
[    0.406297] DMAR: IOMMU feature pasid inconsistent
[    0.406298] DMAR: IOMMU feature eafs inconsistent
[    0.406299] DMAR: IOMMU feature prs inconsistent
[    0.406300] DMAR: IOMMU feature nest inconsistent
[    0.406300] DMAR: IOMMU feature mts inconsistent
[    0.406301] DMAR: IOMMU feature sc_support inconsistent
[    0.406302] DMAR: IOMMU feature dev_iotlb_support inconsistent
So to me everything seems to be enabled and should work but when I try to assign VF, I get following error


Code:
root@pve2:~# echo 4 > /sys/class/net/enp1s0f0/device/sriov_numvfs
-bash: echo: write error: Cannot allocate memory


Aug 26 02:11:52 pve2 kernel: [  109.921447] ixgbe 0000:01:00.0 enp1s0f0: SR-IOV enabled with 4 VFs
Aug 26 02:11:52 pve2 kernel: [  110.244308] ixgbe 0000:01:00.0: Multiqueue Enabled: Rx Queue count = 4, Tx Queue count = 4 XDP Queue count = 0
Aug 26 02:11:52 pve2 kernel: [  110.244319] ixgbe 0000:01:00.0: can't enable 4 VFs (bus 02 out of range of [bus 01])
Aug 26 02:11:52 pve2 kernel: [  110.244348] ixgbe 0000:01:00.0: Failed to enable PCI sriov: -12

Am I missing something? or is just system not capable of?

Thanks
 

kino0924

New Member
Aug 10, 2022
14
1
3
Code:
pci_pt_e820_access=on pci=assign-busses
solved the problem


Code:
01:00.0 Ethernet controller: Intel Corporation Ethernet Controller 10-Gigabit X540-AT2 (rev 01)
01:00.1 Ethernet controller: Intel Corporation Ethernet Controller 10-Gigabit X540-AT2 (rev 01)
02:10.0 Ethernet controller: Intel Corporation X540 Ethernet Controller Virtual Function (rev 01)
02:10.2 Ethernet controller: Intel Corporation X540 Ethernet Controller Virtual Function (rev 01)
02:10.4 Ethernet controller: Intel Corporation X540 Ethernet Controller Virtual Function (rev 01)
02:10.6 Ethernet controller: Intel Corporation X540 Ethernet Controller Virtual Function (rev 01)
02:11.0 Ethernet controller: Intel Corporation X540 Ethernet Controller Virtual Function (rev 01)
02:11.2 Ethernet controller: Intel Corporation X540 Ethernet Controller Virtual Function (rev 01)
02:11.4 Ethernet controller: Intel Corporation X540 Ethernet Controller Virtual Function (rev 01)
02:11.6 Ethernet controller: Intel Corporation X540 Ethernet Controller Virtual Function (rev 01)
02:12.0 Ethernet controller: Intel Corporation X540 Ethernet Controller Virtual Function (rev 01)
02:12.2 Ethernet controller: Intel Corporation X540 Ethernet Controller Virtual Function (rev 01)
02:12.4 Ethernet controller: Intel Corporation X540 Ethernet Controller Virtual Function (rev 01)
02:12.6 Ethernet controller: Intel Corporation X540 Ethernet Controller Virtual Function (rev 01)
02:13.0 Ethernet controller: Intel Corporation X540 Ethernet Controller Virtual Function (rev 01)
02:13.2 Ethernet controller: Intel Corporation X540 Ethernet Controller Virtual Function (rev 01)
02:13.4 Ethernet controller: Intel Corporation X540 Ethernet Controller Virtual Function (rev 01)
02:13.6 Ethernet controller: Intel Corporation X540 Ethernet Controller Virtual Function (rev 01)
 
Last edited:

Nico1320

New Member
Jan 15, 2024
1
0
1
I'm trying to do this on a M720q with an hp 546sfp+(MCX312B). It works as expected but for some reason after every reboot, I have to modprobe mlx4_core. This is the result after blacklisting it which makes sense but if I don't blacklist it then I'd still have to refresh the drivers after boot for some reason. Also, I can't enable trust for the VFs, I'm not sure if that's related to the card's OEM firmware though.
 

AlexGee

Member
Aug 3, 2022
58
27
18
Hi,

I was able to get SR-IOV working with an Intel X520-DA2 on my M720q and I'm able to see the VFs and I'm even able to create a Linux Bridge with the VF BUT
Proxmox WebUI crashes and if I reboot, I cannot access the WebUI...
Loading of Proxmox looks normal - beside no WebUI

Any thoughts?



Code:
IOMMU & SR-IOV setup proxmox

nano /etc/kernel/cmdline
root=ZFS=rpool/ROOT/pve-1 boot=zfs quiet intel_iommu=on iommu=pt pci_pt_e820_access=on pci=assign-busses pcie_acs_override=downstream,multifunction
CTRL-O + CTRL-X


nano /etc/modules
vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd
CTRL-O + CTRL-X


#ixgbe
nano /etc/modprobe.d/ixgbe.conf
options ixgbe allow_unsupported_sfp=1


proxmox-boot-tool refresh


reboot



echo 5 > /sys/class/net/enp1s0f1/device/sriov_numvfs


#check creation of VF interfaces
lspci | grep Ethernet


#Create VFs by copying script
nano /etc/systemd/system/sriov-NIC.service

[Unit]
Description=Script to enable SR-IOV on boot
[Service]
Type=oneshot

ExecStart=/usr/bin/bash -c '/usr/bin/echo 5 > /sys/class/net/enp1s0f1/device/sriov_numvfs'

# Setting static MAC for VFs
#ExecStart=/usr/bin/bash -c '/usr/bin/ip link set enp1s0f1 vf M mac <mac addr of vf M of NIC1>'

## On interface enp1s0f1 mac en:p1:s0:f1:ag:00
ExecStart=/usr/bin/bash -c '/usr/bin/ip link set enp1s0f1 vf 0 mac xx:xx:cc:f1:00:00'
ExecStart=/usr/bin/bash -c '/usr/bin/ip link set enp1s0f1 vf 1 mac xx:xx:cc:f1:00:01'
ExecStart=/usr/bin/bash -c '/usr/bin/ip link set enp1s0f1 vf 2 mac xx:xx:cc:f1:00:02'
ExecStart=/usr/bin/bash -c '/usr/bin/ip link set enp1s0f1 vf 3 mac xx:xx:cc:f1:00:03'
ExecStart=/usr/bin/bash -c '/usr/bin/ip link set enp1s0f1 vf 4 mac xx:xx:cc:f1:00:04'

# Setting trust for VF for promiscuous mode
#ExecStart=/usr/bin/bash -c '/usr/bin/ip link set enp1s0f1 vf M trust on'

## interface enp1s0f1 mac xxx
ExecStart=/usr/bin/bash -c '/usr/bin/ip link set dev enp1s0f1 vf 0 trust on'
ExecStart=/usr/bin/bash -c '/usr/bin/ip link set dev enp1s0f1 vf 1 trust on'
ExecStart=/usr/bin/bash -c '/usr/bin/ip link set dev enp1s0f1 vf 2 trust on'
ExecStart=/usr/bin/bash -c '/usr/bin/ip link set dev enp1s0f1 vf 3 trust on'
ExecStart=/usr/bin/bash -c '/usr/bin/ip link set dev enp1s0f1 vf 4 trust on'
[Install]
WantedBy=multi-user.target


#start that script and test it
systemctl start sriov-NIC
systemctl status sriov-NIC
lspci | grep Ethernet


#Enable script
systemctl enable sriov-NIC


#Check status of script
systemctl daemon-reload


#See IOMMU groups
#!/bin/bash
for d in /sys/kernel/iommu_groups/*/devices/*; do
n=${d#*/iommu_groups/*}; n=${n%%/*}
printf 'IOMMU Group %s ' "$n"
lspci -nns "${d##*/}"
done
 
  • Like
Reactions: dvdwsn

SteveFP

New Member
Jun 25, 2020
8
1
3
Hello all,

I am trying to get IOMMU enabled on my m720q Tiny, with an Intel i350-T4 PCI adapter installed. I have been following both this and the STH web page on enable IOMMU(https://www.servethehome.com/how-to-pass-through-pcie-nics-with-proxmox-ve-on-intel-and-amd/). When I run dmesg | grep -e DMAR -e IOMMU from the STH IOMMU page I see IOMMU enabled, as it states in that document. When I try to run echo 4 > /sys/class/net/enp1s0f0/device/sriov_numvfs I get a Permission denied message. Has anyone run into this?

Thanks,
Steve
 

elseless

New Member
Mar 2, 2021
5
1
3
Hello all,

I am trying to get IOMMU enabled on my m720q Tiny, with an Intel i350-T4 PCI adapter installed. I have been following both this and the STH web page on enable IOMMU(https://www.servethehome.com/how-to-pass-through-pcie-nics-with-proxmox-ve-on-intel-and-amd/). When I run dmesg | grep -e DMAR -e IOMMU from the STH IOMMU page I see IOMMU enabled, as it states in that document. When I try to run echo 4 > /sys/class/net/enp1s0f0/device/sriov_numvfs I get a Permission denied message. Has anyone run into this?

Thanks,
Steve
This is just because you need root permission to write to sysfs. Try sudo bash -c "echo 4 > /sys/class/net/enp1s0f0/device/sriov_numvfs" and see if that works.
 

SteveFP

New Member
Jun 25, 2020
8
1
3
This is a Proxmox implementation. There is no need for sudo, unless its something else in Debian?
 

klui

Well-Known Member
Feb 3, 2019
836
458
63
Not only does your OS need to have SR-IOV enabled, but your card needs to have support for it.

Take a look at
How to enable SR-IOV in I350-T2 NIC and How to Check SRIOV NIC Card and Map Network Interface Name which is using it?.

The first link suggest you ensure the slot you've installed the card in supports SR-IOV. The second link shows how you can check if your device supports the feature. The lspci | awk script doesn't work under Debian in the second link. Use
Code:
lspci -vvv | awk '/^..:..\.. / {device=$0} /SR-IOV/ {print device; found=1} END {if (!found) print "SR-IOV not found."}
There is some chatter that older Dell i350 cards don't support SR-IOV.