Lenovo Tiny (M920q or M720q) with CX3 and SR-IOV

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

sophware

New Member
Nov 1, 2019
4
0
1
I haven't been able to get SR-IOV working on these Tinies. My thought is that the BIOS (motherboard, not just card card) needs to support it and that the Lenovos' BIOS doesn't.

Fairly confident about the former (that actual BIOS support is needed), since there is a plethora of search results saying so, and because my rack servers do have such options.

As to the latter, it seems likely people here would have experience (though maybe not in this forum, Networking). Anyone care to comment?

======

Less important:

Also, getting the FastBoot or the UEFI version isn't working for me to boot to SAN via iSCSI. It's probably a config thing on my part. If I'm doing things correctly, what I'm expecting is to boot to an install CD (like ESXi) and see a drive appear. I get no ctrl+B--the only way UEFI or FastBoot gets its config is from me putting it in the UEFI version when not booting Legacy. This is done by going into the motherboard BIOS and going to the ATA settings. I've had no success going directly to the card from some POST method. Anyway, no drive shows up. I think the target is set up correctly and suspect that the client/initiator side isn't.
 
Last edited:

sug7

New Member
Jul 11, 2022
2
0
1
Hi,

were you able to get this working? I have a M720q and looking to purchase a SR-IOV supported NIC card.
 

zer0sum

Well-Known Member
Mar 8, 2013
756
397
63
I have had it working without any issues on an m920q with an 8500T, using either a CX3 or an AOC-STGN-i2s (82599).
The 82599 based cards work a lot better as you can pass through each port individually, whereas on the CX3 you can only pass through both

Where are you getting stuck with it exactly?
 
  • Like
Reactions: sug7

sug7

New Member
Jul 11, 2022
2
0
1
I have had it working without any issues on an m920q with an 8500T, using either a CX3 or an AOC-STGN-i2s (82599).
The 82599 based cards work a lot better as you can pass through each port individually, whereas on the CX3 you can only pass through both

Where are you getting stuck with it exactly?
Thanks for the info!

I have 2 x M720q running i7-8700Ts, both currently have I340-T4s. I was looking to confirm it would support SR-IOV before purchasing upgraded network cards.
 

mach3.2

Member
Feb 7, 2022
53
36
18
I have had it working without any issues on an m920q with an 8500T, using either a CX3 or an AOC-STGN-i2s (82599).
The 82599 based cards work a lot better as you can pass through each port individually, whereas on the CX3 you can only pass through both

Where are you getting stuck with it exactly?
You seem to be describing hardware passthrough(VT-d) instead of SR-IOV though. Both are distinctly different technologies.

Are you sure you actually got SR-IOV working?
 

lexanam

New Member
Jun 20, 2022
3
0
1
I have an m920q and have been trying to get SR-IOV working but no luck so far. I have AOC-STGN-I2S card which I am trying to enable for this. Found your post and this gives me hope! Would you be able to share the steps on how to get this working.
 

zer0sum

Well-Known Member
Mar 8, 2013
756
397
63
Nothing fancy really...I think it was something like this :)

You want to enable IOMMU at boot time. I'm using Proxmox and zfs so I edit /etc/kernel/cmdline
The process is a little different for grub, but really easy to work out

nano /etc/kernel/cmdline
root=ZFS=rpool/ROOT/pve-1 boot=zfs quiet intel_iommu=on iommu=pt

You may or may not need these other switches: pci_pt_e820_access=on pci=assign-busses pcie_acs_override=downstream,multifunction
It just depends on the hypervisor or operating system and how your IOMMU groups look.

Next, add the vfio modules to /etc/modules
nano /etc/modules

vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd


Update boot
proxmox-boot-tool refresh

REBOOT

Run this command to just check that it is working before creating a script to automate the vf's at boot time
echo 4 > /sys/class/net/enp39s0f0/device/sriov_numvfs

It's not a bad idea to also blacklist the ixgbevf driver so there is no contention, but it's not necessary really.
nano /etc/modprobe.d/pve-blacklist.conf
# blacklist Intel VF driver
blacklist ixgbevf

Then create the script. This one creates 4 VF's on each port.
Probably a good idea to use a MAC address generator for the addresses :)

Code:
nano /etc/systemd/system/sriov-NIC.service

[Unit]
Description=Script to enable SR-IOV on boot

[Service]
Type=oneshot
ExecStart=/usr/bin/bash -c '/usr/bin/echo 4 > /sys/class/net/enp39s0f0/device/sriov_numvfs'
ExecStart=/usr/bin/bash -c '/usr/bin/echo 4 > /sys/class/net/enp39s0f1/device/sriov_numvfs'

# Setting static MAC for VFs
#ExecStart=/usr/bin/bash -c '/usr/bin/ip link set <name of your NIC1> vf M mac <mac addr of vf M of NIC1>'
ExecStart=/usr/bin/bash -c '/usr/bin/ip link set enp39s0f0 vf 0 mac 00:52:44:00:00:00'
ExecStart=/usr/bin/bash -c '/usr/bin/ip link set enp39s0f0 vf 1 mac 00:52:44:00:00:11'
ExecStart=/usr/bin/bash -c '/usr/bin/ip link set enp39s0f0 vf 2 mac 00:52:44:00:00:22'
ExecStart=/usr/bin/bash -c '/usr/bin/ip link set enp39s0f0 vf 3 mac 00:52:44:00:00:33'
ExecStart=/usr/bin/bash -c '/usr/bin/ip link set enp39s0f1 vf 0 mac 00:53:44:44:00:00'
ExecStart=/usr/bin/bash -c '/usr/bin/ip link set enp39s0f1 vf 1 mac 00:53:44:55:00:11'
ExecStart=/usr/bin/bash -c '/usr/bin/ip link set enp39s0f1 vf 2 mac 00:53:44:66:00:22'
ExecStart=/usr/bin/bash -c '/usr/bin/ip link set enp39s0f1 vf 3 mac 00:53:44:77:00:33'

# Setting trust for VF for promiscuous mode
ExecStart=/usr/bin/bash -c '/usr/bin/ip link set dev enp39s0f1 vf 0 trust on'

[Install]
WantedBy=multi-user.target
Then you want to start that script and test it
systemctl start sriov-NIC
systemctl status sriov-NIC


If it all looks good then enable it so it runs at boot time
systemctl enable sriov-NIC


You can check the status of the script by running systemctl daemon-reload

To see the IOMMU groups someone created a really cool little script :)

#!/bin/bash
for d in /sys/kernel/iommu_groups/*/devices/*; do
n=${d#*/iommu_groups/*}; n=${n%%/*}
printf 'IOMMU Group %s ' "$n"
lspci -nns "${d##*/}"
done
 
Last edited:

lexanam

New Member
Jun 20, 2022
3
0
1
This has been super helpful. I finally have SRIOV working. Had some settings a bit different maybe but the service unit file is absolutely amazing. Thanks a ton.

For reference and maybe helping someone in the future, I have a Lenovo ThinkCentre m920q with a SuperMicro AOC-STGN-I2S card installed. Running on Proxmox VE with btrfs filesystem. Here are the steps I took to enable SRIOV. Note that this configuration works for me and I understand I may not have everything correct and can be improved. By no means I am an expert in this area. Let me know if something needs to change and I will give it a try.

Enabling PCI Passthrough and SRIOV

Update `/etc/default/grub`
Add the following flags to *GRUB_CMDLINE_LINUX_DEFAULT* variable

Code:
pci=assign-busses intel_iommu=on iommu=pt pcie_acs_override=downstream,multifunction
Update `/etc/modules` to enable PCI Passthrough

Add the following:
Code:
vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd
Update/Add `/etc/modprobe.d/ixgbe.conf`

Add the following:
Code:
options ixgbe allow_unsupported_sfp=1
Update grub and initramfs and reboot
Code:
update-grub
update-initramfs -u -k all
reboot
Test SRIOV and Virtual Functions
Code:
echo 4 > /sys/class/net/enp1s0f0/device/sriov_numvfs
echo 4 > /sys/class/net/enp1s0f1/device/sriov_numvfs
No error means good :) Check new interfaces added using the following, you should see interfaces with "Virtual Function" in their name.
Code:
lspci | grep Ethernet
Create service unit to enable virtual functions at boot

Create a service unit file
Code:
nano /etc/systemd/system/sriov-nic.service
Paste the following and save and exit
Code:
[Unit]
Description=Script to enable SR-IOV on boot
[Service]
Type=oneshot

ExecStart=/usr/bin/bash -c '/usr/bin/echo 4 > /sys/class/net/enp1s0f0/device/sriov_numvfs'
ExecStart=/usr/bin/bash -c '/usr/bin/echo 4 > /sys/class/net/enp1s0f1/device/sriov_numvfs'

# Setting static MAC for VFs
#ExecStart=/usr/bin/bash -c '/usr/bin/ip link set <name of your NIC1> vf M mac <mac addr of vf M of NIC1>'

## On interface enp1s0f0 mac aa:bb:cc:dd:ee:f1
ExecStart=/usr/bin/bash -c '/usr/bin/ip link set enp1s0f0 vf 0 mac aa:bb:cc:f1:00:00'
ExecStart=/usr/bin/bash -c '/usr/bin/ip link set enp1s0f0 vf 1 mac aa:bb:cc:f1:00:01'
ExecStart=/usr/bin/bash -c '/usr/bin/ip link set enp1s0f0 vf 2 mac aa:bb:cc:f1:00:02'
ExecStart=/usr/bin/bash -c '/usr/bin/ip link set enp1s0f0 vf 3 mac aa:bb:cc:f1:00:03'

## On interface enp1s0f1 mac aa:bb:cc:dd:ee:f2
ExecStart=/usr/bin/bash -c '/usr/bin/ip link set enp1s0f1 vf 0 mac aa:bb:cc:f2:00:00'
ExecStart=/usr/bin/bash -c '/usr/bin/ip link set enp1s0f1 vf 1 mac aa:bb:cc:f2:00:01'
ExecStart=/usr/bin/bash -c '/usr/bin/ip link set enp1s0f1 vf 2 mac aa:bb:cc:f2:00:02'
ExecStart=/usr/bin/bash -c '/usr/bin/ip link set enp1s0f1 vf 3 mac aa:bb:cc:f2:00:03'

# Setting trust for VF for promiscuous mode
#ExecStart=/usr/bin/bash -c '/usr/bin/ip link set <name of your NIC1> vf M trust on'
## interface enp1s0f0 mac aa:bb:cc:dd:ee:f1
ExecStart=/usr/bin/bash -c '/usr/bin/ip link set dev enp1s0f0 vf 0 trust on'
ExecStart=/usr/bin/bash -c '/usr/bin/ip link set dev enp1s0f0 vf 1 trust on'
ExecStart=/usr/bin/bash -c '/usr/bin/ip link set dev enp1s0f0 vf 2 trust on'
ExecStart=/usr/bin/bash -c '/usr/bin/ip link set dev enp1s0f0 vf 3 trust on'

## interface enp1s0f1 mac aa:bb:cc:dd:ee:f2
ExecStart=/usr/bin/bash -c '/usr/bin/ip link set dev enp1s0f1 vf 0 trust on'
ExecStart=/usr/bin/bash -c '/usr/bin/ip link set dev enp1s0f1 vf 1 trust on'
ExecStart=/usr/bin/bash -c '/usr/bin/ip link set dev enp1s0f1 vf 2 trust on'
ExecStart=/usr/bin/bash -c '/usr/bin/ip link set dev enp1s0f1 vf 3 trust on'
[Install]
WantedBy=multi-user.target
Start, verify and then enable
Start service
Code:
systemctl start sriov-nic
systemctl status sriov-nic
lspci | grep Ethernet
If everything looks good enable the service to start at boot
Code:
systemctl enable --now sriov-nic
Enable interfaces in Proxmox VE
In Proxmox VE > Nodes > Network, set each of the VF interface to "autostart" so they can connect to the network and can be brought up.

Edits and Notes
Although the original config did create virtual functions (VF) I was not able to use them in a VM. As soon as the VM started it reset the NIC (VF) and along with it all the devices in the same IOMMU group. Turns out Lenovo ThinkCentre m920q was keeping the physical function and all virtual functions in the same IOMMU group. Moreover, the VF's once created would not connect to network on activation. To resolve these errors the following had to done:

1. Add the flag to allow PCIe ACS override
Add "pcie_acs_override=downstream,multifunction" to /etc/default.grub. This helps create VFs in different IOMMU group

2. Do not blacklist VF driver
Remove the "blacklist ixgbevf" from ixgbe.conf
The driver actually helps Proxmox VE see the interfaces for each of the virtual function that can be then enabled as needed
 
Last edited:

TinyLenovo

New Member
Feb 1, 2023
12
3
3
Thanks to zer0sum and lexanam for pioneering this and capturing for the rest of us to follow.

Would either of you be able to provide some guidance for updates to this for an HBA controller vs a NIC? i.e. There are a number of steps that are nic specific which wouldn't apply, or should be done differently for an HBA.

I have an PCI HBA in IT mode (IBM 81Y4494 H1110) that I'd like to passthrough to a TrueNAS vm on Proxmox, but having trouble getting the passthrough setup. This is on a m920x i5-8500. I can see the drives in Proxmox, and I can find the PCI device to passthrough, but when I assign it, the VM goes into a bootloop instead of loading the from the drive it's stored on.
*My PVE boot drive is an SSD (ext) on the front SATA connector. I'm not sure if there is any conflict using a drive on this connector when trying to pass through PCI HBA through.

I have updated /etc/default/grub and /etc/modules, but haven't added any files or scripts as I'm not sure if they're necessary for the HBA. BIOS is the latest from Lenovo. I believe I have the right settings set there (?), but a refresher for confirmation wouldn't hurt.

Update: I've since been able to add the drives individually to the VM via command line so I can use TrueNAS now. I'd still like to understand how to pass the device successfully, as I I understand it's better for TrueNAS to have control over the HBA vs the drives individually.
 
Last edited: