Lenovo Tiny (M920q or M720q) with CX3 and SR-IOV

[SteveFP]

OK so hold on...I was going to give up but I saw this output from a dmesg command. Can someone comment on what I am seeing? Does this mean my mboard and PCI slot are IOMMU enabled?

root@pve01:~# dmesg | grep -e DMAR -e IOMMU
[ 0.000000] Warning: PCIe ACS overrides enabled; This may allow non-IOMMU protected peer-to-peer DMA
[ 0.008152] ACPI: DMAR 0x000000009AD917F8 0000A8 (v01 LENOVO TC-M1U 00001750 01000013)
[ 0.008190] ACPI: Reserving DMAR table memory at [mem 0x9ad917f8-0x9ad9189f]
[ 0.083568] DMAR: IOMMU enabled
[ 0.230884] DMAR: Host address width 39
[ 0.230886] DMAR: DRHD base: 0x000000fed90000 flags: 0x0
[ 0.230893] DMAR: dmar0: reg_base_addr fed90000 ver 1:0 cap 1c0000c40660462 ecap 19e2ff0505e
[ 0.230897] DMAR: DRHD base: 0x000000fed91000 flags: 0x1
[ 0.230902] DMAR: dmar1: reg_base_addr fed91000 ver 1:0 cap d2008c40660462 ecap f050da
[ 0.230905] DMAR: RMRR base: 0x00000099aae000 end: 0x00000099acdfff
[ 0.230908] DMAR: RMRR base: 0x0000009d000000 end: 0x0000009f7fffff
[ 0.230911] DMAR-IR: IOAPIC id 2 under DRHD base 0xfed91000 IOMMU 1
[ 0.230914] DMAR-IR: HPET id 0 under DRHD base 0xfed91000
[ 0.230917] DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping.
[ 0.232511] DMAR-IR: Enabled IRQ remapping in x2apic mode
[ 0.569987] DMAR: No ATSR found
[ 0.569989] DMAR: No SATC found
[ 0.569991] DMAR: IOMMU feature fl1gp_support inconsistent
[ 0.569992] DMAR: IOMMU feature pgsel_inv inconsistent
[ 0.569995] DMAR: IOMMU feature nwfs inconsistent
[ 0.569998] DMAR: IOMMU feature pasid inconsistent
[ 0.570001] DMAR: IOMMU feature eafs inconsistent
[ 0.570003] DMAR: IOMMU feature prs inconsistent
[ 0.570006] DMAR: IOMMU feature nest inconsistent
[ 0.570008] DMAR: IOMMU feature mts inconsistent
[ 0.570010] DMAR: IOMMU feature sc_support inconsistent
[ 0.570013] DMAR: IOMMU feature dev_iotlb_support inconsistent
[ 0.570016] DMAR: dmar0: Using Queued invalidation
[ 0.570022] DMAR: dmar1: Using Queued invalidation
[ 0.570567] DMAR: Intel(R) Virtualization Technology for Directed I/O
root@pve01:~# dmesg | grep 'remapping'
[ 0.230917] DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping.
[ 0.232511] DMAR-IR: Enabled IRQ remapping in x2apic mode

 

[klui]

I can't comment on your dmesg output. But check out this thread with the use of ethtool to dump and modify the EEPROM from an i350-t2. The data sheet is still available at that link.

[vfio-users] sr-iov support on main boards

 

[SteveFP]

Thanks for this...I am reading through this, to see if I can gleen some info from it. I am not completely convinced I will get this working, but I am going to give it my best effort. I have the Lenovo m720q Tiny, with the I350-T4 installed. I have read ppl who got it working with the m720q but then Lenovo told me the m720q BIOS does not support SR-IOV.

This is way too complicated and confusing...

 

[SteveFP]

Holy crap...this link holds the key to getting the i350 cards to support SR-IOV. You have to play with ETHTOOL but it works! I just flipped my I350-T4 into SR-IOV mode. I am rebuilding my Proxmox server and checking to see how this will react. Thank you @klui!

 

[SteveFP]

It worked....I have 4 virtual interfaces for each physical interface on my I350-T4. Now the question is...what is my limit of the virtual interfaces?

 

[klui]

You have to look at the datasheet, specifically the section referenced in the URL. The maximum is 7 and it's the upper 3 bits of the byte extracted.

 

[SteveFP]

Based on the datasheet on the Intel website it's 8 vf per each physical interface. That's a total of 32 vf on my I350-T4!

This unit is one mighty powerhouse. My unit has an i7-9700, 64 gig of RAM, 1 TB NVMe, I350-T4. It will run Proxmox and I will use SR-IOV for my OPNsense firewall and the vfs for any other vms or containers!

 

[klui]

Yeah it helps if one reads the datasheet correctly.

Max VFs, Defines the value of MaxVF exposed in the IOV structure. Valid values
are 0-7. The value exposed in the PCIe configuration space is the value of this field
+ one.

 

[AlexGee]

I have SR-IOV successfully activated on my M720q and X550-T2 NIC a few weeks ago based on the method posted in this forum.

Internal Gigabit is used as Mgmt port for Proxmox, LAN & WAN with 10 GBit via X550 and one additional virtual NICsfor VMs/LXCs.

If I understand correctly from some YT videos, creating Virtual NICs will split up the bandwidth (in my case 5GBit for physical and 5GBit for virtual NIC)

 

[SteveFP]

Yeah it helps if one reads the datasheet correctly.

Max VFs, Defines the value of MaxVF exposed in the IOV structure. Valid values
are 0-7. The value exposed in the PCIe configuration space is the value of this field
+ one.

Yes I mis-read it...

I dont think I will ever need that many virtual interfaces. I am going to setup one Linux Bond for all 4 physical interfaces and trunk all my vlans across all four. This way I dont need to worry about what vlan is on what interface. I am going to start with 2 vfs on each physical interface and see if I need more.

One other thing...I updated the firmware on my I350, from v29 of the Intel driver pkg. Not sure if that did it but when I run echo 4 > /sys/class/net/enp39s0f0/device/sriov_numvfs the MACs are auto generated. I do not have to assign MACs.

 

[gianry]

Hi,
I've installed a 10Gtek card with Realtek 8125BG chipset in my Lenovo M920q . I use a proxmox node and I'd like to understand if it is possible to pass trough only two port an a VM and other two on another VM.

thanks

 

[Zewkin]

I have SR-IOV successfully activated on my M720q and X550-T2 NIC a few weeks ago based on the method posted in this forum.

Internal Gigabit is used as Mgmt port for Proxmox, LAN & WAN with 10 GBit via X550 and one additional virtual NICsfor VMs/LXCs.

If I understand correctly from some YT videos, creating Virtual NICs will split up the bandwidth (in my case 5GBit for physical and 5GBit for virtual NIC)

Hi, I have the exact same setup and struggling with enabling SR-IOV

root@proxmox:~# dmesg | grep -i sriov
[    1.881538] ixgbe 0000:01:00.0 0000:01:00.0 (uninitialized): Failed to enable PCI sriov: -38
[    2.953768] ixgbe 0000:01:00.1 0000:01:00.1 (uninitialized): Failed to enable PCI sriov: -38
[    7.785308] ixgbe 0000:01:00.0 0000:01:00.0 (uninitialized): Failed to enable PCI sriov: -38
[    8.849332] ixgbe 0000:01:00.1 0000:01:00.1 (uninitialized): Failed to enable PCI sriov: -38
root@proxmox:~#

how did you get this work without support from BIOS?

 

[Midvalley]

Hi, I have the exact same setup and struggling with enabling SR-IOV

root@proxmox:~# dmesg | grep -i sriov
[    1.881538] ixgbe 0000:01:00.0 0000:01:00.0 (uninitialized): Failed to enable PCI sriov: -38
[    2.953768] ixgbe 0000:01:00.1 0000:01:00.1 (uninitialized): Failed to enable PCI sriov: -38
[    7.785308] ixgbe 0000:01:00.0 0000:01:00.0 (uninitialized): Failed to enable PCI sriov: -38
[    8.849332] ixgbe 0000:01:00.1 0000:01:00.1 (uninitialized): Failed to enable PCI sriov: -38
root@proxmox:~#

how did you get this work without support from BIOS?

I've got SRV-IO working with an M720Q with a Mellanox CX3 Pro by changing the grub options.

From my PVE build notes:

Edit /etc/default/grub and modify following line to match this:

GRUB_CMDLINE_LINUX_DEFAULT="quiet pci=assign_busses intel_iommu=on iommu=pt pcie_acs_override=downstream,multifunction"

Edit /etc/modules and add:
vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd

Note that there is debate on what is/isn't necessary for the grub cmdline, and the acs_override command specifically I believe was considered somewhat not-ideal/stable from memory. I remember playing with it for a couple days, and that was all needed to get it working.

 

[Zewkin]

Yes, I've tried all this still nothing, -38 not implemented
which BIOS version do you have installed? I bought mine m720q with the latest one, maybe downgrading will help

 

[Midvalley]

I've got the newest as of when the system was built: 04/10/2024 - 1.119 is the dmidecode reported version.