Lenovo Tiny (M920q or M720q) with CX3 and SR-IOV

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

SteveFP

New Member
Jun 25, 2020
8
1
3
OK so hold on...I was going to give up but I saw this output from a dmesg command. Can someone comment on what I am seeing? Does this mean my mboard and PCI slot are IOMMU enabled?

root@pve01:~# dmesg | grep -e DMAR -e IOMMU
[ 0.000000] Warning: PCIe ACS overrides enabled; This may allow non-IOMMU protected peer-to-peer DMA
[ 0.008152] ACPI: DMAR 0x000000009AD917F8 0000A8 (v01 LENOVO TC-M1U 00001750 01000013)
[ 0.008190] ACPI: Reserving DMAR table memory at [mem 0x9ad917f8-0x9ad9189f]
[ 0.083568] DMAR: IOMMU enabled
[ 0.230884] DMAR: Host address width 39
[ 0.230886] DMAR: DRHD base: 0x000000fed90000 flags: 0x0
[ 0.230893] DMAR: dmar0: reg_base_addr fed90000 ver 1:0 cap 1c0000c40660462 ecap 19e2ff0505e
[ 0.230897] DMAR: DRHD base: 0x000000fed91000 flags: 0x1
[ 0.230902] DMAR: dmar1: reg_base_addr fed91000 ver 1:0 cap d2008c40660462 ecap f050da
[ 0.230905] DMAR: RMRR base: 0x00000099aae000 end: 0x00000099acdfff
[ 0.230908] DMAR: RMRR base: 0x0000009d000000 end: 0x0000009f7fffff
[ 0.230911] DMAR-IR: IOAPIC id 2 under DRHD base 0xfed91000 IOMMU 1
[ 0.230914] DMAR-IR: HPET id 0 under DRHD base 0xfed91000
[ 0.230917] DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping.
[ 0.232511] DMAR-IR: Enabled IRQ remapping in x2apic mode
[ 0.569987] DMAR: No ATSR found
[ 0.569989] DMAR: No SATC found
[ 0.569991] DMAR: IOMMU feature fl1gp_support inconsistent
[ 0.569992] DMAR: IOMMU feature pgsel_inv inconsistent
[ 0.569995] DMAR: IOMMU feature nwfs inconsistent
[ 0.569998] DMAR: IOMMU feature pasid inconsistent
[ 0.570001] DMAR: IOMMU feature eafs inconsistent
[ 0.570003] DMAR: IOMMU feature prs inconsistent
[ 0.570006] DMAR: IOMMU feature nest inconsistent
[ 0.570008] DMAR: IOMMU feature mts inconsistent
[ 0.570010] DMAR: IOMMU feature sc_support inconsistent
[ 0.570013] DMAR: IOMMU feature dev_iotlb_support inconsistent
[ 0.570016] DMAR: dmar0: Using Queued invalidation
[ 0.570022] DMAR: dmar1: Using Queued invalidation
[ 0.570567] DMAR: Intel(R) Virtualization Technology for Directed I/O
root@pve01:~# dmesg | grep 'remapping'
[ 0.230917] DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping.
[ 0.232511] DMAR-IR: Enabled IRQ remapping in x2apic mode
 
Last edited:

SteveFP

New Member
Jun 25, 2020
8
1
3
Thanks for this...I am reading through this, to see if I can gleen some info from it. I am not completely convinced I will get this working, but I am going to give it my best effort. I have the Lenovo m720q Tiny, with the I350-T4 installed. I have read ppl who got it working with the m720q but then Lenovo told me the m720q BIOS does not support SR-IOV.

This is way too complicated and confusing...
 

SteveFP

New Member
Jun 25, 2020
8
1
3
Holy crap...this link holds the key to getting the i350 cards to support SR-IOV. You have to play with ETHTOOL but it works! I just flipped my I350-T4 into SR-IOV mode. I am rebuilding my Proxmox server and checking to see how this will react. Thank you @klui!
 

SteveFP

New Member
Jun 25, 2020
8
1
3
It worked....I have 4 virtual interfaces for each physical interface on my I350-T4. Now the question is...what is my limit of the virtual interfaces?
 

klui

Well-Known Member
Feb 3, 2019
835
457
63
You have to look at the datasheet, specifically the section referenced in the URL. The maximum is 7 and it's the upper 3 bits of the byte extracted.
 

SteveFP

New Member
Jun 25, 2020
8
1
3
Based on the datasheet on the Intel website it's 8 vf per each physical interface. That's a total of 32 vf on my I350-T4!

This unit is one mighty powerhouse. My unit has an i7-9700, 64 gig of RAM, 1 TB NVMe, I350-T4. It will run Proxmox and I will use SR-IOV for my OPNsense firewall and the vfs for any other vms or containers!
 
  • Like
Reactions: klui

klui

Well-Known Member
Feb 3, 2019
835
457
63
Yeah it helps if one reads the datasheet correctly.

Max VFs, Defines the value of MaxVF exposed in the IOV structure. Valid values
are 0-7. The value exposed in the PCIe configuration space is the value of this field
+ one.
 
  • Like
Reactions: SteveFP

AlexGee

Member
Aug 3, 2022
58
24
8
I have SR-IOV successfully activated on my M720q and X550-T2 NIC a few weeks ago based on the method posted in this forum.

Internal Gigabit is used as Mgmt port for Proxmox, LAN & WAN with 10 GBit via X550 and one additional virtual NICsfor VMs/LXCs.

If I understand correctly from some YT videos, creating Virtual NICs will split up the bandwidth (in my case 5GBit for physical and 5GBit for virtual NIC)
 

SteveFP

New Member
Jun 25, 2020
8
1
3
Yeah it helps if one reads the datasheet correctly.

Max VFs, Defines the value of MaxVF exposed in the IOV structure. Valid values
are 0-7. The value exposed in the PCIe configuration space is the value of this field
+ one.
Yes I mis-read it...

I dont think I will ever need that many virtual interfaces. I am going to setup one Linux Bond for all 4 physical interfaces and trunk all my vlans across all four. This way I dont need to worry about what vlan is on what interface. I am going to start with 2 vfs on each physical interface and see if I need more.

One other thing...I updated the firmware on my I350, from v29 of the Intel driver pkg. Not sure if that did it but when I run echo 4 > /sys/class/net/enp39s0f0/device/sriov_numvfs the MACs are auto generated. I do not have to assign MACs.
 
Last edited: