Xeon D-1541 & SR-IOV (Need Help)

Yamabushi · Jun 2, 2016

I'm running the Xeon D1541 based X10SDV-8C-TLNF4, and as indicated by the results of "lspci | grep -i ethernet" believe I have successfully enabled SR-IOV for the two X552/X557-AT NIC's. I did that by modifying GRUB to read:

GRUB_CMDLINE_LINUX_DEFAULT="quiet splash intel_iommu=on iommu=pt net.ifnames=0 biosdevname=0 ixgbe.max_vfs=8"

That being said, I have not yet been able to successfully enable SR-IOV for the two I350 NIC's. FYI, I am running KVM on Kubuntu 16.04, am on BIOS 1.1, and intend to access the VF's via pools. My question is what is the correct/best way to enable SR-IOV for all four NIC's?

Mark Bradley · Jun 2, 2016

I have an X8DN+ board which has dual 82576 NICs that support virtual functions. What I did is as follows and it might not be the best or easiest but it seems to work. Your hardware is newer and probably doesn't have the quirks that mine does.

My kernel command line is:

BOOT_IMAGE=/boot/vmlinuz-4.6.0-1.gaf7ce24-default root=UUID=33f796bc-69b6-4180-9d36-50bf416ef49a resume=/dev/sda2 splash=silent quiet showopts intel_iommu=on iommu=pt

I don't specify the max_vfs there but, instead, I put it in /etc/modprobe.d/99-local.conf. Too lazy to change it...

options igb max_vfs=4
options vfio_iommu_type1 allow_unsafe_interrupts=1

The last option is because Intel shipped faulty silicon [1] and interrupt remapping was disabled. I allow unsafe interrupts per [2] and, so far, I see no ill effects. But read question 8 in [3] because it talks about MSI-based interrupt injection attacks. However remote that is. For me, just getting it working was reason enough.

I assume you've verified that IOMMU and VFIO are enabled in your kernel. Check the /boot/config-<kernel_ver> file:

grep -e IOMMU -e VFIO /boot/config-4.6.0-1.gaf7ce24-default | sort

Anyway, I also blacklist the igbvf driver in /etc/modprobe.d/50-blacklist.conf. You'd probably want to blacklist ixgbevf. It has to do with VFIO which would just unbind the device anyway. I create network pools as well and let KVM manage the devices, MAC addresses, etc [4].

All that said, a MACVLAN might be easier and, in some cases, faster [5] depending upon what you're doing. [5] also mentioned having to specify kernel option pci=assign-busses for an i350-T2. YMMV on that.

Let me know if I've skipped something or just did or said something wrong (it's late).

Good luck.

[1] Support | Faulty Intel chipsets cause problems with interrupt remapping
[2] Pci passthrough - Proxmox VE
[3] VFIO tips and tricks: VFIO+VGA FAQ
[4] Networking - Libvirt Wiki
[5] Exposing Docker containers with SR-IOV | Jason Anderson

Yamabushi · Jun 5, 2016

Thank you Mark for sharing your methodology!

After consulting with EluRex and doing a little experimentation, here is what I found works for me:

To enable the I350's modify igb.conf
$ sudo nano /etc/modprobe.d/igb.conf
options igb max_vfs=4

To enable the X552/X557-AT's modify ixgbe.conf
$ sudo nano /etc/modprobe.d/ixgbe.conf
options ixgbe max_vfs=8

Update
$ sudo update-initramfs -u

Reboot

Check SR-IOV status
$ lspci | grep -i ethernet

Which gave me:

03:00.0 Ethernet controller: Intel Corporation Ethernet Connection X552/X557-AT 10GBASE-T
03:00.1 Ethernet controller: Intel Corporation Ethernet Connection X552/X557-AT 10GBASE-T
03:10.0 Ethernet controller: Intel Corporation Ethernet Connection X552 Virtual Function
03:10.1 Ethernet controller: Intel Corporation Ethernet Connection X552 Virtual Function
03:10.2 Ethernet controller: Intel Corporation Ethernet Connection X552 Virtual Function
03:10.3 Ethernet controller: Intel Corporation Ethernet Connection X552 Virtual Function
03:10.4 Ethernet controller: Intel Corporation Ethernet Connection X552 Virtual Function
03:10.5 Ethernet controller: Intel Corporation Ethernet Connection X552 Virtual Function
03:10.6 Ethernet controller: Intel Corporation Ethernet Connection X552 Virtual Function
03:10.7 Ethernet controller: Intel Corporation Ethernet Connection X552 Virtual Function
03:11.0 Ethernet controller: Intel Corporation Ethernet Connection X552 Virtual Function
03:11.1 Ethernet controller: Intel Corporation Ethernet Connection X552 Virtual Function
03:11.2 Ethernet controller: Intel Corporation Ethernet Connection X552 Virtual Function
03:11.3 Ethernet controller: Intel Corporation Ethernet Connection X552 Virtual Function
03:11.4 Ethernet controller: Intel Corporation Ethernet Connection X552 Virtual Function
03:11.5 Ethernet controller: Intel Corporation Ethernet Connection X552 Virtual Function
03:11.6 Ethernet controller: Intel Corporation Ethernet Connection X552 Virtual Function
03:11.7 Ethernet controller: Intel Corporation Ethernet Connection X552 Virtual Function
06:00.0 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
06:00.1 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
07:10.0 Ethernet controller: Intel Corporation I350 Ethernet Controller Virtual Function (rev 01)
07:10.1 Ethernet controller: Intel Corporation I350 Ethernet Controller Virtual Function (rev 01)
07:10.4 Ethernet controller: Intel Corporation I350 Ethernet Controller Virtual Function (rev 01)
07:10.5 Ethernet controller: Intel Corporation I350 Ethernet Controller Virtual Function (rev 01)
07:11.0 Ethernet controller: Intel Corporation I350 Ethernet Controller Virtual Function (rev 01)
07:11.1 Ethernet controller: Intel Corporation I350 Ethernet Controller Virtual Function (rev 01)
07:11.4 Ethernet controller: Intel Corporation I350 Ethernet Controller Virtual Function (rev 01)
07:11.5 Ethernet controller: Intel Corporation I350 Ethernet Controller Virtual Function (rev 01)

FWIW, my GRUB_CMDLINE_LINUX_DEFAULT="quiet splash intel_iommu=on iommu=pt net.ifnames=0 biosdevname=0"

EluRex · Jun 5, 2016

Excellent! glad you solved it... i didn't pay attention in this thread but only reply with pm.... xeon-d 1540/1541 sr-iov now works flawlessly!
.
I would manually install intel latest driver for both ixgbe and igb from Intel Ethernet Drivers and Utilities (maintained by intel) instead of using intree drivers for better and more stable sr-iov usage.

Yamabushi · Jun 5, 2016

Thank you for your guidance and expertise, EluRex!

ggg · Jun 8, 2016

The 'right way' of configuring the number of vfs is per the driver's module:

echo $NUM_VFS > /sys/class/net/$ETH_DEV/device/sriov_numvfs

One way (not the only way) to run that at boot with systemd is with a a file in /etc/tmpfiles.d/; say ...:

/etc/tmpfiles.d/20-vfs.conf

... with contents:

w /sys/class/net/ens1f[0-1]/device/sriov_numvfs - - - - 7
w /sys/class/net/enp2s0f[0-1]/device/sriov_numvfs - - - - 64

The first line is an example for two I350 interfaces and the second for two X540 interfaces.

Replace the ens1f[0-1] and enp2s0f[0-1] with your interface's names and number(s).

This method also allows for different number of VFs (including none) per interfaces using the same kernel module.

References:
Intel Xeon D-1500 Series Discussion
Intel Xeon D-1500 Series Discussion

EluRex · Jun 8, 2016

i am running pve 4.2 debian 8.4.... the sysfs method no longer works "echo $NUM_VFS > /sys/class/net/$ETH_DEV/device/sriov_numvfs" for i350 or x552. However, it remain working for x540 cards

ggg · Jun 8, 2016

Using the kernel module option has been deprecated for awhile so that seems odd. You get a message noting that when you use it:

"Enabling SR-IOV VFs using the module parameter is deprecated - please use the pci sysfs interface."

linux/igb_main.c at v4.6 · torvalds/linux · GitHub

Perhaps is for some other reason.

EluRex · Jun 8, 2016

Code:

root@pve-kac:~# lspci -nn|grep Eth
03:00.0 Ethernet controller [0200]: Intel Corporation Ethernet Connection X552/X557-AT 10GBASE-T [8086:15ad]
03:00.1 Ethernet controller [0200]: Intel Corporation Ethernet Connection X552/X557-AT 10GBASE-T [8086:15ad]
05:00.0 Ethernet controller [0200]: Intel Corporation 82580 Gigabit Network Connection [8086:150e] (rev 01)
05:00.1 Ethernet controller [0200]: Intel Corporation 82580 Gigabit Network Connection [8086:150e] (rev 01)
05:00.2 Ethernet controller [0200]: Intel Corporation 82580 Gigabit Network Connection [8086:150e] (rev 01)
05:00.3 Ethernet controller [0200]: Intel Corporation 82580 Gigabit Network Connection [8086:150e] (rev 01)
06:00.0 Ethernet controller [0200]: Intel Corporation I350 Gigabit Network Connection [8086:1521] (rev 01)
06:00.1 Ethernet controller [0200]: Intel Corporation I350 Gigabit Network Connection [8086:1521] (rev 01)
07:10.0 Ethernet controller [0200]: Intel Corporation I350 Ethernet Controller Virtual Function [8086:1520] (rev 01)
07:10.1 Ethernet controller [0200]: Intel Corporation I350 Ethernet Controller Virtual Function [8086:1520] (rev 01)
07:10.4 Ethernet controller [0200]: Intel Corporation I350 Ethernet Controller Virtual Function [8086:1520] (rev 01)
07:10.5 Ethernet controller [0200]: Intel Corporation I350 Ethernet Controller Virtual Function [8086:1520] (rev 01)
07:11.0 Ethernet controller [0200]: Intel Corporation I350 Ethernet Controller Virtual Function [8086:1520] (rev 01)
07:11.1 Ethernet controller [0200]: Intel Corporation I350 Ethernet Controller Virtual Function [8086:1520] (rev 01)
07:11.4 Ethernet controller [0200]: Intel Corporation I350 Ethernet Controller Virtual Function [8086:1520] (rev 01)
07:11.5 Ethernet controller [0200]: Intel Corporation I350 Ethernet Controller Virtual Function [8086:1520] (rev 01)

root@pve-kac:~# cd /sys/class/net
root@pve-kac:/sys/class/net# ls -al
total 0
drwxr-xr-x  2 root root 0 Apr 18 16:39 .
drwxr-xr-x 70 root root 0 Apr 18 16:39 ..
lrwxrwxrwx  1 root root 0 Jun  5 07:56 bond0 -> ../../devices/virtual/net/bond0
lrwxrwxrwx  1 root root 0 Apr 18 16:39 eth0 -> ../../devices/pci0000:00/0000:00:1c.0/0000:06:00.0/net/eth0
lrwxrwxrwx  1 root root 0 Apr 18 16:39 eth1 -> ../../devices/pci0000:00/0000:00:1c.0/0000:06:00.1/net/eth1
lrwxrwxrwx  1 root root 0 Apr 18 16:39 eth2 -> ../../devices/pci0000:00/0000:00:02.2/0000:03:00.0/net/eth2
lrwxrwxrwx  1 root root 0 Apr 18 16:39 eth3 -> ../../devices/pci0000:00/0000:00:02.2/0000:03:00.1/net/eth3
lrwxrwxrwx  1 root root 0 Jun  5 07:56 eth4 -> ../../devices/pci0000:00/0000:00:03.0/0000:05:00.3/net/eth4
lrwxrwxrwx  1 root root 0 Apr 18 16:39 lo -> ../../devices/virtual/net/lo
lrwxrwxrwx  1 root root 0 Jun  5 07:56 ovs-system -> ../../devices/virtual/net/ovs-system
lrwxrwxrwx  1 root root 0 Jun  5 07:56 tap200i0 -> ../../devices/virtual/net/tap200i0
lrwxrwxrwx  1 root root 0 Jun  6 08:30 veth205i0 -> ../../devices/virtual/net/veth205i0
lrwxrwxrwx  1 root root 0 Jun  6 08:29 veth206i0 -> ../../devices/virtual/net/veth206i0
lrwxrwxrwx  1 root root 0 Apr 25 09:23 veth207i0 -> ../../devices/virtual/net/veth207i0
lrwxrwxrwx  1 root root 0 Apr 25 09:23 veth208i0 -> ../../devices/virtual/net/veth208i0
lrwxrwxrwx  1 root root 0 May 30 13:18 veth210i0 -> ../../devices/virtual/net/veth210i0
lrwxrwxrwx  1 root root 0 Apr 25 09:23 vmbr0 -> ../../devices/virtual/net/vmbr0
lrwxrwxrwx  1 root root 0 Jun  5 07:56 vmbr0p1 -> ../../devices/virtual/net/vmbr0p1
lrwxrwxrwx  1 root root 0 Apr 18 16:40 vmbr1 -> ../../devices/virtual/net/vmbr1
lrwxrwxrwx  1 root root 0 Jun  5 07:56 vmbr1p1 -> ../../devices/virtual/net/vmbr1p1

root@pve-kac:/sys/class/net# lspci -nn|grep Eth
03:00.0 Ethernet controller [0200]: Intel Corporation Ethernet Connection X552/X557-AT 10GBASE-T [8086:15ad]
03:00.1 Ethernet controller [0200]: Intel Corporation Ethernet Connection X552/X557-AT 10GBASE-T [8086:15ad]
05:00.0 Ethernet controller [0200]: Intel Corporation 82580 Gigabit Network Connection [8086:150e] (rev 01)
05:00.1 Ethernet controller [0200]: Intel Corporation 82580 Gigabit Network Connection [8086:150e] (rev 01)
05:00.2 Ethernet controller [0200]: Intel Corporation 82580 Gigabit Network Connection [8086:150e] (rev 01)
05:00.3 Ethernet controller [0200]: Intel Corporation 82580 Gigabit Network Connection [8086:150e] (rev 01)
06:00.0 Ethernet controller [0200]: Intel Corporation I350 Gigabit Network Connection [8086:1521] (rev 01)
06:00.1 Ethernet controller [0200]: Intel Corporation I350 Gigabit Network Connection [8086:1521] (rev 01)
07:10.0 Ethernet controller [0200]: Intel Corporation I350 Ethernet Controller Virtual Function [8086:1520] (rev 01)
07:10.1 Ethernet controller [0200]: Intel Corporation I350 Ethernet Controller Virtual Function [8086:1520] (rev 01)
07:10.4 Ethernet controller [0200]: Intel Corporation I350 Ethernet Controller Virtual Function [8086:1520] (rev 01)
07:10.5 Ethernet controller [0200]: Intel Corporation I350 Ethernet Controller Virtual Function [8086:1520] (rev 01)
07:11.0 Ethernet controller [0200]: Intel Corporation I350 Ethernet Controller Virtual Function [8086:1520] (rev 01)
07:11.1 Ethernet controller [0200]: Intel Corporation I350 Ethernet Controller Virtual Function [8086:1520] (rev 01)
07:11.4 Ethernet controller [0200]: Intel Corporation I350 Ethernet Controller Virtual Function [8086:1520] (rev 01)
07:11.5 Ethernet controller [0200]: Intel Corporation I350 Ethernet Controller Virtual Function [8086:1520] (rev 01)

root@pve-kac:/sys/class/net# echo  4 > eth2/device/sriov_numvfs
-bash: echo: write error: Operation not supported

Search

Xeon D-1541 & SR-IOV (Need Help)

Yamabushi

Member

Mark Bradley

New Member

Yamabushi

Member

EluRex

Active Member

Yamabushi

Member

ggg

Member

EluRex

Active Member

ggg

Member

EluRex

Active Member