Intel Xeon D-1500 Series Discussion

[FQB]

Muh... absolutely not amused where this discussion is going. Especially:
Is it a hardware firmware issue or a driver issue? Why are a lot of sites touting this as the 'ultimate' virtualization or 'must get' platform (Tinkertry, ServeTheHome, etc)... What is heavy virtualization btw?
PS. Offtopic -> Intel Xeon D-1540 10GbE X552/X557 driver/VIB for VMware ESXi 6 on Supermicro 5028D-TN4T system's X10SDV-TLN4F mobo expected by early Nov. 2015. See post @ Tinkertry

I'm in the same boat - just haven't bought it just yet.

- According to Anandtech's previous review:
"Speaking of networking, the SoC will have bandwidth for two 10GbE connections direct, which will work both in 1G and 2.5G modes. These are optimized for virtualization, allowing 128 Tx and Rx queues per port as well as SR-IOV and VMDq enhancements. With the integration on board, driver support should also be easier to manage rather than external controller solutions."

 

[Jerry Chen]

I think they are talking about the omboard controllers. The i350 is a normal PCIe device. The onboard NICs aren't over PCIe which is why you can't pass them through with VT-d.

Actually, the onboard 1Gb NIC is SR-IOV capable. I had a blog post involving SR-IOV while doing GPU passthrough: GPU Passthrough via vfio-pci with KVM on Ubuntu 15.04

 

[Jeggs101]

Why are a lot of sites touting this as the 'ultimate' virtualization or 'must get' platform (Tinkertry, ServeTheHome, etc)... What is heavy virtualization btw?

I know Paul is on here but he sells those Supermicro machines or has a deal to sell them that he pushes a lot. I don't consider him an independent party with Tinkertry like I would AT, Engadget STH. I'm not saying he doesn't have good stuff, but I consider him as independent as a Supermicro salesperson at this point.

I'm also thinking there's a difference between support for PCIe devices and on die devices in this area. I'm guessing you'll be fine and @PigLover from what I have read is a real world networking guru. I reread everything he renders opinion on.

 

[EluRex]

Muh... absolutely not amused where this discussion is going.....

Neither do I, I have purchased two of these servers and sr-iov is critical for 10 gbe port due to openvswitch network performance limitations or like what @PigLover suggested trading CPU core for network performance.

Like what Patrick said, its not a show stopper, the 10 gbe can be dedicated using for file server (zfs) purpose, used one cpu core as openvswitch + dpdk acceleration for the other 10 gbe for vm to use (maybe reduce max vm from 16 to 14?, just use more containers than vm then). The two 1gbe ports can be passthrough to Pfsense vm as WAN and LAN.

 

[jonaspaulo]

I have also bought one of these and needed SR-IOV. I checked the press release for the boards at the time on SM site and they said they supported it as well , as you mentioned, many other sites.

This is really not a good move by them. I am a little upset..

 

[EluRex]

funny thing happened with kernel 4.2.3..... all sr-iov show up.... how?![emoji7]

 

[FQB]

funny thing happened with kernel 4.2.3..... all sr-iov show up.... how?![emoji7]

as Jerry Chen pointed out before:

" the onboard 1Gb NIC is SR-IOV capable." but not for 10GB

 

[Davewolfs]

If future revisions support it and these boards don't I'd be mad too.

 

[EluRex]

as Jerry Chen pointed out before:

" the onboard 1Gb NIC is SR-IOV capable." but not for 10GB

the lspci shows even the 10gbe card with VF... 03.xx.x are all intel x557 10gbe card's VF device

 

[Jerry Chen]

the lspci shows even the 10gbe card with VF... 03.xx.x are all intel x557 10gbe card's VF device

On my Ubuntu 15.04 kernel 3.9 environment, I'm unable to enable VFs for X552/557. What are your steps to reproduce such result?

 

[EluRex]

I am on ubuntu 14.04.3 and manually upgrade the kernel to 4.2.3 ( I am not able to use 4.2.4 or 4.2.5 due to zfs dkms compile failed, I am on zfs root file system)

root@pdc:~# uname -r
4.2.3-040203-generic

lspci -vnn | grep Ethernet
03:00.0 Ethernet controller [0200]: Intel Corporation Device [8086:15ad]
03:00.1 Ethernet controller [0200]: Intel Corporation Device [8086:15ad]
03:10.0 Ethernet controller [0200]: Intel Corporation Device [8086:15a8]
03:10.1 Ethernet controller [0200]: Intel Corporation Device [8086:15a8]
03:10.2 Ethernet controller [0200]: Intel Corporation Device [8086:15a8]
03:10.3 Ethernet controller [0200]: Intel Corporation Device [8086:15a8]
03:10.4 Ethernet controller [0200]: Intel Corporation Device [8086:15a8]
03:10.5 Ethernet controller [0200]: Intel Corporation Device [8086:15a8]
03:10.6 Ethernet controller [0200]: Intel Corporation Device [8086:15a8]
03:10.7 Ethernet controller [0200]: Intel Corporation Device [8086:15a8]
03:11.0 Ethernet controller [0200]: Intel Corporation Device [8086:15a8]
03:11.1 Ethernet controller [0200]: Intel Corporation Device [8086:15a8]
03:11.2 Ethernet controller [0200]: Intel Corporation Device [8086:15a8]
03:11.3 Ethernet controller [0200]: Intel Corporation Device [8086:15a8]
03:11.4 Ethernet controller [0200]: Intel Corporation Device [8086:15a8]
03:11.5 Ethernet controller [0200]: Intel Corporation Device [8086:15a8]
03:11.6 Ethernet controller [0200]: Intel Corporation Device [8086:15a8]
03:11.7 Ethernet controller [0200]: Intel Corporation Device [8086:15a8]

root@pdc:~# cat /etc/modprobe.d/ixgbe.conf
options ixgbe max_vfs=8

The iommu group is seperated and ready to be use for kvm vfio pci passthrough

/sys/kernel/iommu_groups/45/devices/0000:03:10.0
/sys/kernel/iommu_groups/46/devices/0000:03:10.2
/sys/kernel/iommu_groups/47/devices/0000:03:10.4
/sys/kernel/iommu_groups/48/devices/0000:03:10.6
/sys/kernel/iommu_groups/49/devices/0000:03:11.0
/sys/kernel/iommu_groups/50/devices/0000:03:11.2
/sys/kernel/iommu_groups/51/devices/0000:03:11.4
/sys/kernel/iommu_groups/52/devices/0000:03:11.6
/sys/kernel/iommu_groups/53/devices/0000:03:10.1
/sys/kernel/iommu_groups/54/devices/0000:03:10.3
/sys/kernel/iommu_groups/55/devices/0000:03:10.5
/sys/kernel/iommu_groups/56/devices/0000:03:10.7
/sys/kernel/iommu_groups/57/devices/0000:03:11.1
/sys/kernel/iommu_groups/58/devices/0000:03:11.3
/sys/kernel/iommu_groups/59/devices/0000:03:11.5
/sys/kernel/iommu_groups/60/devices/0000:03:11.7

 

[ggg]

On my Ubuntu 15.04 kernel 3.9 environment, I'm unable to enable VFs for X552/557. What are your steps to reproduce such result?

@Jerry Chen

Use kernel 4.2+ ...

ixgbe: add define for X557 PHY ID · torvalds/linux@c2c78d5 · GitHub

... or build the ixgbe module for your earlier kernel from the code on sourceforge.

Thanks @EluRex and @Jerry Chen for confirming SR-IOV works for you.

I had thought it should from the PCI info provided by @wago.

BTW, the num_vfs kernel module parameter is deprecated per the warning.

This is one way to do it with systemd:

cat /etc/tmpfiles.d/igb.conf
w /sys/class/net/eth[0-1]/device/sriov_numvfs - - - - 7
w /sys/class/net/eth[2-3]/device/sriov_numvfs - - - - 64

The first is for 7 VFs for an I350. The second for an X557 - look at the 64!!!

A better way then with tmpfiles.d would probably be with something specific to udev.

 

[ggg]

@EluRex Thanks!

Finally, the IOMMU groups!

Matching the ZFS for Linux while staying with more recent kernels can be frustrating

I've been considering using this platform and it's nice to know that what I expected would work does indeed work!

 

[ggg]

@EluRex

BTW it took me too long to compose that post with link and example from my config.

I posted some of the same info right after you. Sorry!

 

[EluRex]

Matching the ZFS for Linux while staying with more recent kernels can be frustrating

tell me about it... my configuration also adds mellanox infiniband ofed driver and scst srpt into the mix.... I have to choose my kernel version very very very carefully.

 

[Jerry Chen]

@Jerry Chen

Use kernel 4.2+ ...

ixgbe: add define for X557 PHY ID · torvalds/linux@c2c78d5 · GitHub

... or build the ixgbe module for your earlier kernel from the code on sourceforge.

Thanks @EluRex and @Jerry Chen for confirming SR-IOV works for you.

I had thought it should from the PCI info provided by @wago.

BTW, the num_vfs kernel module parameter is deprecated per the warning.

This is one way to do it with systemd:

cat /etc/tmpfiles.d/igb.conf
w /sys/class/net/eth[0-1]/device/sriov_numvfs - - - - 7
w /sys/class/net/eth[2-3]/device/sriov_numvfs - - - - 64

The first is for 7 VFs for an I350. The second for an X557 - look at the 64!!!

A better way then with tmpfiles.d would probably be with something specific to udev.

root@xeon-kvm-host:/tmp/ixgbe-4.1.5/src# modinfo ixgbe

filename:       /lib/modules/4.0.0-040000-generic/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko

version:        4.1.5

license:        GPL

description:    Intel(R) 10 Gigabit PCI Express Network Driver

author:         Intel Corporation, <linux.nics@intel.com>
...

[  213.798236] ixgbe: I/O Virtualization (IOV) set to 4

[  213.798240] ixgbe: 0000:03:00.0: ixgbe_check_options: IOV is not supported on this hardware.  Disabling IOV.

 

[EluRex]

root@pdc:~# modinfo ixgbe
filename:       /lib/modules/4.2.3-040203-generic/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko
version:        4.0.1-k
license:        GPL
description:    Intel(R) 10 Gigabit PCI Express Network Driver
author:         Intel Corporation, <linux.nics@intel.com>
srcversion:     E08010C0D72963989E9A05E
alias:          pci:v00008086d000015ADsv*sd*bc*sc*i*
....
alias:          pci:v00008086d000010B6sv*sd*bc*sc*i*
depends:        mdio,ptp,dca,vxlan
intree:         Y
vermagic:       4.2.3-040203-generic SMP mod_unload modversions
signer:         Build time autogenerated kernel key
sig_key:        CE:E6:5B:C1:E3:07:7F:6F:F5:58:77:52:AC:20:3F:9D:65:F7:05:DA
sig_hashalgo:   sha512
parm:           max_vfs:Maximum number of virtual functions to allocate per physical function - default is zero and maximum value is 63. (Deprecated) (uint)
parm:           allow_unsupported_sfp:Allow unsupported and untested SFP+ modules on 82599-based adapters (uint)
parm:           debug:Debug level (0=none,...,16=all) (int)

 

[EluRex]

root@pdc:~# dmesg|grep ixgbe
[    0.000000] Command line: BOOT_IMAGE=/ROOT/ubuntu@/boot/vmlinuz-4.2.3-040203-generic root=ZFS=rpool/ROOT/ubuntu ro boot=zfs rpool=rpool bootfs=rpool/ROOT/ubuntu intel_iommu=on ixgbe.max_vfs=8
[    0.000000] Kernel command line: BOOT_IMAGE=/ROOT/ubuntu@/boot/vmlinuz-4.2.3-040203-generic root=ZFS=rpool/ROOT/ubuntu ro boot=zfs rpool=rpool bootfs=rpool/ROOT/ubuntu intel_iommu=on ixgbe.max_vfs=8
[    7.208439] ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver - version 4.0.1-k
[    7.208439] ixgbe: Copyright (c) 1999-2014 Intel Corporation.
[    7.208607] ixgbe 0000:03:00.0: Enabling SR-IOV VFs using the max_vfs module parameter is deprecated - please use the pci sysfs interface instead.
[    9.240064] ixgbe 0000:03:00.0 0000:03:00.0 (uninitialized): SR-IOV enabled with 8 VFs
[    9.512691] ixgbevf: Intel(R) 10 Gigabit PCI Express Virtual Function Network Driver - version 2.12.1-k
[    9.512692] ixgbevf: Copyright (c) 2009 - 2012 Intel Corporation.
[    9.611689] ixgbe 0000:03:00.0: Multiqueue Enabled: Rx Queue count = 4, Tx Queue count = 4
[    9.678012] ixgbe 0000:03:00.0: PCI Express bandwidth of 2GT/s available
[    9.678013] ixgbe 0000:03:00.0: (Speed:2.5GT/s, Width: x1, Encoding Loss:20%)
[    9.678014] ixgbe 0000:03:00.0: This is not sufficient for optimal performance of this card.
[    9.678015] ixgbe 0000:03:00.0: For optimal performance, at least 20GT/s of bandwidth is required.
[    9.678015] ixgbe 0000:03:00.0: A slot with more lanes and/or higher speed is suggested.
[    9.722230] ixgbe 0000:03:00.0: MAC: 5, PHY: 6, PBA No: 020000-000
[    9.722231] ixgbe 0000:03:00.0: 0c:c4:7a:74:09:18
[   15.215889] ixgbe 0000:03:00.0 eth1: IOV is enabled with 8 VFs
[   15.238000] ixgbe 0000:03:00.0: Intel(R) 10 Gigabit Network Connection
[   15.238201] ixgbe 0000:03:00.1: Enabling SR-IOV VFs using the max_vfs module parameter is deprecated - please use the pci sysfs interface instead.
[   16.787177] ixgbevf 0000:03:10.1: enabling device (0000 -> 0002)
[   16.788293] ixgbevf 0000:03:10.1: PF still in reset state.  Is the PF interface up?
[   16.788293] ixgbevf 0000:03:10.1: Assigning random MAC address
[   18.431419] ixgbevf 0000:03:10.1: 86:c7:2f:a5:60:0c
[   18.431420] ixgbevf 0000:03:10.1: MAC: 4
[   18.431420] ixgbevf 0000:03:10.1: Intel(R) 82599 Virtual Function
[   18.431577] ixgbevf 0000:03:10.3: enabling device (0000 -> 0002)
[   18.432709] ixgbevf 0000:03:10.3: PF still in reset state.  Is the PF interface up?
[   18.432709] ixgbevf 0000:03:10.3: Assigning random MAC address
[   19.363463] ixgbevf 0000:03:10.3: 62:5a:23:07:bd:59
[   19.363464] ixgbevf 0000:03:10.3: MAC: 4
[   19.363465] ixgbevf 0000:03:10.3: Intel(R) 82599 Virtual Function
[   19.363609] ixgbevf 0000:03:10.5: enabling device (0000 -> 0002)
[   19.364718] ixgbevf 0000:03:10.5: PF still in reset state.  Is the PF interface up?
[   19.364719] ixgbevf 0000:03:10.5: Assigning random MAC address
[   19.364849] ixgbevf 0000:03:10.5: 92:65:d9:a2:cd:48
[   19.364849] ixgbevf 0000:03:10.5: MAC: 4
[   19.364850] ixgbevf 0000:03:10.5: Intel(R) 82599 Virtual Function
[   19.364978] ixgbevf 0000:03:10.7: enabling device (0000 -> 0002)
[   19.366081] ixgbevf 0000:03:10.7: PF still in reset state.  Is the PF interface up?
[   19.366082] ixgbevf 0000:03:10.7: Assigning random MAC address
[   19.366202] ixgbevf 0000:03:10.7: de:0c:6b:25:63:08
[   19.366203] ixgbevf 0000:03:10.7: MAC: 4
[   19.366203] ixgbevf 0000:03:10.7: Intel(R) 82599 Virtual Function
[   19.366329] ixgbevf 0000:03:11.1: enabling device (0000 -> 0002)
[   19.367439] ixgbevf 0000:03:11.1: PF still in reset state.  Is the PF interface up?
[   19.367440] ixgbevf 0000:03:11.1: Assigning random MAC address
[   19.367568] ixgbevf 0000:03:11.1: 4a:fe:8f:ee:a9:1b
[   19.367569] ixgbevf 0000:03:11.1: MAC: 4
[   19.367569] ixgbevf 0000:03:11.1: Intel(R) 82599 Virtual Function
[   19.367697] ixgbevf 0000:03:11.3: enabling device (0000 -> 0002)
[   19.368804] ixgbevf 0000:03:11.3: PF still in reset state.  Is the PF interface up?
[   19.368804] ixgbevf 0000:03:11.3: Assigning random MAC address
[   19.368925] ixgbevf 0000:03:11.3: ce:85:10:5b:70:dc
[   19.368925] ixgbevf 0000:03:11.3: MAC: 4
[   19.368926] ixgbevf 0000:03:11.3: Intel(R) 82599 Virtual Function
[   19.369050] ixgbevf 0000:03:11.5: enabling device (0000 -> 0002)
[   19.370171] ixgbevf 0000:03:11.5: PF still in reset state.  Is the PF interface up?
[   19.370172] ixgbevf 0000:03:11.5: Assigning random MAC address
[   19.370308] ixgbevf 0000:03:11.5: a6:60:df:d0:d1:34
[   19.370309] ixgbevf 0000:03:11.5: MAC: 4
[   19.370310] ixgbevf 0000:03:11.5: Intel(R) 82599 Virtual Function
[   19.370433] ixgbevf 0000:03:11.7: enabling device (0000 -> 0002)
[   19.371535] ixgbevf 0000:03:11.7: PF still in reset state.  Is the PF interface up?
[   19.371536] ixgbevf 0000:03:11.7: Assigning random MAC address
[   19.371658] ixgbevf 0000:03:11.7: d2:bc:8a:08:3a:8a
[   19.371659] ixgbevf 0000:03:11.7: MAC: 4
[   19.371660] ixgbevf 0000:03:11.7: Intel(R) 82599 Virtual Function
[   19.371673] ixgbe 0000:03:00.1 0000:03:00.1 (uninitialized): SR-IOV enabled with 8 VFs
[   19.748219] ixgbe 0000:03:00.1: Multiqueue Enabled: Rx Queue count = 4, Tx Queue count = 4
[   19.814547] ixgbe 0000:03:00.1: PCI Express bandwidth of 2GT/s available
[   19.814549] ixgbe 0000:03:00.1: (Speed:2.5GT/s, Width: x1, Encoding Loss:20%)
[   19.814549] ixgbe 0000:03:00.1: This is not sufficient for optimal performance of this card.
[   19.814550] ixgbe 0000:03:00.1: For optimal performance, at least 20GT/s of bandwidth is required.
[   19.814551] ixgbe 0000:03:00.1: A slot with more lanes and/or higher speed is suggested.
[   19.858764] ixgbe 0000:03:00.1: MAC: 5, PHY: 6, PBA No: 020000-000
[   19.858765] ixgbe 0000:03:00.1: 0c:c4:7a:74:09:19
[   21.708071] ixgbe 0000:03:00.1 eth6: IOV is enabled with 8 VFs
[   21.727074] ixgbe 0000:03:00.1: Intel(R) 10 Gigabit Network Connection
[   21.727153] ixgbevf 0000:03:10.0: enabling device (0000 -> 0002)
[   21.728283] ixgbevf 0000:03:10.0: PF still in reset state.  Is the PF interface up?
[   21.728284] ixgbevf 0000:03:10.0: Assigning random MAC address
[   22.123348] ixgbevf 0000:03:10.0: 8e:0c:94:c5:aa:f0
[   22.123349] ixgbevf 0000:03:10.0: MAC: 4
[   22.123349] ixgbevf 0000:03:10.0: Intel(R) 82599 Virtual Function
[   22.123360] ixgbevf 0000:03:10.2: enabling device (0000 -> 0002)
[   22.124467] ixgbevf 0000:03:10.2: PF still in reset state.  Is the PF interface up?
[   22.124468] ixgbevf 0000:03:10.2: Assigning random MAC address
[   22.124597] ixgbevf 0000:03:10.2: 9e:a9:c2:aa:1e:18
[   22.124597] ixgbevf 0000:03:10.2: MAC: 4
[   22.124598] ixgbevf 0000:03:10.2: Intel(R) 82599 Virtual Function
[   22.124605] ixgbevf 0000:03:10.4: enabling device (0000 -> 0002)
[   22.125706] ixgbevf 0000:03:10.4: PF still in reset state.  Is the PF interface up?
[   22.125706] ixgbevf 0000:03:10.4: Assigning random MAC address
[   22.125832] ixgbevf 0000:03:10.4: 6e:c3:69:e7:ff:4e
[   22.125832] ixgbevf 0000:03:10.4: MAC: 4
[   22.125833] ixgbevf 0000:03:10.4: Intel(R) 82599 Virtual Function
[   22.125840] ixgbevf 0000:03:10.6: enabling device (0000 -> 0002)
[   22.126941] ixgbevf 0000:03:10.6: PF still in reset state.  Is the PF interface up?
[   22.126942] ixgbevf 0000:03:10.6: Assigning random MAC address
[   22.943349] ixgbevf 0000:03:10.6: 86:df:7b:b6:81:e4
[   22.943350] ixgbevf 0000:03:10.6: MAC: 4
[   22.943351] ixgbevf 0000:03:10.6: Intel(R) 82599 Virtual Function
[   22.943362] ixgbevf 0000:03:11.0: enabling device (0000 -> 0002)
[   22.944516] ixgbevf 0000:03:11.0: PF still in reset state.  Is the PF interface up?
[   22.944517] ixgbevf 0000:03:11.0: Assigning random MAC address
[   22.944643] ixgbevf 0000:03:11.0: a6:41:27:17:7b:f5
[   22.944643] ixgbevf 0000:03:11.0: MAC: 4
[   22.944644] ixgbevf 0000:03:11.0: Intel(R) 82599 Virtual Function
[   22.944649] ixgbevf 0000:03:11.2: enabling device (0000 -> 0002)
[   22.945749] ixgbevf 0000:03:11.2: PF still in reset state.  Is the PF interface up?
[   22.945750] ixgbevf 0000:03:11.2: Assigning random MAC address
[   22.945863] ixgbevf 0000:03:11.2: d6:c7:c2:9f:a3:97
[   22.945864] ixgbevf 0000:03:11.2: MAC: 4
[   22.945865] ixgbevf 0000:03:11.2: Intel(R) 82599 Virtual Function
[   22.945871] ixgbevf 0000:03:11.4: enabling device (0000 -> 0002)
[   22.946978] ixgbevf 0000:03:11.4: PF still in reset state.  Is the PF interface up?
[   22.946979] ixgbevf 0000:03:11.4: Assigning random MAC address
[   22.947093] ixgbevf 0000:03:11.4: 42:3f:49:4f:8f:98
[   22.947094] ixgbevf 0000:03:11.4: MAC: 4
[   22.947094] ixgbevf 0000:03:11.4: Intel(R) 82599 Virtual Function
[   22.947100] ixgbevf 0000:03:11.6: enabling device (0000 -> 0002)
[   22.948245] ixgbevf 0000:03:11.6: PF still in reset state.  Is the PF interface up?
[   22.948246] ixgbevf 0000:03:11.6: Assigning random MAC address
[   22.948354] ixgbevf 0000:03:11.6: 72:02:ed:13:7f:05
[   22.948355] ixgbevf 0000:03:11.6: MAC: 4
[   22.948355] ixgbevf 0000:03:11.6: Intel(R) 82599 Virtual Function
[   23.162283] ixgbevf 0000:03:11.7 em2_7: renamed from eth10
[   23.979141] ixgbevf 0000:03:11.6 em1_7: renamed from eth17
[   34.531118] ixgbe 0000:03:00.1 rename27: renamed from eth6
[   35.502452] ixgbevf: Unable to start - perhaps the PF Driver isn't up yet

the only thing concerns me is
ixgbe 0000:03:00.0: PCI Express bandwidth of 2GT/s available
ixgbe 0000:03:00.0: (Speed:2.5GT/s, Width: x1, Encoding Loss:20%)
ixgbe 0000:03:00.0: For optimal performance, at least 20GT/s of bandwidth is required.

your modinfo shows

4.0.0-040000-generic

I believe kernel version 4.2.x is the key


ixgbe 0000:03:00.0: Enabling SR-IOV VFs using the max_vfs module parameter is deprecated - please use the pci sysfs interface instead.

I need to find out how to use pci sysfs way to enable it

 

[EluRex]

03:10.1 Ethernet controller [0200]: Intel Corporation Device [8086:15a8]
Subsystem: Super Micro Computer Inc Device [15d9:15ad]
Flags: fast devsel
[virtual] Memory at fb300000 (64-bit, non-prefetchable) [size=16K]
[virtual] Memory at fb200000 (64-bit, non-prefetchable) [size=16K]
Capabilities: [70] MSI-X: Enable- Count=3 Masked-
Capabilities: [a0] Express Endpoint, MSI 00
Capabilities: [100] Advanced Error Reporting
Capabilities: [150] Alternative Routing-ID Interpretation (ARI)
Capabilities: [1b0] Access Control Services Kernel driver in use: vfio-pci

03:10.2 Ethernet controller [0200]: Intel Corporation Device [8086:15a8]
Subsystem: Super Micro Computer Inc Device [15d9:15ad]
Flags: fast devsel
[virtual] Memory at fb504000 (64-bit, non-prefetchable) [size=16K]
[virtual] Memory at fb404000 (64-bit, non-prefetchable) [size=16K]
Capabilities: [70] MSI-X: Enable- Count=3 Masked-
Capabilities: [a0] Express Endpoint, MSI 00
Capabilities: [100] Advanced Error Reporting
Capabilities: [150] Alternative Routing-ID Interpretation (ARI)
Capabilities: [1b0] Access Control Services Kernel driver in use: vfio-pci

Confirm pci-passthrough and freebsd able to see the card


pfsense VM sees 10gbe card.

 

[ggg]

I need to find out how to use pci sysfs way to enable it

@EluRex

export NUM_VFS=7
export ETH_NUM=eth0
echo $NUM_VFS > /sys/class/net/$ETH_NUM/device/sriov_numvfs

Change ETH_NUM to match the interface you want to do that for. You should have 4 (2 from I350 and 2 from X557).

Adjust the NUM_VFS to match the number of VFs you want per interface up to the max for the PF.

A way to have this done at boot with tmpfiles.d is in my earlier post.