SM 5028D-TN4T X10SDV-TLN4F X557 sr-iov Does Not Work, i350 Work

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.
EluRex

EluRex

Active Member
Apr 28, 2015
218
80
28
Los Angeles, CA
This thread is fork from https://forums.servethehome.com/ind...500-series-discussion.5036/page-34#post-65094


Ubuntu 14.04.3 with Kernel 4.2.3 with success X557 VF and pci-passthrough but not yet test for a successful ping test. The other NIC Intel i350 confirmed with SR-IOV/pci passthrough/successful ping (also on Proxmox 4)
Code: 
root@pdc:~# uname -r
4.2.3-040203-generic

lspci -vnn | grep Ethernet
03:00.0 Ethernet controller [0200]: Intel Corporation Device [8086:15ad]
03:00.1 Ethernet controller [0200]: Intel Corporation Device [8086:15ad]
03:10.0 Ethernet controller [0200]: Intel Corporation Device [8086:15a8]
03:10.1 Ethernet controller [0200]: Intel Corporation Device [8086:15a8]
03:10.2 Ethernet controller [0200]: Intel Corporation Device [8086:15a8]
03:10.3 Ethernet controller [0200]: Intel Corporation Device [8086:15a8]
03:10.4 Ethernet controller [0200]: Intel Corporation Device [8086:15a8]
03:10.5 Ethernet controller [0200]: Intel Corporation Device [8086:15a8]
03:10.6 Ethernet controller [0200]: Intel Corporation Device [8086:15a8]
03:10.7 Ethernet controller [0200]: Intel Corporation Device [8086:15a8]
03:11.0 Ethernet controller [0200]: Intel Corporation Device [8086:15a8]
03:11.1 Ethernet controller [0200]: Intel Corporation Device [8086:15a8]
03:11.2 Ethernet controller [0200]: Intel Corporation Device [8086:15a8]
03:11.3 Ethernet controller [0200]: Intel Corporation Device [8086:15a8]
03:11.4 Ethernet controller [0200]: Intel Corporation Device [8086:15a8]
03:11.5 Ethernet controller [0200]: Intel Corporation Device [8086:15a8]
03:11.6 Ethernet controller [0200]: Intel Corporation Device [8086:15a8]
03:11.7 Ethernet controller [0200]: Intel Corporation Device [8086:15a8]

root@pdc:~# cat /etc/modprobe.d/ixgbe.conf
options ixgbe max_vfs=8

The iommu group is seperated and ready to be use for kvm vfio pci passthrough

/sys/kernel/iommu_groups/45/devices/0000:03:10.0
/sys/kernel/iommu_groups/46/devices/0000:03:10.2
/sys/kernel/iommu_groups/47/devices/0000:03:10.4
/sys/kernel/iommu_groups/48/devices/0000:03:10.6
/sys/kernel/iommu_groups/49/devices/0000:03:11.0
/sys/kernel/iommu_groups/50/devices/0000:03:11.2
/sys/kernel/iommu_groups/51/devices/0000:03:11.4
/sys/kernel/iommu_groups/52/devices/0000:03:11.6
/sys/kernel/iommu_groups/53/devices/0000:03:10.1
/sys/kernel/iommu_groups/54/devices/0000:03:10.3
/sys/kernel/iommu_groups/55/devices/0000:03:10.5
/sys/kernel/iommu_groups/56/devices/0000:03:10.7
/sys/kernel/iommu_groups/57/devices/0000:03:11.1
/sys/kernel/iommu_groups/58/devices/0000:03:11.3
/sys/kernel/iommu_groups/59/devices/0000:03:11.5
/sys/kernel/iommu_groups/60/devices/0000:03:11.7
x557.jpg
 
  • Like
Reactions: FQB
EluRex

EluRex

Active Member
Apr 28, 2015
218
80
28
Los Angeles, CA
Code: 
root@pdc:~# modinfo ixgbe
filename:       /lib/modules/4.2.3-040203-generic/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko
version:        4.0.1-k
license:        GPL
description:    Intel(R) 10 Gigabit PCI Express Network Driver
author:         Intel Corporation, <linux.nics@intel.com>
srcversion:     E08010C0D72963989E9A05E
alias:          pci:v00008086d000015ADsv*sd*bc*sc*i*
....
alias:          pci:v00008086d000010B6sv*sd*bc*sc*i*
depends:        mdio,ptp,dca,vxlan
intree:         Y
vermagic:       4.2.3-040203-generic SMP mod_unload modversions
signer:         Build time autogenerated kernel key
sig_key:        CE:E6:5B:C1:E3:07:7F:6F:F5:58:77:52:AC:20:3F:9D:65:F7:05:DA
sig_hashalgo:   sha512
parm:           max_vfs:Maximum number of virtual functions to allocate per physical function - default is zero and maximum value is 63. (Deprecated) (uint)
parm:           allow_unsupported_sfp:Allow unsupported and untested SFP+ modules on 82599-based adapters (uint)
parm:           debug:Debug level (0=none,...,16=all) (int)
Code: 
root@pdc:~# dmesg|grep ixgbe
[    0.000000] Command line: BOOT_IMAGE=/ROOT/ubuntu@/boot/vmlinuz-4.2.3-040203-generic root=ZFS=rpool/ROOT/ubuntu ro boot=zfs rpool=rpool bootfs=rpool/ROOT/ubuntu intel_iommu=on ixgbe.max_vfs=8
[    0.000000] Kernel command line: BOOT_IMAGE=/ROOT/ubuntu@/boot/vmlinuz-4.2.3-040203-generic root=ZFS=rpool/ROOT/ubuntu ro boot=zfs rpool=rpool bootfs=rpool/ROOT/ubuntu intel_iommu=on ixgbe.max_vfs=8
[    7.208439] ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver - version 4.0.1-k
[    7.208439] ixgbe: Copyright (c) 1999-2014 Intel Corporation.
[    7.208607] ixgbe 0000:03:00.0: Enabling SR-IOV VFs using the max_vfs module parameter is deprecated - please use the pci sysfs interface instead.
[    9.240064] ixgbe 0000:03:00.0 0000:03:00.0 (uninitialized): SR-IOV enabled with 8 VFs
[    9.512691] ixgbevf: Intel(R) 10 Gigabit PCI Express Virtual Function Network Driver - version 2.12.1-k
[    9.512692] ixgbevf: Copyright (c) 2009 - 2012 Intel Corporation.
[    9.611689] ixgbe 0000:03:00.0: Multiqueue Enabled: Rx Queue count = 4, Tx Queue count = 4
[    9.678012] ixgbe 0000:03:00.0: PCI Express bandwidth of 2GT/s available
[    9.678013] ixgbe 0000:03:00.0: (Speed:2.5GT/s, Width: x1, Encoding Loss:20%)
[    9.678014] ixgbe 0000:03:00.0: This is not sufficient for optimal performance of this card.
[    9.678015] ixgbe 0000:03:00.0: For optimal performance, at least 20GT/s of bandwidth is required.
[    9.678015] ixgbe 0000:03:00.0: A slot with more lanes and/or higher speed is suggested.
[    9.722230] ixgbe 0000:03:00.0: MAC: 5, PHY: 6, PBA No: 020000-000
[    9.722231] ixgbe 0000:03:00.0: 0c:c4:7a:74:09:18
[   15.215889] ixgbe 0000:03:00.0 eth1: IOV is enabled with 8 VFs
[   15.238000] ixgbe 0000:03:00.0: Intel(R) 10 Gigabit Network Connection
[   15.238201] ixgbe 0000:03:00.1: Enabling SR-IOV VFs using the max_vfs module parameter is deprecated - please use the pci sysfs interface instead.
[   16.787177] ixgbevf 0000:03:10.1: enabling device (0000 -> 0002)
[   16.788293] ixgbevf 0000:03:10.1: PF still in reset state.  Is the PF interface up?
[   16.788293] ixgbevf 0000:03:10.1: Assigning random MAC address
[   18.431419] ixgbevf 0000:03:10.1: 86:c7:2f:a5:60:0c
[   18.431420] ixgbevf 0000:03:10.1: MAC: 4
[   18.431420] ixgbevf 0000:03:10.1: Intel(R) 82599 Virtual Function
[   18.431577] ixgbevf 0000:03:10.3: enabling device (0000 -> 0002)
[   18.432709] ixgbevf 0000:03:10.3: PF still in reset state.  Is the PF interface up?
[   18.432709] ixgbevf 0000:03:10.3: Assigning random MAC address
[   19.363463] ixgbevf 0000:03:10.3: 62:5a:23:07:bd:59
[   19.363464] ixgbevf 0000:03:10.3: MAC: 4
[   19.363465] ixgbevf 0000:03:10.3: Intel(R) 82599 Virtual Function
[   19.363609] ixgbevf 0000:03:10.5: enabling device (0000 -> 0002)
[   19.364718] ixgbevf 0000:03:10.5: PF still in reset state.  Is the PF interface up?
[   19.364719] ixgbevf 0000:03:10.5: Assigning random MAC address
[   19.364849] ixgbevf 0000:03:10.5: 92:65:d9:a2:cd:48
[   19.364849] ixgbevf 0000:03:10.5: MAC: 4
[   19.364850] ixgbevf 0000:03:10.5: Intel(R) 82599 Virtual Function
[   19.364978] ixgbevf 0000:03:10.7: enabling device (0000 -> 0002)
[   19.366081] ixgbevf 0000:03:10.7: PF still in reset state.  Is the PF interface up?
[   19.366082] ixgbevf 0000:03:10.7: Assigning random MAC address
[   19.366202] ixgbevf 0000:03:10.7: de:0c:6b:25:63:08
[   19.366203] ixgbevf 0000:03:10.7: MAC: 4
[   19.366203] ixgbevf 0000:03:10.7: Intel(R) 82599 Virtual Function
[   19.366329] ixgbevf 0000:03:11.1: enabling device (0000 -> 0002)
[   19.367439] ixgbevf 0000:03:11.1: PF still in reset state.  Is the PF interface up?
[   19.367440] ixgbevf 0000:03:11.1: Assigning random MAC address
[   19.367568] ixgbevf 0000:03:11.1: 4a:fe:8f:ee:a9:1b
[   19.367569] ixgbevf 0000:03:11.1: MAC: 4
[   19.367569] ixgbevf 0000:03:11.1: Intel(R) 82599 Virtual Function
[   19.367697] ixgbevf 0000:03:11.3: enabling device (0000 -> 0002)
[   19.368804] ixgbevf 0000:03:11.3: PF still in reset state.  Is the PF interface up?
[   19.368804] ixgbevf 0000:03:11.3: Assigning random MAC address
[   19.368925] ixgbevf 0000:03:11.3: ce:85:10:5b:70:dc
[   19.368925] ixgbevf 0000:03:11.3: MAC: 4
[   19.368926] ixgbevf 0000:03:11.3: Intel(R) 82599 Virtual Function
[   19.369050] ixgbevf 0000:03:11.5: enabling device (0000 -> 0002)
[   19.370171] ixgbevf 0000:03:11.5: PF still in reset state.  Is the PF interface up?
[   19.370172] ixgbevf 0000:03:11.5: Assigning random MAC address
[   19.370308] ixgbevf 0000:03:11.5: a6:60:df:d0:d1:34
[   19.370309] ixgbevf 0000:03:11.5: MAC: 4
[   19.370310] ixgbevf 0000:03:11.5: Intel(R) 82599 Virtual Function
[   19.370433] ixgbevf 0000:03:11.7: enabling device (0000 -> 0002)
[   19.371535] ixgbevf 0000:03:11.7: PF still in reset state.  Is the PF interface up?
[   19.371536] ixgbevf 0000:03:11.7: Assigning random MAC address
[   19.371658] ixgbevf 0000:03:11.7: d2:bc:8a:08:3a:8a
[   19.371659] ixgbevf 0000:03:11.7: MAC: 4
[   19.371660] ixgbevf 0000:03:11.7: Intel(R) 82599 Virtual Function
[   19.371673] ixgbe 0000:03:00.1 0000:03:00.1 (uninitialized): SR-IOV enabled with 8 VFs
[   19.748219] ixgbe 0000:03:00.1: Multiqueue Enabled: Rx Queue count = 4, Tx Queue count = 4
[   19.814547] ixgbe 0000:03:00.1: PCI Express bandwidth of 2GT/s available
[   19.814549] ixgbe 0000:03:00.1: (Speed:2.5GT/s, Width: x1, Encoding Loss:20%)
[   19.814549] ixgbe 0000:03:00.1: This is not sufficient for optimal performance of this card.
[   19.814550] ixgbe 0000:03:00.1: For optimal performance, at least 20GT/s of bandwidth is required.
[   19.814551] ixgbe 0000:03:00.1: A slot with more lanes and/or higher speed is suggested.
[   19.858764] ixgbe 0000:03:00.1: MAC: 5, PHY: 6, PBA No: 020000-000
[   19.858765] ixgbe 0000:03:00.1: 0c:c4:7a:74:09:19
 
EluRex

EluRex

Active Member
Apr 28, 2015
218
80
28
Los Angeles, CA
Code: 
[   21.708071] ixgbe 0000:03:00.1 eth6: IOV is enabled with 8 VFs
[   21.727074] ixgbe 0000:03:00.1: Intel(R) 10 Gigabit Network Connection
[   21.727153] ixgbevf 0000:03:10.0: enabling device (0000 -> 0002)
[   21.728283] ixgbevf 0000:03:10.0: PF still in reset state.  Is the PF interface up?
[   21.728284] ixgbevf 0000:03:10.0: Assigning random MAC address
[   22.123348] ixgbevf 0000:03:10.0: 8e:0c:94:c5:aa:f0
[   22.123349] ixgbevf 0000:03:10.0: MAC: 4
[   22.123349] ixgbevf 0000:03:10.0: Intel(R) 82599 Virtual Function
[   22.123360] ixgbevf 0000:03:10.2: enabling device (0000 -> 0002)
[   22.124467] ixgbevf 0000:03:10.2: PF still in reset state.  Is the PF interface up?
[   22.124468] ixgbevf 0000:03:10.2: Assigning random MAC address
[   22.124597] ixgbevf 0000:03:10.2: 9e:a9:c2:aa:1e:18
[   22.124597] ixgbevf 0000:03:10.2: MAC: 4
[   22.124598] ixgbevf 0000:03:10.2: Intel(R) 82599 Virtual Function
[   22.124605] ixgbevf 0000:03:10.4: enabling device (0000 -> 0002)
[   22.125706] ixgbevf 0000:03:10.4: PF still in reset state.  Is the PF interface up?
[   22.125706] ixgbevf 0000:03:10.4: Assigning random MAC address
[   22.125832] ixgbevf 0000:03:10.4: 6e:c3:69:e7:ff:4e
[   22.125832] ixgbevf 0000:03:10.4: MAC: 4
[   22.125833] ixgbevf 0000:03:10.4: Intel(R) 82599 Virtual Function
[   22.125840] ixgbevf 0000:03:10.6: enabling device (0000 -> 0002)
[   22.126941] ixgbevf 0000:03:10.6: PF still in reset state.  Is the PF interface up?
[   22.126942] ixgbevf 0000:03:10.6: Assigning random MAC address
[   22.943349] ixgbevf 0000:03:10.6: 86:df:7b:b6:81:e4
[   22.943350] ixgbevf 0000:03:10.6: MAC: 4
[   22.943351] ixgbevf 0000:03:10.6: Intel(R) 82599 Virtual Function
[   22.943362] ixgbevf 0000:03:11.0: enabling device (0000 -> 0002)
[   22.944516] ixgbevf 0000:03:11.0: PF still in reset state.  Is the PF interface up?
[   22.944517] ixgbevf 0000:03:11.0: Assigning random MAC address
[   22.944643] ixgbevf 0000:03:11.0: a6:41:27:17:7b:f5
[   22.944643] ixgbevf 0000:03:11.0: MAC: 4
[   22.944644] ixgbevf 0000:03:11.0: Intel(R) 82599 Virtual Function
[   22.944649] ixgbevf 0000:03:11.2: enabling device (0000 -> 0002)
[   22.945749] ixgbevf 0000:03:11.2: PF still in reset state.  Is the PF interface up?
[   22.945750] ixgbevf 0000:03:11.2: Assigning random MAC address
[   22.945863] ixgbevf 0000:03:11.2: d6:c7:c2:9f:a3:97
[   22.945864] ixgbevf 0000:03:11.2: MAC: 4
[   22.945865] ixgbevf 0000:03:11.2: Intel(R) 82599 Virtual Function
[   22.945871] ixgbevf 0000:03:11.4: enabling device (0000 -> 0002)
[   22.946978] ixgbevf 0000:03:11.4: PF still in reset state.  Is the PF interface up?
[   22.946979] ixgbevf 0000:03:11.4: Assigning random MAC address
[   22.947093] ixgbevf 0000:03:11.4: 42:3f:49:4f:8f:98
[   22.947094] ixgbevf 0000:03:11.4: MAC: 4
[   22.947094] ixgbevf 0000:03:11.4: Intel(R) 82599 Virtual Function
[   22.947100] ixgbevf 0000:03:11.6: enabling device (0000 -> 0002)
[   22.948245] ixgbevf 0000:03:11.6: PF still in reset state.  Is the PF interface up?
[   22.948246] ixgbevf 0000:03:11.6: Assigning random MAC address
[   22.948354] ixgbevf 0000:03:11.6: 72:02:ed:13:7f:05
[   22.948355] ixgbevf 0000:03:11.6: MAC: 4
[   22.948355] ixgbevf 0000:03:11.6: Intel(R) 82599 Virtual Function
[   23.162283] ixgbevf 0000:03:11.7 em2_7: renamed from eth10
[   23.979141] ixgbevf 0000:03:11.6 em1_7: renamed from eth17
[   34.531118] ixgbe 0000:03:00.1 rename27: renamed from eth6
[   35.502452] ixgbevf: Unable to start - perhaps the PF Driver isn't up yet
loaded by VM
passthrough_succcess.jpg


However, if I use Proxmox VE 4.0 Kernel 4.2.2 failed, complained SR-IOV is not supported

Code: 
root@pve:~# uname -r
4.2.2-1-pve

root@pve:~# dmesg|grep ixgbe
[    4.058315] ixgbe 0000:03:00.0 eth6: MAC: 5, PHY: 6, PBA No: 020000-000
[    4.058324] ixgbe 0000:03:00.0: 0c:c4:7a:74:09:18
[    4.058331] ixgbe 0000:03:00.0 eth6: Enabled Features: RxQ: 16 TxQ: 16 FdirHash RSC vxlan_rx
[    4.080548] ixgbe 0000:03:00.0 eth6: Intel(R) 10 Gigabit Network Connection
[    6.006001] ixgbe 0000:03:00.1 eth7: MAC: 5, PHY: 6, PBA No: 020000-000
[    6.006010] ixgbe 0000:03:00.1: 0c:c4:7a:74:09:19
[    6.006016] ixgbe 0000:03:00.1 eth7: Enabled Features: RxQ: 16 TxQ: 16 FdirHash RSC vxlan_rx
[    6.028236] ixgbe 0000:03:00.1 eth7: Intel(R) 10 Gigabit Network Connection
[    9.621857] ixgbe 0000:03:00.1 rename9: renamed from eth7
[    9.674634] ixgbe 0000:03:00.0 rename8: renamed from eth6
[    9.810747] ixgbe 0000:03:00.1 eth3: renamed from rename9
[    9.866589] ixgbe 0000:03:00.0 eth2: renamed from rename8
[   11.495314] ixgbe 0000:03:00.0: registered PHC device on eth2
[   12.004502] ixgbe 0000:03:00.1: registered PHC device on eth3

root@pve:~# echo 4 > /sys/class/net/eth2/device/sriov_numvfs
-bash: echo: write error: Operation not supported

root@pve:~# ls /sys/class/net/eth2/device/driver/module/drivers
pci:ixgbe

root@pve:~# ls /sys/class/net/eth2/device/driver
0000:03:00.0  0000:03:00.1  bind  module  new_id  remove_id  uevent  unbind

root@pve:~# lspci -s 03:00.0 -vnn
03:00.0 Ethernet controller [0200]: Intel Corporation Ethernet Connection X552/X557-AT 10GBASE-T [8086:15ad]
       Subsystem: Super Micro Computer Inc Device [15d9:15ad]
       Physical Slot: 0-1
       Flags: bus master, fast devsel, latency 0, IRQ 25
       Memory at f9a00000 (64-bit, prefetchable) [size=2M]
       Memory at f9c04000 (64-bit, prefetchable) [size=16K]
       Expansion ROM at fb180000 [disabled] [size=512K]
       Capabilities: [40] Power Management version 3
       Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
       Capabilities: [70] MSI-X: Enable+ Count=64 Masked-
       Capabilities: [a0] Express Endpoint, MSI 00
       Capabilities: [e0] Vital Product Data
       Capabilities: [100] Advanced Error Reporting
       Capabilities: [140] Device Serial Number 00-00-c9-ff-ff-00-00-00
       Capabilities: [150] Alternative Routing-ID Interpretation (ARI)
       Capabilities: [160] Single Root I/O Virtualization (SR-IOV)
       Capabilities: [1b0] Access Control Services
       Capabilities: [1c0] Latency Tolerance Reporting
       Kernel driver in use: ixgbe
 
Last edited:
EluRex

EluRex

Active Member
Apr 28, 2015
218
80
28
Los Angeles, CA
with PVE 4.0 and kernel 4.2.3 failed, complained SR_IOV is not supported

Code: 
root@pve:~# uname -r
4.2.3-1-pve

root@pve:~# dmesg|grep ixgbe
[    4.070688] ixgbe 0000:03:00.0 eth6: MAC: 5, PHY: 6, PBA No: 020000-000
[    4.070692] ixgbe 0000:03:00.0: 0c:c4:7a:74:09:18
[    4.070695] ixgbe 0000:03:00.0 eth6: Enabled Features: RxQ: 16 TxQ: 16 FdirHash RSC vxlan_rx
[    4.092878] ixgbe 0000:03:00.0 eth6: Intel(R) 10 Gigabit Network Connection
[    6.022409] ixgbe 0000:03:00.1 eth7: MAC: 5, PHY: 6, PBA No: 020000-000
[    6.022412] ixgbe 0000:03:00.1: 0c:c4:7a:74:09:19
[    6.022414] ixgbe 0000:03:00.1 eth7: Enabled Features: RxQ: 16 TxQ: 16 FdirHash RSC vxlan_rx
[    6.044585] ixgbe 0000:03:00.1 eth7: Intel(R) 10 Gigabit Network Connection
[   10.199799] ixgbe 0000:03:00.1 rename9: renamed from eth7
[   10.334923] ixgbe 0000:03:00.0 rename8: renamed from eth6
[   10.443240] ixgbe 0000:03:00.1 eth3: renamed from rename9
[   10.475002] ixgbe 0000:03:00.0 eth2: renamed from rename8
[   12.202005] ixgbe 0000:03:00.0: registered PHC device on eth2
[   12.701014] ixgbe 0000:03:00.1: registered PHC device on eth3

root@pve:~# echo 4 > /sys/class/net/eth2/device/sriov_numvfs
-bash: echo: write error: Operation not supported
root@pve:~# dmesg|grep SRIOV
[ 1875.989285] ixgbe 0000:03:00.0: SRIOV not supported on this device

root@pve:~# ls /sys/class/net/eth2/device/driver/module/drivers
pci:ixgbe

root@pve:~# ls /sys/class/net/eth2/device/driver
0000:03:00.0  0000:03:00.1  bind  module  new_id  remove_id  uevent  unbind

root@pve:~# lspci -s 03:00.0 -vnn
03:00.0 Ethernet controller [0200]: Intel Corporation Ethernet Connection X552/X557-AT 10GBASE-T [8086:15ad]
  Subsystem: Super Micro Computer Inc Device [15d9:15ad]
  Physical Slot: 0-1
  Flags: bus master, fast devsel, latency 0, IRQ 25
  Memory at f9a00000 (64-bit, prefetchable) [size=2M]
  Memory at f9c04000 (64-bit, prefetchable) [size=16K]
  Expansion ROM at fb180000 [disabled] [size=512K]
  Capabilities: [40] Power Management version 3
  Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
  Capabilities: [70] MSI-X: Enable+ Count=64 Masked-
  Capabilities: [a0] Express Endpoint, MSI 00
  Capabilities: [e0] Vital Product Data
  Capabilities: [100] Advanced Error Reporting
  Capabilities: [140] Device Serial Number 00-00-c9-ff-ff-00-00-00
  Capabilities: [150] Alternative Routing-ID Interpretation (ARI)
  Capabilities: [160] Single Root I/O Virtualization (SR-IOV)
  Capabilities: [1b0] Access Control Services
  Capabilities: [1c0] Latency Tolerance Reporting
  Kernel driver in use: ixgbe
Code: 
root@pve:~# modinfo ixgbe
filename:  /lib/modules/4.2.3-1-pve/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko
version:  4.1.5
license:  GPL
description:  Intel(R) 10 Gigabit PCI Express Network Driver
author:  Intel Corporation, <linux.nics@intel.com>
...
I think the issue is ixgbe version. I am able to successful to enable SR_IOV is 4.0.1-k using ubuntu 4.2.3, and PVE4 is 4.1.5. Very likely it is Intel purposely disable the driver from access SR_IOV function when using Xeon 1500D processors
 
Last edited:
R

RTM

Well-Known Member
Jan 26, 2014
956
359
63
EluRex said:
Ubuntu 14.04.3 with Kernel 4.2.3 with success X557 VF and pci-passthrough but not yet test for a successful ping test.
Click to expand...
How do you define a successful ping test?

If you are trying to ping some other VM with a VF on the same PF, it should not work unless you have a switch that supports 802.1Qbg. Quote from the ixgbe readme:
Traffic is not passed between Virtual NICs
------------------------------------------
SR-IOV Virtual Functions are unable to send or receive traffic between Virtual
NICs on the same physical port or between emulated connections on a Linux
Software bridge and connections that use SR-IOV VFs. Use an external switch
that is 802.1Qbg (VEPA) capable to workaround this issue.
Click to expand...
 
EluRex

EluRex

Active Member
Apr 28, 2015
218
80
28
Los Angeles, CA
I believe Supermicro is either helping Intel to kill sr-iov on xeon-d platform or they have false advertisement about their product!

Below screenshot is original bios 1.0a bios, there is still sr-iov option
2015-11-05_17-05-09.jpg

but after I updated to the latest bios 1.0b ... its gone!

mmexport1446772139825.jpg

This is outrageous! and even the i350 sr-iov is being disabled as a result!
 
D

Davewolfs

Active Member
Aug 6, 2015
343
32
28
Can you post the custom build you got from SM?

I will say that bios 1.0a is NOT STABLE. I had a complete PSOD on it due to something with C3 states. SM told me that their is a microcode update in 1.0b for this (not sure if they are full of it or just wanted me to leave them alone).
 
D

Davewolfs

Active Member
Aug 6, 2015
343
32
28
Though I would join the fun. I believe I have ping on ESXi 6.0 U1 using latest intel drivers. Note, it is setup as 1GBE I believe because my router which it is plugged into is 1GBE, not sure if I can somehow put this on a virtual switch in ESXi to make it run at 10GBE. Will try if someone would like.

Configured ESXi for SR-IOV.


esxi_1.png



2015-11-06_1925.png

Linux Setup

4.png

3.png

And pinging machine from Windows.

ping.png

Edit: This appears to be a false positive. See notes below.
 
Last edited:
  • Like
Reactions: gigatexal and whitey
D

Davewolfs

Active Member
Aug 6, 2015
343
32
28
I suppose I can set the adapter to full speed in its config and unplug it. Perhaps that will force to 10G. Will try later.
 
D

Davewolfs

Active Member
Aug 6, 2015
343
32
28
Got 10gbe speed. Did not see that before, plugged one port into the other :) Now to create two VM's and have them transfer to each other and see what speeds we hit.

Code: 
02:01.0 SATA controller: VMware SATA AHCI controller
03:00.0 Ethernet controller: Intel Corporation Ethernet Connection X552 Virtual Function
0b:00.0 Ethernet controller: VMware VMXNET3 Ethernet Controller (rev 01)
ubuntu@ubuntu:~$ ethtool eno268436224
Settings for eno268436224:
        Supported ports: [ ]
        Supported link modes:   10000baseT/Full
        Supported pause frame use: No
        Supports auto-negotiation: No
        Advertised link modes:  Not reported
        Advertised pause frame use: No
        Advertised auto-negotiation: No
        Speed: 10000Mb/s
        Duplex: Full
        Port: Other
        PHYAD: 0
        Transceiver: Unknown!
        Auto-negotiation: off
Cannot get wake-on-lan settings: Operation not permitted
        Current message level: 0x00000007 (7)
                               drv probe link
        Link detected: no
ubuntu@ubuntu:~$
 
D

Davewolfs

Active Member
Aug 6, 2015
343
32
28
Noticed something odd. First, the bytes from ifconfig were zero. So took down non VF adapter and couldn't do anything. Looked at vmkernel.log and I see the following. So this could very well be a false positive. Not sure if this is config or something else.

Is there a special way that I am supposed to be bringing the VF NIC up?

Code: 
2015-11-07T15:43:31.052Z cpu2:33924)<4>ixgbe 0000:03:00.0: vmnic2: MACVLANs not supported
2015-11-07T15:43:31.054Z cpu2:32787)<6>ixgbe 0000:03:00.0: vmnic2: Guest OS requesting MTU change to 1518
2015-11-07T15:43:31.054Z cpu2:32838)<3>ixgbe 0000:03:00.0: vmnic2: Unhandled OP 18
2015-11-07T15:43:31.054Z cpu2:32838)WARNING: Uplink: 361: Operation 0x12 failed
2015-11-07T15:43:31.055Z cpu2:32838)<3>ixgbe 0000:03:00.0: vmnic2: Unhandled OP 18
2015-11-07T15:43:31.055Z cpu2:32838)WARNING: Uplink: 361: Operation 0x12 failed
2015-11-07T15:43:31.055Z cpu2:32838)<6>ixgbe 0000:03:00.0: vmnic2: Setting ndo MAC addr 00:50:56:b3:aa:c7 for vf 0
2015-11-07T15:43:31.055Z cpu2:32838)<6>ixgbe 0000:03:00.0: setting MAC 0x4301d0f04a54M on VF 0
2015-11-07T15:43:31.055Z cpu2:32838)<6>ixgbe 0000:03:00.0: Reload the VF driver to make this change effective.
2015-11-07T15:43:31.055Z cpu2:32838)<3>ixgbe 0000:03:00.0: vmnic2: Unhandled OP 14
2015-11-07T15:43:31.055Z cpu2:32838)WARNING: Uplink: 361: Operation 0xe failed
2015-11-07T15:43:31.057Z cpu2:33109)<6>ixgbe 0000:03:00.0: vmnic2: Guest OS requesting MAC addr 00:50:56:b3:aa:c7 for VF 0
2015-11-07T15:43:31.057Z cpu14:32838)<3>ixgbe 0000:03:00.0: vmnic2: Unhandled OP 18
2015-11-07T15:43:31.057Z cpu14:32838)WARNING: Uplink: 361: Operation 0x12 failed
2015-11-07T15:43:31.058Z cpu14:32838)<3>ixgbe 0000:03:00.0: vmnic2: Unhandled OP 18
2015-11-07T15:43:31.058Z cpu14:32838)WARNING: Uplink: 361: Operation 0x12 failed
2015-11-07T15:43:31.058Z cpu14:32838)<6>ixgbe 0000:03:00.0: vmnic2: Setting ndo MAC addr 00:50:56:b3:aa:c7 for vf 0
2015-11-07T15:43:31.058Z cpu14:32838)<6>ixgbe 0000:03:00.0: setting MAC 0x4301d0f04a54M on VF 0
2015-11-07T15:43:31.058Z cpu14:32838)<6>ixgbe 0000:03:00.0: Reload the VF driver to make this change effective.
2015-11-07T15:43:31.058Z cpu14:32838)<3>ixgbe 0000:03:00.0: vmnic2: Unhandled OP 14
2015-11-07T15:43:31.058Z cpu14:32838)WARNING: Uplink: 361: Operation 0xe failed
2015-11-07T15:43:31.059Z cpu2:33109)<4>ixgbe 0000:03:00.0: vmnic2: MACVLANs not supported
2015-11-07T15:43:31.060Z cpu2:33109)<4>ixgbe 0000:03:00.0: vmnic2: MACVLANs not supported
2015-11-07T15:43:31.061Z cpu2:33109)<4>ixgbe 0000:03:00.0: vmnic2: MACVLANs not supported
2015-11-07T15:46:55.492Z cpu1:38046)NetPort: 1780: disabled port 0x200000a
 
Last edited:
You must log in or register to reply here.
Top Bottom