SM 5028D-TN4T X10SDV-TLN4F X557 sr-iov Does Not Work, i350 Work

Discussion in 'Processors and Motherboards' started by EluRex, Oct 30, 2015.

  1. EluRex

    EluRex Active Member

    Joined:
    Apr 28, 2015
    Messages:
    209
    Likes Received:
    70
    This thread is fork from https://forums.servethehome.com/ind...500-series-discussion.5036/page-34#post-65094


    Ubuntu 14.04.3 with Kernel 4.2.3 with success X557 VF and pci-passthrough but not yet test for a successful ping test. The other NIC Intel i350 confirmed with SR-IOV/pci passthrough/successful ping (also on Proxmox 4)
    Code:
    root@pdc:~# uname -r
    4.2.3-040203-generic
    
    lspci -vnn | grep Ethernet
    03:00.0 Ethernet controller [0200]: Intel Corporation Device [8086:15ad]
    03:00.1 Ethernet controller [0200]: Intel Corporation Device [8086:15ad]
    03:10.0 Ethernet controller [0200]: Intel Corporation Device [8086:15a8]
    03:10.1 Ethernet controller [0200]: Intel Corporation Device [8086:15a8]
    03:10.2 Ethernet controller [0200]: Intel Corporation Device [8086:15a8]
    03:10.3 Ethernet controller [0200]: Intel Corporation Device [8086:15a8]
    03:10.4 Ethernet controller [0200]: Intel Corporation Device [8086:15a8]
    03:10.5 Ethernet controller [0200]: Intel Corporation Device [8086:15a8]
    03:10.6 Ethernet controller [0200]: Intel Corporation Device [8086:15a8]
    03:10.7 Ethernet controller [0200]: Intel Corporation Device [8086:15a8]
    03:11.0 Ethernet controller [0200]: Intel Corporation Device [8086:15a8]
    03:11.1 Ethernet controller [0200]: Intel Corporation Device [8086:15a8]
    03:11.2 Ethernet controller [0200]: Intel Corporation Device [8086:15a8]
    03:11.3 Ethernet controller [0200]: Intel Corporation Device [8086:15a8]
    03:11.4 Ethernet controller [0200]: Intel Corporation Device [8086:15a8]
    03:11.5 Ethernet controller [0200]: Intel Corporation Device [8086:15a8]
    03:11.6 Ethernet controller [0200]: Intel Corporation Device [8086:15a8]
    03:11.7 Ethernet controller [0200]: Intel Corporation Device [8086:15a8]
    
    root@pdc:~# cat /etc/modprobe.d/ixgbe.conf
    options ixgbe max_vfs=8
    
    The iommu group is seperated and ready to be use for kvm vfio pci passthrough
    
    /sys/kernel/iommu_groups/45/devices/0000:03:10.0
    /sys/kernel/iommu_groups/46/devices/0000:03:10.2
    /sys/kernel/iommu_groups/47/devices/0000:03:10.4
    /sys/kernel/iommu_groups/48/devices/0000:03:10.6
    /sys/kernel/iommu_groups/49/devices/0000:03:11.0
    /sys/kernel/iommu_groups/50/devices/0000:03:11.2
    /sys/kernel/iommu_groups/51/devices/0000:03:11.4
    /sys/kernel/iommu_groups/52/devices/0000:03:11.6
    /sys/kernel/iommu_groups/53/devices/0000:03:10.1
    /sys/kernel/iommu_groups/54/devices/0000:03:10.3
    /sys/kernel/iommu_groups/55/devices/0000:03:10.5
    /sys/kernel/iommu_groups/56/devices/0000:03:10.7
    /sys/kernel/iommu_groups/57/devices/0000:03:11.1
    /sys/kernel/iommu_groups/58/devices/0000:03:11.3
    /sys/kernel/iommu_groups/59/devices/0000:03:11.5
    /sys/kernel/iommu_groups/60/devices/0000:03:11.7 
    x557.jpg
     
    #1
    FQB likes this.
  2. EluRex

    EluRex Active Member

    Joined:
    Apr 28, 2015
    Messages:
    209
    Likes Received:
    70
    Code:
    root@pdc:~# modinfo ixgbe
    filename:       /lib/modules/4.2.3-040203-generic/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko
    version:        4.0.1-k
    license:        GPL
    description:    Intel(R) 10 Gigabit PCI Express Network Driver
    author:         Intel Corporation, <linux.nics@intel.com>
    srcversion:     E08010C0D72963989E9A05E
    alias:          pci:v00008086d000015ADsv*sd*bc*sc*i*
    ....
    alias:          pci:v00008086d000010B6sv*sd*bc*sc*i*
    depends:        mdio,ptp,dca,vxlan
    intree:         Y
    vermagic:       4.2.3-040203-generic SMP mod_unload modversions
    signer:         Build time autogenerated kernel key
    sig_key:        CE:E6:5B:C1:E3:07:7F:6F:F5:58:77:52:AC:20:3F:9D:65:F7:05:DA
    sig_hashalgo:   sha512
    parm:           max_vfs:Maximum number of virtual functions to allocate per physical function - default is zero and maximum value is 63. (Deprecated) (uint)
    parm:           allow_unsupported_sfp:Allow unsupported and untested SFP+ modules on 82599-based adapters (uint)
    parm:           debug:Debug level (0=none,...,16=all) (int)
    Code:
    root@pdc:~# dmesg|grep ixgbe
    [    0.000000] Command line: BOOT_IMAGE=/ROOT/ubuntu@/boot/vmlinuz-4.2.3-040203-generic root=ZFS=rpool/ROOT/ubuntu ro boot=zfs rpool=rpool bootfs=rpool/ROOT/ubuntu intel_iommu=on ixgbe.max_vfs=8
    [    0.000000] Kernel command line: BOOT_IMAGE=/ROOT/ubuntu@/boot/vmlinuz-4.2.3-040203-generic root=ZFS=rpool/ROOT/ubuntu ro boot=zfs rpool=rpool bootfs=rpool/ROOT/ubuntu intel_iommu=on ixgbe.max_vfs=8
    [    7.208439] ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver - version 4.0.1-k
    [    7.208439] ixgbe: Copyright (c) 1999-2014 Intel Corporation.
    [    7.208607] ixgbe 0000:03:00.0: Enabling SR-IOV VFs using the max_vfs module parameter is deprecated - please use the pci sysfs interface instead.
    [    9.240064] ixgbe 0000:03:00.0 0000:03:00.0 (uninitialized): SR-IOV enabled with 8 VFs
    [    9.512691] ixgbevf: Intel(R) 10 Gigabit PCI Express Virtual Function Network Driver - version 2.12.1-k
    [    9.512692] ixgbevf: Copyright (c) 2009 - 2012 Intel Corporation.
    [    9.611689] ixgbe 0000:03:00.0: Multiqueue Enabled: Rx Queue count = 4, Tx Queue count = 4
    [    9.678012] ixgbe 0000:03:00.0: PCI Express bandwidth of 2GT/s available
    [    9.678013] ixgbe 0000:03:00.0: (Speed:2.5GT/s, Width: x1, Encoding Loss:20%)
    [    9.678014] ixgbe 0000:03:00.0: This is not sufficient for optimal performance of this card.
    [    9.678015] ixgbe 0000:03:00.0: For optimal performance, at least 20GT/s of bandwidth is required.
    [    9.678015] ixgbe 0000:03:00.0: A slot with more lanes and/or higher speed is suggested.
    [    9.722230] ixgbe 0000:03:00.0: MAC: 5, PHY: 6, PBA No: 020000-000
    [    9.722231] ixgbe 0000:03:00.0: 0c:c4:7a:74:09:18
    [   15.215889] ixgbe 0000:03:00.0 eth1: IOV is enabled with 8 VFs
    [   15.238000] ixgbe 0000:03:00.0: Intel(R) 10 Gigabit Network Connection
    [   15.238201] ixgbe 0000:03:00.1: Enabling SR-IOV VFs using the max_vfs module parameter is deprecated - please use the pci sysfs interface instead.
    [   16.787177] ixgbevf 0000:03:10.1: enabling device (0000 -> 0002)
    [   16.788293] ixgbevf 0000:03:10.1: PF still in reset state.  Is the PF interface up?
    [   16.788293] ixgbevf 0000:03:10.1: Assigning random MAC address
    [   18.431419] ixgbevf 0000:03:10.1: 86:c7:2f:a5:60:0c
    [   18.431420] ixgbevf 0000:03:10.1: MAC: 4
    [   18.431420] ixgbevf 0000:03:10.1: Intel(R) 82599 Virtual Function
    [   18.431577] ixgbevf 0000:03:10.3: enabling device (0000 -> 0002)
    [   18.432709] ixgbevf 0000:03:10.3: PF still in reset state.  Is the PF interface up?
    [   18.432709] ixgbevf 0000:03:10.3: Assigning random MAC address
    [   19.363463] ixgbevf 0000:03:10.3: 62:5a:23:07:bd:59
    [   19.363464] ixgbevf 0000:03:10.3: MAC: 4
    [   19.363465] ixgbevf 0000:03:10.3: Intel(R) 82599 Virtual Function
    [   19.363609] ixgbevf 0000:03:10.5: enabling device (0000 -> 0002)
    [   19.364718] ixgbevf 0000:03:10.5: PF still in reset state.  Is the PF interface up?
    [   19.364719] ixgbevf 0000:03:10.5: Assigning random MAC address
    [   19.364849] ixgbevf 0000:03:10.5: 92:65:d9:a2:cd:48
    [   19.364849] ixgbevf 0000:03:10.5: MAC: 4
    [   19.364850] ixgbevf 0000:03:10.5: Intel(R) 82599 Virtual Function
    [   19.364978] ixgbevf 0000:03:10.7: enabling device (0000 -> 0002)
    [   19.366081] ixgbevf 0000:03:10.7: PF still in reset state.  Is the PF interface up?
    [   19.366082] ixgbevf 0000:03:10.7: Assigning random MAC address
    [   19.366202] ixgbevf 0000:03:10.7: de:0c:6b:25:63:08
    [   19.366203] ixgbevf 0000:03:10.7: MAC: 4
    [   19.366203] ixgbevf 0000:03:10.7: Intel(R) 82599 Virtual Function
    [   19.366329] ixgbevf 0000:03:11.1: enabling device (0000 -> 0002)
    [   19.367439] ixgbevf 0000:03:11.1: PF still in reset state.  Is the PF interface up?
    [   19.367440] ixgbevf 0000:03:11.1: Assigning random MAC address
    [   19.367568] ixgbevf 0000:03:11.1: 4a:fe:8f:ee:a9:1b
    [   19.367569] ixgbevf 0000:03:11.1: MAC: 4
    [   19.367569] ixgbevf 0000:03:11.1: Intel(R) 82599 Virtual Function
    [   19.367697] ixgbevf 0000:03:11.3: enabling device (0000 -> 0002)
    [   19.368804] ixgbevf 0000:03:11.3: PF still in reset state.  Is the PF interface up?
    [   19.368804] ixgbevf 0000:03:11.3: Assigning random MAC address
    [   19.368925] ixgbevf 0000:03:11.3: ce:85:10:5b:70:dc
    [   19.368925] ixgbevf 0000:03:11.3: MAC: 4
    [   19.368926] ixgbevf 0000:03:11.3: Intel(R) 82599 Virtual Function
    [   19.369050] ixgbevf 0000:03:11.5: enabling device (0000 -> 0002)
    [   19.370171] ixgbevf 0000:03:11.5: PF still in reset state.  Is the PF interface up?
    [   19.370172] ixgbevf 0000:03:11.5: Assigning random MAC address
    [   19.370308] ixgbevf 0000:03:11.5: a6:60:df:d0:d1:34
    [   19.370309] ixgbevf 0000:03:11.5: MAC: 4
    [   19.370310] ixgbevf 0000:03:11.5: Intel(R) 82599 Virtual Function
    [   19.370433] ixgbevf 0000:03:11.7: enabling device (0000 -> 0002)
    [   19.371535] ixgbevf 0000:03:11.7: PF still in reset state.  Is the PF interface up?
    [   19.371536] ixgbevf 0000:03:11.7: Assigning random MAC address
    [   19.371658] ixgbevf 0000:03:11.7: d2:bc:8a:08:3a:8a
    [   19.371659] ixgbevf 0000:03:11.7: MAC: 4
    [   19.371660] ixgbevf 0000:03:11.7: Intel(R) 82599 Virtual Function
    [   19.371673] ixgbe 0000:03:00.1 0000:03:00.1 (uninitialized): SR-IOV enabled with 8 VFs
    [   19.748219] ixgbe 0000:03:00.1: Multiqueue Enabled: Rx Queue count = 4, Tx Queue count = 4
    [   19.814547] ixgbe 0000:03:00.1: PCI Express bandwidth of 2GT/s available
    [   19.814549] ixgbe 0000:03:00.1: (Speed:2.5GT/s, Width: x1, Encoding Loss:20%)
    [   19.814549] ixgbe 0000:03:00.1: This is not sufficient for optimal performance of this card.
    [   19.814550] ixgbe 0000:03:00.1: For optimal performance, at least 20GT/s of bandwidth is required.
    [   19.814551] ixgbe 0000:03:00.1: A slot with more lanes and/or higher speed is suggested.
    [   19.858764] ixgbe 0000:03:00.1: MAC: 5, PHY: 6, PBA No: 020000-000
    [   19.858765] ixgbe 0000:03:00.1: 0c:c4:7a:74:09:19
     
    #2
  3. EluRex

    EluRex Active Member

    Joined:
    Apr 28, 2015
    Messages:
    209
    Likes Received:
    70
    Code:
    [   21.708071] ixgbe 0000:03:00.1 eth6: IOV is enabled with 8 VFs
    [   21.727074] ixgbe 0000:03:00.1: Intel(R) 10 Gigabit Network Connection
    [   21.727153] ixgbevf 0000:03:10.0: enabling device (0000 -> 0002)
    [   21.728283] ixgbevf 0000:03:10.0: PF still in reset state.  Is the PF interface up?
    [   21.728284] ixgbevf 0000:03:10.0: Assigning random MAC address
    [   22.123348] ixgbevf 0000:03:10.0: 8e:0c:94:c5:aa:f0
    [   22.123349] ixgbevf 0000:03:10.0: MAC: 4
    [   22.123349] ixgbevf 0000:03:10.0: Intel(R) 82599 Virtual Function
    [   22.123360] ixgbevf 0000:03:10.2: enabling device (0000 -> 0002)
    [   22.124467] ixgbevf 0000:03:10.2: PF still in reset state.  Is the PF interface up?
    [   22.124468] ixgbevf 0000:03:10.2: Assigning random MAC address
    [   22.124597] ixgbevf 0000:03:10.2: 9e:a9:c2:aa:1e:18
    [   22.124597] ixgbevf 0000:03:10.2: MAC: 4
    [   22.124598] ixgbevf 0000:03:10.2: Intel(R) 82599 Virtual Function
    [   22.124605] ixgbevf 0000:03:10.4: enabling device (0000 -> 0002)
    [   22.125706] ixgbevf 0000:03:10.4: PF still in reset state.  Is the PF interface up?
    [   22.125706] ixgbevf 0000:03:10.4: Assigning random MAC address
    [   22.125832] ixgbevf 0000:03:10.4: 6e:c3:69:e7:ff:4e
    [   22.125832] ixgbevf 0000:03:10.4: MAC: 4
    [   22.125833] ixgbevf 0000:03:10.4: Intel(R) 82599 Virtual Function
    [   22.125840] ixgbevf 0000:03:10.6: enabling device (0000 -> 0002)
    [   22.126941] ixgbevf 0000:03:10.6: PF still in reset state.  Is the PF interface up?
    [   22.126942] ixgbevf 0000:03:10.6: Assigning random MAC address
    [   22.943349] ixgbevf 0000:03:10.6: 86:df:7b:b6:81:e4
    [   22.943350] ixgbevf 0000:03:10.6: MAC: 4
    [   22.943351] ixgbevf 0000:03:10.6: Intel(R) 82599 Virtual Function
    [   22.943362] ixgbevf 0000:03:11.0: enabling device (0000 -> 0002)
    [   22.944516] ixgbevf 0000:03:11.0: PF still in reset state.  Is the PF interface up?
    [   22.944517] ixgbevf 0000:03:11.0: Assigning random MAC address
    [   22.944643] ixgbevf 0000:03:11.0: a6:41:27:17:7b:f5
    [   22.944643] ixgbevf 0000:03:11.0: MAC: 4
    [   22.944644] ixgbevf 0000:03:11.0: Intel(R) 82599 Virtual Function
    [   22.944649] ixgbevf 0000:03:11.2: enabling device (0000 -> 0002)
    [   22.945749] ixgbevf 0000:03:11.2: PF still in reset state.  Is the PF interface up?
    [   22.945750] ixgbevf 0000:03:11.2: Assigning random MAC address
    [   22.945863] ixgbevf 0000:03:11.2: d6:c7:c2:9f:a3:97
    [   22.945864] ixgbevf 0000:03:11.2: MAC: 4
    [   22.945865] ixgbevf 0000:03:11.2: Intel(R) 82599 Virtual Function
    [   22.945871] ixgbevf 0000:03:11.4: enabling device (0000 -> 0002)
    [   22.946978] ixgbevf 0000:03:11.4: PF still in reset state.  Is the PF interface up?
    [   22.946979] ixgbevf 0000:03:11.4: Assigning random MAC address
    [   22.947093] ixgbevf 0000:03:11.4: 42:3f:49:4f:8f:98
    [   22.947094] ixgbevf 0000:03:11.4: MAC: 4
    [   22.947094] ixgbevf 0000:03:11.4: Intel(R) 82599 Virtual Function
    [   22.947100] ixgbevf 0000:03:11.6: enabling device (0000 -> 0002)
    [   22.948245] ixgbevf 0000:03:11.6: PF still in reset state.  Is the PF interface up?
    [   22.948246] ixgbevf 0000:03:11.6: Assigning random MAC address
    [   22.948354] ixgbevf 0000:03:11.6: 72:02:ed:13:7f:05
    [   22.948355] ixgbevf 0000:03:11.6: MAC: 4
    [   22.948355] ixgbevf 0000:03:11.6: Intel(R) 82599 Virtual Function
    [   23.162283] ixgbevf 0000:03:11.7 em2_7: renamed from eth10
    [   23.979141] ixgbevf 0000:03:11.6 em1_7: renamed from eth17
    [   34.531118] ixgbe 0000:03:00.1 rename27: renamed from eth6
    [   35.502452] ixgbevf: Unable to start - perhaps the PF Driver isn't up yet
    loaded by VM
    passthrough_succcess.jpg


    However, if I use Proxmox VE 4.0 Kernel 4.2.2 failed, complained SR-IOV is not supported

    Code:
    root@pve:~# uname -r
    4.2.2-1-pve
    
    root@pve:~# dmesg|grep ixgbe
    [    4.058315] ixgbe 0000:03:00.0 eth6: MAC: 5, PHY: 6, PBA No: 020000-000
    [    4.058324] ixgbe 0000:03:00.0: 0c:c4:7a:74:09:18
    [    4.058331] ixgbe 0000:03:00.0 eth6: Enabled Features: RxQ: 16 TxQ: 16 FdirHash RSC vxlan_rx
    [    4.080548] ixgbe 0000:03:00.0 eth6: Intel(R) 10 Gigabit Network Connection
    [    6.006001] ixgbe 0000:03:00.1 eth7: MAC: 5, PHY: 6, PBA No: 020000-000
    [    6.006010] ixgbe 0000:03:00.1: 0c:c4:7a:74:09:19
    [    6.006016] ixgbe 0000:03:00.1 eth7: Enabled Features: RxQ: 16 TxQ: 16 FdirHash RSC vxlan_rx
    [    6.028236] ixgbe 0000:03:00.1 eth7: Intel(R) 10 Gigabit Network Connection
    [    9.621857] ixgbe 0000:03:00.1 rename9: renamed from eth7
    [    9.674634] ixgbe 0000:03:00.0 rename8: renamed from eth6
    [    9.810747] ixgbe 0000:03:00.1 eth3: renamed from rename9
    [    9.866589] ixgbe 0000:03:00.0 eth2: renamed from rename8
    [   11.495314] ixgbe 0000:03:00.0: registered PHC device on eth2
    [   12.004502] ixgbe 0000:03:00.1: registered PHC device on eth3
    
    root@pve:~# echo 4 > /sys/class/net/eth2/device/sriov_numvfs
    -bash: echo: write error: Operation not supported
    
    root@pve:~# ls /sys/class/net/eth2/device/driver/module/drivers
    pci:ixgbe
    
    root@pve:~# ls /sys/class/net/eth2/device/driver
    0000:03:00.0  0000:03:00.1  bind  module  new_id  remove_id  uevent  unbind
    
    root@pve:~# lspci -s 03:00.0 -vnn
    03:00.0 Ethernet controller [0200]: Intel Corporation Ethernet Connection X552/X557-AT 10GBASE-T [8086:15ad]
           Subsystem: Super Micro Computer Inc Device [15d9:15ad]
           Physical Slot: 0-1
           Flags: bus master, fast devsel, latency 0, IRQ 25
           Memory at f9a00000 (64-bit, prefetchable) [size=2M]
           Memory at f9c04000 (64-bit, prefetchable) [size=16K]
           Expansion ROM at fb180000 [disabled] [size=512K]
           Capabilities: [40] Power Management version 3
           Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
           Capabilities: [70] MSI-X: Enable+ Count=64 Masked-
           Capabilities: [a0] Express Endpoint, MSI 00
           Capabilities: [e0] Vital Product Data
           Capabilities: [100] Advanced Error Reporting
           Capabilities: [140] Device Serial Number 00-00-c9-ff-ff-00-00-00
           Capabilities: [150] Alternative Routing-ID Interpretation (ARI)
           Capabilities: [160] Single Root I/O Virtualization (SR-IOV)
           Capabilities: [1b0] Access Control Services
           Capabilities: [1c0] Latency Tolerance Reporting
           Kernel driver in use: ixgbe
     
    #3
    Last edited: Oct 30, 2015
  4. EluRex

    EluRex Active Member

    Joined:
    Apr 28, 2015
    Messages:
    209
    Likes Received:
    70
    with PVE 4.0 and kernel 4.2.3 failed, complained SR_IOV is not supported

    Code:
    root@pve:~# uname -r
    4.2.3-1-pve
    
    root@pve:~# dmesg|grep ixgbe
    [    4.070688] ixgbe 0000:03:00.0 eth6: MAC: 5, PHY: 6, PBA No: 020000-000
    [    4.070692] ixgbe 0000:03:00.0: 0c:c4:7a:74:09:18
    [    4.070695] ixgbe 0000:03:00.0 eth6: Enabled Features: RxQ: 16 TxQ: 16 FdirHash RSC vxlan_rx
    [    4.092878] ixgbe 0000:03:00.0 eth6: Intel(R) 10 Gigabit Network Connection
    [    6.022409] ixgbe 0000:03:00.1 eth7: MAC: 5, PHY: 6, PBA No: 020000-000
    [    6.022412] ixgbe 0000:03:00.1: 0c:c4:7a:74:09:19
    [    6.022414] ixgbe 0000:03:00.1 eth7: Enabled Features: RxQ: 16 TxQ: 16 FdirHash RSC vxlan_rx
    [    6.044585] ixgbe 0000:03:00.1 eth7: Intel(R) 10 Gigabit Network Connection
    [   10.199799] ixgbe 0000:03:00.1 rename9: renamed from eth7
    [   10.334923] ixgbe 0000:03:00.0 rename8: renamed from eth6
    [   10.443240] ixgbe 0000:03:00.1 eth3: renamed from rename9
    [   10.475002] ixgbe 0000:03:00.0 eth2: renamed from rename8
    [   12.202005] ixgbe 0000:03:00.0: registered PHC device on eth2
    [   12.701014] ixgbe 0000:03:00.1: registered PHC device on eth3
    
    root@pve:~# echo 4 > /sys/class/net/eth2/device/sriov_numvfs
    -bash: echo: write error: Operation not supported
    root@pve:~# dmesg|grep SRIOV
    [ 1875.989285] ixgbe 0000:03:00.0: SRIOV not supported on this device
    
    root@pve:~# ls /sys/class/net/eth2/device/driver/module/drivers
    pci:ixgbe
    
    root@pve:~# ls /sys/class/net/eth2/device/driver
    0000:03:00.0  0000:03:00.1  bind  module  new_id  remove_id  uevent  unbind
    
    root@pve:~# lspci -s 03:00.0 -vnn
    03:00.0 Ethernet controller [0200]: Intel Corporation Ethernet Connection X552/X557-AT 10GBASE-T [8086:15ad]
      Subsystem: Super Micro Computer Inc Device [15d9:15ad]
      Physical Slot: 0-1
      Flags: bus master, fast devsel, latency 0, IRQ 25
      Memory at f9a00000 (64-bit, prefetchable) [size=2M]
      Memory at f9c04000 (64-bit, prefetchable) [size=16K]
      Expansion ROM at fb180000 [disabled] [size=512K]
      Capabilities: [40] Power Management version 3
      Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
      Capabilities: [70] MSI-X: Enable+ Count=64 Masked-
      Capabilities: [a0] Express Endpoint, MSI 00
      Capabilities: [e0] Vital Product Data
      Capabilities: [100] Advanced Error Reporting
      Capabilities: [140] Device Serial Number 00-00-c9-ff-ff-00-00-00
      Capabilities: [150] Alternative Routing-ID Interpretation (ARI)
      Capabilities: [160] Single Root I/O Virtualization (SR-IOV)
      Capabilities: [1b0] Access Control Services
      Capabilities: [1c0] Latency Tolerance Reporting
      Kernel driver in use: ixgbe
    Code:
    root@pve:~# modinfo ixgbe
    filename:  /lib/modules/4.2.3-1-pve/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko
    version:  4.1.5
    license:  GPL
    description:  Intel(R) 10 Gigabit PCI Express Network Driver
    author:  Intel Corporation, <linux.nics@intel.com>
    ...
    I think the issue is ixgbe version. I am able to successful to enable SR_IOV is 4.0.1-k using ubuntu 4.2.3, and PVE4 is 4.1.5. Very likely it is Intel purposely disable the driver from access SR_IOV function when using Xeon 1500D processors
     
    #4
    Last edited: Oct 30, 2015
  5. RTM

    RTM Active Member

    Joined:
    Jan 26, 2014
    Messages:
    427
    Likes Received:
    142
    How do you define a successful ping test?

    If you are trying to ping some other VM with a VF on the same PF, it should not work unless you have a switch that supports 802.1Qbg. Quote from the ixgbe readme:
     
    #5
  6. EluRex

    EluRex Active Member

    Joined:
    Apr 28, 2015
    Messages:
    209
    Likes Received:
    70
    ping from a actual win10 client to the vm back and forth. or even from the vm trying to ping www.google will be sufficient.
     
    #6
  7. gigatexal

    gigatexal I'm here to learn

    Joined:
    Nov 25, 2012
    Messages:
    2,697
    Likes Received:
    500
    bummer so it's almost but not quite there on the 10gbe nics.
     
    #7
  8. ggg

    ggg Member

    Joined:
    Jul 2, 2015
    Messages:
    35
    Likes Received:
    1
    Could you setup two VMs, each with one of the VFs from the X557, and ping between them?
     
    #8
  9. gigatexal

    gigatexal I'm here to learn

    Joined:
    Nov 25, 2012
    Messages:
    2,697
    Likes Received:
    500
    eager to see if this works
     
    #9
  10. EluRex

    EluRex Active Member

    Joined:
    Apr 28, 2015
    Messages:
    209
    Likes Received:
    70
    I believe Supermicro is either helping Intel to kill sr-iov on xeon-d platform or they have false advertisement about their product!

    Below screenshot is original bios 1.0a bios, there is still sr-iov option
    2015-11-05_17-05-09.jpg

    but after I updated to the latest bios 1.0b ... its gone!

    mmexport1446772139825.jpg

    This is outrageous! and even the i350 sr-iov is being disabled as a result!
     
    #10
  11. Davewolfs

    Davewolfs Active Member

    Joined:
    Aug 6, 2015
    Messages:
    329
    Likes Received:
    31
    Can you post the custom build you got from SM?

    I will say that bios 1.0a is NOT STABLE. I had a complete PSOD on it due to something with C3 states. SM told me that their is a microcode update in 1.0b for this (not sure if they are full of it or just wanted me to leave them alone).
     
    #11
  12. EluRex

    EluRex Active Member

    Joined:
    Apr 28, 2015
    Messages:
    209
    Likes Received:
    70
    I only have the standard one from sm website
     
    #12
  13. Davewolfs

    Davewolfs Active Member

    Joined:
    Aug 6, 2015
    Messages:
    329
    Likes Received:
    31
    Though I would join the fun. I believe I have ping on ESXi 6.0 U1 using latest intel drivers. Note, it is setup as 1GBE I believe because my router which it is plugged into is 1GBE, not sure if I can somehow put this on a virtual switch in ESXi to make it run at 10GBE. Will try if someone would like.

    Configured ESXi for SR-IOV.


    esxi_1.png



    2015-11-06_1925.png

    Linux Setup

    4.png

    3.png

    And pinging machine from Windows.

    ping.png

    Edit: This appears to be a false positive. See notes below.
     
    #13
    Last edited: Nov 8, 2015
    gigatexal and whitey like this.
  14. Davewolfs

    Davewolfs Active Member

    Joined:
    Aug 6, 2015
    Messages:
    329
    Likes Received:
    31
    Here is the VM Setup. Running Ubuntu 15.10.

    esxi_2.png
     
    #14
  15. Davewolfs

    Davewolfs Active Member

    Joined:
    Aug 6, 2015
    Messages:
    329
    Likes Received:
    31
    I suppose I can set the adapter to full speed in its config and unplug it. Perhaps that will force to 10G. Will try later.
     
    #15
  16. Davewolfs

    Davewolfs Active Member

    Joined:
    Aug 6, 2015
    Messages:
    329
    Likes Received:
    31
    Got 10gbe speed. Did not see that before, plugged one port into the other :) Now to create two VM's and have them transfer to each other and see what speeds we hit.

    Code:
    02:01.0 SATA controller: VMware SATA AHCI controller
    03:00.0 Ethernet controller: Intel Corporation Ethernet Connection X552 Virtual Function
    0b:00.0 Ethernet controller: VMware VMXNET3 Ethernet Controller (rev 01)
    ubuntu@ubuntu:~$ ethtool eno268436224
    Settings for eno268436224:
            Supported ports: [ ]
            Supported link modes:   10000baseT/Full
            Supported pause frame use: No
            Supports auto-negotiation: No
            Advertised link modes:  Not reported
            Advertised pause frame use: No
            Advertised auto-negotiation: No
            Speed: 10000Mb/s
            Duplex: Full
            Port: Other
            PHYAD: 0
            Transceiver: Unknown!
            Auto-negotiation: off
    Cannot get wake-on-lan settings: Operation not permitted
            Current message level: 0x00000007 (7)
                                   drv probe link
            Link detected: no
    ubuntu@ubuntu:~$
    
     
    #16
  17. EluRex

    EluRex Active Member

    Joined:
    Apr 28, 2015
    Messages:
    209
    Likes Received:
    70
    So it is intel 4.1.5 driver purposefully disable srv-iov on Linux and on VMware sr-iov work flawlessly
     
    #17
  18. gigatexal

    gigatexal I'm here to learn

    Joined:
    Nov 25, 2012
    Messages:
    2,697
    Likes Received:
    500
    so use vmware to get sr-iov?
     
    #18
  19. EluRex

    EluRex Active Member

    Joined:
    Apr 28, 2015
    Messages:
    209
    Likes Received:
    70
    I want to use pve4 and move away from VMWare... and trying to my environment more closer to openstack framework...
     
    #19
  20. Davewolfs

    Davewolfs Active Member

    Joined:
    Aug 6, 2015
    Messages:
    329
    Likes Received:
    31
    Noticed something odd. First, the bytes from ifconfig were zero. So took down non VF adapter and couldn't do anything. Looked at vmkernel.log and I see the following. So this could very well be a false positive. Not sure if this is config or something else.

    Is there a special way that I am supposed to be bringing the VF NIC up?

    Code:
    2015-11-07T15:43:31.052Z cpu2:33924)<4>ixgbe 0000:03:00.0: vmnic2: MACVLANs not supported
    2015-11-07T15:43:31.054Z cpu2:32787)<6>ixgbe 0000:03:00.0: vmnic2: Guest OS requesting MTU change to 1518
    2015-11-07T15:43:31.054Z cpu2:32838)<3>ixgbe 0000:03:00.0: vmnic2: Unhandled OP 18
    2015-11-07T15:43:31.054Z cpu2:32838)WARNING: Uplink: 361: Operation 0x12 failed
    2015-11-07T15:43:31.055Z cpu2:32838)<3>ixgbe 0000:03:00.0: vmnic2: Unhandled OP 18
    2015-11-07T15:43:31.055Z cpu2:32838)WARNING: Uplink: 361: Operation 0x12 failed
    2015-11-07T15:43:31.055Z cpu2:32838)<6>ixgbe 0000:03:00.0: vmnic2: Setting ndo MAC addr 00:50:56:b3:aa:c7 for vf 0
    2015-11-07T15:43:31.055Z cpu2:32838)<6>ixgbe 0000:03:00.0: setting MAC 0x4301d0f04a54M on VF 0
    2015-11-07T15:43:31.055Z cpu2:32838)<6>ixgbe 0000:03:00.0: Reload the VF driver to make this change effective.
    2015-11-07T15:43:31.055Z cpu2:32838)<3>ixgbe 0000:03:00.0: vmnic2: Unhandled OP 14
    2015-11-07T15:43:31.055Z cpu2:32838)WARNING: Uplink: 361: Operation 0xe failed
    2015-11-07T15:43:31.057Z cpu2:33109)<6>ixgbe 0000:03:00.0: vmnic2: Guest OS requesting MAC addr 00:50:56:b3:aa:c7 for VF 0
    2015-11-07T15:43:31.057Z cpu14:32838)<3>ixgbe 0000:03:00.0: vmnic2: Unhandled OP 18
    2015-11-07T15:43:31.057Z cpu14:32838)WARNING: Uplink: 361: Operation 0x12 failed
    2015-11-07T15:43:31.058Z cpu14:32838)<3>ixgbe 0000:03:00.0: vmnic2: Unhandled OP 18
    2015-11-07T15:43:31.058Z cpu14:32838)WARNING: Uplink: 361: Operation 0x12 failed
    2015-11-07T15:43:31.058Z cpu14:32838)<6>ixgbe 0000:03:00.0: vmnic2: Setting ndo MAC addr 00:50:56:b3:aa:c7 for vf 0
    2015-11-07T15:43:31.058Z cpu14:32838)<6>ixgbe 0000:03:00.0: setting MAC 0x4301d0f04a54M on VF 0
    2015-11-07T15:43:31.058Z cpu14:32838)<6>ixgbe 0000:03:00.0: Reload the VF driver to make this change effective.
    2015-11-07T15:43:31.058Z cpu14:32838)<3>ixgbe 0000:03:00.0: vmnic2: Unhandled OP 14
    2015-11-07T15:43:31.058Z cpu14:32838)WARNING: Uplink: 361: Operation 0xe failed
    2015-11-07T15:43:31.059Z cpu2:33109)<4>ixgbe 0000:03:00.0: vmnic2: MACVLANs not supported
    2015-11-07T15:43:31.060Z cpu2:33109)<4>ixgbe 0000:03:00.0: vmnic2: MACVLANs not supported
    2015-11-07T15:43:31.061Z cpu2:33109)<4>ixgbe 0000:03:00.0: vmnic2: MACVLANs not supported
    2015-11-07T15:46:55.492Z cpu1:38046)NetPort: 1780: disabled port 0x200000a
    
    
     
    #20
    Last edited: Nov 7, 2015
Similar Threads: 5028D-TN4T X10SDV-TLN4F
Forum Title Date
Processors and Motherboards No Boot or VGA on x10sdv-tln4f Feb 5, 2018
Processors and Motherboards Supermicro X10SDV-TLN4F and M.2 drive - is this normal ? Nov 6, 2017
Processors and Motherboards Fan issue with Supermicro X10SDV-TLN4F Sep 15, 2016
Processors and Motherboards X10SDV-TLN4F, Crucial Rdimm, ECC errors Apr 6, 2016
Processors and Motherboards Hot Heak Sink on MBD-X10SDV-TLN4F-O when turned off Nov 6, 2015

Share This Page