Intel Xeon D-1500 Series Discussion

[ggg]

root@xeon-kvm-host:/tmp/ixgbe-4.1.5/src# modinfo ixgbe
filename:       /lib/modules/4.0.0-040000-generic/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko
...

@Jerry Chen

Looks like your working directory is the one you unpacked the downloaded ixgbe driver to.

But you haven't loaded the module you've built (assuming you've built it).

The one that is loaded is the one that is included with your running kernel.

 

[Jerry Chen]

root@pdc:~# dmesg|grep ixgbe
[    0.000000] Command line: BOOT_IMAGE=/ROOT/ubuntu@/boot/vmlinuz-4.2.3-040203-generic root=ZFS=rpool/ROOT/ubuntu ro boot=zfs rpool=rpool bootfs=rpool/ROOT/ubuntu intel_iommu=on ixgbe.max_vfs=8
[    0.000000] Kernel command line: BOOT_IMAGE=/ROOT/ubuntu@/boot/vmlinuz-4.2.3-040203-generic root=ZFS=rpool/ROOT/ubuntu ro boot=zfs rpool=rpool bootfs=rpool/ROOT/ubuntu intel_iommu=on ixgbe.max_vfs=8
[    7.208439] ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver - version 4.0.1-k
[    7.208439] ixgbe: Copyright (c) 1999-2014 Intel Corporation.
[    7.208607] ixgbe 0000:03:00.0: Enabling SR-IOV VFs using the max_vfs module parameter is deprecated - please use the pci sysfs interface instead.
[    9.240064] ixgbe 0000:03:00.0 0000:03:00.0 (uninitialized): SR-IOV enabled with 8 VFs
[    9.512691] ixgbevf: Intel(R) 10 Gigabit PCI Express Virtual Function Network Driver - version 2.12.1-k
[    9.512692] ixgbevf: Copyright (c) 2009 - 2012 Intel Corporation.
[    9.611689] ixgbe 0000:03:00.0: Multiqueue Enabled: Rx Queue count = 4, Tx Queue count = 4
[    9.678012] ixgbe 0000:03:00.0: PCI Express bandwidth of 2GT/s available
[    9.678013] ixgbe 0000:03:00.0: (Speed:2.5GT/s, Width: x1, Encoding Loss:20%)
[    9.678014] ixgbe 0000:03:00.0: This is not sufficient for optimal performance of this card.
[    9.678015] ixgbe 0000:03:00.0: For optimal performance, at least 20GT/s of bandwidth is required.
[    9.678015] ixgbe 0000:03:00.0: A slot with more lanes and/or higher speed is suggested.
[    9.722230] ixgbe 0000:03:00.0: MAC: 5, PHY: 6, PBA No: 020000-000
[    9.722231] ixgbe 0000:03:00.0: 0c:c4:7a:74:09:18
[   15.215889] ixgbe 0000:03:00.0 eth1: IOV is enabled with 8 VFs
[   15.238000] ixgbe 0000:03:00.0: Intel(R) 10 Gigabit Network Connection
[   15.238201] ixgbe 0000:03:00.1: Enabling SR-IOV VFs using the max_vfs module parameter is deprecated - please use the pci sysfs interface instead.
[   16.787177] ixgbevf 0000:03:10.1: enabling device (0000 -> 0002)
[   16.788293] ixgbevf 0000:03:10.1: PF still in reset state.  Is the PF interface up?
[   16.788293] ixgbevf 0000:03:10.1: Assigning random MAC address
[   18.431419] ixgbevf 0000:03:10.1: 86:c7:2f:a5:60:0c
[   18.431420] ixgbevf 0000:03:10.1: MAC: 4
[   18.431420] ixgbevf 0000:03:10.1: Intel(R) 82599 Virtual Function
[   18.431577] ixgbevf 0000:03:10.3: enabling device (0000 -> 0002)
[   18.432709] ixgbevf 0000:03:10.3: PF still in reset state.  Is the PF interface up?
[   18.432709] ixgbevf 0000:03:10.3: Assigning random MAC address
[   19.363463] ixgbevf 0000:03:10.3: 62:5a:23:07:bd:59
[   19.363464] ixgbevf 0000:03:10.3: MAC: 4
[   19.363465] ixgbevf 0000:03:10.3: Intel(R) 82599 Virtual Function
[   19.363609] ixgbevf 0000:03:10.5: enabling device (0000 -> 0002)
[   19.364718] ixgbevf 0000:03:10.5: PF still in reset state.  Is the PF interface up?
[   19.364719] ixgbevf 0000:03:10.5: Assigning random MAC address
[   19.364849] ixgbevf 0000:03:10.5: 92:65:d9:a2:cd:48
[   19.364849] ixgbevf 0000:03:10.5: MAC: 4
[   19.364850] ixgbevf 0000:03:10.5: Intel(R) 82599 Virtual Function
[   19.364978] ixgbevf 0000:03:10.7: enabling device (0000 -> 0002)
[   19.366081] ixgbevf 0000:03:10.7: PF still in reset state.  Is the PF interface up?
[   19.366082] ixgbevf 0000:03:10.7: Assigning random MAC address
[   19.366202] ixgbevf 0000:03:10.7: de:0c:6b:25:63:08
[   19.366203] ixgbevf 0000:03:10.7: MAC: 4
[   19.366203] ixgbevf 0000:03:10.7: Intel(R) 82599 Virtual Function
[   19.366329] ixgbevf 0000:03:11.1: enabling device (0000 -> 0002)
[   19.367439] ixgbevf 0000:03:11.1: PF still in reset state.  Is the PF interface up?
[   19.367440] ixgbevf 0000:03:11.1: Assigning random MAC address
[   19.367568] ixgbevf 0000:03:11.1: 4a:fe:8f:ee:a9:1b
[   19.367569] ixgbevf 0000:03:11.1: MAC: 4
[   19.367569] ixgbevf 0000:03:11.1: Intel(R) 82599 Virtual Function
[   19.367697] ixgbevf 0000:03:11.3: enabling device (0000 -> 0002)
[   19.368804] ixgbevf 0000:03:11.3: PF still in reset state.  Is the PF interface up?
[   19.368804] ixgbevf 0000:03:11.3: Assigning random MAC address
[   19.368925] ixgbevf 0000:03:11.3: ce:85:10:5b:70:dc
[   19.368925] ixgbevf 0000:03:11.3: MAC: 4
[   19.368926] ixgbevf 0000:03:11.3: Intel(R) 82599 Virtual Function
[   19.369050] ixgbevf 0000:03:11.5: enabling device (0000 -> 0002)
[   19.370171] ixgbevf 0000:03:11.5: PF still in reset state.  Is the PF interface up?
[   19.370172] ixgbevf 0000:03:11.5: Assigning random MAC address
[   19.370308] ixgbevf 0000:03:11.5: a6:60:df:d0:d1:34
[   19.370309] ixgbevf 0000:03:11.5: MAC: 4
[   19.370310] ixgbevf 0000:03:11.5: Intel(R) 82599 Virtual Function
[   19.370433] ixgbevf 0000:03:11.7: enabling device (0000 -> 0002)
[   19.371535] ixgbevf 0000:03:11.7: PF still in reset state.  Is the PF interface up?
[   19.371536] ixgbevf 0000:03:11.7: Assigning random MAC address
[   19.371658] ixgbevf 0000:03:11.7: d2:bc:8a:08:3a:8a
[   19.371659] ixgbevf 0000:03:11.7: MAC: 4
[   19.371660] ixgbevf 0000:03:11.7: Intel(R) 82599 Virtual Function
[   19.371673] ixgbe 0000:03:00.1 0000:03:00.1 (uninitialized): SR-IOV enabled with 8 VFs
[   19.748219] ixgbe 0000:03:00.1: Multiqueue Enabled: Rx Queue count = 4, Tx Queue count = 4
[   19.814547] ixgbe 0000:03:00.1: PCI Express bandwidth of 2GT/s available
[   19.814549] ixgbe 0000:03:00.1: (Speed:2.5GT/s, Width: x1, Encoding Loss:20%)
[   19.814549] ixgbe 0000:03:00.1: This is not sufficient for optimal performance of this card.
[   19.814550] ixgbe 0000:03:00.1: For optimal performance, at least 20GT/s of bandwidth is required.
[   19.814551] ixgbe 0000:03:00.1: A slot with more lanes and/or higher speed is suggested.
[   19.858764] ixgbe 0000:03:00.1: MAC: 5, PHY: 6, PBA No: 020000-000
[   19.858765] ixgbe 0000:03:00.1: 0c:c4:7a:74:09:19
[   21.708071] ixgbe 0000:03:00.1 eth6: IOV is enabled with 8 VFs
[   21.727074] ixgbe 0000:03:00.1: Intel(R) 10 Gigabit Network Connection
[   21.727153] ixgbevf 0000:03:10.0: enabling device (0000 -> 0002)
[   21.728283] ixgbevf 0000:03:10.0: PF still in reset state.  Is the PF interface up?
[   21.728284] ixgbevf 0000:03:10.0: Assigning random MAC address
[   22.123348] ixgbevf 0000:03:10.0: 8e:0c:94:c5:aa:f0
[   22.123349] ixgbevf 0000:03:10.0: MAC: 4
[   22.123349] ixgbevf 0000:03:10.0: Intel(R) 82599 Virtual Function
[   22.123360] ixgbevf 0000:03:10.2: enabling device (0000 -> 0002)
[   22.124467] ixgbevf 0000:03:10.2: PF still in reset state.  Is the PF interface up?
[   22.124468] ixgbevf 0000:03:10.2: Assigning random MAC address
[   22.124597] ixgbevf 0000:03:10.2: 9e:a9:c2:aa:1e:18
[   22.124597] ixgbevf 0000:03:10.2: MAC: 4
[   22.124598] ixgbevf 0000:03:10.2: Intel(R) 82599 Virtual Function
[   22.124605] ixgbevf 0000:03:10.4: enabling device (0000 -> 0002)
[   22.125706] ixgbevf 0000:03:10.4: PF still in reset state.  Is the PF interface up?
[   22.125706] ixgbevf 0000:03:10.4: Assigning random MAC address
[   22.125832] ixgbevf 0000:03:10.4: 6e:c3:69:e7:ff:4e
[   22.125832] ixgbevf 0000:03:10.4: MAC: 4
[   22.125833] ixgbevf 0000:03:10.4: Intel(R) 82599 Virtual Function
[   22.125840] ixgbevf 0000:03:10.6: enabling device (0000 -> 0002)
[   22.126941] ixgbevf 0000:03:10.6: PF still in reset state.  Is the PF interface up?
[   22.126942] ixgbevf 0000:03:10.6: Assigning random MAC address
[   22.943349] ixgbevf 0000:03:10.6: 86:df:7b:b6:81:e4
[   22.943350] ixgbevf 0000:03:10.6: MAC: 4
[   22.943351] ixgbevf 0000:03:10.6: Intel(R) 82599 Virtual Function
[   22.943362] ixgbevf 0000:03:11.0: enabling device (0000 -> 0002)
[   22.944516] ixgbevf 0000:03:11.0: PF still in reset state.  Is the PF interface up?
[   22.944517] ixgbevf 0000:03:11.0: Assigning random MAC address
[   22.944643] ixgbevf 0000:03:11.0: a6:41:27:17:7b:f5
[   22.944643] ixgbevf 0000:03:11.0: MAC: 4
[   22.944644] ixgbevf 0000:03:11.0: Intel(R) 82599 Virtual Function
[   22.944649] ixgbevf 0000:03:11.2: enabling device (0000 -> 0002)
[   22.945749] ixgbevf 0000:03:11.2: PF still in reset state.  Is the PF interface up?
[   22.945750] ixgbevf 0000:03:11.2: Assigning random MAC address
[   22.945863] ixgbevf 0000:03:11.2: d6:c7:c2:9f:a3:97
[   22.945864] ixgbevf 0000:03:11.2: MAC: 4
[   22.945865] ixgbevf 0000:03:11.2: Intel(R) 82599 Virtual Function
[   22.945871] ixgbevf 0000:03:11.4: enabling device (0000 -> 0002)
[   22.946978] ixgbevf 0000:03:11.4: PF still in reset state.  Is the PF interface up?
[   22.946979] ixgbevf 0000:03:11.4: Assigning random MAC address
[   22.947093] ixgbevf 0000:03:11.4: 42:3f:49:4f:8f:98
[   22.947094] ixgbevf 0000:03:11.4: MAC: 4
[   22.947094] ixgbevf 0000:03:11.4: Intel(R) 82599 Virtual Function
[   22.947100] ixgbevf 0000:03:11.6: enabling device (0000 -> 0002)
[   22.948245] ixgbevf 0000:03:11.6: PF still in reset state.  Is the PF interface up?
[   22.948246] ixgbevf 0000:03:11.6: Assigning random MAC address
[   22.948354] ixgbevf 0000:03:11.6: 72:02:ed:13:7f:05
[   22.948355] ixgbevf 0000:03:11.6: MAC: 4
[   22.948355] ixgbevf 0000:03:11.6: Intel(R) 82599 Virtual Function
[   23.162283] ixgbevf 0000:03:11.7 em2_7: renamed from eth10
[   23.979141] ixgbevf 0000:03:11.6 em1_7: renamed from eth17
[   34.531118] ixgbe 0000:03:00.1 rename27: renamed from eth6
[   35.502452] ixgbevf: Unable to start - perhaps the PF Driver isn't up yet

the only thing concerns me is
ixgbe 0000:03:00.0: PCI Express bandwidth of 2GT/s available
ixgbe 0000:03:00.0: (Speed:2.5GT/s, Width: x1, Encoding Loss:20%)
ixgbe 0000:03:00.0: For optimal performance, at least 20GT/s of bandwidth is required.

your modinfo shows


I believe kernel version 4.2.x is the key


ixgbe 0000:03:00.0: Enabling SR-IOV VFs using the max_vfs module parameter is deprecated - please use the pci sysfs interface instead.

I need to find out how to use pci sysfs way to enable it

Well I guess I will (try to) use a newer kernel when I have time.

 

[ggg]

Well I guess I will (try to) use a newer kernel when I have time.

You might also use your build module by doing:

cd [your_module_build_dir]
rmmod ixgbe
insmod ./ixgbe

If you use the path of your built module you can load it instead of the one accompanying the kernel.

 

[Jerry Chen]

root@xeon-kvm-host:~# export NUM_VFS=7

root@xeon-kvm-host:~# export ETH_NUM=eth6

root@xeon-kvm-host:~# echo $NUM_VFS > /sys/class/net/$ETH_NUM/device/sriov_numvfs

-bash: echo: write error: Operation not supported

root@xeon-kvm-host:~# tail /var/log/syslog

Oct 27 22:42:16 xeon-kvm-host kernel: [ 173.999157] ixgbe 0000:03:00.0 eth6: Intel(R) 10 Gigabit Network Connection

Oct 27 22:42:16 xeon-kvm-host kernel: [ 173.999292] ACPI Warning: \_SB_.PCI0.BR2C._PRT: Return Package has no elements (empty) (20150204/nsprepkg-126)

Oct 27 22:42:18 xeon-kvm-host systemd[1]: Started ifup for eth7.

Oct 27 22:42:18 xeon-kvm-host systemd[1]: Starting ifup for eth7...

Oct 27 22:42:18 xeon-kvm-host sh[2303]: Unknown interface eth7

Oct 27 22:42:18 xeon-kvm-host kernel: [ 175.970931] ixgbe 0000:03:00.1 eth7: MAC: 5, PHY: 6, PBA No: 020000-000

Oct 27 22:42:18 xeon-kvm-host kernel: [ 175.970942] ixgbe 0000:03:00.1: 0c:c4:7a:77:1e:59

Oct 27 22:42:18 xeon-kvm-host kernel: [ 175.970961] ixgbe 0000:03:00.1 eth7: Enabled Features: RxQ: 16 TxQ: 16 FdirHash RSC vxlan_rx

Oct 27 22:42:18 xeon-kvm-host kernel: [ 175.993136] ixgbe 0000:03:00.1 eth7: Intel(R) 10 Gigabit Network Connection

Oct 27 22:42:50 xeon-kvm-host kernel: [ 207.893354] ixgbe 0000:03:00.0: SRIOV not supported on this device

Nope.

 

[Markus]

Finally did it: Just ordered the 1540D-Board (X10SDV-TLN4F) and 2 x 32 GB ECC (Samsung). Hope this will fit all in my old case.

Plan to virtualize a NAS (IBM 1015), a firewall and several other VMs for testing.

 

[gigatexal]

So wait can anyone confirm a successful ping from a vm with the mic added as a vf

 

[EffrafaxOfWug]

ASRock have just put up their Xeon-D motherboards, finally!

ASRock Rack > D1540D4I-2L2T
ASRock Rack > D1540D4U-2T8R
ASRock Rack > D1540D4U-2O8R
ASRock Rack > D1540D4U-2L+
ASRock Rack > D1540D4I
ASRock Rack > D1520D4I

Sadly no SFP units visible on the mITX boards as expected. D1540D4I-2L2T looking like the closest competition to the X10SDV-8C-TLN4F.

Edit: M2 slots hanging off a Marvell 9172? Bleuch. Bleuch with knobs on. So much for using a nice M2 SSD for disc caching. Don't understand ASRock's continued fascination with those terrible marvell controllers... 9172 was bottom-of-the-barrel SATA controller three years ago...

 

[smokey7722]

ASRock have just put up their Xeon-D motherboards, finally!

ASRock Rack > D1540D4I-2L2T
ASRock Rack > D1540D4U-2T8R
ASRock Rack > D1540D4U-2O8R
ASRock Rack > D1540D4U-2L+
ASRock Rack > D1540D4I
ASRock Rack > D1520D4I

Sadly no SFP units visible on the mITX boards as expected. D1540D4I-2L2T looking like the closest competition to the X10SDV-8C-TLN4F.

I could be wrong but I was pretty sure they were listed on their site previously. The issue is finding them somewhere to actually purchase. I was really hoping for my VM hosts for a mini-itx SFP+ model as 10GBaseT is absolutely useless to me. I could still use a single D1540D4U-2O8R though for another purpose once they ever go on sale and are available.

 

[EffrafaxOfWug]

They'd listed some of the boards in their 2015 catalogue before but this is the first time they've had product pages with actual pictures instead of schematics.

Retail availability will of course be approximately 23 minutes before the heat death of the universe

 

[pyro_]

Now for the important part. What is the price going to be for these

 

[AmshTemp]

@EluRex:
I wish I can give you more likes.
You are the first one on the INTERNET to actually and properly test SR-IOV on Xeon-D
Please keep this stream of info coming.

@everyone:
Some weird things on ASRock D1540D4I-2L2T specifications:

Quad Channel DDR4 memory technology

and

D1540 / D1520: 6 x SATA3 6Gb/s (4 port from mini SAS, 1 port supports SATA DOM)

Edit: You can also spot M.2 in the motherboard picture, but nothing mentioned about it in the specifications.

Edit: the schematics of this motherboard is really needed.

 

[EffrafaxOfWug]

Going from their 2015 PDF, the M2 on the D1540D4I hangs off one of the D15x0's SATA ports, there's no additional storage controllers listed. The mini-SAS port is apparently just four of those aggregated into one. Hopefully the manuals will go up in a few days and we'll get a better explanation.

 

[DavidVermillion]

ASRock have just put up their Xeon-D motherboards, finally!

ASRock Rack > D1540D4I-2L2T
ASRock Rack > D1540D4U-2T8R
ASRock Rack > D1540D4U-2O8R
ASRock Rack > D1540D4U-2L+
ASRock Rack > D1540D4I
ASRock Rack > D1520D4I

Sadly no SFP units visible on the mITX boards as expected. D1540D4I-2L2T looking like the closest competition to the X10SDV-8C-TLN4F.

Edit: M2 slots hanging off a Marvell 9172? Bleuch. Bleuch with knobs on. So much for using a nice M2 SSD for disc caching. Don't understand ASRock's continued fascination with those terrible marvell controllers... 9172 was bottom-of-the-barrel SATA controller three years ago...

I don't understand the fascination with 10gbaseT. It's a shit technology and fiber is so much cheaper than it used to be.

 

[EluRex]

I don't understand the fascination with 10gbaseT. It's a shit technology and fiber is so much cheaper than it used to be.

10 Gbe-T with SR-IOV is great for virtualization platform and remove the bottle of openvswitch/bridge/bond. None of the 10Gb SPF+ can be use by any VM and trying to bridge them will create huge performance hit (reduce about 40% of bandwidth). In addition, 10 Gbe-T also great use as NAS since 4K video are becoming command, if trying to stream 4k video to multiple end client via 1 gbe interface, the bandwidth gets saturated quickly.

10 Gbe-T is much easier to configure and cheaper in comparison to Infiniband solution which should be solely used to interconnect the servers together. 10Gbe-T will be a great interface serving the end clients which only has 1 Gbe-T connection.

 

[DavidVermillion]

10 Gbe-T with SR-IOV is great for virtualization platform and remove the bottle of openvswitch/bridge/bond. None of the 10Gb SPF+ can be use by any VM and trying to bridge them will create huge performance hit (reduce about 40% of bandwidth). In addition, 10 Gbe-T also great use as NAS since 4K video are becoming command, if trying to stream 4k video to multiple end client via 1 gbe interface, the bandwidth gets saturated quickly.

But we can do this all with 10gb fiber ethernet. I concede your point about existing installations, although I would argue it's time to buckle down and run fiber anyway, which will last well into 100gbe. The amount of power needed for cat 6 just doesn't feel worth it (although I am reading that it's gotten better with new technology)

 

[Jerry Chen]

But we can do this all with 10gb fiber ethernet. I concede your point about existing installations, although I would argue it's time to buckle down and run fiber anyway, which will last well into 100gbe. The amount of power needed for cat 6 just doesn't feel worth it (although I am reading that it's gotten better with new technology)

For the sake of argument...

1) Can you vendor lock-in a CAT6/CAT6e?
2) Can you vendor lock-in a SFP+ module?

If you answer "No" and "Yes", congratulation, I just passed the sarcasm test!

 

[EluRex]

I am a die-hard on infiniband and in my work environment, I sees the need to use of

1. infiniband (SAN usage Server to Server),
2. 10GBe-T (NAS usage Server to end client),
3. 10GBe SPF+ (Switch to Switch), <--- its already vendor lock in
   
4. 1GBe SPF+ (Switch to POE Switch) <--- its already vendor lock in

is 10 GBe-T solution expensive? Hell yeah! Before it cost an arm and a leg and now it still cost an arm.

 

[jonaspaulo]

Hi again,

So is there a full procedure for the not-so-expert on linux for getting the SR-IOV working on both network controllers?

Also how can one confirm that this cpu has x2apic support?

Thanks a lot!

 

[hjfr]

ASRock have just put up their Xeon-D motherboards, finally!

ASRock Rack > D1540D4I-2L2T
ASRock Rack > D1540D4U-2T8R
ASRock Rack > D1540D4U-2O8R
ASRock Rack > D1540D4U-2L+
ASRock Rack > D1540D4I
ASRock Rack > D1520D4I

Very strange that Asrock releases now their 1520/1540 boards whereas the new 1521/1541 (and others) come in few days (normally in november)

 

[EffrafaxOfWug]

Very strange that Asrock releases now their 1520/1540 boards whereas the new 1521/1541 (and others) come in few days (normally in november)

"Releases" implies that these boards are actually available somewhere in the retail channel I've certainly seen hide nor hair of them and the SuperMicro xeon-D boards are still like rocking horse poo in europe. The 15x1's might ostensibly be released in november but I expect it'll be mid-2016 at the earliest before you start seeing boards shipping.