Intel Xeon D-1500 Series Discussion

[EluRex]

@ggg Just so you know, the official stance from Intel is that SR-IOV is not supported on the Xeon D NICs.

@Dom apologies I missed this on Thursday. I was actually in Taipei and met with the ASRock Rack folks. I did get to see a version of the board you mentioned and I urged them to get it here as fast as possible. I was told it would be coming soon. I also spoke with the US sales channel person there and stressed the need for availability. Look for more info tomorrow on the STH main site.

Dear Patrick

From my 5028d using ubuntu 14.04.3 kernel 3.19-0.31

root@pdc:~# lspci -vnn -s 03:00.0
03:00.0 Ethernet controller [0200]: Intel Corporation Device [8086:15ad]
Subsystem: Super Micro Computer Inc Device [15d9:15ad]
Physical Slot: 0-1
Flags: bus master, fast devsel, latency 0, IRQ 11
Memory at fbc00000 (64-bit, prefetchable) [size=2M]
Memory at fbe04000 (64-bit, prefetchable) [size=16K]
Expansion ROM at 90100000 [disabled] [size=512K]
Capabilities: [40] Power Management version 3
Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
Capabilities: [70] MSI-X: Enable- Count=64 Masked-
Capabilities: [a0] Express Endpoint, MSI 00
Capabilities: [e0] Vital Product Data
Capabilities: [100] Advanced Error Reporting
Capabilities: [140] Device Serial Number 00-00-c9-ff-ff-00-00-00
Capabilities: [150] Alternative Routing-ID Interpretation (ARI)
Capabilities: [160] Single Root I/O Virtualization (SR-IOV)
Capabilities: [1b0] Access Control Services
Capabilities: [1c0] Latency Tolerance Reporting

root@pdc:~# lspci -vnn -s 05:00.0
05:00.0 Ethernet controller [0200]: Intel Corporation I350 Gigabit Network Connection [8086:1521] (rev 01)
Subsystem: Super Micro Computer Inc Device [15d9:1521]
Flags: bus master, fast devsel, latency 0, IRQ 16
Memory at fb420000 (32-bit, non-prefetchable) [size=128K]
I/O ports at e020
Memory at fb444000 (32-bit, non-prefetchable) [size=16K]
Capabilities: [40] Power Management version 3
Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
Capabilities: [70] MSI-X: Enable+ Count=10 Masked-
Capabilities: [a0] Express Endpoint, MSI 00
Capabilities: [100] Advanced Error Reporting
Capabilities: [140] Device Serial Number 0c-c4-7a-ff-ff-74-07-88
Capabilities: [150] Alternative Routing-ID Interpretation (ARI)
Capabilities: [160] Single Root I/O Virtualization (SR-IOV)
Capabilities: [1a0] Transaction Processing Hints
Capabilities: [1c0] Latency Tolerance Reporting
Capabilities: [1d0] Access Control Services
Kernel driver in use: igb

What is preventing it from enable SR-IOV? the card and also the cpu is definitely capable of vt-d.... its sad thing to hear it cant!

 

[MiniKnight]

Dear Patrick

From my 5028d using ubuntu 14.04.3 kernel 3.19-0.31

root@pdc:~# lspci -vnn -s 03:00.0
03:00.0 Ethernet controller [0200]: Intel Corporation Device [8086:15ad]
Subsystem: Super Micro Computer Inc Device [15d9:15ad]
Physical Slot: 0-1
Flags: bus master, fast devsel, latency 0, IRQ 11
Memory at fbc00000 (64-bit, prefetchable) [size=2M]
Memory at fbe04000 (64-bit, prefetchable) [size=16K]
Expansion ROM at 90100000 [disabled] [size=512K]
Capabilities: [40] Power Management version 3
Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
Capabilities: [70] MSI-X: Enable- Count=64 Masked-
Capabilities: [a0] Express Endpoint, MSI 00
Capabilities: [e0] Vital Product Data
Capabilities: [100] Advanced Error Reporting
Capabilities: [140] Device Serial Number 00-00-c9-ff-ff-00-00-00
Capabilities: [150] Alternative Routing-ID Interpretation (ARI)
Capabilities: [160] Single Root I/O Virtualization (SR-IOV)
Capabilities: [1b0] Access Control Services
Capabilities: [1c0] Latency Tolerance Reporting

root@pdc:~# lspci -vnn -s 05:00.0
05:00.0 Ethernet controller [0200]: Intel Corporation I350 Gigabit Network Connection [8086:1521] (rev 01)
Subsystem: Super Micro Computer Inc Device [15d9:1521]
Flags: bus master, fast devsel, latency 0, IRQ 16
Memory at fb420000 (32-bit, non-prefetchable) [size=128K]
I/O ports at e020
Memory at fb444000 (32-bit, non-prefetchable) [size=16K]
Capabilities: [40] Power Management version 3
Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
Capabilities: [70] MSI-X: Enable+ Count=10 Masked-
Capabilities: [a0] Express Endpoint, MSI 00
Capabilities: [100] Advanced Error Reporting
Capabilities: [140] Device Serial Number 0c-c4-7a-ff-ff-74-07-88
Capabilities: [150] Alternative Routing-ID Interpretation (ARI)
Capabilities: [160] Single Root I/O Virtualization (SR-IOV)
Capabilities: [1a0] Transaction Processing Hints
Capabilities: [1c0] Latency Tolerance Reporting
Capabilities: [1d0] Access Control Services
Kernel driver in use: igb

What is preventing it from enable SR-IOV? the card and also the cpu is definitely capable of vt-d.... its sad thing to hear it cant!

I think they are talking about the omboard controllers. The i350 is a normal PCIe device. The onboard NICs aren't over PCIe which is why you can't pass them through with VT-d.

 

[OBasel]

Just spoke to Asrock in California. No info on the Asrock D1540D4U-2T8R, expected release info will be mid November.

Does that mean the O model is Optical? I think I'd want SFP+ Optical and 10Gbase-T.

 

[EluRex]

I think they are talking about the omboard controllers. The i350 is a normal PCIe device. The onboard NICs aren't over PCIe which is why you can't pass them through with VT-d.

tks miniknight for clear that up. But they have pci sid that's why I am assuming they are connected via PCIe Lane. is this function disabled in bios/firmware or it is physically impossible?

 

[EffrafaxOfWug]

Asrock has a history of poor power efficiency on motherboards relative to Supermicro.

Do they...? I know Patrick hasn't been able to get his hands on any ASRock xeon-D's yet but their E3 and E5 units have always been in the same ballpark from what I've seen.

 

[Patrick]

tks miniknight for clear that up. But they have pci sid that's why I am assuming they are connected via PCIe Lane. is this function disabled in bios/firmware or it is physically impossible?

Here is the official response I got from Intel:

Hi Patrick,
To follow up below, SR-IOV is not enabled in the Intel® Xeon® Processor D-1520 and D1540 products that Intel announced earlier this year.

 

[Dom]

Does that mean the O model is Optical? I think I'd want SFP+ Optical and 10Gbase-T.

It is, using a Cirrus Logic controller I believe.

Here's the 2015 Catalog:
http://www.asrockrack.com/general/Catalog2015.pdf

I apologize if i'm unable to post external links, i'll remove if so.

 

[Dom]

Also @Dom you may want to check this out: http://www.servethehome.com/asrock-rack-xeon-d-ultimate-motherboard-spotted/

Good to see these in the wild, thank you for that. I'd wish there was some more communication via Facebook or announcements regarding the availability from Intel and release time of the MB.

Do you know if you can Passthrough the LSI controller? I'm under the assumption you could!

 

[Dom]

Very nice motherboard, but Asrock has a history of poor power efficiency on motherboards relative to Supermicro. Lets hope that they worked on this issue for the new motherboards.

What are we talking about here, a few watts??

 

[EluRex]

Here is the official response I got from Intel:

http://www.supermicro.com/manuals/superserver/mid-tower/MNL-5028D-TN4T.pdf manual page 95 has option to enable sr_iov (I guess that is only for the pci-e slot device not those on integrated SOC)

So this Xeon D SOC platform in a sense is purposely crippled by Intel so that it cannot run many VM guests due to the networking part will be its bottle neck (it has 10 gbe connections but cannot use it in by VM effectively as
openvswitch/bridge max throughput is about 6gbps only)

So for SM board, 10gbe is only good for serving as NAS, and the VM can only utilize the dual i350 eth via OpenvSwitch (which also use more cpu resources).

Good to see these in the wild, thank you for that. I'd wish there was some more communication via Facebook or announcements regarding the availability from Intel and release time of the MB.

Do you know if you can Passthrough the LSI controller? I'm under the assumption you could!

It should be or otherwise what is the point making the Xeon D has VT-D functionality at all? right now just hope Asrock has not put the LSA controller behind any pci-e switch (which shouldn't)

 

[PigLover]

Does anyone know if the SoC based NICs can be supported via DPDK? I think many folks would tell you that running a DPDK-based v-Switch is more "directional" for Intel than reliance on SR-IOV for network visualization.

 

[EluRex]

What about the other integrated NIC i350-AM2?
On papers, it should support SR-IOV.

Intel® Ethernet Controller I350-AM2 Specifications

i tested /etc/modprobe.d/igb.conf and sr-iov is also disabled as well

 

[EluRex]

Does anyone know if the SoC based NICs can be supported via DPDK? I think many folks would tell you that running a DPDK-based v-Switch is more "directional" for Intel than reliance on SR-IOV for network visualization.

I will do some testing on Open vSwitch over DPDK on Ubuntu It is interesting to see if dpdk acceleration on OVS network performance.

 

[PigLover]

I will do some testing on Open vSwitch over DPDK on Ubuntu It is interesting to see if dpdk acceleration on OVS network performance.

I have test results (that I unfortunately cannot share) with a highly optimized OVS-compatible vSwitch based on DPDK. The results are phenomenal - though it does suck up a core-per-NIC due to DPDK polling activity.

 

[Patrick]

Does anyone know if the SoC based NICs can be supported via DPDK? I think many folks would tell you that running a DPDK-based v-Switch is more "directional" for Intel than reliance on SR-IOV for network visualization.

Not sure offhand. I am 99% sure that the i354 on the Avoton/ Rangeley is supported by DPDK.

The fact that the X550 is listed next to ixbe here - DPDK doc makes me hopeful that it is supported.

 

[EluRex]

I have test results (that I unfortunately cannot share) with a highly optimized OVS-compatible vSwitch based on DPDK. The results are phenomenal - though it does suck up a core-per-NIC due to DPDK polling activity.

So it's a trade off CPU for Network performance, but core per NIC is a hefty price.

 

[PigLover]

So it's a trade off CPU for Network performance, but core per NIC is a hefty price.

Yup. DPDK vSwitch with two 10Gbe NICs pretty much requires two cores dedicated to the vSwitch.

 

[acmcool]

Its good I gave up on Xeon D platform ..
Better off with V3 xeon and Chelsio 10g nic..

 

[PigLover]

So it's a trade off CPU for Network performance, but core per NIC is a hefty price.

Also, remember that the trade off is more complex. When comparing SR-IOV vs DPDK you do have to consider the CPU impacts of DPDKs polling. But remember that with SR-IOV you have to expose DMA+memory management to the NIC, which disables a number of very useful vitalization features.

So to real trade off between two methods of getting high performance network IO under visualization: SR-IOV at the cost of other virtualization features (Migrations, memory ballooning, etc) vs DPDK which keeps those features in tact at the cost of CPU polling overhead.

Note this trade off is not unique to the D-1500 CPU series. It comes up here because one option (SR-IOV) appears to be disabled on the on-SoC NICs.

 

[Remko]

Muh... absolutely not amused where this discussion is going. Especially:

Not necessarily a show stopper but is why I am not overly keen on this as a heavy virtualization platform (e.g. ESXi.)

The fact is I bought a couple of boards specifically for ... you already guessed it ... semi-heavy virtualization. Furthermore, even Supermicro itself still advertises with SR-IOV on their press site @ Supermicro | News | Supermicro® Launches New Line of Low Power, High Density Server Solutions Supporting Intel® Xeon® Processor D-1500. See quote; I highlighted SR-IOV.

Product Specifications

• X10SDV-F/-TLN4F – Mini-ITX Motherboard (6.7” x 6.7”) supports single Intel® Xeon® processor D-1540 SoC (8 core, 45W),VT-d/x, TXT, AES-NI, SR-IOV, Xeon RAS, built in 10GbE. 128GB 2133MHz DDR4 RDIMM or 64GB UDIMM in 4x DIMMs, 6x SATA 3.0, 1x M.2 slot (M key for SSD, 2242/2280, PCIe3.0 x4), 2x USB 3.0, 4x USB 2.0, 1x PCIe 3.0 x16, Quad LAN ports with SoC dual 10GbE and I350-AM2 dual GbE (-F with dual GbE only), IPMI 2.0 with KVM and dedicated port, 0-60°C operating temperature, 4 pin 12V DC and ATX power source

Is it a hardware firmware issue or a driver issue? Why are a lot of sites touting this as the 'ultimate' virtualization or 'must get' platform (Tinkertry, ServeTheHome, etc)... What is heavy virtualization btw?


PS. Offtopic -> Intel Xeon D-1540 10GbE X552/X557 driver/VIB for VMware ESXi 6 on Supermicro 5028D-TN4T system's X10SDV-TLN4F mobo expected by early Nov. 2015. See post @ Tinkertry