Intel Skylake Omni-Path Fabric Does Not Work on Every Server and Motherboard

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

cactus

Moderator
Jan 25, 2011
830
75
28
CA
@Patrick Going to have a piece about the rest of the wiring needed? The gold finger looks like it should just be passive right to the QSFP.
 

zir_blazer

Active Member
Dec 5, 2016
357
128
43
I think that there is something that may require a bit more aclaration, just to make things crystal clear. When you say...

Our initial understanding that every LGA3647 server and motherboard will accept the Omni-Path fabric SKUs is false. The assemblies do fit. For example, here is the same Dynatron/ OPA enabled Xeon assembly in a motherboard that does not support the fabric SKUs.

It physically fits. In this instance, there is no metal socket retention clip for the Omni-Path cable as can be seen on the following motherboard with the same assembly:
I understand that you need a Motherboard that supports these specific Processors to get everything working as intended. However, if you use it in one that doesn't, will the Processor work but OmniPath not, or instead you will be unable to get the Processor working at all? The question is aimed at purchasing Xeons with OmniPath, plugging them in a normal Motherboard, and upgrade Motherboard later.


Also, while looking for Skylake-F support, I noticed this:

Supermicro X11DPH-T

Dual Socket P (LGA 3647) supported, CPU TDP support 205W, 3 UPI up to 10.4 GT/s
CPU1: Skylake-F CPU supported
What I understand is that only one Socket (CPU1) supports Skylake-F. Depending on what the response to the previous question is, it means that you can't expect to buy Dual Processors as identical pairs any longer.

Moreover, in the manual of the X11DPH-T, in the Block Diagram, it mentions that CPU1 has 6+1 phase VRM that supports up to 255W Processors but CPU2 just 5+1, while the website specifications page says up to 205W TDP that should be accurate for CPU2. This is not symmetric, and should account for Skylake-F extra power.
Oh, the manual of the X11DPH-T also mentions the unannounced X11DPH-i and X11DPT-Tq, the difference is that the -i uses Chipset C622, the published -T C624, and -Tq C628, which supports QAT (QuickAssist Technology) but has one less PCIe 8x Slot.

For some reason, I through that it would be fun to use a Skylake-F with a truckload of RAM as a giant external RAMDisk...
 
  • Like
Reactions: BennyT

_alex

Active Member
Jan 28, 2016
866
97
28
Bavaria / Germany
Does OP actually offload from the CPU?
and how does it compare to other fabrics in terms of performance, total system power draw and lately tco ?

I've found a lot of fud about this, both from Intel and mellanox on the web.
So really curious to see the outcome of an independent compare with real-life systems and workload.
 

cactus

Moderator
Jan 25, 2011
830
75
28
CA
Moreover, in the manual of the X11DPH-T, in the Block Diagram, it mentions that CPU1 has 6+1 phase VRM that supports up to 255W Processors but CPU2 just 5+1, while the website specifications page says up to 205W TDP that should be accurate for CPU2.
50W seems like a lot of power for a NIC even with power optics.
 

Patrick

Administrator
Staff member
Dec 21, 2010
12,513
5,805
113
Not 50w. Also, we do have a board with 2x F parts installed if you see the lspci output
 

i386

Well-Known Member
Mar 18, 2016
4,245
1,546
113
34
Germany
and how does it compare to other fabrics in terms of performance, total system power draw and lately tco ?

I've found a lot of fud about this, both from Intel and mellanox on the web.
I'm also interested in this.

@_alex are you talking about these enterprise benchmarks where a 6node edr infiniband cluster outperformed a 12 node omnipath cluster (both using 100gbit/s)?
 

MiniKnight

Well-Known Member
Mar 30, 2012
3,072
973
113
NYC
Mellanox isn't above fudging these numbers either

This is so much bigger than you're making it. It should be called OPA-gate