1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Intel Skylake Omni-Path Fabric Does Not Work on Every Server and Motherboard

Discussion in 'STH Main Site Posts' started by Patrick Kennedy, Sep 12, 2017.

  1. #1
    eva2000 likes this.
  2. cactus

    cactus Moderator

    Joined:
    Jan 25, 2011
    Messages:
    735
    Likes Received:
    42
    @Patrick Going to have a piece about the rest of the wiring needed? The gold finger looks like it should just be passive right to the QSFP.
     
    #2
  3. zir_blazer

    zir_blazer Member

    Joined:
    Dec 5, 2016
    Messages:
    70
    Likes Received:
    30
    I think that there is something that may require a bit more aclaration, just to make things crystal clear. When you say...

    I understand that you need a Motherboard that supports these specific Processors to get everything working as intended. However, if you use it in one that doesn't, will the Processor work but OmniPath not, or instead you will be unable to get the Processor working at all? The question is aimed at purchasing Xeons with OmniPath, plugging them in a normal Motherboard, and upgrade Motherboard later.


    Also, while looking for Skylake-F support, I noticed this:

    Supermicro X11DPH-T

    What I understand is that only one Socket (CPU1) supports Skylake-F. Depending on what the response to the previous question is, it means that you can't expect to buy Dual Processors as identical pairs any longer.

    Moreover, in the manual of the X11DPH-T, in the Block Diagram, it mentions that CPU1 has 6+1 phase VRM that supports up to 255W Processors but CPU2 just 5+1, while the website specifications page says up to 205W TDP that should be accurate for CPU2. This is not symmetric, and should account for Skylake-F extra power.
    Oh, the manual of the X11DPH-T also mentions the unannounced X11DPH-i and X11DPT-Tq, the difference is that the -i uses Chipset C622, the published -T C624, and -Tq C628, which supports QAT (QuickAssist Technology) but has one less PCIe 8x Slot.

    For some reason, I through that it would be fun to use a Skylake-F with a truckload of RAM as a giant external RAMDisk...
     
    #3
  4. _alex

    _alex Active Member

    Joined:
    Jan 28, 2016
    Messages:
    559
    Likes Received:
    45
    Does OP actually offload from the CPU?
    and how does it compare to other fabrics in terms of performance, total system power draw and lately tco ?

    I've found a lot of fud about this, both from Intel and mellanox on the web.
    So really curious to see the outcome of an independent compare with real-life systems and workload.
     
    #4
  5. cactus

    cactus Moderator

    Joined:
    Jan 25, 2011
    Messages:
    735
    Likes Received:
    42
    50W seems like a lot of power for a NIC even with power optics.
     
    #5
  6. Patrick

    Patrick Administrator
    Staff Member

    Joined:
    Dec 21, 2010
    Messages:
    9,764
    Likes Received:
    3,223
    Not 50w. Also, we do have a board with 2x F parts installed if you see the lspci output
     
    #6
  7. i386

    i386 Active Member

    Joined:
    Mar 18, 2016
    Messages:
    667
    Likes Received:
    137
    I'm also interested in this.

    @_alex are you talking about these enterprise benchmarks where a 6node edr infiniband cluster outperformed a 12 node omnipath cluster (both using 100gbit/s)?
     
    #7
  8. MiniKnight

    MiniKnight Well-Known Member

    Joined:
    Mar 30, 2012
    Messages:
    2,188
    Likes Received:
    583
    Mellanox isn't above fudging these numbers either

    This is so much bigger than you're making it. It should be called OPA-gate
     
    #8
  9. _alex

    _alex Active Member

    Joined:
    Jan 28, 2016
    Messages:
    559
    Likes Received:
    45
    #9

Share This Page