DeepLearning11 PCIe switch chips

Discussion in 'Machine Learning, Deep Learning, and AI' started by ryanrs, Mar 19, 2018.

  1. ryanrs

    ryanrs New Member

    Joined:
    Mar 19, 2018
    Messages:
    7
    Likes Received:
    0
    Does anyone know which PCIe switch chips are used in the Supermicro X10DRG-O-PCIE? I'm after the actual PLX part number.

    Maybe Patrick can run a lspci on DeepLearning11 for me?

    Also, if anyone knows if the x16 PCIe slots can be bifurcated to x4,x4,x4,x4 that would be nice to know, too. You know, in case you wanted to make a machine with 40 M.2 sticks in it. I'm guessing it probably can't, but hope springs eternal.
     
    #1
    ryanrs, Mar 19, 2018
  2. ryanrs

    ryanrs New Member

    Joined:
    Mar 19, 2018
    Messages:
    7
    Likes Received:
    0
    A poke through Broadcom's site suggests it could be a PEX8796, but confirmation would be great.
     
    #2
    ryanrs, Mar 19, 2018
  3. pgh5278

    pgh5278 Active Member

    Joined:
    Oct 25, 2012
    Messages:
    474
    Likes Received:
    126
    Why assume that it uses a plex chip?
    CPU 1 provides 2 x 16 and 1 x 8 to slots 4,2,10 = 40 PCIe3
    CPU2 provides 2 x 16 , 1 x 8, and 1 x 4 to slots 8,6,11 =40 PCIe3
    PCH supplies 1 x 4 to slot 9 and plus 1 x 4 to network ports etc = 8 PCIe2
    The 88 PCI lanes come from CPU and 612 chip.
    The plex chip PN mentioned above is often used in motherboards, used by Asus and many others to increase the number of x16 slots in Z270 boards and X99.
    Have a look at the board manual re bifurcation.
     
    #3
    pgh5278, Mar 19, 2018
  4. ryanrs

    ryanrs New Member

    Joined:
    Mar 19, 2018
    Messages:
    7
    Likes Received:
    0
    OK, sure, it could use a Microsemi PCIe switch instead. That's why I'm looking for confirmation on the part number.

    I have read the manual and it was silent on this matter.
     
    #4
    ryanrs, Mar 19, 2018
  5. pgh5278

    pgh5278 Active Member

    Joined:
    Oct 25, 2012
    Messages:
    474
    Likes Received:
    126
     
    #5
    pgh5278, Mar 19, 2018
  6. ryanrs

    ryanrs New Member

    Joined:
    Mar 19, 2018
    Messages:
    7
    Likes Received:
    0
    ...that's not the manual for the X10DRG-O-PCIE
     
    #6
    ryanrs, Mar 19, 2018
  7. pgh5278

    pgh5278 Active Member

    Joined:
    Oct 25, 2012
    Messages:
    474
    Likes Received:
    126
    Have a gander at the SERVER manual, which contain the motherboard and the PCIe Extension board., shows how it all works together.;)
    Always worth reading the manual, my experience is the SM server manuals, have the extra information that is required, as compared to the isolated component parts which fit with many products.

    That 11 PCi Extension board also works with a small range of X9 and X10 mother boards.
     
    #7
    pgh5278, Mar 19, 2018
  8. ryanrs

    ryanrs New Member

    Joined:
    Mar 19, 2018
    Messages:
    7
    Likes Received:
    0
    That 11 slot PCIe expansion board in your GIF, the X9DRG-O-PCI, is NOT the 12 slot PCIe expansion board X10DRG-O-PCIE that is in DeepLearning11. If you remain unconvinced, go read the build article that talks about why DeepLearning11 is designed to be a single-root GPU cluster. The difference between the X9DRG-O-PCI and X10DRG-O-PCI parts is the entire reason ServeTheHome built it.

    Look pgh5278, I get that you are trying to be helpful, but please stop shitting up the thread with wrong guesses.

    What I really need is for someone who has a X10DRG-O-PCIE system (and if you have a 10 GPU cluster running in your home, you will know), to run lspci and post the info re. the PCIe switches. Or just PM me the whole lspci output, which will be pretty long.

    Also, if you've got high-res photos of the board and see any DIP switches or jumpers near the switch chips (so, two identical sets of jumpers), that could be interesting, too.
     
    #8
    ryanrs, Mar 19, 2018
  9. Patrick

    Patrick Administrator
    Staff Member

    Joined:
    Dec 21, 2010
    Messages:
    11,572
    Likes Received:
    4,512
    How about this:
    Code:
    deeplearning11:~$ lspci | grep PLX
02:00.0 PCI bridge: PLX Technology, Inc. Device 9797 (rev aa)
03:04.0 PCI bridge: PLX Technology, Inc. Device 9797 (rev aa)
03:08.0 PCI bridge: PLX Technology, Inc. Device 9797 (rev aa)
03:0c.0 PCI bridge: PLX Technology, Inc. Device 9797 (rev aa)
03:10.0 PCI bridge: PLX Technology, Inc. Device 9797 (rev aa)
03:14.0 PCI bridge: PLX Technology, Inc. Device 9797 (rev aa)
09:00.0 PCI bridge: PLX Technology, Inc. Device 9797 (rev aa)
0a:04.0 PCI bridge: PLX Technology, Inc. Device 9797 (rev aa)
0a:08.0 PCI bridge: PLX Technology, Inc. Device 9797 (rev aa)
0a:0c.0 PCI bridge: PLX Technology, Inc. Device 9797 (rev aa)
0a:10.0 PCI bridge: PLX Technology, Inc. Device 9797 (rev aa)
0a:14.0 PCI bridge: PLX Technology, Inc. Device 9797 (rev aa)
     
    #9
    Patrick, Mar 19, 2018
    ryanrs likes this.
  10. ryanrs

    ryanrs New Member

    Joined:
    Mar 19, 2018
    Messages:
    7
    Likes Received:
    0
    Yes, perfect!

    The PEX9797 is a different chip than I expected, but looking at the datasheet it makes sense.

    Thanks so much, Patrick.
     
    #10
    ryanrs, Mar 19, 2018
  11. zir_blazer

    zir_blazer Active Member

    Joined:
    Dec 5, 2016
    Messages:
    220
    Likes Received:
    66
    It seems to me that it uses TWO PLX PEX9797. Note the PCI Address (02:00.0 then 03:xx.x, 09:00.0 followed by 0a:xx.x).
     
    #11
    zir_blazer, Mar 23, 2018
  12. ryanrs

    ryanrs New Member

    Joined:
    Mar 19, 2018
    Messages:
    7
    Likes Received:
    0
    Yes, 5 GPU slots on each switch.
     
    #12
    ryanrs, Mar 23, 2018
Similar Threads: DeepLearning11 PCIe
Forum Title Date
Machine Learning, Deep Learning, and AI PCIe Risers Feb 15, 2018
Machine Learning, Deep Learning, and AI Any recommendation on motherboard that supports 4 x PCIe 3.0 x16/x16/x16/x16? Aug 27, 2017
Machine Learning, Deep Learning, and AI CUDA Compute and PCIe Lanes May 12, 2017

Share This Page