DeepLearning11 PCIe switch chips

ryanrs

New Member
Mar 19, 2018
7
0
1
51
Does anyone know which PCIe switch chips are used in the Supermicro X10DRG-O-PCIE? I'm after the actual PLX part number.

Maybe Patrick can run a lspci on DeepLearning11 for me?

Also, if anyone knows if the x16 PCIe slots can be bifurcated to x4,x4,x4,x4 that would be nice to know, too. You know, in case you wanted to make a machine with 40 M.2 sticks in it. I'm guessing it probably can't, but hope springs eternal.
 

ryanrs

New Member
Mar 19, 2018
7
0
1
51
A poke through Broadcom's site suggests it could be a PEX8796, but confirmation would be great.
 

pgh5278

Active Member
Oct 25, 2012
479
128
43
Australia
Does anyone know which PCIe switch chips are used in the Supermicro X10DRG-O-PCIE? I'm after the actual PLX part number
Also, if anyone knows if the x16 PCIe slots can be bifurcated to x4,x4,x4,x4 that would be nice to know, too. You know, in case you wanted to make a machine with 40 M.2 sticks in it. I'm guessing it probably can't, but hope springs eternal.
Why assume that it uses a plex chip?
CPU 1 provides 2 x 16 and 1 x 8 to slots 4,2,10 = 40 PCIe3
CPU2 provides 2 x 16 , 1 x 8, and 1 x 4 to slots 8,6,11 =40 PCIe3
PCH supplies 1 x 4 to slot 9 and plus 1 x 4 to network ports etc = 8 PCIe2
The 88 PCI lanes come from CPU and 612 chip.
The plex chip PN mentioned above is often used in motherboards, used by Asus and many others to increase the number of x16 slots in Z270 boards and X99.
Have a look at the board manual re bifurcation.
 

ryanrs

New Member
Mar 19, 2018
7
0
1
51
Why assume that it uses a plex chip?
OK, sure, it could use a Microsemi PCIe switch instead. That's why I'm looking for confirmation on the part number.

I have read the manual and it was silent on this matter.
 

pgh5278

Active Member
Oct 25, 2012
479
128
43
Australia
...that's not the manual for the X10DRG-O-PCIE
Have a gander at the SERVER manual, which contain the motherboard and the PCIe Extension board., shows how it all works together.;)
Always worth reading the manual, my experience is the SM server manuals, have the extra information that is required, as compared to the isolated component parts which fit with many products.

That 11 PCi Extension board also works with a small range of X9 and X10 mother boards.
 

ryanrs

New Member
Mar 19, 2018
7
0
1
51
Always worth reading the manual [..] That 11 PCi Extension board...
That 11 slot PCIe expansion board in your GIF, the X9DRG-O-PCI, is NOT the 12 slot PCIe expansion board X10DRG-O-PCIE that is in DeepLearning11. If you remain unconvinced, go read the build article that talks about why DeepLearning11 is designed to be a single-root GPU cluster. The difference between the X9DRG-O-PCI and X10DRG-O-PCI parts is the entire reason ServeTheHome built it.

Look pgh5278, I get that you are trying to be helpful, but please stop shitting up the thread with wrong guesses.

What I really need is for someone who has a X10DRG-O-PCIE system (and if you have a 10 GPU cluster running in your home, you will know), to run lspci and post the info re. the PCIe switches. Or just PM me the whole lspci output, which will be pretty long.

Also, if you've got high-res photos of the board and see any DIP switches or jumpers near the switch chips (so, two identical sets of jumpers), that could be interesting, too.
 

Patrick

Administrator
Staff member
Dec 21, 2010
12,211
5,297
113
How about this:
Code:
deeplearning11:~$ lspci | grep PLX
02:00.0 PCI bridge: PLX Technology, Inc. Device 9797 (rev aa)
03:04.0 PCI bridge: PLX Technology, Inc. Device 9797 (rev aa)
03:08.0 PCI bridge: PLX Technology, Inc. Device 9797 (rev aa)
03:0c.0 PCI bridge: PLX Technology, Inc. Device 9797 (rev aa)
03:10.0 PCI bridge: PLX Technology, Inc. Device 9797 (rev aa)
03:14.0 PCI bridge: PLX Technology, Inc. Device 9797 (rev aa)
09:00.0 PCI bridge: PLX Technology, Inc. Device 9797 (rev aa)
0a:04.0 PCI bridge: PLX Technology, Inc. Device 9797 (rev aa)
0a:08.0 PCI bridge: PLX Technology, Inc. Device 9797 (rev aa)
0a:0c.0 PCI bridge: PLX Technology, Inc. Device 9797 (rev aa)
0a:10.0 PCI bridge: PLX Technology, Inc. Device 9797 (rev aa)
0a:14.0 PCI bridge: PLX Technology, Inc. Device 9797 (rev aa)
 
  • Like
Reactions: ryanrs

ryanrs

New Member
Mar 19, 2018
7
0
1
51
Yes, perfect!

The PEX9797 is a different chip than I expected, but looking at the datasheet it makes sense.

Thanks so much, Patrick.
 

zir_blazer

Active Member
Dec 5, 2016
303
99
28
It seems to me that it uses TWO PLX PEX9797. Note the PCI Address (02:00.0 then 03:xx.x, 09:00.0 followed by 0a:xx.x).