Is it possible to install 10x Geforce GPU on Supermicro 4028GR-TR?

[matthew_z]

I am trying to build a deep learning server like https://www.servethehome.com/deeple...0-ti-single-root-deep-learning-server-part-1/

Sadly, Supermicro 4028GR-TR2 is somehow much more expensive in my country (asked many vendors), so I am looking at a cheaper choice: 4028GR-TR.

I saw that https://www.servethehome.com/deeplearning10-the-8x-nvidia-gtx-1080-ti-gpu-monster-part-1/ only installed 8x GPU. I wonder if it is possible to install two more GPU. It seems that there are more PCI-e x 8 slots.

 

[Patrick]

That is an interesting idea @matthew_z but there are a few reasons that people do not use the -TR for 10x GPU deep learning servers.

First, the actual daughterboard on the -TR(T) has three middle PCIe x16 physical slots but they are not PCIe 3.0 x16. Instead, there are two PCIe 3.0 x8 and one PCIe 2.0 x4 slots. Here is the diagram:


The second reason is that for nccl to work, you need to be on the same PCIe root/ NVLink complex. If you are doing deep learning training and want to use multiple GPUs, nccl is what you need.

A third reason is that if you ever got GPUs that supported it, and wanted to scale-out, you would want a Mellanox card on the same PCIe root so you can use GPUDirect RDMA.

If you want to go 10x GPU for deep learning, get the single root variants of the Xeon E5 generation https://www.servethehome.com/how-in...acted-single-root-pcie-deep-learning-servers/

 

[matthew_z]

Thank you @Patrick !
Then, I will try to ask more vendors for the price of 4028GR-TR2.