Is it possible to install 10x Geforce GPU on Supermicro 4028GR-TR?

Discussion in 'DIY Server and Workstation Builds' started by matthew_z, Aug 25, 2018.

  1. matthew_z

    matthew_z New Member

    Joined:
    Aug 25, 2018
    Messages:
    3
    Likes Received:
    0
    I am trying to build a deep learning server like https://www.servethehome.com/deeple...0-ti-single-root-deep-learning-server-part-1/

    Sadly, Supermicro 4028GR-TR2 is somehow much more expensive in my country (asked many vendors), so I am looking at a cheaper choice: 4028GR-TR.

    I saw that https://www.servethehome.com/deeplearning10-the-8x-nvidia-gtx-1080-ti-gpu-monster-part-1/ only installed 8x GPU. I wonder if it is possible to install two more GPU. It seems that there are more PCI-e x 8 slots.
     
    #1
    Last edited: Aug 25, 2018
  2. Patrick

    Patrick Administrator
    Staff Member

    Joined:
    Dec 21, 2010
    Messages:
    11,196
    Likes Received:
    4,148
    That is an interesting idea @matthew_z but there are a few reasons that people do not use the -TR for 10x GPU deep learning servers.

    First, the actual daughterboard on the -TR(T) has three middle PCIe x16 physical slots but they are not PCIe 3.0 x16. Instead, there are two PCIe 3.0 x8 and one PCIe 2.0 x4 slots. Here is the diagram:
    upload_2018-8-25_6-17-43.png

    The second reason is that for nccl to work, you need to be on the same PCIe root/ NVLink complex. If you are doing deep learning training and want to use multiple GPUs, nccl is what you need.

    A third reason is that if you ever got GPUs that supported it, and wanted to scale-out, you would want a Mellanox card on the same PCIe root so you can use GPUDirect RDMA.

    If you want to go 10x GPU for deep learning, get the single root variants of the Xeon E5 generation https://www.servethehome.com/how-in...acted-single-root-pcie-deep-learning-servers/
     
    #2
    T_Minus likes this.
  3. matthew_z

    matthew_z New Member

    Joined:
    Aug 25, 2018
    Messages:
    3
    Likes Received:
    0
    Thank you @Patrick !
    Then, I will try to ask more vendors for the price of 4028GR-TR2.
     
    #3
Similar Threads: possible install
Forum Title Date
DIY Server and Workstation Builds Qnap TS-EC1279U-RP with not Qnap Expension possible? Mar 11, 2018
DIY Server and Workstation Builds Lenovo M93P tiny - possible to mount a HSF to? Jul 25, 2017
DIY Server and Workstation Builds Memory question (possible fakes) Feb 22, 2016
DIY Server and Workstation Builds ghetto shelf hack possible? Jun 7, 2013
DIY Server and Workstation Builds Possible Upgrade? May 24, 2013

Share This Page