supermicro H12ssl-i , multi gpu build

[omar-migs]

Hello Guys,
I am building my first server and already running into parts selection problem. The parts i got so far are the supermicro h12ssl-i MOBO and EPYC 7402 CPU.
My plan is to build a system with 3 rtx 3090 for model training, i will just start with one and get more when i find good deals on these cards as they are quite expensive.
The MOBO has 5 PCI-E 4.0 x16 but if i use 3 cards (double slot rtx 3090 ) other PCIE slots won't be usable so the solution from my perspective is to use risers.

Here comes the question. Which case could support this type of setup with possible 2 PSUs later on, at the same time enough HDD bays (8 at least)

 

[unwind-protect]

There are some mining boards on Ebay. They have the appropriate distance between PCIe slots. I don't think they are intended for a case, though.

 

[omar-migs]

@unwind-protect yeah i saw these but as you said they are not intended for a case and i want the system to be enclosed either in a rack or a case.
Thank you for your suggestion

 

[DarthCirrus]

Hello! As I see it you have a two options

1) switch motherboards and use a server whose case is designed for multi gpu.
-that could be 2U 2 node that does 3 GPUs each letting you start with 1 node and add a second in the future (Supermicro A+ Server 2114GT-DNR)
-or a 4U with room for up to 8 GPUs (Supermicro A+ Server 4124GS-TNR)

2) Use water cooled cards to slim down to 1 slot per gpu. It creates the most choice by letting you use the parts you already have and you can use almost any case you want. In terms of actually water cooling the cards it’ll have to be a custom loop with connectors designed allow for multiple 1 slot cards in a row; but there are several commercially available options designed to water cool multiple GPUs.

 

[omar-migs]

Hello! As I see it you have a two options

1) switch motherboards and use a server whose case is designed for multi gpu.
-that could be 2U 2 node that does 3 GPUs each letting you start with 1 node and add a second in the future (Supermicro A+ Server 2114GT-DNR)
-or a 4U with room for up to 8 GPUs (Supermicro A+ Server 4124GS-TNR)

2) Use water cooled cards to slim down to 1 slot per gpu. It creates the most choice by letting you use the parts you already have and you can use almost any case you want. In terms of actually water cooling the cards it’ll have to be a custom loop with connectors designed allow for multiple 1 slot cards in a row; but there are several commercially available options designed to water cool multiple GPUs.

Thank you so much for your reply and recommendations.
for the first option the price tag is a bit high.
From my experience building none server systems, water cooling sometimes caused problems with systems running 24/7, but to be honest it makes the most sense,
Actually currently i'm looking into building a custom case, and the costs of doing so.
Thank you again for your reply

 

[i386]

supermicro 747 has 11 pcie slots. not sure if it supports that mainboard though...

 

[omar-migs]

supermicro 747 has 11 pcie slots. not sure if it supports that mainboard though...

Thank you so much for your reply, i will look into it

 

[luquoo]

Any updates on the project? I'm interested in building a similar system but was trying to get it to work with Genoa chips.

 

[omar-migs]

Any updates on the project? I'm interested in building a similar system but was trying to get it to work with Genoa chips.

Hello , I ended up getting thermaltake wp200 and routed the gpu’S to the other side of the case via pcie risers . I have one huge disadvantage which is the heat is not properly getting out no matter what I do . I wouldn’t recommend this setup, I ended up selling it and got a smaller case with a different setup . Ryzen 7 with rtx 4090 till I figure out something for such a built .

 

[rtech]

Hello , I ended up getting thermaltake wp200 and routed the gpu’S to the other side of the case via pcie risers . I have one huge disadvantage which is the heat is not properly getting out no matter what I do . I wouldn’t recommend this setup, I ended up selling it and got a smaller case with a different setup . Ryzen 7 with rtx 4090 till I figure out something for such a built .

How come? Can you elaborate on this?
If i had guess you used very low static pressure fans.

 

[omar-migs]

How come? Can you elaborate on this?
If i had guess you used very low static pressure fans.

I used 4 fans on the gpu side and 3 on the mobo side . My guess is that the gpu side had more negative pressure and it pulled the hot air from the Mobo side which in turn didn’t cool the gpus down .