Thermal performance of 8x 3090 GPUs in a server chassis?

[ltp410]

Our team is looking to purchase a server with 8x blower-style 3090 GPUs (we have to stick with 3090 even though we prefer A6000). The chassis would be something like ASUS ESC8000A-E11. We're getting mixed response from different vendors:
Some assured there would be no thermal issue.
Some strongly advised against 8x 3090's (even blower-style). Their argument is that the air is already quite toasty by the time it's passed through all the drives, memory, cpus and got to the GPU chamber. And the GPUs don't move sufficient air volume so hot air gets circulated inside the server around the GPUs. It kinda makes sense to me too.

So, if you guys have any experience with this I'd be very interested to hear what you think.
Thank you!

 

[JediAcolyte]

That blower does nothing to cool half of the memory that's on the back of the card.
Idk that chassis but I'm guessing there won't be much airflow in the area around the cards that doesn't end up going through the blowers.

 

[Wasmachineman_NL]

That system is going to turn Chernobyl real quick because of the backside VRAM.

 

[Patrick]

I would also be worried about the back-side memory on those.

 

[nutsnax]

This will be your server.

You have a hot-side of the case post-cpu and then a REALLY hot-side inside those cards and then a blast furnace coming out the back.

You either need to fashion auxiliary intake fan ports post-cpu to cool the exhaust stream hitting the cards.... Or....

Try an idea i had for a similar but less-nuclear project which is to mount a pair of radiators with pump/reservoir as a hot-side heat exchanger to the front of the case (or wherever the pipes go). One radiator sits behind the fans and the other at the front of the case (preferably not directly in the intake path of the CPU's but that might be unavoidable). This would hopefully balance the temps on the hot side with the intake and lower temps without resorting to things like GPU waterblocks.

One could also use a peltier and waterblock to further dispose of waste heat or even pipe the thing into a water chiller negating the second radiator. Peltier and/or second radiator would just be to have a way to keep it compact and not have pipes everywhere.

Or if youre afraid of any amount of water inside of your case, mount a 240mm+ radiator to the front of the chassis, seal it to the front with HVAC foam, and run a water chiller through it to simply chill the intake charge...

watch out for condensation

 

[JediAcolyte]

A lot of crypto miners will replace all the stock thermal pads and paste as soon as they purchase any GPU that use GDDR6X memory because it runs so much hotter. The t-max is 110c, and they all seem to run very hot at stock frequencies.

This is all true for the big 3-4 pci slot gaming cards. Your blower card choices are all 2 slots. There's just not enough heatsink there to dissipate that much heat very quickly.

 

[TangTasteTester]

The honest answer is that it's the wrong card and cooling solution for the chassis.