40KW - 50 KW rack designs. Can you actually cool it?

[Colin]

Currently evaluating 40kw and 50kw rack designs. I was wondering if anyone had any real world experience or thermal data they could share. We always see reviews of 1U or 2U being evaluated but looking at deploying 500kw in 10 racks or so, and want to know that we can actually keep this thing cold. We also would like to run a hot cold isle, so would prefer to be able to get this done without the cold aisle being super cold. If you had any issues could you explain, so we dont get stuck with a non viable design. Currently we have only ran up to 10kw per rack, so stepping to 50kw is a big change for us.

Thanks,

Colin

 

[Colin]

I should also specify we have liquid cooled (rear door) racks at the moment but would like to do this as on all air if possible.

 

[Patrick]

I know some of the deep learning designs are running at 40-50kW+ and have no issue with running aisles of them on air. Usually, they are running 48U racks and have full containment on the cold aisles.

 

[modder man]

Currently evaluating 40kw and 50kw rack designs. I was wondering if anyone had any real world experience or thermal data they could share. We always see reviews of 1U or 2U being evaluated but looking at deploying 500kw in 10 racks or so, and want to know that we can actually keep this thing cold. We also would like to run a hot cold isle, so would prefer to be able to get this done without the cold aisle being super cold. If you had any issues could you explain, so we dont get stuck with a non viable design. Currently we have only ran up to 10kw per rack, so stepping to 50kw is a big change for us.

Thanks,

Colin

Unfortunately I dont know the exact configs of our cooling because I dont work in that area. I do know that our hadoop racks are designed to be 40-50Kw per rack.

 

[Colin]

Patrick we are running something similar. Any contact would be appreciated or really I just need to know what their upper limit on their cold aisle is.

 

[Evan]

40-50 is well beyond anything I have been involved in.
20-25kw on air with contained cold isle, temp is not as important as sufficient volume.

Rear door heat exchangers seem to be the go-to thing when your trying to dispose that much heat.

It’s pretty easy to build a 35kw rack equipment wise but at some point about half way there it’s usually easier to find more racks.... what do you fill a rack with to get to that high spec ?

 

[Colin]

Pretty much any modern 1U or 2U GPU node running flat out will pull over 1kw per U.

 

[Evan]

Pretty much any modern 1U or 2U GPU node running flat out will pull over 1kw per U.

Kind of realised that after I said it but so far in my enterprise work even for HPC the density is not that high.
2 pairs (so total 4 connections rudendant) of 400v 3phase to power it ?

 

[Colin]

Probably not. We are cheap and I try to get 208 dropped before the meter, that way I dont have to pay for the losses. The new raritan PDUs are thin so you can get 4 in a rack. We dont run redundancy. Dont see the point; Ill take the down time vs paying $1000kw/hr for the 1 day we actually use it.

 

[Blinky 42]

We are in several commercial colo facilities that are capable of 50kW+/rack with forced air - they were all (re-)designed with full containment to get the most air through the cabinets as possible. Caged areas vary based on the vendor but typically mirror the open space or are customized to work with the cage occupant's equipment to continue the isolation of hot and cold isle (I have seen some aggressive use of plastic curtains over the years for cages). The cold isle in the locations I have been in are typically warm vs. super cold - the key is cycling the air through often enough to avoid overspending on cooling it too much.

If you are building the data center yourself, more of a question for your cooling people on how to design it to support the load. You should also consider downtime / maintenance of cooling units and have a +1 in there to avoid spinning down a large % of your compute because cooling failed.

 

[Colin]

Thanks Blinky. We use TAS and they have done a great job for us building out the space. We are evaluating using free cooling. I am trying to get away from chilled water. I hear what you are saying about N+1 and if that is how we go that would be a necessity. The more and more I get into HPC, I realize how much infrastructure I have to build to just run one GPU and it makes me sick. I am trying as hard as I can to keep it simple.

For forced air, did they have their own air handler in the space or was it mounted on the rack? I like the concept as long as it is off the rack.

 

[Blinky 42]

I have not seen installs needing special cooling related equipment in the racks with normal looking servers/blades. Just normal 48U (or more) cabinets, careful attention to blanking panels to keep air moving through equipment if not totally full. The hot isle is warm++ and downright windy. Not somewhere you want to hang out, but gets the job done.

The building design dictated what they had to do with ducting or blocking off the ceiling to get the air around, and also what they had to work with for power and ladder racks for fiber etc.

 

[Evan]

If you look at some photos of high density DC’s I have also seen the cold isle like a kind of tunnel where the are is pushed from the end all the way down rather than above or below.
Again it’s all about volume of cooling and separation rather than Low temp, Low temp just buys you a few more minutes when you have a failure.

At least for what I do we looks at n+2 or 2n cooling in terms of both chillers and handling units.

 

[cheezehead]

We've been using drop curtains for a couple years to help force the air front-back through the racks paired with heavy use of blanks. This has allowed us to raise the room temp a couple of degrees while still having the equipment cooler. As a bonus a lot of the 10k humming is absorbed by the curtains so it's quieter now.

Not running that high of a kw density here either, as others have mentioned airflow is key.