Watercooling rack.

[RimBlock]

First off, S0lid has a thread on his build over here. Take a look as there is a lot of good stuff in there.

I am looking to go a different way hence this thread.

I am looking to approach this in a few of phases.

1. Get a basic loop setup for the rack - Pretty simple (hopefully).
2. Integrate a rad system for pre-cooling air going in to the server (Single server).
3. Improve the precooling to cover multiple servers.
4. Integrate direct cooling for a single server.
5. Integrate direct cooling for multiple servers.

My watercooling knowledge is minimal and so some help would be appreciated on flow rates and relevant calculations.

Index of sections (will expand as things progress).

1. Scratchpad of ideas.
2. Interesting links.
3. Parts list.
4. Phase 1.
5. Phase 2.
6. Phase 3.
7. Phase 4.
8. Phase 5.

 

[RimBlock]

Scratch Pad

Initial thoughts and directions for investigation

• Underfloor heating / cooling systems (possibly PEX based).
• Garden automatic water distribution systems.
• Fishtank pumps (Fluval 406 - specs below).
• Fishtank chillers.
• Car radiators (possibily a bit much )

Possible Issues.

• Condensation
  
  Ok, condenstation forms at the dew point which is related to the ambient temps and the humidity. Here in Singapore the humidity is 59% and the ambient tems are around 32DegC (90DegF) which puts the dew point arounf 24DegC (75DegF). So to be safe, the water temps should not be lower than 26DegC.​
• Water Pressure

Initial thoughts on design.
Rad at the top of the rack, possibly on the fan plate.
Top fans blowing vertically out of the top of the rack.
Fish tank canister filter at the bottom of the tack pumping the water. Possibly 2 in series with a top up reservior in between.

Sample Pump (Aquarium) - I have two of these for our turtles (don't ask, long story as to why we have them.... ).
FluVal 406
•Power: 20W
•Max Flow Rate: 1450L/Hr
•Flow Rate: 930L/Hr
•Max Head: 2.25m
•Filter Volume: 8.5L
•Dimensions: 193x368x452mm

Looking at Frozen CPU watercooling pumps spec sheet found on this page, it seems that standard watercooling pumps have a fairly high head (water flow rate) but a lower throughput. For comparison 7.38 Max Head (Feet) on the chart would match the above pumps 2.25m.

Temps / TDP
Intel L5510 - 60W
Intel 5520 chipset - 27.1W
LSI 9202-16e - 17W (ok it is PCI power but best I have found so far).
C6100 Infiniband Mezzanine card - ??????
C6100 Infiniband LSI 1068 SAS card - ?????
C6100 Infiniband LSI 2002 SAS card - ?????
C6100 Infiniband 10GbE card - ?????
C6100 node (dual L5520s) - 175W (CPU & Chipset)
C6100 with 4 nodes (Dual L5520 / node) - 700W

Delta-T (on-line calc here).
Difference between pre and post cooling water temps. 10DegC or less is considered ok, 5->10DegC is considered good.

 

[RimBlock]

Interesting Links

• Calculating the dew point where condensation forms based on ambient temps and humidity here.
• Some PEX heating and cooling system info here.

 

[RimBlock]

Parts List

 

[RimBlock]

Phase 1

Stuff

 

[RimBlock]

Phase 2

Stuff

 

[RimBlock]

Phase 3

Stuff

 

[RimBlock]

Phase 4

Stuff

 

[RimBlock]

Phase 5

Stuff

 

[nry]

Subscribed

 

[vegaman]

Pretty cool idea. I wonder if a 1u chassis would have room to fit the tubing? I guess you'll have similar size constraints with the C6100 blades though.

My first thought was to put the radiator at the bottom because it's cooler. But the top might be better so it can work with convection. This is just musings though as I don't have enough knowledge of these things.

Edit: just reread more carefully and found you're not planning on water blocks, interesting.

 

[rnavarro]

I probably wouldn't do water blocks either....for small rackmounted gear it's hard to route the piping like you mentioned.

I was thinking more along the lines of sealing off my rack and having it a self contained unit, like a fridge almost.

But I haven't figured out a really efficient way of cooling the air properly.....all just musings

 

[RimBlock]

I probably wouldn't do water blocks either....for small rackmounted gear it's hard to route the piping like you mentioned.

I was thinking more along the lines of sealing off my rack and having it a self contained unit, like a fridge almost.

But I haven't figured out a really efficient way of cooling the air properly.....all just musings

Pretty cool idea. I wonder if a 1u chassis would have room to fit the tubing? I guess you'll have similar size constraints with the C6100 blades though.

My first thought was to put the radiator at the bottom because it's cooler. But the top might be better so it can work with convection. This is just musings though as I don't have enough knowledge of these things.

Edit: just reread more carefully and found you're not planning on water blocks, interesting.

I was considering water cooling by blocks for phase 4 & 5 (direct water cooling). I dont really have the tools to build my own waterblocks so would have to source something off the shelf. So far the SwiftTech ApogeeLp seems the best bet (low profile with horizontal entry and exit ports) but the ports look fixed (can't swivel) and it would depend on the amount of extra room was available after the 1U heatsinks were removed and the smaller waterblock was installed.

The sort of thing I would go for with waterblocks is the system used in the SuperMUC Super Computer in Leibniz (Germany). It uses hot water cooling where the water temps can be hotter than room temps because, unlike overclockers, they are trying to keep the system processors running at below but close to their recommended max. The reason is that less effort is required and the systems work fine. The graphics card water blocks look much better with in/outlet positioning but I would probably need to fabricate a bracket them if I went that way.



There is a good pic of the system board of the MUC here and here (neither site is in English) with the thin copper piping.

 

[RimBlock]

After a bit of searching around, it looks like I am going phase 4 first .

It occurs that Phase 2 - Blowing air at the server through a rad with water going through it is not going to do much more to cool the server as the water will be at min room temps without a chiller.

To this end and after much hunting around, I have purchased two Corsair H60 all-in-one units (both very cheap and one not working) to take apart and hopebully utilise the CPU water block. If it is too tall then I will look to re-engineer the base plate in to a smaller top unit without a pump.

I have also ordered the following;

Branch from main rack feed to individual server / node.

Product	Model	Quantity
Fitting Body, 4-Way Brass	ADT-XUTX4	2
Fitting Single, Barb for ID 13mm (1/2in)	NZL-V13B	4
Fitting Single, Barb for ID 10mm (3/8in)	NZL-V10B	2
QD3 Male Quick Disconnect No-Spill Coupling, Panel Barb for ID 10mm (3/8in)	QD3-M10-P	2
QD3 Female Quick Disconnect No-Spill Coupling, Angle Compression for 10mm x 13mm (3/8in x 1/2in)	QD3-F10X13-L	2
Tubing Adapter Single for 10mm x 13mm (3/8in x 1/2in) to 13mm x 16mm (1/2in x 5/8in)	ADT-10T13S	2
Tubing, Clear UV-Reactive PVC, 10mm x 13mm (3/8in x 1/2in)	HOS-13CL	30ft
Tubing, Clear UV-Reactive PVC, 10mm x 13mm (3/8in x 1/2in)	HOS-10CL	15ft
Hose Clamp for OD 13mm (1/2in)	CLM-10	10

Main rack frre to server / node
Fitting Body, 4-Way Brass
Fitting Single, Barb for ID 10mm (3/8in) - To node
Fitting Single, Barb for ID 13mm (1/2in) - Main feed

Feed spur into the server / node via quick disconnect for easier maintenance.
QD3 Male Quick Disconnect No-Spill Coupling, Panel Barb for ID 10mm (3/8in)
QD3 Female Quick Disconnect No-Spill Coupling, Angle Compression for 10mm x 13mm (3/8in x 1/2in)

Feed to radiator (1/2" feed -> 3/8" rad).
Tubing Adapter Single for 10mm x 13mm (3/8in x 1/2in) to 13mm x 16mm (1/2in x 5/8in)

Misc and hose.
Hose Clamp for OD 13mm (1/2in)
Tubing, Clear UV-Reactive PVC, 10mm x 13mm (3/8in x 1/2in)
Tubing, Clear UV-Reactive PVC, 13mm x 16mm (1/2in x 5/8in)

Need to get a few more items I think. Just working the flow through now.​

 

[RimBlock]

It seems from my research, most all-in-one units are amde by either Asetek or CooliT and re-branded.

The Corsair is made by AseTek who also do 1U server kits so I would expect them to be simular.

I am also seriously considering a desktop 4-axis CNC mill . I would then be able to mill my own blocks / cold plates / acrylic, plastic or Alu tops. The machine is not cheap but if I can also pick up a few small custom part milling jobs then I may be able to make it pay for itself.

I have always loved CNC machines so it is a great excuse but it will be a steep learning curve with the CAD and Mach3 control software.

Most importantly.... the wife has not said no . Not sure if she understands what I will be bringing in to our apartment (no basement / attic space or shed).

 

[RimBlock]

I hope to have the parts arrive in the next week or so.

I have bought a 4-axis CNC milling machine (X6-1500GT) and am planning on milling my own alu waterblocks utilising the Corsair all-in-one copper heat plates with it. It is still a challenge with the 25mm height restriction but at least I can have flater, wider channels between the cpus inside the block (plate) rather than hacing to mess around with barbs and standard fittings.

The CNC machine is a X6-1500GT.

 

[OBasel]

I can't wait to see this. What happens if something breaks and how do you stop it from taking out other machines in a rack?

 

[RimBlock]

The blocks (plates would probably be more accurate) will be two piece (top & bottom) with a rubber seal or gasket between them. The end will be connected to a small pipe which would link to the quick release bulkhead connenctors exiting the server. As long as the gasket is in tact then all shouild be good. The PoC server will be on the bottom of the rack and initially only a single node will be connected (one of the bottom nodes ). The C6100 will not be turned on as the water will be coming from a pump external to the chassis so I can have an extensive burn in.

Building in some sort of water setection alarm would also probably be worth while and I understand there are also pressure drop alarms available. Will any alarm stop a catastrophic flood fubaring everything.... nope, probably now but if I do put in a pressure drop sensor and connect it to a IPMI based shutdown system for the servers and a dump valve allowing the water to drain safely but quickly out of the system if a leak occurs maybe possible. Maybe a sensor / shutdown / dump option on a per server basis.

Another option would be to have a module for each server that featured a pump and heat exchanger with a cold feed going in to the other side. The amount of water in any given servers closed circuit would be fairly minimal and the external flow of cold water int eh heat exchanger would cool the internal circuits water. It would be more expensive but less chance of seeing a cabinet full of sparks if something developed a leak.

It may also be possible to have a thin tray for eack server with a drain hole to take any leaked water away rathe than it dripping on the machine below.

Lots of options but that will be for later. First I need a basic setup and the blocks built to see how well it works.

RB

 

[vv111y]

Nice Rimblock , I've been thinking of DIY watercooling too. Subbed

- pex vs copper? and why use that vs smaller braided tubing with clamps on the end for both rack and in server boxes. I think flexible tubing is the way to go. fyi Pex is pricey vs copper.

- quickconnects? how good is the quality/risk of leak? are they really necessary?

- when I was looking into it I found info on block design. one manufacturer showed an animation of turbulence flow with claimed improved heat removal vs a winding channel. I'll look up the links

- why Alu block vs copper?

- the MUCS system is neat. Their advanced, fractal like micro-channels is beyond us. I wonder how much that is actually necessary.

- the milling machine $$ ouch. I'll be looking for a cheaper solution.

- under trays per box does seems like a good idea, depending on confidence of finished product.

I seriously wonder how performant the blocks need to be. I suspect even at it's worst, watercooling is better than any # of fans.

Thanks for all the great info

EDIT: saw the car quickconnects. attractive idea, want to find cheaper.
EDIT: saw pex quickconnects. cheap.

 

[RimBlock]

Nice Rimblock , I've been thinking of DIY watercooling too. Subbed

Cool, the more people that join in, the more info that will be available for others who wish to take a look in the future .

- pex vs copper? and why use that vs smaller braided tubing with clamps on the end for both rack and in server boxes. I think flexible tubing is the way to go. fyi Pex is pricey vs copper.

The main reason I would use copper tubing would be to remove the need for barb/compression fittings but then I would have to solder them in and I trust barbs / compression fittings more than my soldering of pipe work at the moment. That may change in the future though.

- quickconnects? how good is the quality/risk of leak? are they really necessary?

I think so for a couple of reasons.
I can disconnect a server / node without affecting the loop in the rack and so the other servers.
Neater than having a shutoff valve for each feed to each machine / node.

The quality of the QRCs is still to be seen but I will assess on the initial build. Other options are welcome .

- when I was looking into it I found info on block design. one manufacturer showed an animation of turbulence flow with claimed improved heat removal vs a winding channel. I'll look up the links

Autodesk does a piece of fluid dynamics simulation software that would have been interesting. Way outside of my budget though . If anyone has any other ideas of free / open source fluid dynamics modelling software (could happen ) then please let us know.

There are a number of Corsair H60 teardowns on Youtube. From those it looks as if the cool water enters from above the center of copper plate and they use a flash plate to distribute the water in a horizontal line into the shaped surface of the copper plate where it is then squeezed out to the edges and then out of the outlet. I will probably do the same but without the bulk for the waterblocks pump needed with the H60 AiO setup so it should hopefully fit the 25mm space.

- why Alu block vs copper?

Price and availability. If you are able to source 1/2" copper bar cheaper than Alu then please let me know. I am still searching (locally and internationally) for a cheap supplier of offcuts I can use. I have found a few on EBay but a lot have high shipping prices ($8 for the alu block + $24 for US shipping).

Copper tends to corrode more easily. Alu is a little easier to work with (so I have read).

- the MUCS system is neat. Their advanced, fractal like micro-channels is beyond us. I wonder how much that is actually necessary.

Yes agree on both counts.

- the milling machine $$ ouch. I'll be looking for a cheaper solution.

Well I have always wanted one of those Mechwarrior (Battletech) robots as a model so this is just an excuse to get a machine and practice to I may be able to one day build a scale version myself (maybe 1 mtr tall). Someone has done one via 3D printer that looks good but I would prefer one with Alu. I love small detailed models, not that I would claim any ability of being able to create them, and with a CNC mill, a number of options open up. I could even start looking at PC cases for a couple of other ideas I had but got stuck on due to there not being any cases out there.

- under trays per box does seems like a good idea, depending on confidence of finished product.

Thin, low wall, cheap and with a drain would do I think. I think it has advantages and disadvantages of a commercial product based around confidence in the product over being serious about protection over your clients other machines in the rack.

I seriously wonder how performant the blocks need to be. I suspect even at it's worst, watercooling is better than any # of fans.

I would also tend to agree. Especially for my L5520s, I think there is minimal design needed to achieve minimal requirements. A good place to start. This also then brings forward the question of how much water needs to flow to allow for adequate cooling... The less water, the smaller the tubing. Looking at the MUC, it seems Intel believes that not much is needed considering the size of the piping in the nodes. Of course, using brazed / soldered piping may mean they are using higher pressures for faster flow rates .

Another consideration is having enough tubing to handle sliding rails and a way of managing it in the rack. A cable tidy extending arm may be needed for servers on rails.

Lots of challenges beyond the initial waterblocks.

RB