Watercooling rack.

[vv111y]

couple links broken on Tom's sticky. Couldn't find the spreadsheets easily.

Here's a spreadsheet from an xtremesystems user. You have to click the 'accetto' button before the download button Download

source: Water Cooling Estimator Spreadsheet

looking through the pump guides, I like this
still good guide.

[mods: can we upload files to the forums?]

 

[vv111y]

All you ever wanted to know about water

PART I
PC Water Coolant Chemistry – Part I | Overclockers

PART II
PC Water Coolant Chemistry – Part II | Overclockers

 

[RimBlock]

Thanks for all the links. The more info the better.

I have come across Toms guide but have issues viewing from the office .

The CNC machine has arrived and I have now assembled it.

It is in the cardboard box in order to catch the chips for the short term but I visited a plastics distributor and bought some acrylic including parts to make a cover for the machine.

I am now looking to cut slots in to 20mm X 20mm bars to slot the sheets in to but the clamps I have will not go to 20mm so I need to get some longer bolts tomorrow.

I have spent the rest of the day getting the machine setup and watching videos on using the Mach3 CNC software. The CNC machine comes with no instructions or getting started guide, which I knew before buying but considering anything else close is a lot more expensive I still decided to get.

I am fairly close to being able to actually get it to do something now but I need to home the axis without endpoint switches (a later upgrade) and the jog is ridiculously slow. I also do not seem to be able to change the feedrate (which would increase the job) for some reason.

Learning is fun .

RB

 

[RimBlock]

The Corsair H60s have come in.

I have now taken the broken one apart and this is what is found inside.

Bottom plate

Other side of the bottom plate

Remove the seal and silicone water guide. Lots of corrosion.

Inside the plastic block. More corrosion. The wires in plastic on the bottom right are a temp probe.

A bit of work with a scouring pad and we are looking a bit better.

Taking the top off, the impellers electronics is revealed.

Taking out the screws we can remove the hub and see the impeller blades.

Removing the impeller out of the way we can see in to the chamber. More corrosion.

Underneath the block with the silicone piece back shows how the feed from the top is directed by the silicone in to the slit and then forces through the slits in the copper plate out to the sides and exits through the bottom left.

A comparison between the 1U heatsink and the H60. The green liquid in the bowl is the antifreeze that came out of the closed loop. The H60 is 3mm taller, which in the 1U world is just too tall.

A real shame about the height, but if I can take a bit off the top and sort out a new mounting bracket and seal the impeller chamber without the impeller. On the other hand the design seems to be a little flawed. The cheap sealed box I got from Ebay was from New Egg and is actually a refurbished unit.

RB

 

[Patrick]

That is really cool. May stick that up on the main site. That is a refurbished H60 in all of those pictures?

 

[RimBlock]

Nope that is the broken one I purchased for around US$30.

The refurbished was a bit more expensive and is still boxed up.

RB

 

[mackle]

I'm probably a little late to this, but for your 1U servers I'm curious why you didn't look at smaller od/id connections and tubing? 1/4" with a rack mounted manifold wouldn't be too restrictive if they were in parallel, but would be a lot easier to accommodate internally... In previous builds I've used 19mm outside the case (between computer and cooling unit) and 10mm inside the case (I doubt there is very much Venturi effect either way).

I'm a while off undertaking a rather similar rack mounted project (I have an international move to undertake in the new year), inspired by Yes, Another Car Radiator Thread--56K Warning! - [H]ard|Forum and The Hole…Geothermal Loop…56K Warning! - [H]ard|Forum. But with 1U being so difficult to work in and no existing hardware to accommodate I'll probably limit myself to cooling 2U and 4U environments, with 4U ideal for multi-GPU cooling. My 1U's, for firewalls etc, will be short, with atom type CPUs that really don't justify liquid cooling.

Have you thought about a Rasp pi monitoring node with sensors, automated valves, relays? I plan to have fail-safes and a plate exchange to isolate computer loops from cooling loops.

 

[RimBlock]

I'm probably a little late to this, but for your 1U servers I'm curious why you didn't look at smaller od/id connections and tubing? 1/4" with a rack mounted manifold wouldn't be too restrictive if they were in parallel, but would be a lot easier to accommodate internally... In previous builds I've used 19mm outside the case (between computer and cooling unit) and 10mm inside the case (I doubt there is very much Venturi effect either way).

I may look at going smaller due to the flexability issues with placement, Intel demonstrating it can be done with the SuperMUC and after taking apart the H60 and seeing how small the internal pipe / channel dimentions are and how small the section of the copper cold plate exposed to the liquid coolent is. Current thoughts were to use 3/8" (10mm) ID for internel to the server and 1/2" (12.5mm) ID for the rack. I will measure tonight but the H60 looks to have been using around 2/8" (6.35mm).

I am wondering if, rather than forcing the liquid coolent in to the centre of the cool plate, forcing it from one side to the other through the slits would be good enough. It would make it much simpler to build a very low profile acrylic top. The only real problem with most of the barbs etc are that they usually come with G 1/4 threads which along with the collars used to grip and screw them in take up a big chunk of the 25mm max height. Maybe a central pyramid so the G 1.4 threads can be mounted at an angle so their size is not all vertical and using one of the pivoting barbs, a 30 degree angle would still allow the pipe to run horizontally. Bit more Design Spark tonight I think.

I'm a while off undertaking a rather similar rack mounted project (I have an international move to undertake in the new year), inspired by Yes, Another Car Radiator Thread--56K Warning! - [H]ard|Forum and The Hole…Geothermal Loop…56K Warning! - [H]ard|Forum. But with 1U being so difficult to work in and no existing hardware to accommodate I'll probably limit myself to cooling 2U and 4U environments, with 4U ideal for multi-GPU cooling. My 1U's, for firewalls etc, will be short, with atom type CPUs that really don't justify liquid cooling.

That would make a lot of sense. I will have a good read of that thread. A car rad was a thought I had originally. My 2U DL380 G6 would be a dream to work with for watercooling. there is a pretty unobstructed path from the two, side by side, cpus to the rear of the case, but if I do the difficult ones first, the hope is that the rest will become quite a bit easier . I also have had some interest for a system that would fit the C6100 (all 4 nodes) and would go in my custom made 9U server racks for a wheel around demo unit. That would probably be an interesting project to expand on in future.

Have you thought about a Rasp pi monitoring node with sensors, automated valves, relays? I plan to have fail-safes and a plate exchange to isolate computer loops from cooling loops.

Sounds like a good idea although I am not much of an electronics person. I have to do a bit of playing around in order to work out how to attach limit switches to the USB CNC controller yet and that is meant to be pretty simple. I am sure I can get it done but it is not an area I have explored much in the past.

RB

 

[RimBlock]

A small update.

I have got the CNC to make a small cut in acrylic.

The CNC bed is not level to the gantry. I need to either put a piece of acrylic or aluminium there and then face it to make sure it is all level, but this would mean I would also loose the ablility to use the slots on the bed for holding the work pieces and would have to drill holes to screw down holding blocks. THe other option is to put a packer on the lower side of the gantry to lift it up slightly. This would be easier, less messy but possibly slightly less accurate.

The spindle does not have a rpm feed. If I want to know how fast it is going I will probably need to wire up a hall sensor and put a magnet on the spindle so the sensor can count the revs.

I am looking for replacement motor cable as the stuff supplied is not very good quality. I am also looking for the limit switches, possibly using hall sensors again, so it can auto home.

I also need to cut some holding blocks as I can not hold the pieces steady very easily.

The 3D cad stuff I am getting to grips with but the CAM software that comes with Mach3 is 2.5D only. This means it will generate the tool path code to cut out only flat shapes. To cut out more complex 3D shapes I need a better set of CAM software. I am looking at MeshCam and CamBam. MeshCAM also have a version that will take a 2D drawing and help you convert it in to 3D which sounds funky but how good it works is another matter. Both of these sets of CAM software are a few hundred $$.

I also need to make the box around the CNC. My 2yo is facinated by it so it is needed for safety first and foremost but also to try and contain the chips. I did get some 6mm acrylic to make a case but I now think polycarbinate twice as thick would have been a better choice with some alu channels to join all the bits together. Oh well, lesson learned.

RB

 

[RimBlock]

So, I am trying out MeshCam and decided to try an do an acrylic plate with two indents where the CPUs would be and a channel between them and an entry / exit channel. It was only a quick and dirty design and will not work as there is no in/out flow seperation but it was purly to get a feel for the CAD / CAM and milling software and to try to work out how fast I can mill acrylic.

I started at around 2mm/minute. It was so slooooooooooow. Each pass does 2mm and I was looking for a 9mm depth (12mm acrylic block). I stoped it and changed the rates. ramped up to 20,000 rpm on the spindle and 2,800mm/min. Now we were flying. It was looking like the youtube cnc videos....

Observations.
Meshcam allows you to set exclusion zones so you can 'off-limits' the areas you are using to clamp the material down which is great but if you dont want to mill the whole block you have to make sure everything apart from where you want to cut is excluded. I have some fine grooves along the side of the material where it was trying to cut the outside edge of the block down to the size of my fairly inaccurate CAD model.

It creates a lot of mess. It was like it had been snowing. I really need to get the machine cover built to contain the chips as well as for safety. It will also hopefully cut down on noise. at 20,000 rpm it sounds like a circular saw table cutting wood, constantly for a few hours.

Unfortunately the finishing cut it does at the end which is very fine goes very slowly and I left it running overnight. When I woke up I found that my new PC I build to control the mill had rebooted after a Microsoft update and left the CNC stationary but with the spindle still spinning at 20,000rpm..... all night..... I will be setting it to the 'download and tell me' Windows update scheme rather than the current "install and reboot".

I have a tapping set on the way so I can put threads on drilled holes. I also have a set of collets (for holding different sized endmills) and a selection of endmills (cutting bits) on the way.

RB

 

[vv111y]

What size reservoir are you using? I saw a [H] thread where he used a 55gal water barrel. That seems unnecessary to me.

 

[gigatexal]

Man, this looks way cool. C6100 watercooled is just awesome.

 

[mackle]

Nice updates.

About vv111y's comment on the size of the reservoir - I had been wondering about what sort of load your system has, as it is important for determining the cooling. The guy on [H] was using his 55 gallon drum as a large heat store IIRC. A res that large takes a seriously long time to heat up and cool down, which evens out the daily fluctuations (and intermittent variations, although from memory in his case they were running constantly at 100% load though). Not dissimilar to the Nest thermostat (I think it is), which has their demand management function that lets them pre-cool your house prior to the forecast AC peak and then shut it down while everyone else is loading theirs.

For my old i7 920 WC set up (OC'd to 4.2GHz and folding 24/7) I had a 3 litre res, along with about ~10m of 20mm of tubing. It took a good 15 minutes of 100% load from stable idle for the temperature to stabilise at its operating temperature. There was also an interesting lag between air temperature and system temps over the course of the day. A larger tank would have allowed me to pre-cool the system overnight while air temperatures were lower and soak up the heat during the day. I had a cooling unit consisting of a free second hand heater core and ended up running the fan only intermittently during most of the day (eg 5 mins on/15 off) as it kept the CPU that much cooler than my Noctua. It was more efficient that way as the fan ran while the differential between ambient & fluid temps was higher (by a couple of C) than if it was constantly running. It was a manual config using a 24hour timer, based on my historical monitoring log data (at the time I didn't have the money or time to set up an automated interface for it).

Thinking about the mobile rack you mentioned, mounting your rads at the bottom back of your rack with a res occupying some of the space in front (still letting enough air through) would look slick. Should only take up 3U as well. Fabricating a shroud to hold the fans back from radiator would help eliminate the centre deadzone where the fan motor/hub sits. I'd definitely recommend mounting the res and radiators at the bottom if it has a good seal, even if it means you can't have an upward venting radiator top of stack. If your load usage is intermittent then a decent sized res should allow you to ride out the fluctuations better and potentially get by on a smaller/fewer radiator(s) (sized only for worst-case scenario), should your cooling requirements increase/be insufficient you could look at adding an additional radiator and re orientating to the sides of that rack level.

That said, a smaller res would allow you to potentially mount a switch and/or short depth chassis in front, but might limit your ability to add additional radiator capacity (short of sandwiching).

 

[RimBlock]

What size reservoir are you using? I saw a [H] thread where he used a 55gal water barrel. That seems unnecessary to me.

I am looking at using an aquarium canister filter. Res and pump all in one with shutoff and disconnect all built in. Probably a Fluval 406 to start as I have a couple for our turtles (don't ask how we ended up with them ... lets just say my wife can be impulsive and they looks so cute when they were the size of a quarter) so I have some experience of them. Think I have already posted the specs but here they are again.

Flow Rate: 1450 LPH (383 US GPH)
Head Height (max.): 2.25 m (7.5 ft)
Media Capacity: 8.5 L
Wattage: 120 V/60 Hz – 23 W and 230-240 V/50 Hz – 20 W
Dimensions (L x W x H): 25 x 20 x 50.8 cm (10 x 8 x 20 in)

So it has 8.5L canister capacity with enough pressure to push to the top of rack give or take any pressure reducing bits I put in). I still have to check the hose diameter but the other advantage for me is all the filters etc I will take out and use for spares in my own turtle filters.

One thing about these canster filters is that they are tall. The 406 is around 10U plus connectors and pipes coming out of the top. This is fine for me as my rack is fairly empty but I may have to look at a wide / deep solution after the initial PoC is done.

My main effort at the moment is getting a safety case built around the CNC machine. This would seem to be fairly trivial to most but in Singapore, I have no workshop, shed, loft, basement etc so I am very limited as to where I can do any cutting and building. Materials are not so easy to get either. No big DIY Superstores where I can get the acrylic, alu channel, silicon etc, everything has to be sourced from multiple specialist shops all over the island.... It is a pretty big PITA, especially without a car (for those not aware, to give an idea, a BMW 320i over here is approx US$215,000 new. An entry level Chevolet Cruze is around US$91,000. Cars are also scrapped after 10 years unless they receive 'Classic' status).

Man, this looks way cool. C6100 watercooled is just awesome.

. Some way to go yet though. Have some more endmills (CNC cutting bits) on the way. Feeling more confident witht he tool now though and the CAD / CAM workflow.

Nice updates.

Thinking about the mobile rack you mentioned, mounting your rads at the bottom back of your rack with a res occupying some of the space in front (still letting enough air through) would look slick. Should only take up 3U as well. Fabricating a shroud to hold the fans back from radiator would help eliminate the centre deadzone where the fan motor/hub sits. I'd definitely recommend mounting the res and radiators at the bottom if it has a good seal, even if it means you can't have an upward venting radiator top of stack. If your load usage is intermittent then a decent sized res should allow you to ride out the fluctuations better and potentially get by on a smaller/fewer radiator(s) (sized only for worst-case scenario), should your cooling requirements increase/be insufficient you could look at adding an additional radiator and re orientating to the sides of that rack level.

That said, a smaller res would allow you to potentially mount a switch and/or short depth chassis in front, but might limit your ability to add additional radiator capacity (short of sandwiching).

I can see the main problem being the rear door blocking the hot air escaping where as having the rads at the top would allow venting vertically out through the vents for the top fan panel. Having said that, I do like the idea of having the water as low as possible to try to avoid a shower situation due to any critical failures.

Oh and my Makita cordless hammer drill (like this but clean and looked after Makita 8400D), given to me by my father when I moved over here 8 years ago, has finally given up. Well TBH it is probably just the batteries that have died after serving my father and then me pretty well for lots of years. Whilst I could get a replacement battery, I deviced to take a look at what is currently on offer. I ended up getting a Bosch GSR 10,8-2-LI Pro which is tiny but is sort of like a Star Trek phaser (it even has a light) so I just had to buy it . Seriously though it is quite nice, tiny compared to the Makita with 10.8v rather than the Makitas 9.6v. The Makita has the hammer action for concrete but if I am going to drill concrete I will use my GSR 21-2 RE Pro.

RB

 

[RimBlock]

Finally got some sort of decent protection around the CNC.

The top side and front panels are removable.

My kill coil finally came for the water cooling of the spindle so that is now in a sealed container with the water pump.

I also got a jigsaw for cutting up the large pieces of acrylic in to pieces the CNC can deal with. So much easier than using a hand saw and much more accurate... for me at least.

Finally I can start to think of getting some water blocks cut.

I am currently considering building a single node watercooled unit to get this started and a bit more experience with watercooling before shrinking it down to the 1U size. I have a few ideas including a double loop with different coloured coolant for each loop.

RB

 

[Lost-Benji]

Fark me RB, when you said other thread, I didn't realize you had this big bugger already.....


Forgive me if I repeat something that may have already been said.

Head pressure is the key here. I am not talking about the pumps either. What I am talking about and needs to be considered in great detail is the average 42-47U cabinet means that you will have water height of a couple of meters at best. This may not sound dire but if you do a little test by getting say 3M of PVC hose, fill it with water and then try keeping a finger on one end while the other is raised to 2M. Feel the pressure? Weak joints or blocks will pop.

I have seen blocks pop open under 2M of head.

I have mentioned copper tube as to be the best option for multiple reasons, the foremost is its strength. Cheap hoses over time can swell and with added pressure and heat, it will happen sooner. You are also adding more fittings inside the system where leaks become a bigger threat. Brazed joints and water blocks FTW.

The reason for the copper is the corrosion issue. As RB indirectly discovered with his sealed system pull-down, dissimilar metal corrosion is the biggest killer in WC systems or anything for that fact. You need to use the same metal throughout the systems. Copper blocks, copper tube and copper header lines. Plastics are fine for pumps, tubes outside of server and the reservoirs. Radiators are harder to deal with as making sure the grade of copper is actually copper and not a copper alloy.
The system that RB show was classic corrosion, I would say the pump shaft is metal and the top and bottom plates maybe different alloys.




OK, now my reasoning for WC a rack.
My little DC lives in my back shed that is 60M away from my front office and is also exposed to the humid air of Queensland but also gets some sea breezes as we are only 30KM inland of the coast. The shed is dry and solid brick but the roof is Iron and no insulation as of yet. What my goals are is to actually seal the rack (air tight as much as possible) and run a heat exchanger style system. Heat vented from rear of servers goes to rear of cabinet, this then rises up to where I would have a top mounted heat exchanger that strips out most of the heat and then directs cooler air down over the front of the cabinet back into the servers. This would mean full air cooling of the units and no mods. The heated water would then go into a heat exchanger that would allow me to pump coolant out through a stainless loop sitting in the bottom of my swimming pool. The trick will be getting my metals right to avoid corrosion.


There was another concept I came up with a while back to eliminate the head-pressure issue. it was to run a small loop through the server across the CPU's and other hot areas (DIMM's and Chip-sets) then out the rear to a small pump which then fed it through a heat exchange coil/loop inside a bigger header tank made out of 4" copper pipe in vertical fashion. This bigger pipe would act as a thermal mass but also as the res. Hot water at top would be sent off the forced air-cooled radiators to get the heat away.

 

[mackle]

I had a WC rig with over a metre of head on it – the computer is in the ceiling of the garage below the living room TV and because it gets quite hot up there I took the cooling unit down under the house. For pumps, the thing to remember with a closed loop is that it only exhibits friction head, with the water being pushed up offset by the water ‘falling’ back down, so pump head is not very important. Plus flow rates don’t really have a huge impact on temps.

Because parts are so expensive down here I actually just used regular home irrigation components for the build (never mind how damn expensive copper is these days). I was able to find clear tubing at the hardware store. With 10mm tubing inside the case and 19mm outside it, I had perfectly fine flow with a small aquapro pond pump. The only downside about it was that it wasn’t quiet - even with padding most of its noise coming from vibrations.
I had a mix of hose clamps (simple plastic barbed ones and also some hose band clamps). All my components seem to do fine with mains pressure in a garden set up (>12m head), and the pressure of my loop is far below that. The radiator is the heater core from a Ford Falcon that I managed to get for free, set in a cooling unit box I fabricated. Coolant was filtered rainwater with a small amount of antifreeze added to help keep the water free from biotic build up and any corrosion (of which there shouldn’t have been any though).

I was measuring the coolant temperatures at the res and it had a maximum thermal range in operation of ~20c, if I really pushed it at both extremes for sustained periods. For my system I found that maxed out @100% CPU utilisation (and a >50%OC): 80c core temps = 70c CPU temp = ~35c deg coolant temps. As I was drawing on some outside air, coolant temps could drop down to about 15c at idle loads with full fan in winter, but that usually meant I was running the fan too hard…

Because I had a relatively large quantity of water in the loop and res, I had the fan set to run intermittently, for example at 45m off/1 5m on at night with the cooler temps and 15m on/15m off at lunchtime*. A decent amount of heat would have radiated from the tubing too, but that constant cycling should have placed a higher than usual strain on the tubing and connectors, yet it didn’t seem to, as I ran it for years with nothing untoward found during inspections.

If you’re cooling a full 42u rack to a remote cooling location, I’d recommend breaking it up into a series of loops. Subloops for a section of the rack, a ‘rack distribution’ loop circulating the height of the rack, and transit loop. Find appropriate heat exchangers, such as those used by fireplaces with wetbacks, and tap the distribution loop with a regular WC pump loop, each cooling a few u of rackspace each. Two small computer pumps behind a half-depth 1u chassis or switch, with one pump servicing 4 or 5+U above and the other 4 or 5+U below, would work nicely I’d say, depending on the rack density.

I’m not sure how effective a sealed rack system would work, because the delta between coolant and ambient air (inside the rack) would be pretty low. You’d need a pretty large radiator at the top pushing a lot of air. It’d certainly be cheaper to set up than buying waterblocks for all of the CPUs/GPUs, but you’re missing out on exploiting the high temperature delta between CPUs/GPUs and coolant (which are also the largest heat sources). If you do though, I’d love to see the results and I am happy to be proven wrong

*Before I decided to migrate, the plan was to have the coolant temps monitored using a rasp pi and the fan automated. The benefit with the cycling is that it engaged the fan when the differential between ambient and coolant was highest, therefore improving the PUE. Pity life gets in the way… (I am only delayed... )

 

[RimBlock]

Thanks Lost-Benji & mackle for your suggestions and experiences.

I am currently working on a single node workstation to get a bit more familiar with getting the watercooling set-ups and the CNC mill and will take the information from that experience to enhance this project.

RB

 

[Lost-Benji]

Making the blocks would be fairly easy for you (machining copper will be fun for you if you haven't done it before) and I am sure you can figure out the copper tube. Braze the two together and have a leak free system.

 

[RimBlock]

I am currently prototyping in acrylic which seems to be fairly forgiving, I will then move on the aluminium which may present a bit more for a challenge but I understand copper tends to be 'sticky' and can cause even more issues.

Not sure about being able to braze. I don't have the equipment or space to really get a brazing set-up running. It will be bad enough finding the space for a powder coating box .

RB