Custom Water Cooled Rack (Fun with Prototyping)

Discussion in 'DIY and Makers Spot' started by funkywizard, Aug 22, 2018.

  1. funkywizard

    funkywizard mmm.... bandwidth.

    Joined:
    Jan 15, 2017
    Messages:
    543
    Likes Received:
    230
    As mentioned in another thread, I recently got a monster radiator. It was suggested I make a thread for the project it's related to. So here we are.

    A bit of a loose prototype kind of project at this point.

    The general theme is rackmount server cooling efficiency.

    One major thrust of this is related to my "outdoor server torture test" that's currently running. I have two servers (dual e5-2680v2) outdoors, in the full 115F Arizona heat, mining Monero. So far so good on this front, but more on that later.

    The other major project at this point is custom water cooled servers. I've bought a bunch of different water blocks, fittings, tubes, radiators, and a couple pumps thus far. Initial testing is for "physical fit" / component compatibility on a DOA motherboard.

    Ultimately, the goal is to find a better way to cool servers than to supply them with large amounts of very cold air. It seems really unnecessary given that CPUs these days are designed to operate at temperatures approaching the boiling point. The second goal is to have fun!

    On that note, I decided to buy a massive overkill radiator:

    radiator.jpg

    This takes up to 9x 120mm (or 4x 180mm) fans on each side. For scale, you can see 1x 120mm Noctua fan in the picture.

    I've got 6 of these mounted now and waiting on 3 more to arrive. They also make a model that takes 9x 140mm fans (or 4x 200mm)

    It was suggested that I consider 14" / 16" car radiator fans instead. Looking into it, must say, not a terrible idea.

    Now, how to cool the servers?

    Playing around with different blocks, fittings, etc, I think it's probably achieveable to get something that performs well and is reliable for $150 per dual cpu server. Maybe less depending on what improvements can be made over time.

    Working theory at this point is to have a split system. Each smaller water cooling loop would service 1 - 4 servers with a pump (maybe two), fittings, splitters, etc. This small loop would be connected to a larger loop using a water->water plate heat exchanger. . This reduces the failure domain compared to cooling a whole rack from one water loop.

    plate exchangers.jpg

    ^ Plate heat exchangers (SSD for scale). The left one cost $30 (20 plate exchanger), right one only $20 (10 plates). Idea is one of these would be the interface from the "small loop" to the "main loop". You could potentially have dual redundant "main loops" -- this would require two plate exchangers per "small loop".

    The amount of liquid in each smaller loop would be small, so, if a server leaked, it shouldn't damage more than one server (knock on wood). Cooling 1 - 4 servers from one loop, if a given loop failed, a small number of servers would be impacted. The main loop would be sufficiently far from the servers that if it leaked, it shouldn't damage them (other than possible overheating). Servicing more than 1 server per loop would keep costs down. This has diminishing returns beyond a couple of servers, especially if you end up needing more powerful pumps to compensate. My guess is that each small loop will have 2 - 4 servers, but I can't exclude the possibility of 1 server per loop.

    I did find that a cheap ebay 200mm "water block" will actually cover both cpus in a dual cpu system quite nicely (see pic). Ultimately, I'll probably go for two smaller water blocks, but I do want to try it both ways to see.

    dual block.jpg

    ^ I went with the above just to see what could be physically installed. You've got a 200mm ebay waterblock ($20), an EKWB "narrow ILM" mounting plate ($6), various fittings and adapters (I think I can get this down to $30-40, but as pictured is closer to double that), and a PCI passthrough plate ($10-15).

    The one mounting plate on CPU1 puts enough pressure on the water block to provide solid contact on both CPUs. At least, it sure seems that way. When I test this out on a working board, I'll check the temperatures to be sure.

    Really wish the pcie passthrough was half height, but only full height is in stock anywhere.

    Pictured is the southbridge heatsink and second-cpu vrm heatsink removed. I noticed the first-cpu vrm's had no heatsink by default, so I assume the second CPU only needs the heatsink because of cpus 1 and 2 heating up the air before it gets to the VRMs. Either way, I think I can do this without removing the VRM heatsink if I use two smaller water blocks instead of the one big one. The southbridge heatsink was removed just to see if it's feasible to put a water block on it (it isn't). This board was DOA anyway, so who cares.

    Thoughts? Any recommended parts (to get, or avoid)?
     
    #1
    Last edited: Aug 22, 2018
    pgh5278, leebo_28, K D and 2 others like this.
  2. funkywizard

    funkywizard mmm.... bandwidth.

    Joined:
    Jan 15, 2017
    Messages:
    543
    Likes Received:
    230
    Radiator with 6 fans attached. (waiting for 3 more to arrive)

    six-fan.jpg
     
    #2
    pgh5278 and William like this.
  3. alex_stief

    alex_stief Active Member

    Joined:
    May 31, 2016
    Messages:
    303
    Likes Received:
    91
    If this works so far so good for the CPUs.
    How are you going to cool the other components inside that need active cooling? Will the normal fans stay in the server? Some chipsets, VRMs and RAM could get pretty toasty without plenty of airflow.
     
    #3
  4. funkywizard

    funkywizard mmm.... bandwidth.

    Joined:
    Jan 15, 2017
    Messages:
    543
    Likes Received:
    230
    Planning to keep the normal fans in there, just hopefully will keep a low rpm. Ultimately would like to have the "cold aisle" at up to 90f in the summer.
     
    #4
  5. BLinux

    BLinux Well-Known Member

    Joined:
    Jul 7, 2016
    Messages:
    1,854
    Likes Received:
    519
    I assume AZ is pretty dry right? What's the humidity level in your area? If it is low enough, you can build some plumbing around the radiator to mist water on the radiator. It's a technique used on intercoolers to chill the intake air from a supercharger before it gets into an engine. Same principal as those swamp coolers.
     
    #5
  6. funkywizard

    funkywizard mmm.... bandwidth.

    Joined:
    Jan 15, 2017
    Messages:
    543
    Likes Received:
    230
    Yup, definitely doable. Very dry outside.

    That said, shouldn't be necessary unless the radiator is running close to maximim capacity. Most water cooling equipment is rated for liquid temperatures up to 60-65C or thereabouts. At peak temperatures, that gives you a 20C delta to work with vs the weather. Chips can easily run another 20C above that. Those deltas should allow for pretty substantial heat dissipation. Evap can probably get you another 10C, if you need it.
     
    #6
  7. funkywizard

    funkywizard mmm.... bandwidth.

    Joined:
    Jan 15, 2017
    Messages:
    543
    Likes Received:
    230
    Got the last 3 fans installed, and the fan grill.

    full radiator.jpg

    Set up a temporary water loop (with some extra fill / drain tubes) to flush the radiator:

    flush radiator.jpg
     
    #7
    pgh5278 likes this.
Similar Threads: Custom Water
Forum Title Date
DIY and Makers Spot Custom power-supply mounting in a server case Aug 31, 2018
DIY and Makers Spot Custom cooling for the 9361-8i Aug 7, 2017
DIY and Makers Spot Custom STH-Themed Parts Apr 20, 2016

Share This Page