As mentioned in another thread, I recently got a monster radiator. It was suggested I make a thread for the project it's related to. So here we are.
A bit of a loose prototype kind of project at this point.
The general theme is rackmount server cooling efficiency.
One major thrust of this is related to my "outdoor server torture test" that's currently running. I have two servers (dual e5-2680v2) outdoors, in the full 115F Arizona heat, mining Monero. So far so good on this front, but more on that later.
The other major project at this point is custom water cooled servers. I've bought a bunch of different water blocks, fittings, tubes, radiators, and a couple pumps thus far. Initial testing is for "physical fit" / component compatibility on a DOA motherboard.
Ultimately, the goal is to find a better way to cool servers than to supply them with large amounts of very cold air. It seems really unnecessary given that CPUs these days are designed to operate at temperatures approaching the boiling point. The second goal is to have fun!
On that note, I decided to buy a massive overkill radiator:
This takes up to 9x 120mm (or 4x 180mm) fans on each side. For scale, you can see 1x 120mm Noctua fan in the picture.
I've got 6 of these mounted now and waiting on 3 more to arrive. They also make a model that takes 9x 140mm fans (or 4x 200mm)
It was suggested that I consider 14" / 16" car radiator fans instead. Looking into it, must say, not a terrible idea.
Now, how to cool the servers?
Playing around with different blocks, fittings, etc, I think it's probably achieveable to get something that performs well and is reliable for $150 per dual cpu server. Maybe less depending on what improvements can be made over time.
Working theory at this point is to have a split system. Each smaller water cooling loop would service 1 - 4 servers with a pump (maybe two), fittings, splitters, etc. This small loop would be connected to a larger loop using a water->water plate heat exchanger. . This reduces the failure domain compared to cooling a whole rack from one water loop.
^ Plate heat exchangers (SSD for scale). The left one cost $30 (20 plate exchanger), right one only $20 (10 plates). Idea is one of these would be the interface from the "small loop" to the "main loop". You could potentially have dual redundant "main loops" -- this would require two plate exchangers per "small loop".
The amount of liquid in each smaller loop would be small, so, if a server leaked, it shouldn't damage more than one server (knock on wood). Cooling 1 - 4 servers from one loop, if a given loop failed, a small number of servers would be impacted. The main loop would be sufficiently far from the servers that if it leaked, it shouldn't damage them (other than possible overheating). Servicing more than 1 server per loop would keep costs down. This has diminishing returns beyond a couple of servers, especially if you end up needing more powerful pumps to compensate. My guess is that each small loop will have 2 - 4 servers, but I can't exclude the possibility of 1 server per loop.
I did find that a cheap ebay 200mm "water block" will actually cover both cpus in a dual cpu system quite nicely (see pic). Ultimately, I'll probably go for two smaller water blocks, but I do want to try it both ways to see.
^ I went with the above just to see what could be physically installed. You've got a 200mm ebay waterblock ($20), an EKWB "narrow ILM" mounting plate ($6), various fittings and adapters (I think I can get this down to $30-40, but as pictured is closer to double that), and a PCI passthrough plate ($10-15).
The one mounting plate on CPU1 puts enough pressure on the water block to provide solid contact on both CPUs. At least, it sure seems that way. When I test this out on a working board, I'll check the temperatures to be sure.
Really wish the pcie passthrough was half height, but only full height is in stock anywhere.
Pictured is the southbridge heatsink and second-cpu vrm heatsink removed. I noticed the first-cpu vrm's had no heatsink by default, so I assume the second CPU only needs the heatsink because of cpus 1 and 2 heating up the air before it gets to the VRMs. Either way, I think I can do this without removing the VRM heatsink if I use two smaller water blocks instead of the one big one. The southbridge heatsink was removed just to see if it's feasible to put a water block on it (it isn't). This board was DOA anyway, so who cares.
Thoughts? Any recommended parts (to get, or avoid)?
A bit of a loose prototype kind of project at this point.
The general theme is rackmount server cooling efficiency.
One major thrust of this is related to my "outdoor server torture test" that's currently running. I have two servers (dual e5-2680v2) outdoors, in the full 115F Arizona heat, mining Monero. So far so good on this front, but more on that later.
The other major project at this point is custom water cooled servers. I've bought a bunch of different water blocks, fittings, tubes, radiators, and a couple pumps thus far. Initial testing is for "physical fit" / component compatibility on a DOA motherboard.
Ultimately, the goal is to find a better way to cool servers than to supply them with large amounts of very cold air. It seems really unnecessary given that CPUs these days are designed to operate at temperatures approaching the boiling point. The second goal is to have fun!
On that note, I decided to buy a massive overkill radiator:
This takes up to 9x 120mm (or 4x 180mm) fans on each side. For scale, you can see 1x 120mm Noctua fan in the picture.
I've got 6 of these mounted now and waiting on 3 more to arrive. They also make a model that takes 9x 140mm fans (or 4x 200mm)
It was suggested that I consider 14" / 16" car radiator fans instead. Looking into it, must say, not a terrible idea.
Now, how to cool the servers?
Playing around with different blocks, fittings, etc, I think it's probably achieveable to get something that performs well and is reliable for $150 per dual cpu server. Maybe less depending on what improvements can be made over time.
Working theory at this point is to have a split system. Each smaller water cooling loop would service 1 - 4 servers with a pump (maybe two), fittings, splitters, etc. This small loop would be connected to a larger loop using a water->water plate heat exchanger. . This reduces the failure domain compared to cooling a whole rack from one water loop.
^ Plate heat exchangers (SSD for scale). The left one cost $30 (20 plate exchanger), right one only $20 (10 plates). Idea is one of these would be the interface from the "small loop" to the "main loop". You could potentially have dual redundant "main loops" -- this would require two plate exchangers per "small loop".
The amount of liquid in each smaller loop would be small, so, if a server leaked, it shouldn't damage more than one server (knock on wood). Cooling 1 - 4 servers from one loop, if a given loop failed, a small number of servers would be impacted. The main loop would be sufficiently far from the servers that if it leaked, it shouldn't damage them (other than possible overheating). Servicing more than 1 server per loop would keep costs down. This has diminishing returns beyond a couple of servers, especially if you end up needing more powerful pumps to compensate. My guess is that each small loop will have 2 - 4 servers, but I can't exclude the possibility of 1 server per loop.
I did find that a cheap ebay 200mm "water block" will actually cover both cpus in a dual cpu system quite nicely (see pic). Ultimately, I'll probably go for two smaller water blocks, but I do want to try it both ways to see.
^ I went with the above just to see what could be physically installed. You've got a 200mm ebay waterblock ($20), an EKWB "narrow ILM" mounting plate ($6), various fittings and adapters (I think I can get this down to $30-40, but as pictured is closer to double that), and a PCI passthrough plate ($10-15).
The one mounting plate on CPU1 puts enough pressure on the water block to provide solid contact on both CPUs. At least, it sure seems that way. When I test this out on a working board, I'll check the temperatures to be sure.
Really wish the pcie passthrough was half height, but only full height is in stock anywhere.
Pictured is the southbridge heatsink and second-cpu vrm heatsink removed. I noticed the first-cpu vrm's had no heatsink by default, so I assume the second CPU only needs the heatsink because of cpus 1 and 2 heating up the air before it gets to the VRMs. Either way, I think I can do this without removing the VRM heatsink if I use two smaller water blocks instead of the one big one. The southbridge heatsink was removed just to see if it's feasible to put a water block on it (it isn't). This board was DOA anyway, so who cares.
Thoughts? Any recommended parts (to get, or avoid)?
Last edited:
