Upgrade time: more cooling, more cores, more RAM, more better

alex_stief

Active Member
May 31, 2016
666
211
43
35
This build started out here: https://forums.servethehome.com/ind...ild-for-fluid-simulation-now-with-pics.17703/
With upgrades accumulating over time, it is time to show off once more :cool:

Build’s Name:
AMD is still back
Working title: Noctua fans. How many are enough
Operating System/ Storage Platform: OpenSUSE leap 15.1
CPU: 2x AMD Epyc 7551 retail
Motherboard: Supermicro H11DSi rev. 1
Chassis: Phanteks Enthoo Pro
Drives: Samsung 970 Evo Plus 500GB, HGST 7.68TB SN200 Ultrastar NVMe, 3x WD red 6TB (software RAID5)
RAM: 512GB populated as 16x32GB Samsung DDR4-2666 reg ECC
Add-in Cards: the cheapest AMD RX 570 8GB I could find
Power Supply: Seasonic Focus Plus Platinum 850W
Other Bits: Now with water cooling for CPUs and VRMs. And a few 3D-printed bits like the memory coolers and standoffs for the top cover
Usage Profile: Code development, CFD simulations

The most time-consuming upgrade was obviously water cooling. Doing your first custom loop on expensive hardware can be quite exciting. Here is a list of parts, if someone is interested:
Radiator: Alphacool Nexxxos ST30 full copper 420mm
Pump/reservoir: Alphacool Eisfach - Single Laing D5 Eispumpe VPP755
Tubing: EK Water Blocks EK-Tube ZMT 16/10
Fittings: Mostly Alphacool Eiszapfen
CPU blocks: EK-Velocity sTR4 - Nickel + Acetal
VRM block: Anfi-tec UPC002 13x88mm UPC002 (13 x 82-120mm) - Universal
Liquid: Aqua Computer Double Protect Ultra. 1l was enough.

More info
All fans are now connected to the fan controller on the front, that is also equipped with temperature sensors. It's not perfect because control is not very sensitive at low RPMs. But still miles ahead of Supermicros overly aggressive fan control.
I haven't dialed in a stable CPU overclock yet. But temps are pretty good, with CPUs temps around 40°C under full load, and CPU VRMs staying below 60°C. From a few tests it seems like the motherboards OCP will be the limiting factor.
VRMs other than the CPU got a little copper heatsink glued on with 2k thermal adhesive: Alphacool GPU RAM Copper Heatsinks 6,5x6,5mm - 10 Stk. Phobya 2-Komponenten-Wärmeleitkleber
All fans are obviously Noctua: 3x Noctua NF-A14 PWM intake front and bottom, 3x Noctua NF-A14 PWM exhaust/radiator, 1x Noctua NF-A14 PWM on the PCIe slots and 4x Noctua NF-A6x25 PWM for memory.

What I would have done differently
Not try to order a monoblock here: Monoblock HCM PRO MBD-H11DSI-NT-B
After almost 3 months of excuses, I gave up and cancelled the order. Also, a lone thermal pad arrived in the mail at some point, that was supposed to go on the VRMs. At a thickness of 5mm, I have my doubts that this will perform any better than the stock aluminium heatsink. Also, it was severely damaged during shipping.
Use a thicker radiator. I ordered only 30mm thick because I had hopes of getting both radiator and fans inside the case. It might work, but only with all power cables routed inside the case. With the fans outside, a thicker radiator would fit. But anyway, temps are still good.
Leave out the fill/drain port. With the reservoir/pump combo I used, filling and draining the loop is nightmare. The port I added is no help at all. But once I had the loop filled and free of air bubbles, I was just too lazy to do it all again. Definitely a task on the list for the first loop maintenance.
Use a different case with more room to build in. The only problem is: I still haven't found one. Since I am not a fan tempered glass side panels, I guess no amount of money I am willing to spend on a case will ever change that.

Pictures
Epyc Water cooling upgrade
IMG_20200405_160041.jpg IMG_20200405_160057.jpg IMG_20200405_160157.jpg IMG_20200405_160208.jpg IMG_20200405_160335.jpg IMG_20200405_155913.jpg
 
Last edited:

Subatomic

New Member
Mar 23, 2020
8
1
3
Very nice build! As far as the fan control, I mentioned in a previous post on my EPYC overclock that I modified a shell script I stole from someone that does a good job of controlling my low rpm Noctua fans without the constant revving that Supermicro boards always seem to suffer from. I am running Kubuntu and just start this at boot. I thought about adding an external controller as well, but I am on air and needed the system to rev up the fans as needed based on temps. I will be happy to post this if anyone is interested.
 

alex_stief

Active Member
May 31, 2016
666
211
43
35
I had the revving up under control using lower threshold values set through IPMI. Still, the fan speeds applied by the board under load were too high for my taste. Can your script do more than that? I.e. setting fan curves?
 

Subatomic

New Member
Mar 23, 2020
8
1
3
I had the revving up under control using lower threshold values set through IPMI. Still, the fan speeds applied by the board under load were too high for my taste. Can your script do more than that? I.e. setting fan curves?

Yep. I had the same issue, and it drove me crazy. The script (written by Glorious1 for a X10 board I think) will poll ipmi at regular intervals and adjust fanspeeds based on min and max speeds. The formula can be adjusted, but I didn't feel any need. When idling the 6 fans in my box idle between 400-600 rpm. The original script from Glorious1 can be found on Github at

glorious1/spin_scripts
 

Rand__

Well-Known Member
Mar 6, 2014
4,578
911
113
Why?
The hardware interaction is done by ipmitool if I saw it correctly on a quick scan of the script.
The script is just the automation layer around it - so as long as windows ipmitool behaves similar to linux ipmitool it should run
 

bashdan

New Member
Mar 29, 2020
4
0
1
This is a piece of art to me! I have the same case (although in white, and same mobo) for my setup. Is your 12-pin ATX long enough to make it around without an extension? I feel like I could get an extra 100MHz or so out of my overclocks with some VRM cooling. Did you find any air-based mounting solutions to cooling it in your research?
 

alex_stief

Active Member
May 31, 2016
666
211
43
35
The 8-pin power connectors are plugged in directly. I used an extension for easier routing of the 24-pin cable though.
I did not put much effort into searching for better air-cooling of the CPU VRMs. You can hook up two 40mm fans to the existing heatsink if they fit between/under your CPU coolers. I did that while air-cooling with Noctua NH-U14S. Maybe you could replace the thermal pad with non-conductive thermal paste. But check for good contact, there is a reason why pads are usually used for this application: they better compensate for poor flatness of the components and heatsink.
 

ari2asem

Active Member
Dec 26, 2018
547
87
28
The Netherlands, Groningen
the vrm heatsink is 60mm long. so putting 2* 40mm will not fit. and 40mm*10mm fans have not so much air flow. max air flow what i know for 40mm*10mm is about 7-9cfm. and you cann't fit thicker than 10mm fans because you have limitation cuased by noctua sp3 coolers
 

alex_stief

Active Member
May 31, 2016
666
211
43
35
Are you referring to the geometry for these memory fan holders?
If you really want them, I can ask my brother for the files. He did all the 3D printing and modeling.
 

Subatomic

New Member
Mar 23, 2020
8
1
3
Trust me, it fits. And not much airflow is still better than no airflow.
I use the same fan, but only one and it is centered. I assume that you are only using 2 of the mounting holes and the end of the fans stick off the ends of the heatsink. I also have a fan mounted to my Rosewill Blackhawk Ultra case that blows onto the back of the motherboard between the cpu mounts. This seems to help with the VRM temps from my experience.
 

Patriot

Moderator
Apr 18, 2011
1,311
695
113
Are you referring to the geometry for these memory fan holders?
If you really want them, I can ask my brother for the files. He did all the 3D printing and modeling.

Mhmm, I am pro printed fan baffles, I likey your setup. Be careful on the overclock, the motherboard traces are not designed for it.
 
  • Like
Reactions: nasi