Upgrade time: more cooling, more cores, more RAM, more better

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

alex_stief

Well-Known Member
May 31, 2016
884
312
63
38
Unfortunately not. With a pump and reservoir installed, there is very little room for hard drives. If it even fits hard drives at all with an SSI-EEB motherboard installed.
Also, I had similar Fractal cases in the past. In my opinion, they take noise insulation too far, choking on air flow.
 

bashdan

New Member
Mar 29, 2020
5
0
1
The Phanteks case (for me) was the one that could be clearly modified to be an upward-air motion instead of a front-to-back air motion, because of the orientation of the sockets on H11DSi and, advertised clear SSI-EEB support. I only found 4-5 good alternatives in my decision. There are other cases that support EATX and could've potentially worked, but I went with the safe option (and bought on Amazon.)
 

alex_stief

Well-Known Member
May 31, 2016
884
312
63
38
Hard to define 'worth it' with something as unnecessary as water cooling. Probably not, but I did not want a half-assed solution.
 

edge

Active Member
Apr 22, 2013
203
71
28
Interesting build.

Flushing, filling and draining your loop should be easy and not hard. No matter what fluid I used in my last wc, it always developed crap in the loop over a period of months. I had my drain line at the low point of the loop and my top fill reservoir was highest in the loop with a side mounted radiator (large passive). Drain from the bottom, fill from the top, all the bubbles accumulate in the reservoir.

The radiator lines connected through the case with quick connects. Flushing meant adding the flush source and run out using the quick connects so they were between the radiator and case.

I did water cooling for quiet, not oc. That was in the athlon days with small heat sinks and 40mm screamers. Due to too much time in data centers, I have tinnitus and fan noise makes it a whole lot worse. 140mm noctua as slow as they will spin and huge heatsinks have made wc no longer necessary for me.

I highly recommend quick connects where your loop goes through the case, it will make filling and flushing your loop easier.
 

maxermaxer

Active Member
Oct 28, 2016
289
48
28
48
Hi Alex, I am planning to build a dual EPYC 7742 rendering system and did a lot of research in this forum. People's discussion around the overheating issue of SM motherboard makes me feel it is probably not so wise to use SM motherboard. I was thinking to change to Gigabyte MZ72 HB0 motherboard but it is very expensive and difficult to buy.

So I saw your post here. Would appreciate if you can give me some insight of resolving the SM overheating issue. Let me tell you a bit about the current system I have which I would like to upgrade:
- This build is going to be sitting in my Caselabs THW10 (which is a huge box for watercooling) as photo shown below
- Currently it has 2x E5 2696 v3 CPUs and 7x 1080Ti (I do both CPU and GPU rendering)
- It has 4x 560 radiator (1x for CPU loop and the other 3x for GPU loop)
After the upgrade the 7x 1080Ti GPU will be replaced by 3x or 4x 30 series GPUs which are also watercooled

PHOTO-2020-12-08-23-21-07.jpg

So my questions are:
1) Are you running CPU intensive program? I will use this system to render 3D graphics and it can be kept running at 100% for hours. I am not sure if your test (which is really nice temperature) is based on short time task
2) How do you mount the 3d-printed memory cooler onto the motherboard? I have 3D printer and would like to do the same if necessary.
3) Do I really need to cool the RAM if what I am using are not RAM intensive? I usually will do 3D modeling in another workstation (faster single CPU clock) and use network rendering to render the graphics in this water-cooled build. The rendering is not so RAM intensive.

Thank you in advance!
 
  • Like
Reactions: nnunn

mirrormax

Active Member
Apr 10, 2020
225
83
28
Get the gigabyte board if you can especially if you plan to use gpus. The sm board can't fit long gpus in some slots because of storage ports bering in the way. It will just save you a lot of trouble.

Or check out the asrock rome2d16-2t I think it should be out now.

Love your build BTW that 1080ti stack is sick, how's the temp difference from First to last gpu?
 

maxermaxer

Active Member
Oct 28, 2016
289
48
28
48
Hi Mirrormax, glad to see your reply. I have also seen a great post from you by using the Gigabyte MB to build a dual EPYC system.

The temp is very good. Under full load the average temp of the cards is around 40c. I didn't really look at the difference between the 1st and the last card but the difference should not be major. However compared with the 30 series card the power draw is actually bigger. The performance of these 7x 1080Ti under Octane is equal to 3x RTX 3080.

Actually I have sold all the 7x 1080Ti when seeing the price of GPUs are rising due to mining. Saving funds for this new build.

It is a bit difficult to find that Gigabyte board here. @ what price did you buy it? The one listed on eBay is 1000 USD.

Yes asrock rome2d16-2t and it is said it will be out in Feb. Maybe worth waiting for a while.

Do you think the price of 7742 ES CPU will probably go down a bit considering the new Zen 3 EPYC is coming out soon?

{Edit} I just saw this - looks like asrock rome2d16-2t is more expensive


asrock.PNG
 

alex_stief

Well-Known Member
May 31, 2016
884
312
63
38
1) Are you running CPU intensive program? I will use this system to render 3D graphics and it can be kept running at 100% for hours. I am not sure if your test (which is really nice temperature) is based on short time task
My usual workload is a mix of CPU intensive and memory intensive. But I also tested purely synthetic workloads for both, and temps were fine. It's really not hard to cool these CPUs with their relatively low heat flux density. My build could do with a little more radiator surface for virtually silent operation, but I don't really care if the CPUs are running at 40°C or 50°C.
Since your CPUs have higher power draw and twice the amount of cores, your mileage may vary. But CPU temperatures are generally nothing to worry about with water cooling on Epyc CPUs. You definitely have the radiator surface for it. VRM temperature is a bigger concern, especially when overclocking the CPUs with relatively little airflow over the board. As is usually the case with water cooling in a large workstation case.

2) How do you mount the 3d-printed memory cooler onto the motherboard? I have 3D printer and would like to do the same if necessary.
They clip onto the DIMM slots/latching mechanism themselves. Took 2 iterations to get it right, but now they are really a snug fit. Be aware that these are taylor-made for the H11DSi board. They might not fit on different boards with slightly different latching mechanisms for the DIMMs.

3) Do I really need to cool the RAM if what I am using are not RAM intensive? I usually will do 3D modeling in another workstation (faster single CPU clock) and use network rendering to render the graphics in this water-cooled build. The rendering is not so RAM intensive.
For me, it was more a gimmick than a necessity. Memory temperatures were fine without the fans. But there are cases where additional cooling might come in handy. Again, a huge workstation case with relatively slow airflow over the board can lead to higher temperatures. And if I am not mistaken, LRDIMM tends to run hotter than the RDIMM I am using here.
At least this guy had some trouble cooling LRDIMM and related VRMs: https://forums.servethehome.com/ind...x-epyc-7742-in-define-7-xl-1tb-2tb-ram.30142/
 
Last edited:

maxermaxer

Active Member
Oct 28, 2016
289
48
28
48
Thanks Alex for replying to my questions. Very helpful! I don't really worry about the CPUs temp since I am pretty sure a 560 radiator and waterblock from EK can handle them pretty well. My concerns are really more about the temperature of the VRMs. I am just worrying if the VRM can get overheated easily when running in 100%. As you mentioned the 64 cores would probably get hotter I guess something has to be done to cool them down.

Have you heard about that the Rev 2 of H11Dsi motherboard has resolved the overheating issue? Some people say that but I am not sure.
 

mirrormax

Active Member
Apr 10, 2020
225
83
28
it also depends on the memory sticks, if you use really large one it might become an issue where they need lots of airflow over them like in a rack chassis, remember that the ramsticks and vrm heatsinks all sit horizontal for front to back airflow. not top/bottom like your current setup and most desktop motherboards.
for more normal 8/16/32gb sticks it should be fine though.

and even on the 7742 the cpus stay fairly cool, i can run 100% load overnight overclocked fine on air. i would recommend getting a vrm block if you plan to watercool though, since thats gonna be the hottest part in the system.
 

alex_stief

Well-Known Member
May 31, 2016
884
312
63
38
Have you heard about that the Rev 2 of H11Dsi motherboard has resolved the overheating issue? Some people say that but I am not sure.
I haven't seen an H11DSi rev.2 without its VRM heatsink.
But if I had to guess, the only hardware change between rev.1 and 2 is the larger bios chip.
With water cooling for the CPUs, you have ample space to mount a fan to the CPU VRM heatsink. Or just replace it with a water cooler while you are at it.
Not sure if I would get any Supermicro H11 variant at this point in time. They lack PCIe 4.0. And given their track record, I would not count on them supporting Epyc Milan.
 

captainblie

New Member
Feb 12, 2021
3
5
3
Roaming the country
Have you heard about that the Rev 2 of H11Dsi motherboard has resolved the overheating issue? Some people say that but I am not sure.
I have the Rev 2 H11DSi-NT and I can attest that the CPU VRMs still overheat when in a workstation setup with high CPU usage on dual 7742's. I've not been able to test the memory VRM's yet.

Oh, do any of you guys know the dimensions of the heatsink used for the memory VRM's?

BTW, OP, thanks for the info about the VRM water-block. Going to try the fans on the heatsink while waiting for the block to arrive. Lots of good info in this thread. Thank you all.
 

shpitz461

Member
Sep 29, 2017
109
19
18
50
- This build is going to be sitting in my Caselabs THW10 (which is a huge box for watercooling) as photo shown below
- Currently it has 2x E5 2696 v3 CPUs and 7x 1080Ti (I do both CPU and GPU rendering)
- It has 4x 560 radiator (1x for CPU loop and the other 3x for GPU loop)
After the upgrade the 7x 1080Ti GPU will be replaced by 3x or 4x 30 series GPUs which are also watercooled

View attachment 17152

Thank you in advance!
Hi, which waterblocks are you using for the GPU's? Can you use the same blocks on 30xx cards?
Which power supply do you use? Does it power the whole system or do you need to use two power supplies?
What part # is that big block that connects all the GPU's?
Great build!
Thanks!