Dual AMD EPYC 7742 workstation build

[shadowstorm]

Hi, I'm looking for some advice on building a powerful workstation for some specific use cases that my team has.

I came here after watching this video and checking this detailed post about the same. I really liked the details provided and how this setup can absolutely annihilate similar ones from Intel at that price point. I have a few questions about this setup:

1. I'm looking at dual 7742 CPUs and the only motherboard I've found that supports this the Supermicro H11DSI-NT. Newegg only sells with dual 7702 processors installed, but not with the 7742s. It looks like v2.0 of the motherboard supports 7002 platform, so the higher power processors should still work. Is this true? Are there other motherboards you would recommend for this type of setup?
2. I don't want to use a 2U or 4U chassis for this setup due to some physical limitations we have in our office space, so I'm looking at full tower cases instead that can support a full-sized dual socket ATX board. Something like the Phantek Enthoo Pro for example. Does this work as a good case for such a powerful setup? My concern is airflow mostly, as the primary use case for this is lots of immediate processing power as opposed to large amounts of data storage.
3. Instead of using dual 4U CUP coolers, can I use dual Noctua NH-U14S coolers instead? I've read that it works quite well for cooling Threadripper and EPYC processors if a second fan is added. Main reason for wanting to use this cooler is for a quieter system. I'm not 100% sure if it's possible to use them physically in the case or on the motherboard, but based on specs alone, it seems theoretically possible. Just curious if you would advice using this over dedicated server heatsinks with large and noisy fans.
4. Do you use Windows 10 Pro only or are you using Windows 10 Pro for Workstations? Is there any difference apart from higher memory and core count support? My specific application is for running design simulations and calculations, so I will require the maximum use of all cores in the machine.

The setup is obviously not going to be used as a daily workstation, but only when intense processing power is required by the team.

Having said all this, I'm also open to the idea of buying a pre-built system, so if there are full systems available that we can purchase with this kind of setup (say, with 1TB of RAM), I'd be interested in hearing about them. I'd like to work on building this myself because we are able to save quite a bit on cost if we source the components ourselves. I have quite a bit of experience building desktop workstations and servers (Intel-based), but just a bit unsure about the new AMD EPYCs in their dual socket configuration. There doesn't seem to be a lot of variety of components for AMDs either.

I would really appreciate any kind of guidelines/recommendations anyone here can provide.
Thanks!

 

[Patrick]

Hi there.

This is a very interesting idea. Frankly, there are not too many who have gone down this route simply because as a workstation.

Personally, I would go with a H12 generation motherboard if possible e.g. one that supports DDR4-3200. That H11 motherboard you link only supports DDR4-2666. I also do not know if that supports the 7742 as those are higher TDP parts.

For that video, I believe we used Windows Server not Windows 10. We actually, at the time of launch, needed a special patch from Microsoft for the Windows Server OS to handle those 64 core / 128 thread parts.

What I might suggest is to think about this as a rack-mounted workstation. Even with 4U or larger, the CPUs use so much power that noise is going to be too much for the office. Even the Threadripper 3970X will turn my NZXT water cooler fans to very high speeds at 60% load. Personally, if you are going to use it as an offload machine anyway, it may make sense to rackmount it and stick it in a data closet or even colocation then RDP into the desktop if you have to use Windows. That gives you access to servers that are specifically designed for these higher TDP parts. If you can use Linux, it will be easier to remotely manage and frankly does better with higher thread counts.

If you did want to cobble something together, look at what people are using for the Threadripper 3990X then use one per socket. My sense is that you will end up running two loops one for each CPU which will make case selection slightly more challenging as well.

 

[IamSpartacus]

@Patrick Just wanted to clarify that the H11DSi-NT does support both DDR4-3200 (with Rev. 2 of the board).

H11DSi-NT | Motherboards | Super Micro Computer, Inc.

Also, it's not listed on their site but pretty sure they support full TDP 7002 chips including the 7742. I'll have to find where I saw that.

 

[alex_stief]

Noctuas NH-U14s TR4-SP3 coolers have no clearance issues on H11DSi motherboards. And they will probably handle cooling of these CPUs with ease. In my experience, high fan rpms applied by Supermicro motherboards are not necessarily linked to CPU cooling being inadequate. My H11DSi ramped up the fans to over 70%, despite reporting CPU temperatures below 50°C. However, these CPU coolers are too tall for a 4U form factor, so you will have to get a proper (i.e. wider) case when using these air coolers.
I am still using the Phanteks Enthoo Pro. For me, it is one of the cheapest cases that tick all the boxes I need. Plenty of airflow, supports SSI-EEB without modifications, and comes in a flavor without RGB or glass side panels. I have switched to water cooling since I started using this case. It could provide a little more room to build in, but I still haven't found any alternatives because I really don't want glass side panels. I prefer hiding the cable mess I produce.

Edit: not sure where you are located. In Europe, this shop offers Epyc workstations with a good selection of components, at reasonable prices: D20z-ULN-ZN
No option to configure more than 512GB of memory, but I am sure you can ask for a special quote.
They won't provide the kind of service you would expect from OEMs like Dell or HP though.

 

[shadowstorm]

@Patrick Thanks for your response.

Personally, I would go with a H12 generation motherboard if possible e.g. one that supports DDR4-3200. That H11 motherboard you link only supports DDR4-2666. I also do not know if that supports the 7742 as those are higher TDP parts.

Is there any other major difference between the H11 and H12 generation boards? From what I can see, the H12 are using their custom form factor and require their own chassis, which is something we want to avoid. Supermicro also sells rev 2.0 of H11 boards which fully support EPYC 7002 processors (with DDR-3200 support, as @IamSpartacus mentioned, and here's an article from them about it). The only reason this is relevant is because they specifically did a collaboration with Newegg talking about supporting all 7002 processors on H11 boards.

For that video, I believe we used Windows Server not Windows 10. We actually, at the time of launch, needed a special patch from Microsoft for the Windows Server OS to handle those 64 core / 128 thread parts.

Do you or anyone else here know if this has changed today? Does Windows still need a patch to support higher core counts, or have they released a firmware update yet? We unfortunately need to use Windows since the software that will be doing the simulations only runs on Windows, otherwise it would have been Linux all the way.

Related - does anyone know the difference between Windows 10 Pro and Windows 10 Pro for Workstations? As far as I can tell, it supports higher core counts and higher memory, while unlocking some other internal features that's not available on the consumer version. Basically an intermediate between Windows 10 Pro and Windows Server OS.

Noctuas NH-U14s TR4-SP3 coolers have no clearance issues on H11DSi motherboards. And they will probably handle cooling of these CPUs with ease. In my experience, high fan rpms applied by Supermicro motherboards are not necessarily linked to CPU cooling being inadequate. My H11DSi ramped up the fans to over 70%, despite reporting CPU temperatures below 50°C. However, these CPU coolers are too tall for a 4U form factor, so you will have to get a proper (i.e. wider) case when using these air coolers.

@alex_stief thanks for the confirmation! I believe the sheet amount of airflow from the high RPM fans by Supermicro will help with any kind of heat these processors would generate, that's why they work well in a 4U setup.

I am still using the Phanteks Enthoo Pro. For me, it is one of the cheapest cases that tick all the boxes I need. Plenty of airflow, supports SSI-EEB without modifications, and comes in a flavor without RGB or glass side panels. I have switched to water cooling since I started using this case. It could provide a little more room to build in, but I still haven't found any alternatives because I really don't want glass side panels. I prefer hiding the cable mess I produce.

This is exactly why I was looking at the Phanteks Enthoo Pro. My preferences are very similar to yours - no RGB, glass panels and case that supports the motherboard without any issues. It also appears to support the dual Noctua heatsinks, since it appears to be wide enough, but I don't have the case in person to verify this (just going by the dimensions listed online)

Edit: not sure where you are located. In Europe, this shop offers Epyc workstations with a good selection of components, at reasonable prices: D20z-ULN-ZN
No option to configure more than 512GB of memory, but I am sure you can ask for a special quote.
They won't provide the kind of service you would expect from OEMs like Dell or HP though.

I'm located in the US. There are similar shops here, but I haven't encountered one with any reasonable pricing yet; they're just expensive. There is one by Velocity Micro that offers something very similar to what I'm looking for but with a custom water loop instead (check it out here), but not sure of the performance benefit of using water cooling over simply dual Noctuas. My guess is water cooling would perform better, but the Noctua is probably good enough.

 

[alex_stief]

I previously had two Noctua NH-U14S TR4-SP3 coolers in that case. Their mounting mechanism allows to adjust the position of the cooler by a few mm. Just enough to make them fit perfectly.

I don't think that water cooling would make a performance difference. Said the guy who just switched to water cooling
But I did it mostly for quiet operation, and because I wanted water cooling. You can get a bit lower temperatures with a proper custom water cooling setup, but that alone will not translate to higher performance.

 

[shadowstorm]

@alex_stief what about RAM clearance?

Also, do you use EPYC 7742 processors or similar? I'm assuming the water cooling is a custom dual loop as opposed to two separate AIOs.

I was thinking the Noctua fans are already very quiet, how much quieter can a water cooling setup get? Did using Noctua fans have that much of a noise issue?

 

[alex_stief]

No issues with RAM clearance.
I am using Epyc 7551 CPUs. It's a single loop for both CPUs and VRMs with a 420mm radiator.
Noctua fans are relatively quiet, but physics still apply. They become audible at higher rpm, just like any other quality fans. I am kind of sensitive to computer noises. To put that into perspective: from an acoustics point of view, I could never settle for one of the big OEM workstations from HP or others.
A decently sized water cooling just makes it easier to get the heat out of the case at low rpm. I can set the fan speed to some setting I am comfortable with, and leave it there regardless of system load.

 

[RMac]

I just built a system with the H11DSi and the Noctua NH-U14S TR4-SP3 coolers you referenced. They were tight, but fit fine with the exception of a bit of clearance issues on the P2-DIMME1 RAM slot, which necessitated mounting the fan a bit higher than I liked (the fan casing sticks a bit above the cooler's heat pipes and is actually the highest point in my case).

I've found them to be nice and quiet once I reset the minimum fan speed to 200rpm with ipmitool as described in this reddit comment here.

Even at full load they're pretty quiet, and don't tend to need full fan speeds to keep things cool, but without resetting the fan minimums, they will cycle between 300rpm and full load constantly as the bmc panics thinking the CPU fans are failing and drives all fans to 100% duty cycle . And THAT is very annoying!

I can also confirm that the RAM runs fine at DDR4-3200 speeds with my dual 7402's.

H12 motherboards should support PCI-E 4.0, while the H11DSi version 2 board I have is PCI-E 3.0 only. However, I haven't seen an E-ATX Dual Epyc H12, so I settled on the H11 board.

 

[ari2asem]

I just built a system with the H11DSi and the Noctua NH-U14S TR4-SP3 coolers you referenced. They were tight, but fit fine with the exception of a bit of clearance issues on the P2-DIMME1 RAM slot, which necessitated mounting the fan a bit higher than I liked (the fan casing sticks a bit above the cooler's heat pipes and is actually the highest point in my case).

I've found them to be nice and quiet once I reset the minimum fan speed to 200rpm with ipmitool as described in this reddit comment here.

Even at full load they're pretty quiet, and don't tend to need full fan speeds to keep things cool, but without resetting the fan minimums, they will cycle between 300rpm and full load constantly as the bmc panics thinking the CPU fans are failing and drives all fans to 100% duty cycle . And THAT is very annoying!

I can also confirm that the RAM runs fine at DDR4-3200 speeds with my dual 7402's.

H12 motherboards should support PCI-E 4.0, while the H11DSi version 2 board I have is PCI-E 3.0 only. However, I haven't seen an E-ATX Dual Epyc H12, so I settled on the H11 board.

do you have dual 7742 setup? what OS are you using ??

i have dual epyc 7742, windows 10 v2004 enterprise. i cann't get booted in windows when i enable SMT in bios (h11dsi-nt rev.2.board).

 

[RMac]

I'm only using EPYC 7402s on Windows 10 Pro 1909, but haven't had any trouble boot with SMT on, though I'm running considerably fewer threads than you are.

There's another thread where user Layla was struggling to get Windows Server 2019 to boot and eventually succeeded by turning IOMMU on in BIOS, might be worth reading through:

I thought I'd updated this post with the fix, but apparently not... :/

https://forums.servethehome.com/index.php?threads/finally-overclocking-epyc-rome-es.28111/post-272812

 

[KarelG]

I am using Epyc 7551 CPUs. It's a single loop for both CPUs and VRMs with a 420mm radiator.
Noctua fans are relatively quiet, but physics still apply. They become audible at higher rpm, just like any other quality fans. I am kind of sensitive to computer noises. To put that into perspective: from an acoustics point of view, I could never settle for one of the big OEM workstations from HP or others.
A decently sized water cooling just makes it easier to get the heat out of the case at low rpm. I can set the fan speed to some setting I am comfortable with, and leave it there regardless of system load.

Fans are fans, but what about pump noise? Isn't it quite noticeable too? Thanks!

 

[alex_stief]

I reduced the pump speed to a noise level that I can not hear. Same as fans, pumps can be annoying at higher rpm, but tuned down to lower rpm and noise levels.

 

[traderjay]

I currently use the dark rock pro 4 TR edition for the my EPYC 7702 and it is VERY VERY quiet with no ram clearance issues.

 

[yorkman]

Hi, I'm looking for some advice on building a powerful workstation for some specific use cases that my team has.

I came here after watching this video and checking this detailed post about the same. I really liked the details provided and how this setup can absolutely annihilate similar ones from Intel at that price point. I have a few questions about this setup:

1. I'm looking at dual 7742 CPUs and the only motherboard I've found that supports this the Supermicro H11DSI-NT. Newegg only sells with dual 7702 processors installed, but not with the 7742s. It looks like v2.0 of the motherboard supports 7002 platform, so the higher power processors should still work. Is this true? Are there other motherboards you would recommend for this type of setup?
2. I don't want to use a 2U or 4U chassis for this setup due to some physical limitations we have in our office space, so I'm looking at full tower cases instead that can support a full-sized dual socket ATX board. Something like the Phantek Enthoo Pro for example. Does this work as a good case for such a powerful setup? My concern is airflow mostly, as the primary use case for this is lots of immediate processing power as opposed to large amounts of data storage.
3. Instead of using dual 4U CUP coolers, can I use dual Noctua NH-U14S coolers instead? I've read that it works quite well for cooling Threadripper and EPYC processors if a second fan is added. Main reason for wanting to use this cooler is for a quieter system. I'm not 100% sure if it's possible to use them physically in the case or on the motherboard, but based on specs alone, it seems theoretically possible. Just curious if you would advice using this over dedicated server heatsinks with large and noisy fans.
4. Do you use Windows 10 Pro only or are you using Windows 10 Pro for Workstations? Is there any difference apart from higher memory and core count support? My specific application is for running design simulations and calculations, so I will require the maximum use of all cores in the machine.

I have this exact motherboard, dual 7742s and using 8x 3200MHz ram with dual Noctua NH-U14S TR4-SP3 coolers inside a Phanteks Enthoo Pro Series PH-ES614P case. The coolers barely fit as the fins are slightly pressed against the rear of the drive bay but otherwise fit just fine.

No issues cooling the cpu's with these coolers. But i don't think the case is great for airflow. After using it for about 3 months without issues in terms of heat, I've encounted a program that I absolutely must use 24x7 and this program makes the VrmCPU1 and VrmCPU2 temps go through the roof. Motherboard goes off with a constant buzzing sound when these vrmcpu's hit 100C. My cpu temps are usually no more than 50-55C but this motherboard seems it cannot handle some software. I have as many high airflow fans plugged into this motherboard as possible, even with the side open it keeps hitting 100C and cpu's are automatically throttled. What's annoying too is when the buzzing starts there's no way to stop it unless you shut the system down. Even if you power it back up the buzzing continues so I have to actually flip the PSU switch off and on before the buzzing alarm finally resets. Which means if you're working remotely then you'll never be able to clear that buzzer!!

Also, I bought this motherboard because the benchmarks I saw with this program are excellent, but I can only get about 70% of those speeds that others get. I totally regret spending the thousands of dollars on the cpu's, ram and especially motherboard. I will be trying a lot harder in avoiding Supermicro motherboards, at least for AMD processors.

 

[alex_stief]

VRMs overheating on the H11DSi with two 64-core CPUs is not so much the fault of the case, but the motherboard design.
Sure, it would probably fare better in a 1U server case with a wall of 40m screamers forcing air over the tiny VRM cooler. But You can't blame that on a workstation case. Have you tried squeezing two 40x10mm fans under the CPU coolers, on top of the VRM heatsink? It doesn't do much, but maybe just enough to avoid throttling.
By the way, I had the same CPU coolers, on the same board, in the same case. They don't necessarily rub against the HDD cage. Noctuas coolers have three different mounting positions that slightly adjust where the cooler sits relative to the socket. Out of the box, the coolers are mounted in the "wrong" position.
Lastly, what program exactly is under-performing for you? Maybe it's not just the CPUs throttling, but a memory bandwidth limit with only 8 DIMMs installed.

 

[ari2asem]

you can control VRM heating by lowering the TDP and PPT of the cpu's in bios. consequently you will loose performance when you lower TDP to 180 Watt from 225 Watt.

epyc 7742 at room temp of 29°C, TDP 225 Watt

http://imgur.com/a/5ufo64y

dual epyc 7742 temps and voltages when my TDP was 225 Watt

http://imgur.com/a/wOkVf1S

now i lowered TDP and PPT to 180 Watt and have VRM-temps of 86-88°C with 252 threads of monero mining, room temps sometimes at 40°C. notice 2 small fans of 40*40*10mm on cool block of vrm and bigger black fan blowing air towards sockets and vrm.

with cold (low temp's) state of motherbord i run xmrig (monero miner) at 93kH/s. but now with my setup as described with above settings in bios and all fans i am running xmrig at 75 kH/s.

short story, like @alex_stief said, it is the fault of supermicro board. and the case....this board is designed to have airflow over the vrm cool block when it is in server-chassis with air shrouder

 

[yorkman]

Thanks for the replies guys. But you misunderstood my point. I wasn't blaming the case but the motherboard design. I mentioned the case and fans to explain that it has as good as airflow as I can give it without splitting into some of the fan headers to be able to add more fans. I have two fans already blowing air at the vrm's and it's still not enough to keep it under 100 degrees C.

Also, I'd hate to have to lower the performance of this system. It already under performs according to the benchmarks other get with different motherboard (Patrick used the AMD Daytona, which is not possible to purchase). The thing is that I use almost exactly the same program at 100% cpu without any vrm temp issues. The program that causes it is actually the same program but with some code modification to support some new features and running this one at 100% cpu causes this problem. It's called SFNNUE (stockfish chess engine).

 

[yorkman]

VRMs overheating on the H11DSi with two 64-core CPUs is not so much the fault of the case, but the motherboard design.
Sure, it would probably fare better in a 1U server case with a wall of 40m screamers forcing air over the tiny VRM cooler. But You can't blame that on a workstation case. Have you tried squeezing two 40x10mm fans under the CPU coolers, on top of the VRM heatsink? It doesn't do much, but maybe just enough to avoid throttling.

I already have the top of the case all covered with 3 fans. Tried pull/push but same issue.

And how do you mean add fans under the cpu coolers on top of the vrm heatsinks? I don't remember now but I think the vrm's were above the cpu's/coolers and I believe that's where I have the 2 fans blowing air from inside the case. It's just not enough though. Only place left where I can add more fans is by modifying the case's side plastic window which would require some time and effort. Would also have to split into a fan header as I've already used all the fan headers on the motherboard. Not sure that'd be enough still seeing as how fast these vrms overheat with so many fans in the case already.

 

[alex_stief]

The CPU VRM on that board is located right between both CPU sockets. Hiding under a small aluminium heatsink. On top of that heatsink, tucked under the CPU coolers, is where those 40mm fans could go. They fit, I tried.


If all is lost, you can still go water cooling. Shameless plug: https://forums.servethehome.com/ind...ooling-more-cores-more-ram-more-better.28077/
As for fan headers: 3-pin or 4-pin splitter cables are pretty cheap. Some fans even come with them as an accessory.