128 thread workstation rebuild on the gigabyte MZ72-HB0

mirrormax

Member
Apr 10, 2020
61
27
18
Build’s Name: 128core epyc windows Build​
Operating System: Win 10 Pro workstation​
CPU: 2 x 7742 epyc 64cores​
Cooler: NH-U14S TR4-SP3​
Motherboard: gigabyte MZ72-HB0​
Chassis: Fractal Design 7 XL​
Drives: 900p optane, samsung 970​
RAM: 256gb 16x16gb 3200mhz samsung dual rank M393A2K43DB3-CWE​
GPU: 1080ti(upgrading to either 3080/3090 or rdna2)​
Add-in Cards:
pcie usb3 card​
asus hyper m.2 x16 card for some extra slots.​
Power Supply: corsair i1600w​

Usage Profile: workstation for compute/virtualization/occasional game.

Other information
This build was inspired by the epyc overclocking thread on this forum, the plan was to replace my dual workstation setup (7960x and a 1950x) with one system, the sane thing would be to go for a 3970x, but i wanted some more headroom, so then looking at the 3990x its closer closer to 5k euro here and I realized i could build a dual epyc system with 2nd hand ES/oem cpus for less. With the downside that there's no real workstation board for dual epycs, no official windows 10 support and generally very little information on it except in this forum!

Initial supermicro build
Being fully aware i would probably run into problems and this build requiring a lot of tinkering i ordered the cpus on ebay for 3500$ wich still feels like a steal for pretty close to stock speeds and the ability to OC(AMD EPYC 64 Core ROME 7742 100-100000053-04 2.00GHz(3.2GHz Turbo) 225W)
initially I got a h11dsi rev1 board i found locally, but i could never get it to work even with the custom bios(i suspect i might have shorted it in the case or it was just doa, more on that later)
Replaced it with a new h11dsi-nt rev2 board where after a lot of fiddling with bios settings, using the correct bios and windows10 version reading countless pages on here about this board, i finally had a working system a few days later(oculink ports only detected nvm disks with the latest bios). But i was never completely happy with this board, my gfx card barely fit, i could not get 2gpus to work, no pcie 4.0, the m.2 slot is x2.

I had a full writeup of all my issues getting the supermicro to work but managed to lose it before posting it, doh. but there are a few other threads on here where people have figured it out. Overall though the supermicro board was solid at stock speeds when i finally got it working and performed well for my tasks for the few months i've been using it, i was just looking for those extra features it lacked. the only caveat is for people planning 100% cpu usage with avx/fpu load on top where the vrm will overheat almost instantly and cause throttling/shutdowns, several users on here can verify this issue.

Gigabyte MZ72-HB0 rebuild
So i started looking for a new board with pcie 4.0, the asrock rome2d16-2t board looked like the best option but its still months away and the gigabyte board was out first and looks very similar, so i put in an order to switch over. Surprisingly it worked right out of the box, i didn't even go into the bios it just booted straight into the same windows install after an automatic configuration by windows, ram at correct speeds all nvme disks detected correctly. way to go gigabyte!

All the IO ports are positioned cleverly to not block any long pcie cards, there's a usb3 header, hurray! The IPMI software is a little more modern with gui for custom fan curves based on whatever temp sensor you want, so you could have a specific vrm fan ramp up based on vrm temps etc.(another issue with the sm board was the fan would ramp up and down all the time because of the default low RPM alarm, although it was fixable with console commands). bios also has the full AMD CBS menu if you feel adventurous with tweaking.

To test vrm temps i ran some monero mining at stock speeds and without any vrm fans, and my kinda temporary case fan setup(waiting for some more front fans) the vrms settled at around 88c doing 77khs, this would overheat the SM board. the cpus are chilling at 64c with the noctua u14s tr4 single fan. pulling about 750w from the wall stock full load, during one of the OC runs I saw 950w :eek:

I have not played around too much with overclocking yet as i won't need it for my day to day use, but the little i tried i could not recreate the 47k corona benchmark i managed on the SM board(that was def not running stable though and scores would fluctuate massively) or the 34866 C20 score. best i've seen so far is 43k corona and 31hk ish cinebench, fine by me!

Future plans:
I want to eventually watercool this, but i am waiting to decide on the gpu setup, 3080/3090 most likely unless amd comes up with something better soon, i did try a radeon vii but had some driver issues and the 1080ti was working better for me.

Before switching out the motherboard, the AMD reference cards sure look good.
IMG_20200702_125103.jpg

You can see two issues with the first build here, gfx card longer than 268mm will rest on the storage ports or be blocked completely if something is plugged in, both oculink and sata. Can also see the electrical tape I had to use on the lip of the motherboard tray as it would come in contact with the board if not, case says it supports SSI-EEB so its abit strange there's a raised lip at the end that potentially can come in contact with long boards. I didnt wanna take any chances after the first board wouldn't boot, so I made sure to use plenty of electrical tape.

IMG_20200829_220612.jpg

Upgrade time, everything feels more modern about this motherboard including the colors :cool:

IMG_20200922_100459 (1).jpg

assembled
IMG_20200922_152729.jpg

Management
chrome_Uh9eLpMWOO.png
chrome_V6Nnvre9iQ.png
 

ari2asem

Active Member
Dec 26, 2018
560
89
28
The Netherlands, Groningen
so, when you run monero you get 77khs, with you open case ?? and in bios no any tweaks for cTDP or PPL ?

my settings:
h11dsi-nt rev.2 board with dual 7742 with cTDP and PPL set to 180 watt in bios
all 16 momery channels using 32gb at 3200mhz speed memory
2 little fans (40*10mm) on vrm-heatsink
2* 120mm noctua tr4 cpu coolers with 4*120mm in push-pull
mining rig case (open rig)
room temp at 26°C
vrm-temps are 73-74°C, cpu temps are 53-54°C

i get monero speed of 81khs


then i want to know in which way is this gigabyte board better than h11dsi rev.2 board. just curious,
 

mirrormax

Member
Apr 10, 2020
61
27
18
that was all stock settings using xmrig on windows(I don't really do montero mining this was mostly to test if vrms overheat on stock settings)

pros over the h11dsi board for me is mostly pcie clearance(I will use multi gpu), pcie4.0(future proof).
for pure montero mining iam sure the h11dsi is fine
 

gabe-a

New Member
Sep 10, 2020
24
3
3
I have the h11dsi (the only other mobo available). I am suffering from overheating problems left and right inside a workstation/desktop chassis. For me, the RAM VRMs (and sometimes the CPU VRMs) would overheat when doing high-intensity computation.

I am curious if this new motherboard is better and doesn't throttle all the time.
 

mirrormax

Member
Apr 10, 2020
61
27
18
Yea I've been reading your thread but its hard to say for sure without testing since you have different and much larger capacity ram sticks, and I'm not sure what your load is like. running aida64 memory stress test dimm0 vrm goes to 72, and dimm1 vrm goes to 92 after one hour, with fairly poor cooling, no direct fans towards vrms or ram, and low rpm case fans.

what i can say is there looks to be four extra phases for the memory vrms(one for each set of 4 dimms)so that might help with heat dissipation on the gigabyte board and the coming asrock board.
 
  • Like
Reactions: gabe-a

gsrcrxsi

Member
Dec 12, 2018
53
3
8
where did you buy that gigabyte board? and how much was it?

i can't seem to find it for sale anywhere.
 

mirrormax

Member
Apr 10, 2020
61
27
18
had some spare time to play around a bit more with overclocking since I was kind of disappointed with motherboard performing worse than the h11dsi in my initial tests. It did not disappoint this time around. set the world record corona benchmark score, highest cinebench r23 ive seen and crushed my aida64 mem scores.

chrome_gDIsLklQq1.png

xnview_NflRX6S2yn.png
XSplit.Core_oZTUBE3clA.png
 
  • Like
Reactions: writerinserepeat

neobenedict

New Member
Oct 2, 2020
10
1
3
I'm looking at an overclocked naples system myself, do you think the h11dsi would be fine for 2x7551? Maybe if I add some heatsinks to the VRMs? This would be in a server chassis with plenty of airflow
 

mirrormax

Member
Apr 10, 2020
61
27
18
i have no experience with naples, but seeing its a lower tdp cpu iam sure its fine. probabaly better to ask in the naples OC thread though
 

mirrormax

Member
Apr 10, 2020
61
27
18
Been testing some monero mining in my offtime, and iam hitting 92kh/s at stock with xmrig running all night, vrm at stable 90/91c without fans blowing on it. cpus chilling at a cool 65c hardly breaking a sweat! seems like theres been improvements on the miner software side since last time i tested on the h11dsi board, pretty sure i struggled breaking 80khs.

Ordered some noctua 40mm fans to see how much it helps the vrm temps, until i decide if i am gonna do a custom loop or not. it would be a fun a project but i will have to make my own vrm block for it to be worth it.

Also tested with a rtx3090 but turns out the latest drivers that support 3000 series were unstable for my use case so switched to a 2080ti for now, ill probabaly test the 6900xt when thats out but dont have too high hopes as the radeon VII had drivers issues for me aswell.
IMG_20201008_094014.jpg
 

ari2asem

Active Member
Dec 26, 2018
560
89
28
The Netherlands, Groningen
Been testing some monero mining in my offtime, and iam hitting 92kh/s at stock with xmrig running all night, vrm at stable 90/91c without fans blowing on it. cpus chilling at a cool 65c hardly breaking a sweat! seems like theres been improvements on the miner software side since last time i tested on the h11dsi board, pretty sure i struggled breaking 80khs.

Ordered some noctua 40mm fans to see how much it helps the vrm temps, until i decide if i am gonna do a custom loop or not. it would be a fun a project but i will have to make my own vrm block for it to be worth it.
your vrm-temps will go down with 4-5 Celsius with 2 small fans on vrm-block

I'm looking at an overclocked naples system myself, do you think the h11dsi would be fine for 2x7551? Maybe if I add some heatsinks to the VRMs? This would be in a server chassis with plenty of airflow
7551-cpu (dual socket) is fine for h11dsi-board. with 1 fan (40*40*10mm) on the vrm-block and room temp at 25-26 Celsius, i am running xmrig 6.5.2 with cpu-temp at 45-46 C, vrm-temp 56-57 C.
all bios settings are at DEFAULT. no modded bios, no cpu-overclock. cpu is dual RETAIL 7551.

if you use air shroud in your server rack case with enough air flow, then no worry about vrm-temps
 
  • Like
Reactions: mirrormax

neobenedict

New Member
Oct 2, 2020
10
1
3
your vrm-temps will go down with 4-5 Celsius with 2 small fans on vrm-block

7551-cpu (dual socket) is fine for h11dsi-board. with 1 fan (40*40*10mm) on the vrm-block and room temp at 25-26 Celsius, i am running xmrig 6.5.2 with cpu-temp at 45-46 C, vrm-temp 56-57 C.
all bios settings are at DEFAULT. no modded bios, no cpu-overclock. cpu is dual RETAIL 7551.

if you use air shroud in your server rack case with enough air flow, then no worry about vrm-temps
It'll be a server case, so no issue with the airflow, I think. Have you tried overclocking, how well do the chips perform?
 

ari2asem

Active Member
Dec 26, 2018
560
89
28
The Netherlands, Groningen
It'll be a server case, so no issue with the airflow, I think. Have you tried overclocking, how well do the chips perform?
i cann't overclock because overclock-tool is not working on my system.... and it is very unstable software. too many hassle for me and not worth to do it. naples overclock tool here: