Dual AMD EPYC 7742 workstation build

yorkman

New Member
Dec 31, 2019
28
0
1
Thanks alex_stief, and for the picture as well! This will really help I'm sure. I will probably order one of these since I already use Noctua coolers: Noctua NF-A4x10 FLX, Premium Quiet Fan, 3-Pin (40x10mm). The link you included to your build is also very informative. I'm almost sure I won't be upgrading to water cooling though. My other Supermicro X10DRL-i system has AIO water cooling and I think it was overkill, and they don't last as long as air coolers. Then again that system doesn't have any cpu vrm issues.

Btw, how hard is it to remove and replace that heatsink? Is it just a matter of unscrewing it? Do you think replacing it with some other one will help a lot?
 

alex_stief

Active Member
May 31, 2016
650
203
43
35
Removing the heatsink is the easy part. It is held in place by two push-pins. Similar to the ones Intel uses for their boxed CPU coolers. Just way flimsier, and not designed to be mounted more than once. But with a bit of patience, you can get them out without damaging them.
Replacing the heatsink with a better one is the hard part. I am not aware of any off-the-shelf part that would fit, and do a better job at cooling. It would probably need to be custom-made, by someone who knows a thing or two about heat transfer. At that point, water cooling might be cheaper and easier.
 

ari2asem

Active Member
Dec 26, 2018
539
86
28
The Netherlands, Groningen
......I will probably order one of these since I already use Noctua coolers: Noctua NF-A4x10 FLX, Premium Quiet Fan, 3-Pin (40x10mm).....
the air flow of NF-A4x10 FLX is about 8,20 m³/h (or 4,83 CFM). noise level 17,9 dB(A)

i use this one, it has airflow of 11,73 m³/h (or 6,90 CFM), noise level 19dB(A).
 

alex_stief

Active Member
May 31, 2016
650
203
43
35
Ah yes, I am absolutely certain that those specs are comparable, since testing is done under standardized conditions by an independent third party ;)
SCNR
 

yorkman

New Member
Dec 31, 2019
28
0
1
Oh one more thing. Do you think that having only 8 of the 16 memory slots filled would cause my benchmarks to be so low? I don't think so since I believe I saw on this site that Patrick was testing stockfish or asmFish with 8 dimms and I just can't come close to that.

Also found this: https://www.servethehome.com/amd-epyc-7002-rome-cpus-with-half-memory-bandwidth/

Not sure if I understand it right though whether that's really hurting me or not.
 

ari2asem

Active Member
Dec 26, 2018
539
86
28
The Netherlands, Groningen
Oh one more thing. Do you think that having only 8 of the 16 memory slots filled would cause my benchmarks to be so low? I don't think so since I believe I saw on this site that Patrick was testing stockfish or asmFish with 8 dimms and I just can't come close to that.

Also found this: https://www.servethehome.com/amd-epyc-7002-rome-cpus-with-half-memory-bandwidth/

Not sure if I understand it right though whether that's really hurting me or not.
that is another category of cpu. you have 7742, for you is the vrm temp more important than quantity of memory. if you have 8dimms per cpu you are fine.

once again, try to get vrm-temps down...
 

alex_stief

Active Member
May 31, 2016
650
203
43
35
Unfortunately, I have no idea what the algorithm of stockfish looks like. I read discussions that memory latency might matter a lot. But those were mostly for mainstream CPUs with 8 cores on 2 memory channels. Since you are running 64 cores on only 4 memory channels, you might run into a bandwidth limit, on top of CPUs throttling due to VRM temps. But again, it's impossible for me to tell, I work in a different field where memory bandwidth matters a lot.
Do you have a link to the test Patrick did?
 

yorkman

New Member
Dec 31, 2019
28
0
1
Actually here's a thread we were both involved in: https://forums.servethehome.com/index.php?threads/dual-epyc-7742-and-stockfish.27053/

As you can see he was using 8x 32GB 3200MHz ram and still got 275,000 kNps. Best I can do is like 200,000 and I think I only got that once by some fluke. Usually it's like 175,000 which is close to what Patric got on just one 7742: https://www.servethehome.com/amd-epyc-7742-benchmarks-and-review-simply-peerless/2/

Also here: Ipman Chess

On this link you can see Patrick privately submitted tests to this site. One other owner of the 7742 (vondele) got almost the same benchmark although I don't know the details on his system. Either way, as you can see I should be getting around 270,000 with 256 threads, not 175K or 200K. And if Patrick can get 275Knps with 8x 32GB 3200MHz then I should be able to get it too.

Recently I did manage to raise my bench score slightly by forcing 3200Mhz ram in bios and Memory Interleaving to disabled instead of both on Auto but that's still not even close to what those guys are getting. Worse, I don't think they used large pages for those tests where as I am. Without LP my bench scores drop significantly to about 130Knps. This is a problem I have had ever since I built this system. Only thing I know is neither one of them are using the Supermicro H11DSi-NT motherboard like I am but I find it hard to believe that'd be the sole reason....unless there's some other magical bios setting I haven't tried that makes sense to try, and I've tried a lot of them already. Originally I was using 8x 8GB 2133Mhz in this system and my scores hardly improved by upgrading (my ass) to 3200MHz for almost $2000 extra. To buy another eight 256GB 3200Mhz dimms at what will cost $2500 today would be insane to try unless I was rich. This has utlimately become my 2nd worst investment ever.
 
Last edited:

alex_stief

Active Member
May 31, 2016
650
203
43
35
I would say let's tackle one issue at a time. Once you get rid of VRM-induced CPU throttling, we can look at other possible issues.

One of them might be hinted at in the benchmark link you posted:
Note:every Xeon cpu who get used in a dual or quad system has in Bios a cluster-on-die setting
and so long they are not working in a server as a cluster system it's better to put cluster-on-die ->Off and get higher
speeds in this bench! (Thanks DeeDs) ->All benches from Patrick are done with COD=Off confirmed!
For AMD Epyc Rome, the equivalent to COD=disabled would be setting NPS1 in bios. But I think that should be the default setting anyway.
 

yorkman

New Member
Dec 31, 2019
28
0
1
Sure, however I believe I've already alleviated the vrm issue by repositioning the extra fans to be more direct to the heatsink. I had it running overnight and didn't wake up to an alarm or any throttling. VrmCpu* temps got to 95C which is still high but that's probably the best i can do without that 40mm fan coming in. The one you mentioned is not available anywhere in my area or even Amazon though I could order it from Amazon UK where it only costs like $2 pounds though a lot more for shipping. There are other fans I could order with a lot more air flow but I don't want to hear it as my case is quite silent and I'd like to keep it that way if possible. So I may go with your fan.

Just about every setting in this bios by default was Auto. I don't have a COD option since that's Intel only. But I did try NUMA Nodes Per Socket from NPS0-NPS4 and I currently have it at NPS0 which I believe gave me the best performance. So I've already tried NPS1 which made no difference. Also, I'm very familiar with that site as I've been following it for years.

One strange phenoma that keeps occuring, and I don't expect you to understand, is whenever I load Stockfish and start analysis the nps (speed) is 50% slower than it can be. I have to keep unloading the engine and then loading it again until maybe the 3rd or 10th try where the speed is finally 100% of what I know it can be. Strange but anyway.

Btw I appreciate your help.
 
Last edited:

norcimo

Member
Aug 11, 2020
37
1
8
Greetings,

Does the motherboard support engineering samples? I see alot of these on EBAY: AMD EPYC 64-Core 100-000000053-04 , supposedly its a 7742 but at slightly different clock speeds.
 

norcimo

Member
Aug 11, 2020
37
1
8
rev2 h11dsi for sure does as iam using that exact setup with the 53-04 cpus
Excellent. Thank two more question.

The best performance is occupying all the slots with 1r memory correct? This does the 8 channel memory mode...

any power supply you recommend?
 

ari2asem

Active Member
Dec 26, 2018
539
86
28
The Netherlands, Groningen
Excellent. Thank two more question.

The best performance is occupying all the slots with 1r memory correct? This does the 8 channel memory mode...

any power supply you recommend?
how do you come to 1r as best performance ?

as for PSU....if you use your system without any pci-e expansion hardware (i.e. storage card, gpu card, nic card), 650-700 watt PSU is good enough. choose a decent brand, like corsair, coolermaster, enermax etc etc.
 

norcimo

Member
Aug 11, 2020
37
1
8
how do you come to 1r as best performance ?

as for PSU....if you use your system without any pci-e expansion hardware (i.e. storage card, gpu card, nic card), 650-700 watt PSU is good enough. choose a decent brand, like corsair, coolermaster, enermax etc etc.
Ok, thank you.

Concerning 1R, I read it somewhere. However, my point being to achieve 8 channel memory mode, once has to occupy all slots? is this correct?
 

mirrormax

Member
Apr 10, 2020
41
17
8
dual rank has best performance although not by much afaik. and you wanna populate all 16 slots for 8 channels on each cpu.
psu depends on rest of the system, full load on cpus my system pulls around 600w, include gpu and its above 800w. if you add a lot of disks etc you def want 1000w +
 
  • Like
Reactions: ari2asem

norcimo

Member
Aug 11, 2020
37
1
8
dual rank has best performance although not by much afaik. and you wanna populate all 16 slots for 8 channels on each cpu.
psu depends on rest of the system, full load on cpus my system pulls around 600w, include gpu and its above 800w. if you add a lot of disks etc you def want 1000w +
Are there any windows 10 drivers or compatible for the unknown devices in the system?