AMD Epyc, memory and motherboards

alex_stief

Active Member
May 31, 2016
653
205
43
35
My plans on trying AMDs new Epyc processors for my new workstation are in their final stage. But a few questions remain.

1) Memory compatibility - frequency
Right now I can't really afford buying 16x16GB DDR4-2666 memory. But I have enough 16GB DDR4-2133 reg ECC waiting to be put to use. I know that slow memory will negatively affect performance of Epyc in general and for my applications in particular, but until memory prices drop significantly this is the only option for me.
So I contacted Supermicro just to make sure. Their support person tried to tell me that Epyc only supports DDR4-2400/2666, implying that their motherboards would not even post with DDR4-2133. I would normally call BS on that, but since this will be my first AMD setup in years I am now slightly concerned.
Any experience on that?

2) Memory compatibility - organization
I read some comments that even with 1DPC RDIMM dual-rank, maximum memory speed drops to DDR4-2133 anyway. Is that true? And what about performance? For the Desktop Ryzen processors, dual-rank DIMMs provide slightly better performance than single-rank DIMMs with the same transfer rate. Is the same true for Epyc?

3) Memory tweaking
A few of the dual-socket workstation boards for Intel that I had in the past allowed increasing memory frequency beyond the specifications of the memory. I know for a fact that the DDR4-2133 modules I have are capable of running at DDR4-2666. Any chance that there will be a dual-socket workstation motherboard for Epyc processors that allows adjusting memory frequency?

4) CPU turbo frequency
While the base clock speeds are rather low, all-core turbo frequencies listed in AMDs data sheets are pretty decent. For example 2.7GHz for the Epyc 7301. Is this a long-term turbo speed that the processor can sustain over long periods of time provided adequate cooling or is this just for a few seconds?
 

Patrick

Administrator
Staff member
Dec 21, 2010
11,938
4,892
113
1) If you can swing 2666 after selling 2133 then I highly recommend doing so. Infinity fabric speeds are tied to memory clocks. On EPYC, 2400 to 2666 has a demonstrable impact.

2) Depends on the motherboard. Check the manual of what you are buying. Seems like a drastic drop for 1DPC.

3) Setting the POR and memory speeds works on Skylake-SP but we have had a lot of trouble with this on EPYC systems.

4) Power plays a role in this. I was surprised recently by the delta between the 7351P and 7301 as I was reviewing data.
 

alex_stief

Active Member
May 31, 2016
653
205
43
35
Thanks for your input.

I am aware of the importance of memory speed for the Zen architecture and the reasons behind it, so the plan is definitely to step-up memory speeds as soon as the prices drop again. But right now it is DDR4-2133 or no Epyc.
So my question boils down to this: will a dual-socket Epyc setup even boot up properly with DDR4-2133 installed?

I peeked into the manual of the Supermicro H11DSi. They report a maximum frequency of DDR4-2400 with 1 dual-rank RDIMM per channel. How did you run dual-rank DIMMs at DDR4-2666 in your tests?
And the question remains: would it be better to use single-rank DDR4-2666 or dual-rank DDR4-2400? All tests I read for the Ryzen desktop CPUs would imply that dual-rank DDR4-2400 would yield better performance. When replacing my 16GB DIMMs I can choose both options.

Pardon my ignorance: what is POR? And if you say you were having trouble adjusting memory transfer rates on Epyc, are you implying that the user can at least try to adjust it or is it completely locked?

Since most of my workload is memory bandwidth limited, I hope that CPU power consumption will not exceed TDP thus sustaining maximum turbo frequency. Would you subscribe to this assumption?
 

Patrick

Administrator
Staff member
Dec 21, 2010
11,938
4,892
113
We are using a Supermicro suggested setup in that box. On the POR - there is a setting you can use where you force POR off and can manually adjust DDR4 speed in a small range. Skylake systems seem to be having little to no issue with this. EPYC systems will not boot/ POST.

On the DDR4-2133 - I would guess it will boot if the board works with the DIMMs. Memory compatibility is a bigger concern on the EPYC side. Total guess that it will work.
 

alex_stief

Active Member
May 31, 2016
653
205
43
35
So I see there is no way around a little trial&error. I guess that's the price you pay as an early adopter, especially with AMD.
I will have to wait for a few more weeks anyway until the parts are available, maybe I can at least gather some new information in the meantime.
Again, thanks sharing your knowledge.
 

alex_stief

Active Member
May 31, 2016
653
205
43
35
So first update. I got to try my DIMMs in a Supermicro H11DSi-NT and they work. At least with DDR4-2133. DDR4-2666 did not post as expected, I am still hoping to get DDR4-2400 running once I get my own hardware in a few weeks.
Single-core performance per clock (assuming 2.7GHz) is already better than with Haswell-E for my workloads. Will test multi-core performance in the next few days, there were some issues with the compiler flags used for Intel fortran compiler...
lscpu only shows a CPU max frequency of 2.2GHz and current CPU frequency of 2.2GHz even under single-core load. But I guess this is only a wrong reading, CPU turbo was enabled in the bios. And if the CPU were only running at 2.2GHz, that single-core performance would be outright spectacular :cool:
 

alex_stief

Active Member
May 31, 2016
653
205
43
35
It goes for around 600€ here in germany.
I guess their machine cost somewhere around 7000€ in total.
 

ruckb

New Member
Feb 15, 2018
1
0
1
50
Hi, I'm just looking in buying a similar machine.. I know this thread is already a bit older and hope there are new informaitons..
Either it will be a single processor with the Gigabyte mainboard, or dual processor with the Supermicro board.
I think i have seen a nice table with memory speed vs. number of ranks, but I can not find this any more.
as I would like to go with 512GB it seems difficult to get 2666 based on single Rank DIMMs .. (not sure if it is possible in the supermicro case where I would have 16 slots available.. need to do some more checks of my dimms.. )
But anyhow: is there in the meantime more information available on the ability to run dual rank dimms at 2666 ?
Or maybe on the performance impact on 2666 vs. 2400 ?

thanks for any feedback..

Hermann
 

Evan

Well-Known Member
Jan 6, 2016
3,101
517
113
Yep it seems most 16gb can be single rank, but for 32gb I see mostly dual rank. I am sure there is probably single rank 32gb just not come across it myself.
 

yorkman

New Member
Dec 31, 2019
28
0
1
I got to try my DIMMs in a Supermicro H11DSi-NT and they work. At least with DDR4-2133. DDR4-2666 did not post as expected, I am still hoping to get DDR4-2400 running once I get my own hardware in a few weeks.
Why wouldn't DDR4-2666 post as you had expected? This motherboard supports it. I too have DDR4-2133 Micron ram (8x8GB 1R 2133) in my E5 Xeon system so I am wondering now if they'll also work with the dual Epyc 7742 on the H11DSi-NT board I plan to get soon? What do you think? At least I could being some testing before I decide on what ram I should actually buy.
 

alex_stief

Active Member
May 31, 2016
653
205
43
35
To say the least, memory overclocking is not the strong suit of Supermicro motherboards for Epyc.
Want to run DDR4-2666? Then buy DDR4-2666.
Keep in mind that this was all written during a time when memory cost 2-3 times as much as it does nowadays. And I already had the DDR4-2133, so there was no way I was going to spend 3000$ for memory on a budget build. I started with 2x7301, the whole build cost way less than a single Epyc 7742.
 
  • Like
Reactions: yorkman

alex_stief

Active Member
May 31, 2016
653
205
43
35
I know, right :rolleyes:
There are reasons why I try to avoid Supermicro as much as possible. Most support experiences I had with them were questionable.
 

balnazzar

Member
Mar 6, 2019
103
7
18
It seems they are top quality boards, but yes, I heard people complaining about their support here and there on fora...
 

sno.cn

Active Member
Sep 23, 2016
206
73
28
36
I haven't come across any ECC RDIMM that won't run in my H11DSi with Rome, and I haven't run any QVL sticks.

I think if you're running memory higher than its rated speed, you'll want a modified BIOS to adjust timings.