AMD EPYC Build - Memory Questions

Erick

New Member
Nov 16, 2017
28
19
3
Saint Paul, MN
Hello all,

I'm new here, and I'm looking to build a home server using the AMD EPYC platform. I have decided to go with an EPYC 7401P and a Gigabyte MZ31-AR0 board. It's been about 10 years since I last built an enterprise grade server, but I am very fluent with desktop components. I am having a hard time understanding how to populate the memory on this board to get the best performance I can, with a minimum of 64 GB of memory.

As I understand, the faster the memory, the better with these new EPYC chips. Can I get by with using 2 X 32 GB RDIMM's @ 2666 mHz? I am looking into this kit: Crucial 64GB Kit (2 x 32GB) DDR4-2666 RDIMM CT11010536 | MZ31-AR0 | Crucial.com

Here is the quick reference for the board: http://download.gigabyte.us/FileList/Manual/server_qrg_mz31-ar0_e_v10.pdf

From what I can tell, using a pair of dual ranked RDIMM's would drop my speed down to 2400 mHz. Any insight would be greatly appreciated!

Thank you,

Erick
 
  • Like
Reactions: Subsonic

Patrick

Administrator
Staff member
Dec 21, 2010
11,885
4,846
113
You can run 8x DIMMs at 2666. I would strongly recommend this as well. With two DIMMs you will have poor memory bandwidth
 
  • Like
Reactions: Subsonic

Erick

New Member
Nov 16, 2017
28
19
3
Saint Paul, MN
Yes I want to do whatever I need to do to get the maximum memory performance possible. So, I will go with 8x DIMMs @ 2666 MHz.

Can anyone comment on if these will work for the maximum 2666 speed on the Gigabyte board listed above? Crucial 16GB DDR4-2666 RDIMM | CT16G4RFS4266 | Crucial.com If so, I will pick up 8 of these.

Based on the documentation I should be able to achieve the full 2666 MHz, if I am reading it correctly. These are single ranked DIMMs.
 

alex_stief

Active Member
May 31, 2016
606
183
43
35
Since it is single-rank, they should run at DDR4-2666 if the run.
Epyc officially supports RDIMM DDR4-2666 single-rank and DDR4-2400 dual-rank.
But I am still waiting for benchmarks single vs. dual. The benchmarks for Ryzen CPUs suggest that 2400 dual-rank delivers better performance than 2666 single-rank. And since it is basically the same CPU and memory controller, just 4 of them, I suspect that the results for Epyc would be similar.
If I had to buy new memory for my dual-Epyc build, I would probably choose DDR4-2666 dual-rank based on the little information I found so far. Then either try to run it at DDR4-2666 or wait for a few more bios-updates that should increase memory compatibility. Until then I would run the memory at DDR4-2400.
STH would really do us a favor testing single- vs. dual-rank memory on Epyc ;)
 
Last edited:

Erick

New Member
Nov 16, 2017
28
19
3
Saint Paul, MN
I agree, there is very little information out there and I have probably read through everything there is on single vs dual rank with Ryzen. There are a lot of unknowns. I understand that the potential for better performance exists with dual rank but it is more difficult to run at higher speeds. Performance gains on Ryzen appear to be minimal, in the 1% to 3% range, but maybe this increases with more cores. Don't really know as there are no test results out there I can find.

I think if price was equal, it would be a no brainer to go with dual rank, but I can get some single rank Crucial ram with the same specs as dual rank Crucial ram for ~25% less cost, so I might go that route, and use that extra $450 on SSD's. This is "just" a home server, something I'm going to be using for moderate video encoding tasks, nothing mission critical.
 

alex_stief

Active Member
May 31, 2016
606
183
43
35
If $450 is 25% saved that means you are paying a total of $1800 for 8x16GB DDR4.
Even with todays ridiculous prices I can find 16GB DDR4-2666 reg ECC (single or dual-rank) for ~180€. € with taxes in Germany is usually the same as US$.
Is memory even more expensive in the US?
 
Last edited:

Erick

New Member
Nov 16, 2017
28
19
3
Saint Paul, MN
Alex, Yes the cheapest dual rank 8x16 GB DDR4 I could find is roughly $1800 USD after tax. If you know of anything cheaper out there, I'm all ears :)
 

Patrick

Administrator
Staff member
Dec 21, 2010
11,885
4,846
113
Good point on SR v. DR RAM. Generally 1DPC you want DR with EPYC.

And for someone searching this thread later, the reason you do not want 2 DIMMs per socket on the AMD EPYC is that the AMD EPYC package is four die. Each die has dual channel memory. If you have two DIMMs that means that two die will have half of their memory channels full while the other two will have no attached RAM.

Always use 8 DIMMs with AMD EPYC. I have been told SR v. DR is measurable but not huge.
 

alex_stief

Active Member
May 31, 2016
606
183
43
35
Ryzen benchmarks suggest that dual-rank vs. single-rank increases performance even slightly more than DDR4-2666 vs. DDR4-2400. In other words: Ryzen with dual-rank DDR4-2400 performs similar or slightly better than Ryzen with single-rank DDR4-2666 across a number of various benchmarks.
Since you emphasized the importance of using DDR4-2666 instead of DDR4-2400 both here and in the reviews, I think this is worth further investigation.
 

Patrick

Administrator
Staff member
Dec 21, 2010
11,885
4,846
113
Ryzen benchmarks suggest that dual-rank vs. single-rank increases performance even slightly more than DDR4-2666 vs. DDR4-2400. In other words: Ryzen with dual-rank DDR4-2400 performs similar or slightly better than Ryzen with single-rank DDR4-2666 across a number of various benchmarks.
Since you emphasized the importance of using DDR4-2666 instead of DDR4-2400 both here and in the reviews, I think this is worth further investigation.
The bigger impact on EPYC is that Infinity Fabric works at RAM speed. So when you have multiple NUMA nodes die-to-die transfers are a bigger deal than RAM throughput. That impacts memory access across die as well as PCIe/ disk/ network access. That is not as big of an issue on Ryzen which is why I would treat the two differently in this regard.
 

alex_stief

Active Member
May 31, 2016
606
183
43
35
I am with you so far in theory.
But I won't be satisfied until I see some actual benchmarks ;)
Because there may be a lot of applications where inter-die transfers are not a bottleneck.
 

alex_stief

Active Member
May 31, 2016
606
183
43
35
Epyc has an 8-channel memory interface. With only 4 DIMMs, you are wasting performance in bandwidth-bound applications.