Dell EPYC R7425 DIMM Operating Speed

[jtku]

Hi All,

Just purchased our laboratory a brand new EPYC server from Dell - A R7425 chassis with dual EPYC 7451.

I also purchased 8x32 GB DIMM modules rated at 2666. The BIOS is only recognizing them as 2400. In looking through the manual (https://topics-cdn.dell.com/pdf/poweredge-r7425_owners-manual_en-us.pdf) I noticed that while the EPYC 7000 series recognizes a max DDR Freq of 2666, for some reason dual rank DIMMs at 16 or 32 GB capacity only operate at 2400 (See Table 42).

Does anyone know technically why that is the case? And similarly does any one know how to get the operating speed back up to where it is supposed to? I don't know if this is a Dell specific thing, or an EPYC specification. I can't find any hints online that it is due to EPYC hardware.

At the end of the day, I spent more money on the 2666, but I'll eat that if I'm stuck at 2400. I'm trying to suck as much speed out of this unit as possible.

Thanks,

J

 

[101]

The gigabyte mz31 board is consistent with the dell data in that table. Dual rank modules only run at 2400 MHz even with only 1 installed per channel. I guess they still look like two to the controller.
http://download.gigabyte.asia/FileList/Manual/server_qrg_mz31-ar0_e_v10.pdf

 

[jtku]

Well we just found this from AMD, hot off the press for July 2018:

https://developer.amd.com/wp-content/resources/56301_0.70.pdf

Apparently specifically 32 GB Dual Rank RDIMMs are going to run at 2400, when populated at 1 DIMM per channel in systems with a 2 slots per channel configuration. All DIMMs will drop to 2133 when populated at 2 DIMM per channel in systems with a 2 slots per channel configuration. What I can read from Dell, however, is potentially 64 GB 4R RDIMMs might jump back up to 2666, even in systems with 2 slots per channel (as the R7425 has).

Though Figure 4 makes it all very confusing, because I read it as directly contradicting Figure 3 until you catch that it only is true for systems with a 1 slot per channel configuration.

It all gives me a headache and it all makes me question why both vendors are still suggesting we buy 2666 RDIMMs, even if the chances of them operating at that speed are low.

 

[101]

Well 64GB LRDIMMs DO run at 2666, but I would be suprised if standard registered quad ranks would work when duals already don't. It's not exactly like epyc has an amazing memory controller on board. If you really need the capacity/speed use load reduced 32s or 64s, although I have no data regarding the impact of the additional buffer.

Great datasheet BTW, they should've issued that last year.

 

[alex4nder]

I ran into a smiliar problem, with a Gigabyte MZ31-AR0. I bought 8x CT16G4RFD8266 because it was on the QVL as 2666 MHz. I missed the fine print on Gigabyte's documentation, because it turns out that dual rank RDIMMs are only supported at 2400.

This isn't a production machine, so I decided to try my hand at manually setting the RAM speed to 1333 MHz. After 48 hours of mprime95 running on ~126 GB of the RAM, at 50C (with ambients in the 30C) I haven't had any issues. YMMV.

 

[alex_stief]

Does anyone know technically why that is the case? And similarly does any one know how to get the operating speed back up to where it is supposed to? I don't know if this is a Dell specific thing, or an EPYC specification. I can't find any hints online that it is due to EPYC hardware.

It is not a Dell thing, the problem here is AMD. The IMCs they used for Zen have been a source of constant grief ever since it's release.
For Epyc first gen, AMD specifies DDR4-2400 with one dual-rank DIMM per channel and DDR4-2666 only with one single-rank DIMM per channel. Allegedly, some folks got lucky and were able to increase dual-rank DIMMs to DDR4-2666. Try updating the bios to the latest version and see if you can change memory speeds manually.
A word of caution: if this kind of "overclocking" fails, you need to do a full bios reset.

Edit: If your Dell-specific bios does not provide an option to adjust memory frequency, the problem is part AMD, part Dell

 

[jtku]

Thanks to everyone for their experiences; I'm relieved to know it's not just isolated to this design.

We are considering LRDIMMs, but are leaning towards just funneling that money into buying more RDIMMs to fill up the memory channels on both sockets (1 DIMM/channel). We needed at minimum 256 GB, which is what we have now. If RAM wasn't so obnoxiously expensive right now, we would have gotten a lot more.

We think we are at the latest BIOS release (1.3.6). There is an option to force the DIMMs to run at 2666, but it's not sticking. Maybe down the line it will.

 

[alex_stief]

LRDIMMs won't give you better performance, just the ability to get much more RAM (in this case more than 512GB) crammed into one machine. For the same rated frequency, LRDIMM performs a bit worse than regular RDIMM. Switching to LRDIMM just to get DDR4-2666 running probably won't increase performance.

 

[jtku]

Ok, well, that makes that decision. Thanks for the heads up. I did not know that.