Memory support on X99 (2011-3) motherboards with Xeon E5v4

[kingmouf]

Hi all,

I am upgrading my workstation from a Xeon E5v3 CPU to recently bought E5v4. My motherboard is based on X99 chipset (AsRock X99 Extreme6) and it lists support for up to 128GB RAM with 16GB memory sticks.
Since the new Xeon is said to support up to 1.5TB of DDR4-2400 memory, I am wondering for the following:
1. does it make sense (under no circumstances I want to get into overclocking and such of the CPU), to buy sticks with speed grades higher than 2400MHz? Are there any performance benefits? I can see a few things that trouble me:
- buying a 2400Mhz stick at the moment is either more expensive or the same price with 3000MHz/3200MHz sticks and this means that I can reuse those sticks more easily in the future if I upgrade to Ryzen 9 / Threadripper CPU (or even if I try to sell them in the future).
- most 2400MHz sticks have lower CL latencies (15) than 3000/3200MHz sticks (typically CL16 or CL17), so does this mean that I risk getting lower performance if the extra frequency cannot be exploited?
- 32GB sticks are a sweet spot at the moment, but in the QVL list of the motherboard, no such sticks are tested. If I buy 32GB sticks, do I risk having not working sticks on my system? Has anyone used (non-ECC UDIMMs) such a configuration on an X99 board? Getting 4 sticks to find out they are not working is quite costly...

Thank you in advance!!
Andreas

 

[ericloewe]

1. does it make sense (under no circumstances I want to get into overclocking and such of the CPU), to buy sticks with speed grades higher than 2400MHz? Are there any performance benefits? I can see a few things that trouble me:

Latency could conceivably be lower.

- most 2400MHz sticks have lower CL latencies (15) than 3000/3200MHz sticks (typically CL16 or CL17), so does this mean that I risk getting lower performance if the extra frequency cannot be exploited?

No, if you do the math, absolute latency (i.e. time, not cycles) tends to go down as frequency goes up. 3200 MHz/CL17 is lower latency than 2400 MHz/CL15

 

[kingmouf]

No, if you do the math, absolute latency (i.e. time, not cycles) tends to go down as frequency goes up. 3200 MHz/CL17 is lower latency than 2400 MHz/CL15

The real question is whether the processor will be able to work with these kind of frequencies. If the processor clocks down the 3200/CL17 memory to 2400/CL17, this is going to be worse. I know I may be missing or understanding something wrong...

 

[alex_stief]

In general: don't buy lower binned memory modules, just because your current CPU only supports a certain transfer rate. If it's not a huge price difference and you already know that you will upgrade at some point, get faster modules. Only caveat here: if your next upgrade is to a platform that requires DDR5, spending a single cent more on faster DDR4 is pointless, unless you account for resale value.

ericlowe already explained how memory latency works, so I'll give my take on last question:

32GB sticks are a sweet spot at the moment, but in the QVL list of the motherboard, no such sticks are tested. If I buy 32GB sticks, do I risk having not working sticks on my system? Has anyone used (non-ECC UDIMMs) such a configuration on an X99 board? Getting 4 sticks to find out they are not working is quite costly...

QVLs are prepared when a product launches, and are rarely updated later on. And 16GB was the max capacity UDIMM at the time. So it is possible that 32GB UDIMMs might work with this board. Then again, I can't recall seeing anyone bragging about 256GB UDIMM on X99... I'd say chances are rather slim.
From my experience, it is actually worth contacting ASRock support about this. Every time I had to deal with them, I quickly got someone who actually gave a **** and could answer questions like these. Instead of just pointing me towards the QVL, as many other vendors do.

The real question is whether the processor will be able to work with these kind of frequencies. If the processor clocks down the 3200/CL17 memory to 2400/CL17, this is going to be worse. I know I may be missing or understanding something wrong...

That's not quite how this works. DDR4-3200 CL17 will have more profiles in SPD for lower transfer rates. With tables for latency values. I.e. if you clock down DDR4-3200 CL17 to let's say DDR4-2400, you will get lower latency values with auto settings.

 

[larrysb]

You should look at your board's QVL closely. I use Asus X99 workstation boards with Xeon E5-V4 processors. They will boot and run with non-ECC memory. But, why bother? A set of used, ECC RDIMM server memory is usually less expensive and you can usually buy enough of the same part number to populate the machine.

Fully populating the memory in some cases might mean using LRDIMM...

The Xeon E5-v4 will not run the memory any faster than 2400. I wouldn't spend extra for faster, but of course, there's future use to consider down the road, so getting ECC RDIMM of higher speed grade is a good idea as you can preserve the investment a for future build.

I would not bother with commodity memory for desktops or "gamer" memory with XMP clock ratings. Usually, the XMP will have some crazy high number, and the standard JEDEC profile will have some 'safe' very low number, like 2133. That's so the overclockers have something to fall back on.

But your Xeon will NOT run at the XMP profile. It will only run a standard JEDEC setting embedded on the DIMMs.

 

[RageBone]

i second the "don't go higher then 2400mhz" because nearly all e5 xeons are locked and especially locked on the memory side.
The few unlocked E5 xeons will most likely do more though. Had a E51650V3 that took 3200mhz Gskill udimms.

So, larrysb is right.

Some X99 boards do not work with higher capacity memory, others don't give a ****.
Had issues on EVGA boards, Asus didn't give me any problems, don't have Asrock or Gigglebyte.

 

[kingmouf]

From my experience, it is actually worth contacting ASRock support about this. Every time I had to deal with them, I quickly got someone who actually gave a **** and could answer questions like these. Instead of just pointing me towards the QVL, as many other vendors do.


That's not quite how this works. DDR4-3200 CL17 will have more profiles in SPD for lower transfer rates. With tables for latency values. I.e. if you clock down DDR4-3200 CL17 to let's say DDR4-2400, you will get lower latency values with auto settings.

I guess I can check with Asrock support and also verify (if such info is available) the specific profiles for the DDR modules that I can find.

 

[kingmouf]

You should look at your board's QVL closely. I use Asus X99 workstation boards with Xeon E5-V4 processors. They will boot and run with non-ECC memory. But, why bother? A set of used, ECC RDIMM server memory is usually less expensive and you can usually buy enough of the same part number to populate the machine.

Fully populating the memory in some cases might mean using LRDIMM...

The Xeon E5-v4 will not run the memory any faster than 2400. I wouldn't spend extra for faster, but of course, there's future use to consider down the road, so getting ECC RDIMM of higher speed grade is a good idea as you can preserve the investment a for future build.

I would not bother with commodity memory for desktops or "gamer" memory with XMP clock ratings. Usually, the XMP will have some crazy high number, and the standard JEDEC profile will have some 'safe' very low number, like 2133. That's so the overclockers have something to fall back on.

But your Xeon will NOT run at the XMP profile. It will only run a standard JEDEC setting embedded on the DIMMs.

I always thought that ECC memory would cost me much more, but what you suggest opens new possibilities that are worth exploring.

 

[larrysb]

I always thought that ECC memory would cost me much more, but what you suggest opens new possibilities that are worth exploring.

Yeah, I've got dozens of ECC RDIMMs from Flea-Bay and what not. I've found it pretty easy to buy 8-sticks of used quality server memory, all from the same production lots. I have some 2400 that originally went into a E5-v3 that only ran at 2133. Buying a little faster than you need makes a bit of sense if you think you'll re-use it. Often, it is only maybe a couple of bucks a stick difference in price.

When server farms get upgraded, they'll pull hundreds of sticks of memory at a time. I bought 16 sticks of 32gb DDR4 2666 not too long ago, all Samsung from the same production lot. 24 hour burn in on Memtest86 and none of them had any issues. The seller had hundreds of them available.

 

[kingmouf]

Yeah, I've got dozens of ECC RDIMMs from Flea-Bay and what not. I've found it pretty easy to buy 8-sticks of used quality server memory, all from the same production lots. I have some 2400 that originally went into a E5-v3 that only ran at 2133. Buying a little faster than you need makes a bit of sense if you think you'll re-use it. Often, it is only maybe a couple of bucks a stick difference in price.

When server farms get upgraded, they'll pull hundreds of sticks of memory at a time. I bought 16 sticks of 32gb DDR4 2666 not too long ago, all Samsung from the same production lot. 24 hour burn in on Memtest86 and none of them had any issues. The seller had hundreds of them available.

I am off to ebay!!!!!! I contacted also the Asrock support and waiting their answer on the 32GB sticks...

 

[kingmouf]

For anyone considering similar things as I am, I post here the answer I got from Asrock support Europe:

Hello,

32GB modules should be okay with Xeon on this motherboard. But when this board was released, 32GB unbuffered RAM was not available yet. As far as I know this has not been tested.
So RDIMM should be fine. LRDIMM and UDIMM not sure. I will ask a colleague in Taiwan if he can provide more precise information. Thanks.
Kind regards,

ASRock Support

 

[kingmouf]

So I managed to gather four (4) 32GB HyperX DDR4-3200 Predator modules (HX432C16PB3/32) from some work PCs in order to test and they were working perfectly. I couldn't gather any more 32GB sticks, so I cannot verify if populating all 8 slots would make any difference.

 

[RageBone]

so I cannot verify if populating all 8 slots would make any difference.

it does not in my experience.
HyperX aren't Reg ECC sticks right?
Your mileage might vary with Reg ECC.