Epyc Milan/Rome memory compatibility (3200 v 2400) + overclocking ok?

[robc]

I am wondering about memory speed compatibility with epyc 7003/7002 server I am getting which only lists 3200 as compatible. Server is a Asus
RS500A-E11-RS12U and due to budget reasons I am wanting to fill it with 2400 ram vs 3200 ($60 vs $110 per stick) as the memory performance is not critical (expecting ~20% less memory bandwidth which I can live with). Anything I should be aware of when buying slower ram or just avoid anything slower than 2933 which I think is manufacturer approved?

I have seen that lower memory speeds are generally fine but wondering if anyone has any direct experience with running slower memory and if there are any stability issues I will have to worry about? There will be 3 of these servers in a kubernetes cluster and so while the applications will be setup with fault tolerance in mind, they will be in production and so stability is very important.

And also considering stability, I have also seen many people talk about overclocking memory in epyc platforms which sounds like a horrible idea if anything is to be used in production. Is overclocking really ok and how much if anyone has experience with that?

Thank you for your time.

 

[bayleyw]

I ran 2400 at 2666 on a Rome system for two years and didn't run into any memory related crashes. I think RAM is qualified at high temperatures so you can get away with overclocking if your workload is not memory intensive or your temperatures are low.

 

[robc]

@bayleyw - Thank you for your insight.

 

[ano]

2133 even works fine! 8x 2133 is faster than 4x3200 for msut stuff as well

 

[robc]

Thanks @ano - What if I am eventually planning on running 2dpc (ie 16 dimms) at 2133? Do you think there could be some instability there?

 

[gb00s]

From my experience, you can OC memory on ROME two increments up only, meaning 2133 >> 2666, 2400 >> 2933 or 2666 >> 3200. Nothing else is stable at the least. Also, the motherboard itself is a huge factor here. Supermicro boards always left me with one increment only while Gigabyte and Tyan boards allowed two. I have no specific experience with ASUS board in this specific matter.

 

[robc]

@gb00s - Thank you for your insight.

 

[Advin]

Avoid LRDIMM at all costs

 

[robc]

@Advin - Why is that? Already got 2400 lrdimms. About to start testing and was hoping to overclock 1 step. Memory bandwidth / compute is not a bottleneck in my application so not super concerned.

 

[Advin]

@Advin - Why is that? Already got 2400 lrdimms. About to start testing and was hoping to overclock 1 step. Memory bandwidth / compute is not a bottleneck in my application so not super concerned.

Here's my experience with Samsung 2400 MHz LRDIMM: https://forums.servethehome.com/ind...-no-ipmi-or-no-post-solved.40440/#post-402928 (there is also another user in the thread having similar problems)

I think you might have trouble getting them to even POST, or if you do, you might run into memory errors, DIMM's missing, sometimes not even being able to boot, etc. That's why I would suggest to avoid them if possible. My testing was done on a Gigabyte server. I also did try an ASUS motherboard (KRPA-U16) and I couldn't get LRDIMM to boot/POST at all with Samsung 2400 MHz 4Rx4 LRDIMM no matter which DIMM I tried. Hynix 2400 MHz 4Rx4 LRDIMM I could never get to even POST.

@RolloZ170 also put his experience here with Hynix 64GB 2400 LRDIMM: https://forums.servethehome.com/index.php?threads/ram-for-7003-epyc.38902/#post-364263

Highly would suggest going with either 2Rx4 memory and eating the cost, or finding 64GB 2S2Rx4 2400 MHz / 2666 MHz which should work a lot better (and 2S2Rx4 are similar in costs, usually $55-$65/DIMM).

 

[slidermike]

Are there generally accepted brands to get when wanting to OC 2600 < 3200 rdimm, populating all 8 slots?
I have a romed8-2t and was considering the hynix because for home grade they make great chips but with @Advin comment above and my lack of experience with server grade quality. What I do know in my limited experience is the Samsung rdimm I would steer away from.

Also, whats the deal with warranty?
Samsung is next to impossible to find if there even is one for the rdimms I have.
I see some resellers offer 30day, 1yr or even lifetime through themselves but do manufacturers have warranties you can exercise as a home/prosumer?

I would consider value in either a vendor or reseller with a warranty.
Thanks

 

[zachj]

If I were your boss and I found out you overclocked production I’d fire you for taking needless risk.

The only time overclocking in a workplace setting makes any sense is when everyone in your leadership chain understands what it is, why it’s necessary and are telling you to do it—so maybe .000001% of the time.

 

[robc]

@Advin - Not going to lie, your post freeked me out as I had already bought 3tb of samsung 64gb lrdimms - M386A8K40BM1 which is 4rx4 but so far after about 10 hours of memtesting have not seen an issue. Going to do another day or so of testing but I think I should be ok. I'm running on an asus rs500-e11 with a 7532 cpu.

 

[robc]

If I were your boss and I found out you overclocked production I’d fire you for taking needless risk.

@zachj - I think you are totally right. That was my thought going into this but then I saw a lot of people doing it saying it was fine up to 2 steps above. Anyways, not worth it really.

 

[zachj]

I run two 2933mhz dimms at 3200 in my homelab. I’m not opposed to doing it and neither are a bunch of people here—just not at work.

There are some high frequency trading shops that run on overclocked CPUs. It’s not unheard of. Just rare that anyone has need to or willingness to take the risk.

 

[robc]

Yeah this is going to be in production - no need to take the risk. Total memory capacity is more important. Anyways, main concern was using slower memory and have done several passes in memtest now so all is good.

 

[Advin]

@Advin - Not going to lie, your post freeked me out as I had already bought 3tb of samsung 64gb lrdimms - M386A8K40BM1 which is 4rx4 but so far after about 10 hours of memtesting have not seen an issue. Going to do another day or so of testing but I think I should be ok. I'm running on an asus rs500-e11 with a 7532 cpu.

The thing is, memtest isn't the deciding factor. I was also able to get it to where sometimes all LRDIMM's would show up and then pass memtest, but then when I rebooted the server it just wouldn't boot anymore. One time I built a server, passed memtest, completely tested, but then when it arrived at the datacenter it just wouldn't boot anymore or had DIMM's missing or ECC memory errors in the system. In fact, I had this problem twice. Once with MZ32-AR0 and Samsung LRDIMM with EPYC 7B13 and once with Tyan motherboard with EPYC 7502. Replacing with RDIMM fixed problem.



However, I also have 2 servers in production with 1TB Samsung LRDIMM each which also has been working perfectly fine, so I think it might be dependent on the kit that you get from your seller (maybe based on age due to timings? don't exactly know). I am still stressed that one day I will reboot these systems and it will just never boot again.

Try warm rebooting the server (using `reboot -n` or IPMI) a bunch of times, ideally just make a Linux install and have it run the reboot command on startup and let it go for a few hours. If it passes that, then I think you might be alright, just avoid buying LRDIMM in the future and spend the little bit extra (or even the same) on either 2S2Rx4, or spend the extra on RDIMM.

 

[Advin]

Are there generally accepted brands to get when wanting to OC 2600 < 3200 rdimm, populating all 8 slots?
I have a romed8-2t and was considering the hynix because for home grade they make great chips but with @Advin comment above and my lack of experience with server grade quality. What I do know in my limited experience is the Samsung rdimm I would steer away from.

Also, whats the deal with warranty?
Samsung is next to impossible to find if there even is one for the rdimms I have.
I see some resellers offer 30day, 1yr or even lifetime through themselves but do manufacturers have warranties you can exercise as a home/prosumer?

I would consider value in either a vendor or reseller with a warranty.
Thanks

I don't know that much about overclocking server RAM, but as sorta mentioned before you might be better off just getting 3200 MHz RDIMM instead of overclocking. If it's a homelab, then it might be OK if you have an easy way to clear the CMOS and can risk an outage from it going down, but going from 2666 MHz to 3200 MHz on server RAM is probably not worth it compared to the risk of problems or weird behavior happening especially since the performance advantage is probably more minor.

For warranty, I'm not entirely sure but some eBay sellers and such will provide their own warranty, such as A-Tech which could maybe honor it after a while. I haven't checked if used RAM can be claimed under Samsung warranty but I'd say probably not unless you buy directly since a lot of the modules on eBay are probably older and probably don't come with invoices.

 

[robc]

@Advin - I really hope I don't have to deal with that. Been running tests and restarting machines a lot and no issues thus far. I'll remember this though but unfortunately money has been spent and these guys need to go into a data center.

Thanks again for your comment though.