Move from Xeon to EPYC (Virtualization)

[usthm]

Hi,

currently i'm running 13 VMs on a Xeon 6C@2.4GHz E5-2620 v3 (Haswell), 32GB DDR4 memory, ESXi 6.5 + latest patches from 03-2018. As storage i use a RAID10 (Megaraid + Samsung SSDs)

- 3 VMs have 2 vCPUs (2x BSD, 1x Windows) +4 GB mem
- 10 VMs have only 1 vCPU + 2GB mem

Besides adding two additional VMs in the next few weeks....reasons for an upgrade:

- after the latest patches ESXi + microcode, at least two disk intense VMs are 'laggy'. This affects a postgresql system + Windows (lags when switching folders, opening files). HDTune shows higher random access ms. -> less issues with AMD according to CPU, IO tests.

- 3-4 systems could benefit from having access to 2 vCPUs, or even 4 vCPUs (as i'm planning to go for a EPYC 7401P with 24C there won't be any overprovisioning at all)

- my backup hypervisor is getting old, so i could use this Xeon 6C as backup

- damn, i like to play with new hardware

Planned new system:
- Supermicro board
- EPYC 7401P
- 64 GB Samsung dual rank 2666Mhz memory
- Supermicro 2U heatsink

What's your opinion on this? My main concern with respect to virtualization is the higher base clock speed of my current Xeon. Will the EPYC still perform better/equal with 1 vCPU assigned (only 2.0 GHz but better overall performance)? Imo the big benefit is, i can run multiple systems with 2 vCPUs assigned without any overprovisioing issues, so if all systems use their ressources at the same time, there won't be any bottlenecks. That's not the case now. 13-15 VMs on an 6C.

Thanks

 

[alex_stief]

Epyc CPUs (just as any other CPU I ever owned or used) rarely run at base clock speeds.
AMD Epyc CPUs have all-core turbo clock speeds between 2.55 and 2.9 GHz https://www.amd.com/system/files/2017-06/AMD-EPYC-Data-Sheet.pdf
And at least in my experience, they beat 3rd gen Intel Xeons in terms of performance per frequency. So you should be fine.

 

[usthm]

Thank you very much for your reply. One memory related thing. EPYC seems to be pretty picky when it comes to memory (dual rank vs single rank, B-Die, speed). I plan on picking eight 2Rx8 DDR4 2666 from Samsung (M393A1G43EB1-CTD). I hope that fits

 

[alex_stief]

It should work, you might just have to dial it down to DDR4-2400. With 1DPC dual-rank, AMD Epyc is only certified for DDR4-2400.
If you are lucky they might run at DDR4-2666 with the latest bios version. Nobody knows for sure
For applications that are not memory bandwidth limited, single-rank DDR4-2666 might be slightly better than dual-rank DDR4-2400 thanks to lower IF latencies. But again, this is more of an extrapolation from what I know so far, I never saw any actual benchmarks to back up my hypothesis.

 

[usthm]

So if it drops to 2400 i have no benefit compared to 2666 single rank? Hmmm...well as long as it doesn't get worse.

 

[alex_stief]

Just to be clear: Infinity Fabric latencies are lower with higher RAM frequencies: https://www.servethehome.com/amd-epyc-infinity-fabric-latency-ddr4-2400-v-2666-a-snapshot/
My point is just that AMDs Zen Architecture favors dual-rank memory. Just as other architectures to some lower degree where dual-rank can yield higher memory throughput than single-rank. All benchmarks I saw for Ryzen CPUs reported similar or slightly higher performance with dual-rank 2400 vs single-rank 2666 over a wide variety of applications.
Unfortunately, I am still waiting for benchmarks that confirm this for Epyc CPUs.

Edit: just don't overthink it. The performance differences are less than 5%.

 

[usthm]

I thought i've read everything on sth regarding EPYC...thanks for the link. According to Patricks benchmarks it'll be safer to order 2666 single rank...otherwise i could loose up to 8% because of slower infinity fabric speeds. I think contacting AMD/Supermicro would be best here.

 

[alex_stief]

Infinity fabric latencies are not everything that matters here.
Like I said, other benchmark suggest higher overall performance with dual-rank than with single-rank, at least for Ryzen CPUs.

Contacting AMD or Supermicro: Good luck with that, let us know what they are willing to share

 

[usthm]

Haha...thanks (mail went out)

 

[usthm]

I just found out, that the only Samsung B-Die (most recommended) Dimm with 8GB @ 2666 is the single ranked

M393A1K43BB1-CTD

Both the M393A1G43EB1-CTD / M393A1G40EB2-CTD (listed @Supermicro) are E-Dies

btw: no response from Supermicro/AMD yet (not that i've expected that in the short time)

 

[usthm]

Feedback from SM. Most SM boards will only have 1DPC because of space restrictions and that's why it should almost always be possible to run @2666MHz...nonetheless running memory listed on the SM page is recommended.

Dunno what to do now. In this thread on sth you can clearly see that 2666MHz dual rank drops to 2400MHz on a Gigabyte board. EDIT: ...and manually setting memory speed via bios @2666MHz is obviously not a good idea.

 

[alex_stief]

Well they are currently listing some dual-rank DDR4-2666 in their tested memory list. Hard to tell if that means tested at DDR4-2666 though
I have very little trust in this particular company.

 

[Patrick]

Dunno what to do now.

Do single rank with EPYC. I just looked at two machines where I know AMD checked the configs. Both are using Samsung M393A2K40BB2-CTD.

Memory Device
        Array Handle: 0x0034
        Error Information Handle: 0x005B
        Total Width: 128 bits
        Data Width: 64 bits
        Size: 16384 MB
        Form Factor: DIMM
        Set: None
        Locator: DIMM_P0_G1
        Bank Locator: BANK 1
        Type: DDR4
        Type Detail: Synchronous Registered (Buffered)
        Speed: 2666 MHz
        Manufacturer: Samsung
        Serial Number: 030F1259
        Asset Tag: Not Specified
        Part Number: M393A2K40BB2-CTD
        Rank: 1
        Configured Clock Speed: 2666 MHz
        Minimum Voltage: 1.2 V
        Maximum Voltage: 1.2 V
        Configured Voltage: 1.2 V

 

[usthm]

Ok, thanks. I'll go for the B-Die M393A1K43BB1-CTD then.

 

[jerrytsao]

EPYC - Samsung SR 16GB RDIMM, because of weak IMC and Octa Channel even 2R RDIMM in 1DPC configuration will slow down to 2400, would have to give up interleaving benefit from 2R and 4R DIMMs to favor frequency due to IF frequency plays a huge role in Zen performance.

Xeon SP - Samsung DR 32GB RDIMM or better wait for the recently showed 2R 64GB RDIMM based on 16Gb M-die on OCP Summit, Intel got rid of 3SPC configuration, and with better IMC all six channels will stay on 2666 for both 1 & 2DPC no matter how many ranks each DIMM has.

 

[alex_stief]

Still waiting for real-world performance benchmarks AMD Epyc dual-rank DDR4-2400 vs single-rank DDR4-2666

 

[usthm]

@jerrytsao: 32GB or even 64GB for a single dimm is too much if i need 8x in total.

@alex_stief : yes...and hopefully (if i bought the wrong ones) memory is going to be a lot cheaper in near future.

To make the confusion perfect....are there any important EPYC related differences between

(1G x 4) x 18 - (1G x 8) x 9 (logical organization)