Rome Epyc 16, 24 and 32 cores sub 200w TDP - 2133 vs 3200 effect on workloads that require multiprocessing over compute

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

nickwalt

Member
Oct 4, 2023
47
14
8
Brisbane
Hi Everyone,

When looking to buy an Epyc system from ebay I noticed that Gen 2 Rome CPUs and motherboards (for example: AMD EPYC 7452 CPU + Supermicro H12SSL-i) were often sold bundled with 2133Mhz DDR4. My initial reaction was to completely disregard these offerings because the standard on desktop is now 3200 and above for AM4. 2133 DDR is unthinkable these days.

I could not find 3200 DDR4 included in the CPU and motherboard kits at a price to fit my budget. So I reconsidered my aversion to 2133 and looked again at my use case. It seems like 2133 is not so much of an issue on multi-core server systems with eight channels that aren't going to perform under high utilisation.

My aim is to build a power efficient virtualisation platform that will host a few persistent services (as either VMs or System Containers aka LXD) and labs for Cisco/Juniper/Arista infrastructure and linux based cloud infrastructure. None of these VMs will place compute or bandwidth demand on the system and really they just require free cores and RAM. The network labs spin up numerous virtual instances which will likely make the biggest demand.

This server will also virtualise a modest NVME based Truenas installation that will install ZFS on NVME SSDs passed through from the host. This will pretty much idle most of the time and most operations will be light.

Because this server will be configured as much as possible to be power efficient at idle and low use it may be that 2133 DDR will provide all the memory performance it needs, especially if all eight DIMM slots are populated.

The motherboard will be a Supermicro H12SSL-i and the CPUs I'm considering are Rome:
- Epyc 7352
- Epyc 7452
- Epyc 7402

The other consideration is whether or not TDP and Base clock should be factors to use to help determine low power efficiency of a particular SKU. Part of my reasoning has been to consider a low Base clock to be more desirable than a higher Base. Same with TDP. However, I'm not sure that it is valid reasoning to think that two CPUs with the same Core count but different TDPs cannot both idle at low power. A 32 core 200w CPU and a 32 core 155w CPU might both have similar idle and low utilisation power behaviour but one will boost higher when demand requires it (more headroom).

My questions then are:
1. for a modest duty (lots of small VM/System Containers) network/cloud "homelab" would cheap 2133 DDR4 be an excellent value proposition?
2. when considering CPUs with desirable core count and cache (looking at cache it seems that 7002 SKUs share similar cache/core design) is it incorrect to disregard higher TDP SKUs as being incompatible for low power applications (again, a homelab running lots of small virtualized loads)?

If YES to both questions I will be able to consider more CPU, Motherboard and RAM combinations that might fit my budget. It seems that on the Chinese Ebay sites (like tugm4470) a few of the high TDP SKUs are more plentiful and therefore cheaper). Thanks!
 
Last edited:

nickwalt

Member
Oct 4, 2023
47
14
8
Brisbane
I guess one way to frame these two considerations, and perhaps think of them as inherently related, is that given the same Epyc architecture (Zen 2) and core count an SKU with a higher TDP/ has more headroom. Give it faster DDR (3200) and load it up with a more demanding workload and it will keep going long after the low TDP SKU has reached its limit.

But if we give them both a workload that the lower TDP SKU can handle without any problem, and the higher TDP SKU has 3200, over the lower TDP SKU's 2133, we might not see any meaningful performance delta.

Is this an adequate understanding of the relationship and between TDP and available memory frequency?

Looking at two similar SKUs:

SKUCoresThreadsWattsL2L3BaseTurbo

74523264155 W16 MiB128 MiB2.35 GHz3.35 GHz
75423264225 W16 MiB128 MiB2.90 GHz3.40 GHz

Will these behave similarly (in terms of power draw and multiprocessing switching/responsiveness) at low to moderate "homelab" loads: 7452 with 2133 DDR and 7542 with 3200 DDR?
 
Last edited:

nickwalt

Member
Oct 4, 2023
47
14
8
Brisbane
I'm also wondering if an Epyc platform for a virtualization platform homelab is going to be a better proposition than a Ryzen AM4 5900/5950x platform based on a motherboard like ASRock Rack X570D4U. This would be the 12 and 16 core Ryzen SKUs against 16, 24 or 32 core Epyc SKUs. Interestingly, pricing for a 5950x (16-cores), X570D4U (ASRock Rack PCIE 4.0) and 128GB non-ECC 3200 DDR4 is similar to a Epyc 7352 (24-cores), H12SSL-i (Supermicro PCIE 4.0) and 128GB ECC 3200 DDR4 platform.
 

bayleyw

Active Member
Jan 8, 2014
311
103
43
Epyc uncore frequency is tied to DDR4 frequency so you take a 30% hit in the fabric latency by using slower memory. Because of the disaggregated design it matters more than on Xeon, but its really hard to predict - the platform gets a bit pathological if you use it outside of its intended 1dpc DDR4-3200 with all four/eight channels populated (some SKUs are optimized for quad-channel memory).

As one data point I had a scientific workload which was not bandwidth bound, but due to a series of unfortunate events wound up with asymmetric memory population on my machine (something like 6x32GB + 2x64GB). It booted fine, and Cinebench performance was great, by this particular workload tanked in performance by almost 2x and to this day I don't know why. Replacing the rogue 2x64GB with 2x32GB solved the problem.

For light home server duty I'm a fan of Skylake-SP, you can get 24 cores at a good price and the monolithic design makes a lot more sense for the smaller processors. Epyc has a full sized IO die no matter what SKU you're running, and the overhead from driving the IF links and keeping the IO die running cuts into your power budget.
 
  • Like
Reactions: nickwalt

nickwalt

Member
Oct 4, 2023
47
14
8
Brisbane
The other option I'm now considering (or, reconsidering) is the Epyc PCIE 3.0 platform with 2133 DDR4. Would it be fair to reason that a virtualization/nas server that has low nas and low to moderate virtualisation requirements would be fine on only PCIE 3.0 and 2133 DDR4? Once the VMs are started the storage access will likely be minimal, requiring loading of switch and router images and file access - ah, IOPs is something that will be required more than raw bandwidth I think.
 

bayleyw

Active Member
Jan 8, 2014
311
103
43
It sounds like you can get by with vanilla Ryzen then, if it's really that lighweight. As long as you aren't using legacy HBAs that need 8 lanes or plan on doing faster than 25 gbps networking you can probably make it fit in the 16+8 lanes that the desktop platforms have to offer.
The desktop parts also idle *a lot* lower than the enterprise parts.
 

drdepasquale

Member
Dec 1, 2022
77
31
18
I recommend only using DDR4 3200 MHz for EPYC 7002 and 7003 processors. DDR4 2133 MHz was the standard in the Intel X99 era when DDR4 memory was introduced. DDR4 has plummeted in price providing a great value for most types of memory. DDR4 3200 MHz is the latest and greatest DDR4 memory standard (excluding gaming memory). This is a comparison of the slowest DDR4 memory versus the fastest.
 
  • Like
Reactions: nickwalt

nickwalt

Member
Oct 4, 2023
47
14
8
Brisbane
Great points, thanks. Checking out Skylake SP. The AM4 platform is a great platform and very mature.

I like the idea of having a larger pool of CPU cores and RAM to carve up amongst VMs and containers so having Epyc's 24 or 32 cores over Ryzen's 16 is appealing. Lots of cores, bigger cache and more memory and memory channels (8 vs 2 on Ryzen) feels like virtualization's best friend and PCIE 3.0 might be fine for what I need if modern NVME M.2 SSDs still provide a lot of IOPs on the lanes.
 

nickwalt

Member
Oct 4, 2023
47
14
8
Brisbane
I ended up stretching my budget to:
- Epyc 7452 32 core 155w
- Supermicro H12SSL-i PCIE 4.0 rome/milan motherboard
- 4 x 32GB DDR4 ECC 3200 RDIMMs
- 4U CPU cooler for SP3 sockets.

Because every component on this system is likely certified for ESXi 8+ I'll build the server on the VMware platform. The Samsung 990 Pro looks like a great consumer drive that is fully recognised by ESXi.
 

DaveLTX

Active Member
Dec 5, 2021
170
40
28
For me, i had to go with 2666 as 3200 was pricey but now 3200 is basically the same price so I would advise AGAINST going for anything less than 2666 or even 3200
My poor epyc 7713 is struggling with 4 channels of 2666
 

ano

Well-Known Member
Nov 7, 2022
657
273
63
we found 8x 2133 will beat 4x 3200 for most stuff, even with 7763
 

DaveLTX

Active Member
Dec 5, 2021
170
40
28
we found 8x 2133 will beat 4x 3200 for most stuff, even with 7763
Bandwidth? Sure. Latency? No.
2133 tends to be way older RAM too, buying 3200 doesn't guarantee it will keep trucking but you got a higher chance of it constantly trucking. 2133 been around since 2015... I know I bought some way back on X99 when ;)
 

mikegrok

New Member
Feb 26, 2023
16
4
3
Alabama, USA
Have you considered epyc 8xx4?
each core is about 60% faster than 7xx2. and 30% slower than 9xx4.
8 cores under $450 90 watt tdp
16 cores under $650 125 watt tdp

6 ddr5 ecc channels
96 pcie5 lanes
8-64 cores

I have seen these showing up on retail sites this week.


anandtech page has all of the prices, and some other details:
 
  • Like
Reactions: RealPjotr

DaveLTX

Active Member
Dec 5, 2021
170
40
28
Have you considered epyc 8xx4?
each core is about 60% faster than 7xx2. and 30% slower than 9xx4.
8 cores under $450 90 watt tdp
16 cores under $650 125 watt tdp

6 ddr5 ecc channels
96 pcie5 lanes
8-64 cores

I have seen these showing up on retail sites this week.


anandtech page has all of the prices, and some other details:
That will also depend on the cost of the motherboards since they have a different socket. And also DDR5 rdimms are particularly painful
Lastly, it's pretty easy and cheap to buy discounted Zen 2 and 3 CPUs
 
  • Like
Reactions: nickwalt

nickwalt

Member
Oct 4, 2023
47
14
8
Brisbane
I think this really is the question that needs answering by those who don't have experience of the extent of the difference in performance — for a given application — between 2133, 2666 and 3200.

If a 32 core CPU has some kind of load (such as VMs or Containers) across the majority of cores and is constantly dipping into memory without stressing bandwidth perhaps 2133 memory populating all slots is going to be fine for a 3200 CPU.

When less slots are populated and or the load stresses bandwidth perhaps this is only where 2133 performance loss is noticeably significant?

Perhaps, single threaded workloads that stress bandwidth on fewer cores will see significant benefit from 3200 memory.

These are the questions I have, although moot for my purchases decision now as I bought 3200 memory.
 

nickwalt

Member
Oct 4, 2023
47
14
8
Brisbane
PCIE 5.0 and DDR5 will be an interesting platform for years to come but most people (as opposed to companies) are probably buying Epyc Rome and Milan on a tight budget, like me, and for modest application.
 
Last edited:

mikegrok

New Member
Feb 26, 2023
16
4
3
Alabama, USA
Yeah,
Ryzen is the sports car
Epyc 9004 is the semi truck
Rome is the schoolie
And 8004 is the delivery van.
Not as big an engine, but efficient and able to carry a load long after the wheels fall off the schoolie, while being price competitive with a schoolie.
Schoolie is a school bus converted after end of life to general cargo use.
 

nickwalt

Member
Oct 4, 2023
47
14
8
Brisbane
I paid for an entire 7452 32 cores what the current generation cost for one core. For a home lab it will be awesome and likely chew through everything I throw at it.
 
  • Like
Reactions: mikegrok

nickwalt

Member
Oct 4, 2023
47
14
8
Brisbane
I didn't get a choice to specify what kind of 3200 or brand. Pretty standard ECC timings and 2Rx4 I would guess. I'm ok with whatever because it was very inexpensive — about 25% more than the cheapest consumer 3200 here in Australia.