Epyc Rome, L3 Cache and CCX-es and CCDs

[Gordan]

I've been reading around this subject and I'm trying to figure something out.
My understanding that the L3 cache on Epyc Rome consists of 16MB of L3 cache per CCX, and there are 2xCCX-es per CCD.

Epyc 7402 has 24 cores, which implies 3x CCDs of 8 cores and 32MB of L3 cache each. Which would be fine except that adds up to 96MB of L3 cache and the spec says 128MB of L3 cache.

I can see only two possible explanations, both of which involve the presence of 4x CCDs on the 7402 CPU package. So either
1) each CCD has only 6 enabled CPU cores, or
2) there are three CCDs that are fully enabled and one CCD that only has the L3 cache enabled.

I think option 1) is more likely.

How thoroughly are the missing cores disabled? What are the chances of a BIOS edit being or a MSR setting, or soldering the missing SMD components on the bottom of the CPU getting the missing cores back online?

 

[RageBone]

I don't think it is completely known that it is 4 CCDs though that makes sense.

Except that i think 8 CCDs with 3Cores each would make more sense.
But the amount of cache does say otherwhise.

I guess we will only know, once someone, probably der8auer delids one.


Chances of enabling those deactivated cores are very very slim.
Mainly because those will be disabled for a reason. Mainly defects that would make the core unusable, and unreliable.

Some would like you to believe that such things get "lasered off" which I don't think to be true at all.
Not in the mass production at least.

Another factor is that AMD is pretty sparse on public documentation compared to Intel, even though Intel ist still the king of NDAs.

That makes it kind of hard to figure such things out. Work to understand the PSP deeper has basically just started, and that would be one of those places that could maybe maybe maybe do it.

The components on the bottom and between the CCDs are normally Multi Layer Ceramic Capacitors in various sizes and packages. I doubt that those even can be responsible for disabling a core.



On the second theory just to destroy it, if you look at mantissa, 3700x, 3800x, those are all Single CCD CPUs, so if it were 3 full CCDs with one disabled one, why then even have it?
Not "just for the cache"
And good, good performing 8C CCDs are that what's rare and in very high demand for Rome.

 

[Gordan]

Laser cutting the power inputs for the core is the usual way of disabling them. It is rather rare that the cores are disabled via firmware (e.g. in early Vega 56 that could be turned into Vega 64 by flashing the Vega 64 firmware onto it).

Doing it for defect management purposes would make sense, given the disproportional price jump from 24 to 32 core CPUs, but the silicon quality seems to be extremely good. I am seeing my 7402P boost to maximum boost clocks on all cores 100% of the time under load, with regular air cooling, and it never gets to 50C, implying very low current leakage.

 

[zir_blazer]

As far that I know there is no list of any sort that tells you the internal composition of each EPYC Rome model. It would be fun to do so for someone that has the parts.

Keep in mind that while a CCD has two CCXs, because they are not interconnected at the CCD level as each CCX has a totally independent Bus to the IO die, a single CCD with 2 CCX and two CCDs with 1 CCX each are functionally equivalent. For this reason, I don't know if it is possible to check how many CCDs an EPYC model has via Software. I expect that due to physical symmetry and Heatsink load pressure it makes sense that the Processors uses either 4 or 8 CCDs and not any odd numbers.
Is possible to get an imaging of sorts like x-rays that allows to see how many CCDs a particular EPYC has? It would be the best confirmation.

 

[diogin]

A single CCD with 2 CCXs share the same L3 cache, while two CCDs with 1 CCX each do not. This may be the key to differentiate them.
But why AMD choose not to connect two CCXs in the same CCD for inter-CCX communication? It should be faster then going through IO die..

 

[zir_blazer]

A single CCD with 2 CCXs share the same L3 cache, while two CCDs with 1 CCX each do not. This may be the key to differentiate them.
But why AMD choose not to connect two CCXs in the same CCD for inter-CCX communication? It should be faster then going through IO die..

The Cache L3 in each CCX is exclusive. They are NOT shared between CCXs in a CCD. Otherwise the CCX-to-CCX latency wouldn't be the same across all CCXs in the entire Processor.