TDP & turbo limits vs. actual frequency under load (Xeon E5 v1)

[Stereodude]

In a given CPU family Intel sells CPUs at a variety of frequencies that have the same TDP. As far as I understand it for Sandy Bridge CPUs:

• The BIOS/motherboard/processor will enforce the TDP limit, but allows the CPU to temporarily exceed by 1.2-1.3x it for some amount of time.
• With everything else being equal power consumption increases as frequency goes up.
• TDP != power consumption

So, for example from the v1 Xeon E5 family:

• Xeon E5-2665 has a base clock of 2.4gHz and turbo limits of 4/4/5/5/6/6/7/7 with a TDP of 115W
• Xeon E5-2670 has a base clock of 2.6gHz and turbo limits of 4/4/5/5/6/6/7/7 with a TDP of 115W
• Xeon E5-2689 has a base clock of 2.6gHz and turbo limits of 7/7/7/7/8/8/10/10 with a TDP of 115W

How can they all have the same TDP limit with such different potential capabilities? Does the E5-2689 throttle far more from it's theoretical capability (all 8 cores at 3.3gHz) due to the enforced TDP limit than the E6-2665 (all 8 cores at 2.8gHz)? Do the E5-2689 chips have a lower VID at a given frequency (from binning the better dies) vs. the E5-2665 which allows them to perform much faster despite the same TDP limit, with the E5-2670 somewhere in the middle?

Ultimately, I guess the main question is: Do the two higher end 115W TDP chips really run appreciably faster than the 2665 for sustained workloads stressing all 8 cores or do they simply offer higher short term performance and higher performance when only a few cores are loaded?

Then the next logical question is: Can the E5-2689 really offer the same performance under a long term 8 core load as the E5-2690 despite a 20W lower TDP? Both are supposedly capable of 3.3gHz turbo on all 8 cores if Wikipedia is correct. That seems hard to believe unless the E5-2689 has a better "bin" of silicon with much better VID vs. frequency characteristics than the E5-2690.

I read through this Intel presentation, but it doesn't really answer my questions. I did glean from it that the sustained clock frequency (once the TDP limit is being enforced) can exceed the base clock, but isn't guaranteed. It makes the whole thing seem like a big, "Well, it depends..." with a lot of potential chip to chip variation even with two identical model CPUs.

Can anyone explain to me how this works, or point me to something that explains this?

 

[T_Minus]

If memory serves me the E5-2689 is a specific part like the L5639 was made for a certain systems builder.

The 2690 can perform on some cores at higher frequency (3.8) and the base frequency for all cores is also 2.9 which are both improvements over the E5-2689.

I didn't look into # of cores at which frequency but what you said would make sense.

Until recently the E5-2689 were selling for $300+ ... I know I sold a couple to replace with 2670s since we're not running 100% often, and for the cash why not

If you have a compute work load that utilizes all cores and you can benefit from the higher frequency then maybe it's ok for you? But $600 for 2 CPUs is enough to get 4 2670s + 1 - 2 motherboard, so depending no your compute work load if you can spread it out you'll likely see better performance by doubling the cores.

 

[Patrick]

This is a HUGE topic. The processors are basically making decisions on clock frequencies based on overall power limits for the chip. It is also one of the technologies that has seen major revisions in each generation.

I have a piece idea in the April 2016 box for more about how this works.

 

[Stereodude]

If memory serves me the E5-2689 is a specific part like the L5639 was made for a certain systems builder.

The 2690 can perform on some cores at higher frequency (3.8) and the base frequency for all cores is also 2.9 which are both improvements over the E5-2689.

I didn't look into # of cores at which frequency but what you said would make sense.

Until recently the E5-2689 were selling for $300+ ... I know I sold a couple to replace with 2670s since we're not running 100% often, and for the cash why not

If you have a compute work load that utilizes all cores and you can benefit from the higher frequency then maybe it's ok for you? But $600 for 2 CPUs is enough to get 4 2670s + 1 - 2 motherboard, so depending no your compute work load if you can spread it out you'll likely see better performance by doubling the cores.

Well, that's the key question. If the TDP limit is going to clamp the 2670 and 2689 to the same frequency for tasks that take hours it's not worth any more money let alone several times as much money. Perhaps the customer who drove the existence of the part had some odd workload that needed all 8 cores at as high of a frequency as possible for only a few seconds at a time. The 2689 would be better at that than the 2670 despite the same TDP.

The 2689 does have better Passmark scores than the 2670, and is closer to the 2690 than the 2670, but if the Passmark test only take a few seconds to run (I don't know how long it takes) it would be testing the 1.2-1.3x TDP window, not long term sustained performance.

 

[Patrick]

The 2689 does have better Passmark scores than the 2670, and is closer to the 2690 than the 2670, but if the Passmark test only take a few seconds to run (I don't know how long it takes) it would be testing the 1.2-1.3x TDP window, not long term sustained performance.

Passmark and Geekbench will not get you anywhere near the heat soak needed for getting real use benchmark numbers from these Xeons. TBH those tests, and especially user-submitted tests of those are borderline useless for these chips.

Just to give you an idea, I load all threads for 5 hours straight before starting the official Linux-Bench run loops and have to ensure cooling is working during that time.

 

[T_Minus]

Well, that's the key question. If the TDP limit is going to clamp the 2670 and 2689 to the same frequency for tasks that take hours it's not worth any more money let alone several times as much money. Perhaps the customer who drove the existence of the part had some odd workload that needed all 8 cores at as high of a frequency as possible for only a few seconds at a time. The 2689 would be better at that than the 2670 despite the same TDP.

The 2689 does have better Passmark scores than the 2670, and is closer to the 2690 than the 2670, but if the Passmark test only take a few seconds to run (I don't know how long it takes) it would be testing the 1.2-1.3x TDP window, not long term sustained performance.

Well, I monitored each core in linux on the 2x E5-2670 as I pushed them all to 100% for nearly an hour.
I believe the cores all were shuffling between 2.99Ghz and 3.01Ghz during that time.

This is with temps in the mid/high 70*C on each core too. Depending on the fan setting I could push them t0 80*C in <20 minutes too. With 100% FAN on CPU you can keep them <70*C at 100% load though too...

If that helps you at all.

 

[Stereodude]

Passmark and Geekbench will not get you anywhere near the heat soak needed for getting real use benchmark numbers from these Xeons. TBH those tests, and especially user-submitted tests of those are borderline useless for these chips.

Just to give you an idea, I load all threads for 5 hours straight before starting the official Linux-Bench run loops and have to ensure cooling is working during that time.

So then short of buying one of each and testing, how can I tell if the 2689 is going to be faster than the 2670 for what I want to do (x264 compression)?

Well, I monitored each core in linux on the 2x E5-2670 as I pushed them all to 100% for nearly an hour.
I believe the cores all were shuffling between 2.99Ghz and 3.01Ghz during that time.

Well, that would suggest that you're not hitting the 115W TDP with the max turbo settings (all core loaded). Otherwise it would be throttling it down. Of course how much headroom is left under the 115W TDP and if there's enough headroom to let the 2689 do that same thing at 3.3gHz, making it 10% faster, is the big unknown.

I'd like to know before plopping down 3x as much for a 10% boost that isn't really there.

 

[T_Minus]

@Stereodude will your work load have tasks in queue for the CPU to deliver instantly upon completion of the previous? If not the CPU will cool and slow near instantly core-by-core as they wait (even split second) for the next data to crunch.

 

[Stereodude]

@Stereodude will your work load have tasks in queue for the CPU to deliver instantly upon completion of the previous? If not the CPU will cool and slow near instantly core-by-core as they wait (even split second) for the next data to crunch.

I'm not 100% sure. I expect the CPU pipeline stays full. The CPU utilization graphs in Windows 7 show the logical cores stay saturated with x264 on my i7-2600k and i7-4770k, but perhaps it's not sampling fast enough.