E5-2640 v3 running warmer than expected

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

nezach

Active Member
Oct 14, 2012
210
128
43
I have an ESXi system running in a Supermicro 826B 2U chassis with 4x FAN-0074L4 midplane fans.

It was running E5-2650 v2 / X9DRD-7LN4F-JBOD (single CPU) with passive 2U heatsink and was happily idling at 30C. Now I upgraded to E5-2640 v3 / X10SRH-CF and with passive heatsink it was idling at 43C. I switched to active 2U SM cooler and it is still idling at 42C. My expectation was that upgrading to v3 from v2 will keep CPU temp the same or maybe even drop by a degree or two, but instead it jumped up by 10C. I am using optimal fan mode, but switching to full fan mode only drops temp to 40C.

lm-sensors is showing me that Package Id 0 temp is 43C (matching what IPMI is reporting), but hottest core is only at 34C, now I haven't paid much attention to the difference between package and core temp in the past so I don't know if this is normal or not.

I don't have another v3 CPU that I could try, but I am leaning towards there being something wrong with this CPU. Any ideas?
 

alex_stief

Well-Known Member
May 31, 2016
884
312
63
38
Any particular reason why idle temps are so important to you? They are hard to judge, can be all over the place, and most importantly: they are mostly meaningless, unless you hit really high temperatures. 43°C doesn't count here.
Load temperatures are where it's at.
 

gb00s

Well-Known Member
Jul 25, 2018
1,175
586
113
Poland
.... Load temperatures are where it's at.
Absolutely with you.

In my opinion, there's no real reason to give package temp any attention until core temp figures are seriously out of 'sight'. I think to provide just the 43C isn't enough to judge about this number. It depends on the delta to your room temp. The problem with these cases is sucking 'warmed up' air from the HDD's etc .. Your ram sticks add up and the cases are not well known for superior airflow for semi-passive solutions. Your BIOS settings play a role if you have power or energy efficiency set up ... We can go further and further.
 

nezach

Active Member
Oct 14, 2012
210
128
43
Normally I wouldn't care about idle temps, but this server is in a home environment and higher temps cause higher fan speed, thus higher noise. From what I have seen IPMI uses CPU package temp for fan speed control, so even though core temps are low, fans ramp up anyway.

I removed all HDDs from the front of the chassis for testing, I made sure all energy efficiency settings are enabled in BIOS and ambient temp is fairly constant so it does not explain 10c jump in CPU temp.
 

gb00s

Well-Known Member
Jul 25, 2018
1,175
586
113
Poland
Don't forget your CPU on the X9DRD-7LN4F was sitting much closer to the fans in CPU1 socket compared to the X10SRH-CF (I assume it was sitting in socket 1 and not 2). Passive heatsinks need lots of static air pressure and flow. If you set the same fan just 5cm more away from the heatsink, the efficiency decreases rapidly.

What's your distro? Did you detect sensors again after the switch of the CPU or you re-installed the distro again? Same kernel version and config? Do you compile your kernel by yourself? I mean, they are two different processor generations aka Ivybridge vs Haswell. I've seen differences while using corrected CFLAGS.

kernel_config.png

Is the heatsink sitting well after re-installation? Do you use other thermal paste ... etc etc.
 
Last edited:

nezach

Active Member
Oct 14, 2012
210
128
43
Don't forget your CPU on the X9DRD-7LN4F was sitting much closer to the fans in CPU1 socket compared to the X10SRH-CF (I assume it was sitting in socket 1 and not 2). Passive heatsinks need lots of static air pressure and flow. If you set the same fan just 5cm more away from the heatsink, the efficiency decreases rapidly.
Yes, it was sitting in socket 1. I thought about that, that's why I switched to an active cooler, but that made little difference. I also tried active cooler in push as well as pull configurations.

What's your distro? Did you detect sensors again after the switch of the CPU or you re-installed the distro again? Same kernel version and config? Do you compile your kernel by yourself? I mean, they are two different processor generations aka Ivybridge vs Haswell. I've seen differences while using corrected CFLAGS.
Using Ubuntu and ESXi both fresh installs. What lm-sensors is reporting under Ubuntu matches what IPMI is reporting. Running Ubuntu vs ESXi does not affect idle temp much, so I doubt this is software related.

Is the heatsink sitting well after re-installation? Do you use other thermal paste ... etc etc.
I checked heatsink after installation and thermal paste distribution was fine and re-seating it did not make any difference.
 

gb00s

Well-Known Member
Jul 25, 2018
1,175
586
113
Poland
To be honest I have no other idea. I mean, Haswell does run with higher temps, but 13C in idle? Never experienced such 'high' idle temps with Haswell processors in 3/4U environments and active cooling with Noctuas or water. My average delta in idle is always 5-7C around 22C room temp. Becomes lower the higher the room temp is. But to make a correct comparison, you need factors like room temp when you measured the Ivy Bridge install and the Haswell install. Different boards makes it a different environment with different voltages etc.. But as you said:
.... I am using optimal fan mode, but switching to full fan mode only drops temp to 40C.
it might by really an air flow issue or the cooler doesn't sit well on the IHS of the new CPU. But if you wouldn't have the requirement to have the system quiet, I wouldn't bother at all with 43C untill the high peak temps in stress tests are still acceptable.
 

nezach

Active Member
Oct 14, 2012
210
128
43
Yesterday I did some load testing and the CPU temp did not go over 62C while running mprime/prime95 small FFTs (16 threads), so there is plenty of cooling capacity from CPU hsf and system fans. During testing I noticed that reported CPU package temp matches temp of the hottest core, unlike during idle.

So I googled a bit to figure out what exactly that package temp is and how it is measured and I found the answer on HWiNFO forums:
CPU Package: Shown on Intel CPUs represents a 256-millisecond average value (calculated by CPU) of the hottest temperature sensor within the CPU package.
Core #n (n=any number): Actual temperature of a particular CPU core.
I figured that means that during idle there is a spot within the CPU that is hotter than any of the cores. Maybe heatsink really does not sit properly on the CPU and there is a hotspot somewhere? So I removed heatsink again and started looking to make sure there is really a good contact between heatsink and the CPU and of course Supermicro heatsink surface is far from being even. As a matter of fact there is a huge valley where the CPU contacts the heatsink. Well, I had to do something I haven't done since AMD Duron Spitfire days and lap the heatsink.

After lapping temps only dropped maybe by 1C, but now package temp even during idle matches hottest core.

For now I just installed Noctua low noise adapter on the CPU fan and will look into setting PWM duty via IPMI raw command in the future.
 
  • Like
Reactions: alex_stief

gb00s

Well-Known Member
Jul 25, 2018
1,175
586
113
Poland
Sometimes the IHS doesn't fit properly and is not straight horizontal. Btw, you can check temps with i7z under root in realtime. I don't use lm-sensors. Check if you see a proc running on the specific core who's temp is spiking. I still think you have an air flow issue if under load and ramped up fans you have proper cooling, but not in idle and low fan revs. It shows that only with ramped up fans you have enough flow and pressure between the cooling fins. I know you wrote you installed an active 2U SM cooler. Doesn't mean you have the right cooler. If the active cooler (fan) doesn't have the power to get rid of the hot air, you can ramp the fans up as long as you want. It's not that simple.

PS: What SM cooler are you using?
 

nezach

Active Member
Oct 14, 2012
210
128
43
Sometimes the IHS doesn't fit properly and is not straight horizontal. Btw, you can check temps with i7z under root in realtime. I don't use lm-sensors. Check if you see a proc running on the specific core who's temp is spiking. I still think you have an air flow issue if under load and ramped up fans you have proper cooling, but not in idle and low fan revs. It shows that only with ramped up fans you have enough flow and pressure between the cooling fins. I know you wrote you installed an active 2U SM cooler. Doesn't mean you have the right cooler. If the active cooler (fan) doesn't have the power to get rid of the hot air, you can ramp the fans up as long as you want. It's not that simple.

PS: What SM cooler are you using?
I am running SNK-P0048AP4. Thanks for the tip on i7z, I will have to try it.

I don't see how it can be airflow issue since Ivy Bridge CPU was running fine with lower fan speeds and passive CPU cooler and now Haswell is running 10C hotter with active cooler and higher fan speeds. Also, if it was airflow problem then setting fan mode to full speed would drop the CPU temp significantly, which is also not happening.

I think I have a dud CPU, maybe there is an issue with heat transfer between CPU die and IHS, or maybe there is an issue with temp sensor, they are not exactly accurate at below 50C from what I read.

Also, on the consumer CPU side there are a lot of reports of early Haswells running hotter than Ivys, don't really know if it applies to Xeons though.