Is Xeon scaleable 1st gen BAD?

[xayide]

I have a Supermicro with:

1. 2 x xeon gold 6144 (8core 3.5ghz)
2. 32G x 16 (512GB)
3. 8x SAS 3.5"
4. 2x hs PSU 1.2kw
5. LSI sas raid
7. SUpermicro MB with C627 and QAT (X11DPH-TQ)

This thing gets really hot all the time even at idle and uses like 300watts doing nothing. BIOS is all set to standard and uses latest version. Fans are extemly noisy and varies up/down all the time, even for small / short bursts of cpu-usage.

I have a Dell R730 with 2 x 2667v4 and 512GB ram + 8x SAS 3.5" which is very quiet and uses at idle 110 watts. Why the large difference?

Is Xeon Gold / Scalable 1st gen really that bad? It's not really that much faster. Using like 3x the power at idle. Any insights are welcome!

 

[RolloZ170]

Using like 3x the power at idle

what is IDLE for you ? have you checked the core clocks ? idle is about 800mhz per core.
you can check the package power in windows with HWinfo i.e.

Is Xeon Gold / Scalable 1st gen really that bad?

maybe yours are BAD, silicon lottery.

 

[SimonBarker]

I don't think that's unusual for that setup, I would have expected closer to 200W but it's not so far out I'd assume something is wrong.

If you really want to figure out what the CPUs are consuming, just disconnect everything then measure the power consumption, remove one of the CPUs and compare. There's probably a smarter approach but this'll give you a quick answer and it'll probably be less than you think but more than you'd like. You also need to consider things like using dual 1.2kw PSUs at low loads is going to be inefficient.

Noise is down to your choice of cooling solution though.

 

[ano]

you can tune power profiles in bios usually (or via idrac)

 

[i386]

Fans are extemly noisy and varies up/down all the time, even for small / short bursts of cpu-usage.

What Chassis are you using? Fans modded?

 

[RolloZ170]

pls check the core clocks and cpu package power in windows with HWinfo.
if it is 2x 100 watts @800...1200mhz per core you know the 1st gens are bad.
if the package power is 20W per CPU, 1st gens are good and the thread can be closed.

 

[sko]

Are C-states enabled/used? This will give 95% of power savings for the CPU - downclocking usually hasn't that much of an impact: ~50% of using C states and almost no additional savings with C-states enabled.

My E5 v4 systems use drop to ~20% power usage when 'idling' (i.e. normal daily load) compared to high load (e.g. poudriere running package builds)

 

[mrpasc]

I have a Supermicro with:

1. 2 x xeon gold 6144 (8core 3.5ghz)
2. 32G x 16 (512GB)
3. 8x SAS 3.5"
4. 2x hs PSU 1.2kw
5. LSI sas raid
7. SUpermicro MB with C627 and QAT (X11DPH-TQ)

This thing gets really hot all the time even at idle and uses like 300watts doing nothing. BIOS is all set to standard and uses latest version. Fans are extemly noisy and varies up/down all the time, even for small / short bursts of cpu-usage.

There must be something wrong with your setup. I do run a somehow similiar system Cisco UCS C220 M5 with

1. 2 x Xeon Gold 6134 (8 Core 3.2GHz)
2. 32GB x 16 (512GB)
3. 10x SAS 2.5'' SSDs
4. 2x HS PSU 650W
5. Cisco MRaid 12G 4GB Cache (basically a branded LSI 12G RAID card)
6. 2x 40G NIC + 2x 10G NIC + 4x 1G NIC

This system consumes less than 300W under load (ESXi 7.U3 with ~ 65% load). And ESXi isn't known for being very power-efficent....

 

[xayide]

Thank you everyone. I will check up on all the bios-settings and hwinfo in Windows to see if something is wrong. I'll get back here with results tomorrow.

 

[CyklonDX]

The Intel Xeon Skylake & Kabby Lake do run hot, like a lot when under load.

Take a look at supermicro bios -> turbo (as many mobo's have been overriding standard intel turbo, and been running turbo on all cores.)


Your best bet is to limit their turbo - cut off last 200MHz when you do have some load on the system, the cpu's should run much cooler after doing that. Those extra 100-300MHz really punches the heat up...

On one of my systems doing compute load (*dual gold 6254 - 18c @ turbo 4GHz)
Sits at avg 72'C without thermal downclocking when turbo is locked to 3.8GHz (with fans running at 70%)

ex. of load


prior with stock turbo setting @ 4GHz limit, the server was thermally throttling ~ and sitting in 90'C /s per package. (with fans running at 100%)

 

[RolloZ170]

I will check up on all the bios-settings and hwinfo in Windows to see if something is wrong.

check first the HWInfo sensors window, core clocks(800..1200mhz = idle) and cpu package-power(idle up to 30Watts) .

 

[xayide]

Had a look in BIOS and changed from Energy Efficient to Custom the n went into Windows. It is now a bit quieter but still using 340 watts at idle. The first CPU seems low temp but the second is nearly hot at idle. The one the furtest from the fans ofcourse. I can see it only goes down to 1200mhz.

 

[RolloZ170]

Had a look in BIOS and changed from Energy Efficient to Custom the n went into Windows

in windows no need to touch this, OS controls ... is OK.
25 Watts package power is idle.
your DRAM needs 25W per CPU.
some other unit draws watts in your system.
Intel QAT can draw much in if other units r in idle.

 

[RolloZ170]

but the second is nearly hot at idle

sure, 2nd gets the hot air from the first CPU right ?

 

[CyklonDX]

Ask yourself if you need turbo, if you don't - just turn it off. It will help with temps.

 

[RolloZ170]

HWinfo: 2nd CPU is not visible in sensors window.

 

[RolloZ170]

Ask yourself if you need turbo, if you don't - just turn it off. It will help with temps.

not in idle.

 

[xayide]

in windows no need to touch this, OS controls ... is OK.
25 Watts package power is idle.
your DRAM needs 25W per CPU.
some other unit draws watts in your system.
Intel QAT can draw much in if other units r in idle.

Yeah disks are blinking but they do not generate much heat and I guess the LSI is doing some light patrol read. The second cpu is behind the first sorta at an angle/verticalshift so half of it. I will see if I can turn off QAT somehow, do not use it and windows driver got uninstalled so the devices are not showing up again. See if I can get it down to 800mhz and lso remove turbo bins.

 

[CyklonDX]

not in idle.

since you are running on windows; look up your system power profile. (Minimum Processor State)

 

[xayide]

since you are running on windows; look up your system power profile. (Minimum Processor State)

Good point! Will do! Should actually install Windows on new NVME and see what gives me.

Just amazed that the R730 is so much more efficient and quiet.