Idle power consumption Intel LGA 1700 vs AMD AM5 platform

[JanR]

I'm in the process of replacing my trusted server system after 13 years. For all this time, the Xeon X3470 run perfectly on an ASUS P7F-E in 24/7 operation. Three years ago I even upgraded to 32 GB UDIMM ECC (officially not supported) and NVME system disk giving this old Lynnfield CPU (first generation Core-i processor in server version) some more useful time.

The replacement machine should be of similar quality (here I'm unsure since I regret that Intel discontinued entry level server Xeons and AMD never really entered this market segment).

The machine has to drive two displays (iGPU is sufficient, currently an old nVidia Quadro NVS does the job), has plenty of memory (96 GB is the goal once modules are available), obviously ECC, three NVME-SSD and a fast processors. Since I will run it in Germany with high energy prices 24/7, idle consumption is of much more importance than full power consumption.

Since there is no real entry level server platform anymore (I would prefer an Raptor Lake Xeon or an AM5 based EPYC), I see the main choice between LGA 1700 or AM5. For both platforms ECC capable boards (server class or workstation class) are available and both platforms offer a wide range of CPUs (I will most likely use the top SKU and reduce power limits according to my needs in order to maximize flexibility).

On LGA 1700 side, I have experiences from work with an Supermicro X13SAE-F. BMC is not needed here, so the question is how much power that safes. The board at work idles around 35-37 W (at wall) with 13900K, 128 GB RAM, 2 NVME SSDs and a 10 GBe card. I will also measure this for two memory modules and without the 10 GBe card.

On AMD side, my last real experience (beside some experimental machines at work) dates back into Socket 939 times which is really LONG ago (this was the predecessor of the current Xeon).

So far, I figured out the following (which does not make decision simpler):

LGA 1700:
pro: lower idle, boards available with success reports (X13SAE, ASUS WS680...)
con: very high power at high load, no 4x4 PCIe bifurcation (I want to have the option of having more NVME connected to PCIe lanes of CPU)

AM5:
pro: lower power consumption at load, CPU supports 4x4 bifurcation (of board also supports it), fast USB ports directly from SoC
con: no real experiences on idle consumption found so far, seems to depend on a number of aspects, less reports on boards found.

On the AMD side, I like the options of the Supermicro H13SAE (escpecially the PCI-e layout giving two NVME connected to CPU and 16 more lanes on CPU hopefully with bifurcation in the 16x slot), but so far I have not found real experiences with that board and no figures on power consumption. It has an BMC but it is unclear if disabling it reduces power consumption. Furthermore, there are some boards from ASRock Rack but they also have BMC and do not expose all USB from CPU as Supermicro does.

What are your experiences? Is there any argument that I missed? Is there a real (same components) comparison between entry level server or workstation boards based on LGA 1700 and AM5 with respect to idle power consumption?

 

[unwind-protect]

All I can say is that my AMD4 system has bad idle power, 30 watts more than the Intel 9700k system before it.

 

[twin_savage]

here I'm unsure since I regret that Intel discontinued entry level server Xeons

Intel is actually bringing these back under the E-2400 series, best part about it is that they are going to be P-core only and raptor lake based. I wouldn't expect this to release before Q1 2024 though.

(I will most likely use the top SKU and reduce power limits according to my needs in order to maximize flexibility).

This might not be an option. Some of the server boards do not have bioses that will allow you to set TDP as is common in consumer boards. This is the case for the H13SAE-MF, there is no option to run a higher SKU CPU as a lower TDP.

In my experience disabling the most modern aspeed BMC chipset results in ~4 watts of power savings, it isn't as much as it use to be in the past because aspeed is on a 28nm process now instead of the 40nm process the previous AST2500 chips were made with.

H13SAE-MF has no bifurcation support within a single PCIe slot.

I'm able to get a H13SAE-MF, Ryzen 7900 and Samsung PM9A3 to idle as 41 watts on a platinum efficiency power supply.

 

[JanR]

Intel is actually bringing these back under the E-2400 series, best part about it is that they are going to be P-core only and raptor lake based. I wouldn't expect this to release before Q1 2024 though.

Thanks... I completely missed that information. After some googling I found details... top level SKU is 95 W and 8C/16T. I'm not sure if this is sufficient to compete against TR Pro especially with respect to multi-core.

If the Raptor Lake cores are more or less similar to those in the 13th or 14th Core-i-generation, this can easily be simulated on the X13SAE at work with 13900K CPU (even remotely ):

With PL1 and PL2 reduced to 95 W and taskset to P cores only it needs 2:05 for compiling my test kernel. At 125 W with P + E this goes down to 1:12 and 1:21 with P + E at 95 W. Fun fact: The 16 E cores need 2:01 for the same compilation at 95 W. Therefore, with respect to multi-core performance the E cores contribute a lot...

However, it is really great to see the return of these Xeons. This is a good argument to wait for some more weeks with the final decision. Since it is LGA 1700, these Xeons should run on current W680 boards with BIOS update, I hope.

Additionally, the existence of these Xeons lets me hope that somebody will implement an EDAC-driver for Linux supporting these memory controllers.

This might not be an option. Some of the server boards do not have bioses that will allow you to set TDP as is common in consumer boards. This is the case for the H13SAE-MF, there is no option to run a higher SKU CPU as a lower TDP.

This is really bad (and the information on no PCIe bifurcation also)... but thanks for the information! Do you run Linux or Windows on the H13SAE?

The X13SAE for LGA 1700 has this option. Furthermore, the power limits can also be changed at runtime with simple commands on Linux (this way I produced the benchmark results stated above).

In my experience disabling the most modern aspeed BMC chipset results in ~4 watts of power savings

Thanks for this number. Do you have any experiences if disabling has the same effect on power as "not being there" (so xxx-F vs. xxx in Supermicro scheme)?

I'm able to get a H13SAE-MF, Ryzen 7900 and Samsung PM9A3 to idle as 41 watts on a platinum efficiency power supply.

Hmm... at work I have 35-37 W with X13SAE-F with i9-13900K, 128 GB RAM, 2 Samsung SSD Pro 980 (seem to have the same PCI-ID as PM9A3 according to Linux lspci) and 10 GbE on a standard bronze power supply and with no further tuning. Next, I will remove 10 GbE and disable BMC to see the impact.

How much memory do you have on that H13SAE-F?

 

[twin_savage]

I'm not sure if this is sufficient to compete against TR Pro especially with respect to multi-core.

I think Intel is using it's W790 platform to compete with TR pro.... although it is a power hog, I'm lucky to get the W5-3435X alone to idle at even 70 watts; Right now my whole W790 system idles at ~200 watts because of some overclocking.

This is really bad (and the information on no PCIe bifurcation also)... but thanks for the information! Do you run Linux or Windows on the H13SAE?

I don't know why, but it seems like all the "professional" AMD systems have less BIOS options than comparable Intel systems.
I plan to eventually run Windows on the H13SAE, but I don't have alot time on the system yet because I am having trouble getting a good case that holds enough hdds.

Thanks for this number. Do you have any experiences if disabling has the same effect on power as "not being there" (so xxx-F vs. xxx in Supermicro scheme)?

Yes, the absence of BMC on a SKU seems to be about the same as disabling it... that being said some motherboards don't actually allow you to easily disable the BMC. I haven't actually tried to disable BMC on my H13SAE.


That 41w power draw figure for H13SAE was with only 2 48GB non-ECC DIMMs installed, eventually I want to get 192GB of ECC on the system but I really don't want to buy nemix ram so I'm waiting.

 

[JanR]

I think Intel is using it's W790 platform to compete with TR pro.... although it is a power hog, I'm lucky to get the W5-3435X alone to idle at even 70 watts; Right now my whole W790 system idles at ~200 watts because of some overclocking.

This is... very high. I also considered this platform for a moment but board prices and the expectation of high idle consumption were clearly against that idea.

I don't know why, but it seems like all the "professional" AMD systems have less BIOS options than comparable Intel systems.

Therefore, this is another "pro" for Intel and "contra" for AMD on my list.

That 41w power draw figure for H13SAE was with only 2 48GB non-ECC DIMMs installed

And another "contra" against AMD since I expect that X13SAE with no BMC and only two instead of four DIMMs and without 10 GbE will again save some Watt.

With respect to Xeon E-24xx: Maybe AVX512 is the real reason behind dropping the E-cores since they do not support it. Nevertheless, I would prefer to have them so the user can decide to enable or disable them in the BIOS.

According to my compilation tests, an E core at 4,3 GHz is faster than a 4 GHz Broadwell-EP core in a Xeon E5-1650-v4 (23:04 vs. 28:05 in single core compilation). i9-13900 has 16 of those... this gives a lot of performance.

 

[twin_savage]

This is... very high. I also considered this platform for a moment but board prices and the expectation of high idle consumption were clearly against that idea.

The non-chiplet LGA-4677 Xeons are alot better than the chiplet ones with idle draw, I remember hearing third hand that they are ~40 watts idle, that is CPU power draw, not whole system.

According to my compilation tests, an E core at 4,3 GHz is faster than a 4 GHz Broadwell-EP core in a Xeon E5-1650-v4 (23:04 vs. 28:05 in single core compilation). i9-13900 has 16 of those... this gives a lot of performance.

E-cores are actually fairly potent, at least compared to CPUs of the recent past; If you wanted to go all out for efficiency, you could run the alderlake-n platform, the N300 is about as performant as a E5-2620 v4 but uses only 7 watts as opposed to 85 watts of the 2620.

 

[RolloZ170]

--

The non-chiplet LGA-4677 Xeons are alot better than the chiplet ones with idle draw, I remember hearing third hand that they are ~40 watts idle, that is CPU power draw, not whole system.

non-chiplet = MCC E1B package. indeed the ones with one DIE (SPR) need around 30-40Watts package power(C6)
Emerald Rapids use up to 2 DIE.
we should simply not buy low core count SKUs with 4 tiles configuration.
high core count ones can replace a dual socket system, the IDLE power comparable.

 

[JanR]

40 watts idle, that is CPU power draw, not whole system.

This results in too much idle consumption...

If you wanted to go all out for efficiency, you could run the alderlake-n platform, the N300 is about as performant as a E5-2620 v4 but uses only 7 watts as opposed to 85 watts of the 2620.

As far as I figured out, these CPUs do not support ECC - a server with no ECC is a no-go for me.

WIth respect to the original question I made some more measurements to have a comparison to the 41 W idle of the H13SAE-MF measured by @twin_savage .

At work, I removed one of the three SSD and the 10 GbE card from the X13SAE-F + i9-13900k machine. This dropped idle consumption to 29 W with active BMC, one 1 Gbit link (having two of them increases power by 1-1.5 W), 4 DIMMs, two Samsung SSD 980 Pro and one active graphic output in Linux idle. Compared to the AMD measurement by @twin_savage this is one M2 SSD and two DIMMs more. I also tested the impact of removing 2 DIMMs (128 GB -> 64 GB) but this was very minor (0.5 W less). I guess this is due to the fact that 4 DIMMs run at DDR5-3600 while two DIMMs run at DDR5-4400 so the energy saved by having less DIMMs is nearly compensated by running the memory controller at higher speed. The impact of one M2 SSD more at idle is rather low, I would guess.

All measurements were taken at wall (230 V). The power supply is 80plus bronze with 650 W (Enermax CyberBron ECB600AWT ) so I guess that its efficiency at this level is not really good (Enermax gives 81% at 10% load which equals to 65 W at output side so we are much lower here). However, with no BMC 25 W idle seems to be possible with that platform (3-4 W more in my application since I need two active gigabit ethernet links and two active graphic outputs). Carefully selecting a power supply with excellent efficiency at very low load might improve situation further.

On the other end of spectrum, the machine (with 3 SSD and 10 GbE card - idle at 38 W) requires 182 W while compiling kernel with 2 DIMMs and 185 W with 4 DIMMs. Obviously, I set PL1=PL2=125 W for this test.

 

[JanR]

Now, the system is build and I'm nearly ready with setup. Therefore, I can give numbers related to my original question:

Here was the measurement from AM5 (H13SAE):

I'm able to get a H13SAE-MF, Ryzen 7900 and Samsung PM9A3 to idle as 41 watts on a platinum efficiency power supply.

Now, I have an X13SAE with i9-14900K, 2x32 GB ECC DDR5, 2 x M2 NVME (Samsung Pro 990 + WD SN850X, 4 TB each), one active GbE connection and a platinum efficiency power supply (550W). Cooling is done semi-passive with two low RPM 14 cm fans (300 rpm at idle, up to 1000 at full load). In Linux idle, I measure between 19-22 W (fluctuating) at wall (230 V).

Therefore, in idle the Intel platform needs HALF the power of the AMD platform. Obviously, the AMD platform has BMC while my Intel board does not but there is no competing AMD server/workstation board with no BMC.

 

[T_Minus]

Now, the system is build and I'm nearly ready with setup. Therefore, I can give numbers related to my original question:

Here was the measurement from AM5 (H13SAE):



Now, I have an X13SAE with i9-14900K, 2x32 GB ECC DDR5, 2 x M2 NVME (Samsung Pro 990 + WD SN850X, 4 TB each), one active GbE connection and a platinum efficiency power supply (550W). Cooling is done semi-passive with two low RPM 14 cm fans (300 rpm at idle, up to 1000 at full load). In Linux idle, I measure between 19-22 W (fluctuating) at wall (230 V).

Therefore, in idle the Intel platform needs HALF the power of the AMD platform. Obviously, the AMD platform has BMC while my Intel board does not but there is no competing AMD server/workstation board with no BMC.

That's kind of a big deal considering the BMC is likely 50% (9-10w) of the intel power at idle... putting the AMD system not that far off.

 

[JanR]

That's kind of a big deal considering the BMC is likely 50% (9-10w) of the intel power at idle

I don't think that is that much... I'm more with this:

In my experience disabling the most modern aspeed BMC chipset results in ~4 watts of power savings, it isn't as much as it use to be in the past because aspeed is on a 28nm process now instead of the 40nm process the previous AST2500 chips were made with.

Please note that you can NOT compare my numbers of X13SAE-F (system at work) and X13SAE (system at home). This X13SAE-F system uses a 650 W gold efficiency PSU while my X13SAE build features a 550W platinum efficiency PSU that I selected based on reviews that measured consumption not only at 10% but also at 5%.

Therefore, I guess that the 9-10 W difference goes 50% to the BMC and 50% to the PSU.

 

[RolloZ170]

Therefore, I guess that the 9-10 W difference goes 50% to the BMC and 50% to the PSU.

note the BMC is powered by +5VSB not by mail rails.

 

[twin_savage]

For reference, the 41 watt power figure on the H13SAE system was with a 850watt corsair hx850i platinum power supply. I was previously running the exact same system on a 1500w Silverstone ST1500 silver power supply (because I thought I needed more 5v rail amps) and the idle draw was 61watts. Power supply can definitely make a big difference.

 

[JanR]

In my case, the PSU of the X13SAE system is a Seasonic Focus Plus 550 Platinum.

And... the X13SAE-F at work uses a bronze PSU, not gold as written above (Enermax CyberBron ECB600AWT).

What we really need for such systems is a platinum PSU with 200-250 W. That would be completely sufficient as the X13SAE system needs only 165 W at full CPU load (obviously restricted to 125 W PL1 and PL2).

 

[Sven Nilsson]

Now, I have an X13SAE with i9-14900K, 2x32 GB ECC DDR5, 2 x M2 NVME (Samsung Pro 990 + WD SN850X, 4 TB each), one active GbE connection and a platinum efficiency power supply (550W). Cooling is done semi-passive with two low RPM 14 cm fans (300 rpm at idle, up to 1000 at full load). In Linux idle, I measure between 19-22 W (fluctuating) at wall (230 V).

The interesting thing is that it does not seem to pay off to get a lower core count CPU if you want to achieve the lowest possible idle consumption.

I have an ASRock H610M-ITX/ac with the Pentium G7400 quad core, which at the time was one of the lowest end parts you could buy.
Power consumption at idle is 18W, measured after the PSU (12V 1.5A), so pretty much identical to your high end 14900k.

If anyone knows an LGA1700 CPU that can reduce this consumption to around 10W, please speak up.
The Intel 300 and 300T dual cores were released just few days ago but I doubt they are any better than the G7400.

 

[JanR]

The interesting thing is that it does not seem to pay off to get a lower core count CPU if you want to achieve the lowest possible idle consumption.

The question is if the "low core count" CPU is really a low core count die or a "standard" die with some cores disabled. Additionally, the idle consumption is determined mainly by the uncore. Idle cores are power gated. A power gated core needs the same as a disabled core - just nothing.

If anyone knows an LGA1700 CPU that can reduce this consumption to around 10W, please speak up.

I don't think you will find that... a better approach is to search for a power optimized boards with nothing that is not needed.

The X13SAE, for example, features an PCI-e-to-PCI-bridge in order to have a PCI slot from the stone age (the 32 Bit, 33 MHz version). Obviously, this bridge needs some energy but it is useless for most scenarios today.

 

[FingerBlaster]

search for a power optimized boards with nothing that is not needed.

How does one search for power optimized boards is my question?

 

[JanR]

How does one search for power optimized boards is my question?

Reading reports from others, starting threads like this one...

For me, this thread answered my LGA1700 vs AM5 question - AM5 gives more efficiency at load while LGA1700 is more efficient at idle. Since my setting is dominated by idle, the answer was clear.