AMD EPYC Milan Idle Power Consumption

vcc3 · Sep 18, 2021

From reading at this forum and other places of the internet I got the feeling that the idle power consumption of AMD EPYC Milan CPUs must be really bad. I can not remember all pages where I rad that the idle power consumption of Milan CPUs is "so high" but my impression is that hardly anyone is providing some real number but only things like: two time as much as Intel Scalable XY, or worser than the previous AMD EPYC generation Rome.
[https://forums.servethehome.com/index.php?threads/amd-epyc-7003-milan-the-fast-gets-faster.32215/, https://forums.servethehome.com/ind...d-file-server-upgrade-time.32281/#post-298372]

Anandtech did provide some real measuring of the "Package Idle Power" consumption (AMD 3rd Gen EPYC Milan Review: A Peak vs Per Core Performance Balance).
However, they measured the "Package Power"? In another paragraph the stated "We’re using AMD’s energy counters to achieve this, "... but I'm not sure what exactly that means.
For this "Package Power" package power the stated 99 and 103 Watts for the 64 core non "F" SKUs.
In a newer article [AMD EPYC Milan Review Part 2: Testing 8 to 64 Cores in a Production Platform] they state 65 Watt for the 64 core 7763 and 50 Watts for the 24 core 7443. If the "Package Power" also includes the power draw of 8 memory DIMMs the new values sounds a lot more reasonable then the numbers in the first review. For me, the Anandtech articles raises more questions than they answers (in the context of idle power draw).

I want to get an idea about how much power a whole EPYC Milan single socket system draws from the wall. Therefore, I did my own measurements with the following minimal configurations.

Mainboard: Supermicro H12SSL-C - BMC Firmware Version 01.00.34; BIOS Firmware Version: 2.1; The onboard SAS3 HBA was disable by a jumper;
CPU: AMD EPYC 7443P 24-Core
Memory: Only one 32GB RDIMM ECC 3200MT was installed.
PSU: 2U Redundant 800 Watts (ASPOWER R2A-DV0800-N) - labeled with 94% efficiency
Boot drive: M.2 240GB Kingston DC1000B
All fans was set to 0x32 duty (50.0%) fixed. (1 x 60mm for the CPU at 5880rpm, and 3 x 120mm midspan fans at 1260rpm)
The SFP+ NICs did not have a transceiver / SFP Module installed.

Component Info	C-State	one PSU	PSU-1	PSU-2
Shut down IPMI only	-	10.1W	7.8W	6.0W
boot ssd only	Disabled	77.0W	46.9W	43.7W
boot ssd only	Enabled	55.1W	36.6W	32.6W
boot ssd + U.2 2.0TB Intel P4510	Disabled	81.9W	48.7W	45.2W
boot ssd + U.2 2.0TB Intel P4510	Enabled	59.4W	39.3W	34.6W
boot ssd + 2x Solarflare S7120 (2xSFP+) NIC	Disabled	91.9W	52.3W	49.8W
boot ssd + 2x Solarflare S7120 (2xSFP+) NIC	Enabled	69.5W	44.5W	41.0W
boot ssd + 2x Solarflare S7120 (2xSFP+) NIC + U.2 2.0TB Intel P4510	Disabled	96.7W	55.5W	51.0W
boot ssd + 2x Solarflare S7120 (2xSFP+) NIC + U.2 2.0TB Intel P4510	Enabled	75.6W	46.4W	43.4W

Since I often benefit from the knowledge others contribute to this forum I want to give something back and I hope it is useful for someone else. I would greatly appreciated more test result on other, different configurations. Also any thoughts about the AMD EPYC power consumptions would be highly welcome.

RolloZ170 · Sep 19, 2021

vcc3 said:
If the "Package Power" also includes the power draw of 8 memory DIMMs the new values sounds a lot more reasonable then the numbers in the first review

package power does NOT include any Memory outside the package.
the new sIOD(not in 7nm) with the infinity fabric never sleeps.

i386 · Sep 19, 2021

vcc3 said:
I want to get an idea about how much power a whole EPYC Milan single socket system draws from the wall. Therefore, I did my own measurements with the following minimal configurations.

Mainboard: Supermicro H12SSL-C - BMC Firmware Version 01.00.34; BIOS Firmware Version: 2.1; The onboard SAS3 HBA was disable by a jumper;

CPU: AMD EPYC 7443P 24-Core

Memory: Only one 32GB RDIMM ECC 3200MT was installed.

PSU: 2U Redundant 800 Watts (ASPOWER R2A-DV0800-N) - labeled with 94% efficiency

Boot drive: M.2 240GB Kingston DC1000B

All fans was set to 0x32 duty (50.0%) fixed. (1 x 60mm for the CPU at 5880rpm, and 3 x 120mm midspan fans at 1260rpm)

The SFP+ NICs did not have a transceiver / SFP Module installed.

Component Info C-State one PSU PSU-1 PSU-2
Shut down IPMI only - 10.1W 7.8W 6.0W
boot ssd only Disabled 77.0W 46.9W 43.7W
boot ssd only Enabled 55.1W 36.6W 32.6W
boot ssd + U.2 2.0TB Intel P4510 Disabled 81.9W 48.7W 45.2W
boot ssd + U.2 2.0TB Intel P4510 Enabled 59.4W 39.3W 34.6W
boot ssd + 2x Solarflare S7120 (2xSFP+) NIC Disabled 91.9W 52.3W 49.8W
boot ssd + 2x Solarflare S7120 (2xSFP+) NIC Enabled 69.5W 44.5W 41.0W
boot ssd + 2x Solarflare S7120 (2xSFP+) NIC + U.2 2.0TB Intel P4510 Disabled 96.7W 55.5W 51.0W
boot ssd + 2x Solarflare S7120 (2xSFP+) NIC + U.2 2.0TB Intel P4510 Enabled 75.6W 46.4W 43.4W

At what room temperatures and humidity?
Server or tower chassis?
Was an air shroud used?

zir_blazer · Sep 19, 2021

Milan idle power consumption was initially measured on an AMD reference platform that wasn't optimized for Milan. After reviews of third party Motherboards started to pop up it seemed to idle at the same level than Rome.

vcc3 · Sep 19, 2021

i386 said:
At what room temperatures and humidity?
Server or tower chassis?
Was an air shroud used?

room temperature: about 24°C
humidity: about 65%
19" 4U rack case with 16 x 3.5" hot swap HDD trays and 8 x 2.5" drives in the front. However, only 2 2.5" NVMe SSDs was physically installed during the test and only one connected for some tests - see my instal post.
There is no air shroud in the case. The used CPU cooler is a Dynatron A26. It is designed for 2U cases and has its own 60mm x 20mm fan. (A26 | Dynatron)

nasbdh9 · Sep 19, 2021

vcc3 said:
There is no air shroud in the case. The used CPU cooler is a Dynatron A26. It is designed for 2U cases and has its own 60mm x 20mm fan. (A26 | Dynatron)

What is the temperature of A26 under full-core load and single-core load? (For example, run the performance test in CPU-Z)

zer0sum · Sep 20, 2021

I just built a Ryzen system so you can compare it's baby brother

Mainboard: Asrock Rack X470D4U (MATX)
CPU: Ryzen 5600X ( 6C / 12T)
Cooler: Artic Liquid Freezer II 120mm AIO
Case Fan: Noctua 120mm
Memory: 2 x 32GB UDIMMS
PSU: Corsair SF450 SFX (80+ Gold certified)
Boot drive: Sandisk USB (ESXi 6.7)
Storage: AOC-SLG3-2M2 PCIe Add-On card, with an Intel 660p 2TB NVME and ADATA SX8200 1TB NVME
Network: Mellanox CX3

Idle power consumption measured with a Kill-a-watt when fully booted with ESXi 6.7 and no VM's running is ~38W

vcc3 · Sep 21, 2021

@nasbdh9 CPU-Z is a Windows Program right? I will not install windows. In order to stress the CPU I just used yes > /dev/null &. I think this should be fine for testing the cooling system.

CPU Temperature
With 3 times CUI Devices - CFM-A225-13-22 (120mm x 20mm, 4100rpm) as midspan case fans and a Dynatron A26 with a 60mm x 20mm fan as CPU cooler. (During the idle power consumption test above, 3 Noctua - NF-P12 redux-1700 PWM was used as midspan case fans.) Alle 16 3.5" hot swap HDD trays were empty and only 2 2.5" NVMe SSDs were installed in the front. All test are performed at a room temperature of about 24°C and about 57% humidity. The idle power consumption above was measured with a watt meter at the wall. The PSU Power vales hire are the values reported by the PSU itself and presumable not as precise.

Mainboard: Supermicro H12SSL-C - BMC Firmware Version 01.00.34; BIOS Firmware Version: 2.1; The onboard SAS3 HBA was disable by a jumper;
CPU: AMD EPYC 7443P 24-Core (Global C-state Control was enabled)
Memory: Only one 32GB RDIMM ECC 3200MT was installed.
PSU: 2U Redundant 800 Watts (ASPOWER R2A-DV0800-N) - labeled with 94% efficiency
2x Solarflare S7120 (2xSFP+) NIC
Boot drive: M.2 240GB Kingston DC1000B
Other Storage: 2 x 2,5" NVMe SSDs with power loos protection
OS: Ubuntu v20.04.1, kernel: 5.11.0-34-generic

Idle Temperature

Fan Control - Standard Speed
- CPU Fan Speed: 1680rpm
- Case Fan Speed (3x): 840rpm
- Max. CPU Frequency: 1500MHz
- CPU Temp: 37°C
- System Temp: 32°C
- PSU:
  - AC input Power: 82W
  - DC Main Output Power: 69W
Fan Control - Full Speed
- CPU Fan Speed: 8540rpm
- Case Fan Speed (3x): 4060rpm
- Max. CPU Frequency: 1500MHz
- CPU Temp: 27°C
- System Temp: 26°C
- PSU:
  - AC input Power: 114W
  - DC Main Output Power: 99W

Single Core Stress Test

Code:

    yes > /dev/null &
    watch -n 1 "cat /proc/cpuinfo | grep MHz"
    killall yes

Fan Control - Standard Speed
- CPU Fan Speed: 2520rpm
- Case Fan Speed (3x): 1540rpm
- Max. CPU Frequency: 4024MHz
- CPU Temp: 53°C
- System Temp: 30°C
- PSU:
  - AC input Power: 116W
  - DC Main Output Power: 101W
Fan Control - Full Speed
- CPU Fan Speed: 8540rpm
- Case Fan Speed (3x): 4060rpm
- Max. CPU Frequency: 4024MHz
- CPU Temp: 39°C
- System Temp: 26°C
- PSU:
  - AC input Power: 144W
  - DC Main Output Power: 127W

All Cores Stress Test

Code:

    for i in $(seq $(getconf _NPROCESSORS_ONLN)); do yes > /dev/null & done
    watch -n 1 "cat /proc/cpuinfo | grep MHz"
    killall yes

Fan Control - Standard Speed
- CPU Fan Speed: 5460rpm
- Case Fan Speed (3x): 2800rpm
- Max. CPU Frequency: 3592MHz
- CPU Temp: 68°C
- System Temp: 31°C
- PSU:
  - AC input Power: 266W
  - DC Main Output Power: 240W
Fan Control - Full Speed
- CPU Fan Speed: 8680rpm
- Case Fan Speed (3x): 4060rpm
- Max. CPU Frequency: 3596MHz
- CPU Temp: 59°C
- System Temp: 29°C
- PSU:
  - AC input Power: 280W
  - DC Main Output Power: 254W

EffrafaxOfWug · Sep 21, 2021

FWIW I don't think yes in an infinite loop will give much in the way of stressing the system - you're better off using a real-world util with a synthetic workload to make sure more than just a few instructions are used. For CPU and memory I like to use pxz (parallel xz) since it's on all our servers anyway and is easy to configure on a per-thread basis.

Code:

pxz -T 4 -c /dev/urandom > /dev/null

vcc3 · Sep 21, 2021

yes does for sure not stress the entire system, but I think for testing the CPU cooling system it is at least as good as pxy. From what I understand, the CPU Cores (Register, ALU, Instruction decoder....) at very high clock rates consumes a lot more energy than the IO parts of the CPUs with a relative low clock speed. I know that it is very hard to keep CPU cores or shading units on GPUs fed with enough data. This is often the biggest problem if you want to make an algorithm fast. If the pipeline of a core can not be fed, a cash miss will happens. A cache miss will unavoidable lead to CPU stall and presumable the energy saving features of the CPU kick in and hopefully reduce the required energy. At least the no-/halt-operation is designed so that it requires less energy the any other instruction.

The lossless compression algorithm LZMA requires a dictionary that stores bit patterns. This dictionary can have a size in the range of hundreds of megabytes or even giga bytes. I assume that especially for pseudo random data the dictionary will grow pretty quick.
Since new bit patterns need to be compared to the entries in that dictionary, the dictionary must use a data structures that allows an efficient search (e.g. binary trees). However, I'm pretty sure that the dictionary lookups still leads to a lot of cache misses.

While the arithmetics required for yes is simple, I think the arithmetics required for the compression with pxy is not much more complicated. However, yes has few instructions and needs hardly an data to read. Therefore, it should be good in keeping the CPU cores busy and from what I understand busy CPU cores should result in a perfect wast of energy.

But I'm not a Hardware engineer, I know only as much about hardware as required for the software development I need to do. ;-)

Never the less, thank you for the tip! I think it is very useful as simple test for the CPU and the memory sub system.

Keith Myers · Sep 21, 2021

The best torture test for finding the max temps for Ryzen/TR/Epyc is y-cruncher. Select the number of cores you want to stress and the amount of memory if not accepting defaults and then run through all six standard tests. Run through 4 iteration passes to stress and find stability.
y-cruncher
The AVX2 Float tests will really hammer the cpu. The BBP Digit Extraction test no.11 will make your cpu whimper. The Mixed Workload No. 16 test is a great balance of typical work on a server. You will see 200-350W of package power being used.

The best Zen2(3)/Epyc/TR modules for monitoring voltages, power and temps I have found is the zenpower driver and mating zenmonitor driver. I use the Zen3 forks found here.
zenpower3
zenmonitor3

vcc3 · Sep 21, 2021

zer0sum said:
I just built a Ryzen system so you can compare it's baby brother

Mainboard: Asrock Rack X470D4U (MATX)

CPU: Ryzen 5600X ( 6C / 12T)

Cooler: Artic Liquid Freezer II 120mm AIO

Case Fan: Noctua 120mm

Memory: 2 x 32GB UDIMMS

PSU: Corsair SF450 SFX (80+ Gold certified)

Boot drive: Sandisk USB (ESXi 6.7)

Storage: AOC-SLG3-2M2 PCIe Add-On card, with an Intel 660p 2TB NVME and ADATA SX8200 1TB NVME

Network: Mellanox CX3

Idle power consumption measured with a Kill-a-watt when fully booted with ESXi 6.7 and no VM's running is ~38W

Sounds like a cool little server! do you have pictures? Does the UDIMMS support ECC? Does the Mellanox CX3 two SFP+ ports, or does it have interfaces with higher bandwidth? I have also build a 2U rack server for 12 3.5" HDDs and 2 2.5" NVMe SSDs around the Asrock Rack X470D4U but your server has a lower idle power consumption then my build.

Mainboard: Asrock Rack X470D4U (UEFI Version P3.50)
CPU: AMD Ryzen 3900X ( 12C / 24T)
CPU Dynatron A24 (Fan: 60mm x 25mm, 7000rpm)
Case Fan: Delta Electronics - AFB0812SH-TP12 (80mm x 25mm, 4500rpm)
Memory: 2 x 32GB UDIMM ECC 3200MT/s
PSU: 2U Redundant 550 Watts (ASPOWER R2A-DV0550-N)
Boot drive: M.2 240GB Kingston DC1000B
OS: Ubuntu v20.04.1, kernel: 5.11.0-34-generic

The SFP+ NICs did not have a transiver / SFP Module installed.

Desc	C-State	one PSU	PSU-1	PSU-2
Shut down IPMI only	-	5.9 W	4.4W	3.9W
M.2 240GB Kingston DC1000B boot ssd only	Disabled	55.8W	28.0W	34.7W
M.2 240GB Kingston DC1000B boot ssd only	Enabled	35.4W	20.9W	27.5W
M.2 240GB Kingston DC1000B + U.2 2.0TB Intel P4510	Disabled	61.0W	30.9W	37.7W
M.2 240GB Kingston DC1000B + U.2 2.0TB Intel P4510	Enabled	40.7W	22.1W	28.8W
M.2 240GB Kingston DC1000B + 2x Solarflare S7120 (2xSFP+) NIC	Disabled	74.2W	34.9W	45.6W
M.2 240GB Kingston DC1000B + 2x Solarflare S7120 (2xSFP+) NIC	Enabled	53.5W	27.5W	34.6W
M.2 240GB Kingston DC1000B + 2x Solarflare S7120 (2xSFP+) NIC + U.2 2.0TB Intel P4510	Disabled	77.9W	37.3W	47.7W
M.2 240GB Kingston DC1000B + 2x Solarflare S7120 (2xSFP+) NIC + U.2 2.0TB Intel P4510	Enabled	58.1W	29.4W	36.5W

Now the title of the thread does not fit any more.... May be "AMD ZEN CPUs Idle Power Consumption" would have been a better fit.

P.S.: This forum software really should support markdown for text formatting. This RTE drives me nuts.

vcc3 · Sep 25, 2021

Keith Myers said:
The best torture test for finding the max temps for Ryzen/TR/Epyc is y-cruncher. Select the number of cores you want to stress and the amount of memory if not accepting defaults and then run through all six standard tests. Run through 4 iteration passes to stress and find stability.
y-cruncher
The AVX2 Float tests will really hammer the cpu. The BBP Digit Extraction test no.11 will make your cpu whimper. The Mixed Workload No. 16 test is a great balance of typical work on a server. You will see 200-350W of package power being used.

The best Zen2(3)/Epyc/TR modules for monitoring voltages, power and temps I have found is the zenpower driver and mating zenmonitor driver. I use the Zen3 forks found here.
zenpower3
zenmonitor3

As suggested by Keith Myers, I did a test with y-cruncher. I observed the power consumption reported by the PSU as I did during my tests with yes. However, this time both PSUs was plugged in and the values represents the sum of booth PSUs. I did not try the suggested zen power tools, I'm sure they can deliver interesting values but I need to pay for the Watt/h at the wall ;-) and I'm a bit sort on time at the moment.
The air temperature was a bit higher to day, about 26.5°C. The Fan Control was set to "Standard Speed" in the BMC.
The values below are the maximum power draw I was able to observer during each test. While the power draw was very constant during the test with yes, the values from the y-cruncher are often 40Watt bellow the maximum values I observed.

yes: 277W
BKT: 275W
BBP: 259W
SFT: 281W (281W appears at only one single sample, the next highest sample was 275Watt)
N64: 249W
HNT: 263W
VST: 229W

Fan speed:

3 x 120mm midspan case: 2500 - 3000rpm
1 x 60mm CPU: 4500 - 6000rpm

For me this numbers makes perfect sense since the maximum power consumption is limited by the TDB of the CPU and while yes can permanently stress all cores since it never needs to wait for data, on the other hand the y-cruncher does a real job that needs lots of data which likely leads to CPU stalls. However, this bottleneck can probable be reduced by populating all 8 memory channels of the CPU.

Code:

Component Stress Tester

  1   Logical Cores:      48
  2   Memory:             27.4 GiB  (  584 MiB per thread )
  3   NUMA Mode:          Local - Memory allocated from local thread.
  4/5 Time Limit:         120 seconds per test / Run Forever
  6   Stop on Error:      Enabled

 7/8  Enable All Tests / Disable All Tests
 9/10 Load/Save Configuration File

  #   Tag - Test Name               Mem/Thread   Component        CPU------Mem
 11   BKT - Basecase + Karatsuba      27.8 KiB   Scalar Integer    -|--------
 12   BBP - BBP Digit Extraction        small    AVX2 Float        |---------
 13   SFT - Small In-Cache FFT         255 KiB   AVX2 Float        -|--------
 14   FFT - Fast Fourier Transform    Disabled   AVX2 Float        ---------|
 15   N32 - Classic NTT (32-bit)      Disabled   AVX2 Integer      -----|----
 16   N64 - Classic NTT (64-bit)       560 MiB   AVX2 Integer      ---|------
 17   HNT - Hybrid NTT                 574 MiB   Mixed Workload    -----|----
 18   VST - Vector Transform           584 MiB   AVX2 Float        ------|---
 19   C17 - Code 17 Experiment        Disabled   AVX2 Mixed        ---|------

  0   Start Stress-Testing!

option: 0


Allocating Memory...
  Core   1:   584 MiB  node 0 (100%)
  Core   0:   584 MiB  node 0 (100%)
  Core   2:   584 MiB  node 0 (100%)
  Core   6:   584 MiB  node 0 (100%)
  Core   4:   584 MiB  node 0 (100%)
  Core   9:   584 MiB  node 0 (100%)
  Core   8:   584 MiB  node 0 (100%)
  Core  11:   584 MiB  node 0 (100%)
  Core   7:   584 MiB  node 0 (100%)
  Core   3:   584 MiB  node 0 (100%)
  Core  10:   584 MiB  node 0 (100%)
  Core  12:   584 MiB  node 0 (100%)
  Core  13:   584 MiB  node 0 (100%)
  Core   5:   584 MiB  node 0 (100%)
  Core  14:   584 MiB  node 0 (100%)
  Core  15:   584 MiB  node 0 (100%)
  Core  17:   584 MiB  node 0 (100%)
  Core  16:   584 MiB  node 0 (100%)
  Core  18:   584 MiB  node 0 (100%)
  Core  19:   584 MiB  node 0 (100%)
  Core  21:   584 MiB  node 0 (100%)
  Core  20:   584 MiB  node 0 (100%)
  Core  22:   584 MiB  node 0 (100%)
  Core  23:   584 MiB  node 0 (100%)
  Core  25:   584 MiB  node 0 (100%)
  Core  24:   584 MiB  node 0 (100%)
  Core  26:   584 MiB  node 0 (100%)
  Core  27:   584 MiB  node 0 (100%)
  Core  29:   584 MiB  node 0 (100%)
  Core  28:   584 MiB  node 0 (100%)
  Core  31:   584 MiB  node 0 (100%)
  Core  30:   584 MiB  node 0 (100%)
  Core  32:   584 MiB  node 0 (100%)
  Core  33:   584 MiB  node 0 (100%)
  Core  36:   584 MiB  node 0 (100%)
  Core  34:   584 MiB  node 0 (100%)
  Core  37:   584 MiB  node 0 (100%)
  Core  38:   584 MiB  node 0 (100%)
  Core  39:   584 MiB  node 0 (100%)
  Core  40:   584 MiB  node 0 (100%)
  Core  41:   584 MiB  node 0 (100%)
  Core  42:   584 MiB  node 0 (100%)
  Core  43:   584 MiB  node 0 (100%)
  Core  44:   584 MiB  node 0 (100%)
  Core  46:   584 MiB  node 0 (100%)
  Core  45:   584 MiB  node 0 (100%)
  Core  35:   584 MiB  node 0 (100%)
  Core  47:   584 MiB  node 0 (100%)

Iteration: 0  Total Elapsed Time: 6.031 seconds  ( 0.101 minutes )
Running BKT: Passed  Test Time:  120.000 seconds  ( 2.000 minutes )
Running BBP: Passed  Test Time:  122.352 seconds  ( 2.039 minutes )
Running SFT: Passed  Test Time:  120.000 seconds  ( 2.000 minutes )
Running N64: Passed  Test Time:  125.459 seconds  ( 2.091 minutes )
Running HNT: Passed  Test Time:  125.108 seconds  ( 2.085 minutes )
Running VST: Passed  Test Time:  122.524 seconds  ( 2.042 minutes )

Iteration: 1  Total Elapsed Time: 741.475 seconds  ( 12.358 minutes )
Running BKT: Passed  Test Time:  120.000 seconds  ( 2.000 minutes )
Running BBP: Passed  Test Time:  122.328 seconds  ( 2.039 minutes )
Running SFT: Passed  Test Time:  120.000 seconds  ( 2.000 minutes )
Running N64: Passed  Test Time:  125.811 seconds  ( 2.097 minutes )
Running HNT: Passed  Test Time:  127.354 seconds  ( 2.123 minutes )
Running VST: Passed  Test Time:  122.008 seconds  ( 2.033 minutes )

Iteration: 2  Total Elapsed Time: 1478.976 seconds  ( 24.650 minutes )
Running BKT: Passed  Test Time:  120.000 seconds  ( 2.000 minutes )
Running BBP: Passed  Test Time:  122.316 seconds  ( 2.039 minutes )
Running SFT: Passed  Test Time:  120.000 seconds  ( 2.000 minutes )
Running N64: Passed  Test Time:  126.030 seconds  ( 2.101 minutes )
Running HNT: Passed  Test Time:  124.607 seconds  ( 2.077 minutes )
Running VST: Passed  Test Time:  122.104 seconds  ( 2.035 minutes )

Iteration: 3  Total Elapsed Time: 2214.035 seconds  ( 36.901 minutes )
Running BKT: Passed  Test Time:  120.000 seconds  ( 2.000 minutes )
Running BBP: Passed  Test Time:  121.147 seconds  ( 2.019 minutes )
Running SFT: Passed  Test Time:  120.000 seconds  ( 2.000 minutes )
Running N64: Passed  Test Time:  125.347 seconds  ( 2.089 minutes )
Running HNT: Passed  Test Time:  125.136 seconds  ( 2.086 minutes )
Running VST: Passed  Test Time:  122.335 seconds  ( 2.039 minutes )

Iteration: 4  Total Elapsed Time: 2948.002 seconds  ( 49.133 minutes )
Running BKT: Passed  Test Time:  120.000 seconds  ( 2.000 minutes )
Running BBP: Passed  Test Time:  122.355 seconds  ( 2.039 minutes )
Running SFT: Passed  Test Time:  120.000 seconds  ( 2.000 minutes )
Running N64: Passed  Test Time:  125.916 seconds  ( 2.099 minutes )
Running HNT: Passed  Test Time:  126.995 seconds  ( 2.117 minutes )
Running VST: Passed  Test Time:  123.042 seconds  ( 2.051 minutes )

thimplicity · Jan 19, 2022

This is great!

I am trying to decide between a Ryzen or an Epyc build. As the Ryzen CPU and motherboard prices are totally nuts at the moment, the Epyc built is roughly the same. It seems like the Epyc is about 20w more on average, which is not too much. As I would be looking at the Ryzen 3600/3700x vs Epyc 7251/7252, I assume the numbers are lower in total, but the difference would be about the same.

Does that logic make sense?

111alan · Jan 19, 2022

The reason why this happened is mainly due to these two points.
1. The power consumption of the North Bridge (or IO die) isn't low. sIOD is basically four cIOD put together. Previously in Rome the north bridge runs at 1466MHz fclk, but in Milan it went up to 1600MHz, and seems to sleep less.

2. There is a linear power loss for each core due to the LDO power design, when those cores are trying to lower the voltage.

That means the core will not save as much power as we've expected when idling.

tinfoil3d · Jan 20, 2022

EffrafaxOfWug said:
Code:

pxz -T 4 -c /dev/urandom > /dev/null

Just a side note, /dev/urandom does not provide enough data quickly enough. It's much better to just aes-encrypt /dev/zero with random key to get as much as possible random data fast. And then xz it.
Or shaXsum /dev/zero as well for easy single-core load test.

Lleo_ · Feb 26, 2022

thank you all for this informative thread.
I am in the process of estimating the power budget needed for a Milan system, but I am having difficulty identifying what impact the amount of RAM installed may have on [idle|max] power use.
any information or thoughts on the difference between say 64GB -> 128GB or 128GB -> 256GB?
thanks in advance

Evan · Feb 26, 2022

It very much depends on what memory is being used, what process node are the chips created on and how big size they are.
Larger dimms are usually on a more modern process node so a 64gb rdimm uses similar to a 32gb rdimm (ddr4)
I would imagine say 16gb single rank rdimm ddr4 probably uses less by a reasonable margin but I didn’t ever test.

I allow about ~2w or ~2.5w per rdimm.
Note that ddr3 uses more like 3w per rdimm and ddr5 I haven’t tested yet.

Jellyfish · Mar 23, 2022

Hi everyone,

I bought a very similar setup but got what I think are quite different numbers. I tested the following basic configuration:

Mainboard: Supermicro H12SSL-C - BMC Firmware Version 01.00.35; BIOS Firmware Version: 2.3 (Note: HBA not disabled with jumper)
CPU: AMD EPYC 7443P 24-Core
Memory: 2 x 64 GB DDR4-3200
PSU: 1200W PWS-1K21P-1R PSUs
Boot drive: USB key with Ubuntu for testing
SSD for VM storage: 2 x 2TB Samsung PM9A1 NVMe
Disconnected all fans except one that was hard to get to, that wasn't making much noise.

Measurements were taken with a kill-a-watt type device, the values in the BMC were way off so I won't bother listing them here. All measurements are with C-State enabled as it was by default and no BIOS tweaks.

Basic configuration as listed above, switched off to BMC only, 1 x PWS-1K21P-1R PSU: 13W
Basic configuration as listed above, switched off to BMC only, 2 x PWS-1K21P-1R PSU: 21W
Basic configuration as listed above, idling booted into Ubuntu, 1 x PWS-1K21P-1R PSU: 80W
Basic configuration as listed above, idling booted into Ubuntu, 1 x PWS-1K21P-1R PSU + all fans: 85W

When I look at the comparable configuration in the opening post however, I would expect to be below 50W when idling.

After connecting everything I wanted (BPN-SAS3-846EL1 backplane (+15W), NVIDIA Grid K1 (+15W), 2x SSD (too low to measure), 8x HDD (6W per drive, +48W total), 2nd PSU (+20W)), I got to 196W. Strangely, the sum of the parts resulted in a higher combined power usage than when I tested them all one by one to get the numbers in the brackets.
In an attempt to lower this I changed the stock Supermicro 1200W PWS-1K21P-1R Gold PSUs for more appropriate 920W PWS-920P-SQ Platinum PSUs, still oversized but better and more quiet. With everything that I want connected, it got me down from 196W to 185W. So that brought a small improvement, but I still feel like my numbers are too high and I'm missing something.

Any ideas why my idle power usage is so high compared to vcc3's?

zir_blazer said:
Milan idle power consumption was initially measured on an AMD reference platform that wasn't optimized for Milan. After reviews of third party Motherboards started to pop up it seemed to idle at the same level than Rome.

So it does sound like if it isn't optimized, Milan power consumption can be unexpectedly high. Is there something that I can do to optimize for Milan, or is this something a firmware update needs to fix?

vincococka · Mar 28, 2022

FYI

While using following setup:
- H12SSL-i , BIOS 2.3 , BMC 01.01.04
- AMD Epyc 7313p 16c, SMT Enabled
- 128GB RAM (8x 16GB DDR4 3200MT/s Samsung Single Rank sticks)
- FibreChannel QLogic dual-port 8Gbps
- NIC Chelsio T6225
- NIC Mellanox CX5 EN 100GbE single-port
- SSD 2x Intel Optane P5800x
- SSD 1x Intel P5510 3.84TB
- SSD NVMe m.2 128GB Transcend
- SSD NVMe m.2 512GB Samsung
- FANs: 3x120mm front & back & side fan for PCIe cooling, CPU cooler from SuperMicro for SP3 socket
- PSU: 1x Seasonic Focus GX 550W
- OS: RHEL 8.5 with RHEL kernel 4.18.0-348.20.1.

... here are power consumptions observed on NETIO PDU which have very precise measurements:
Consumption OFF /w IPMI active: 4W
Consumption idle OS: ~105W
Consumption Prime95 /w HT Enabled Test#4: ~260W

AMD EPYC Milan Idle Power Consumption

Member

Well-Known Member

Well-Known Member

Active Member

Member

Active Member

Well-Known Member

Member

Radioactive Member

Member

Active Member

Member

Member

Member

Active Member

QSFP28

New Member

Well-Known Member

New Member

Member