PSU considerations low load / power off

[_alex]

With this post i want to show some results i got trying different Gold PSU's

I also might get Platinum PWS-741P-1R or even the smaller PWS-501P-R for this system.

The motivation behind this is that i'm planning to deploy two very similar systems (in SC826BE1-XXXXLBP that will come without PSU). These will be near idle for estimated 30-40% of the time. So, there is quite some potential how much energy could be saved over time. Also, even if i like the tech-stuff, i also feel somehow committed to the planet we live on


This will be a bit 'work in progress', as i'm currently missing some gear, more PSU's to test and also time to test everything was a bit short today.
(see: next steps)

So, currently the most worse thing is that all the wattages are taken from IPMI.

Will do the best to get a TRMS Power Meter asap and update the numbers i gathered so far.


System under Test
i will have this exact system until October in the labs, and won't change anything:

X10SRL-f BIOS 2.0b IPMI FW 3.58
Xeon E5 1620v3
2x 4GB DDR4 Micron
CSE-825, different PSU, each single and dual

1x ConnectX2 Dual Port
2x 500GB WD RE3

OS: Proxmox 5.0

All Power Draw numbers from IPMI, 'Server Health' => 'Power Source'

kW/h based on 8670 hrs/year

No values for savings given as power plans differ.
Left to each reader to estimate it's savings.

Notice: after adding/removing a PSU:
System must be taken from Power/unplugged (all PSU) and booted to see changes in IPMI.
IPMI might be unavailable for a minute or so.

One of the Asssertions would be that a PSU with smaller nominal load performs more efficient, as efficiency generally is bad in low load situation. 80 Plus requirements start at 20% load, optimization is done towards 50% load.

Supermicro gives an informative number for 10% load in their 80 Plus verification sheets. I added links to the PDF's for the PSU's tested so far.

In them here is a number how much the psu-fans draw, that is not included (!) in the calculated efficiency of the 80 Plus standard, but your power bill. The power-draw from these fans can be quite different.



PSU's tested so far:

PWS-1K21P-1R
1200 W 80+ Gold
http://www.supermicro.com.tw/products/powersupply/80PLUS/80PLUS_PWS-1K21P-1R.pdf

Single PSU:

                      DC 12V Output    AC Input    efficiency

Power off:            0                16W         n/a
Idle:                 48W              90W         53%

Dual PSU:

Power off:            0                16W / 24W   n/a
(total)               0                40W         n/a
Idle:                 24W / 18W        48W / 60W      
(total)               42W              108W        38%

PWS-721P-1R
720 W 80+ Gold
https://www.supermicro.com/products/powersupply/80PLUS/80PLUS_PWS-721P-1R.pdf

Single PSU:

                     DC 12V Output    AC Input    efficiency
Power off:           0                48W         n/a
Idle:                54W              90W         60%

Dual PSU:

Power off:           0                50W / 49W   n/a
(total)              0                99W         n/a
Idle:                57W / 59W        61W / 84W          
(total)              116W             145W        80%

Observations so far:

1.) The Measurements of DC12V Output Power in IPMI 'Power Sources' must be wrong, so the effectiveness simply says noting.
2.) Even so, the total AC-Draw (what will be accounted) can be compared.
3.) In total, the smaller PSU (PWS-721P-1R) performs worse in powered-off state and idle than the larger.
Powered off add's up 32W for single PSU and 59W for dual PSU.

This are the differences in kW/h per Year:

=> Single: 280 kW/h per Year
=> Dual: 517 kW/h per Year



Idle is the same with single PSU, but 37W more with dual PSU.

=> Dual: 324 kW/h per Year

4.) The Difference in total idle AC Input in the two dual PSU setups can only be explained by PSU fan's power draw, as the OS idling wouldn't draw more power.



Conclusions:
1.) if you have PWS-721P-1R and your systems are often under low load or even powered-off but IPMI running:
get rid of them and replace, it will payout fast, especially when you manage to get some money selling them
2.) PSU with smaller nominal load not always have higher efficiency at lower load, keep the fans in mind as they are not part of the efficiency they are sold for.



next steps:

- get / borrow a true-rms power meter to validate IPMI totals / estimate error range
- get some Platinum 500W and 740W PSU's to compare
- test under 50% and 100% load
- add informative results from FatTwin Platinum PSU - even if no alternative PSU's available.


Any comments, critics, suggestions how to change testing are really wellcome,
Alex

 

[_alex]

reserved for more PSU / test data

 

[_alex]

reserved for more PSU / test data

 

[Evan]

What about the 200w/250w units ? Your system as configured won't need that much. Pity you some have X10srm that can run 12v from a power brick for comparison.
Those power off figures are ugly would have hoped for about 10w

I was hoping to build a home x10 e5 v4 system shortly that I hoped would idle at ~30w or less. (Maybe X10srm-f and e5-2640v4) so maybe I have a tough task ahead.

 

[_alex]

Haven't found any 200/250W redundant PSU's yet.
And the final build will draw more, there will be 2x ConnectX2 IB HCA, up to 12x HDD/SSD, more Ram, eventually 12G HBA + SAS Expander and maybe somewhen also NVMe.

Guess the 500W Platinum could be a huge improvement above those two gold PSU and fit the power needs, even if not at their best with 100% load but should be able to handle it.

I really wonder why there are no lower-specced PSU's, as a lot of Servers for sure spend most of their time in the lower loads.
Starting with 500w is somehow weird, as everybody is working towards lower TDP parts, and then you need to run your PSU in 10% load range what is absolutely terrible in terms of efficiency.

Another idea is putting them with 1 PSU on difference Feeds and just let it go down if a feed fails - this is what clusters are made for
Something between would be single PSU on different Feeds + Battery in the second slot.

For the numbers, also at power-off:
i wouldn't take them too serious atm, maybe the PSU just report totally wrong to PMBUS.
Will get this confirmed (or not) when i managed to have a TRMS Power Meter.

 

[T_Minus]

What about the 200w/250w units ? Your system as configured won't need that much. Pity you some have X10srm that can run 12v from a power brick for comparison.
Those power off figures are ugly would have hoped for about 10w

I was hoping to build a home x10 e5 v4 system shortly that I hoped would idle at ~30w or less. (Maybe X10srm-f and e5-2640v4) so maybe I have a tough task ahead.

SuperMicro x10 board w.onboard sound, 4x16gb rdimms, pcie wifi, intel 750 pcie aic version, supermicro 4u HSF no other fans, 1x s3510 480gb intel ssd, evga g2 750 gold psu, and E5-1650 v3 idle at wall: 44w
Added EVGA GTX 1080 and 2x 2TB WD RE HDD and 4 1200rpm fans plugged in now idle 74w.

Fresh in my head from a couple days ago

 

[Evan]

SuperMicro x10 board w.onboard sound, 4x16gb rdimms, pcie wifi, intel 750 pcie aic version, supermicro 4u HSF no other fans, 1x s3510 480gb intel ssd, evga g2 750 gold psu, and E5-1650 v3 idle at wall: 44w
Added EVGA GTX 1080 and 2x 2TB WD RE HDD and 4 1200rpm fans plugged in now idle 74w.

Fresh in my head from a couple days ago

44w -> ~30w with a far more optimised psu and v4 cpu seems within reach

Back to the topic I agree, why we don't have some ~300w redundant options ?? I guess that the logic is redundant = datacenter = don't want to stuff around with many PSU types and the base say 500w should work for most configs except the largest ones.

Technically HPE DL380, 15 x 8tb 3.5" SAS disks, 2 x e5-2680v4, 128/256gb ram and just the flexlom 10g card can run on the 500w psu, I would normally spec the 800w to have some headroom but gives you an idea what can be powered by 500w.

 

[Rand__]

Those results are weird indeed.
-The 721's fan is rated with 8w, the 1k21 with 19W (probably two fans).
So with the same load (70-100W) and nearly same 10% efficiency level (83,1/84,7%) the fan impact should be significantly higher (10%) than the efficiency difference - meaning that the 1k21 would consume more.
The tests show the other way round...

-The fan's are rated at near constant usage which is rather unlikely - maybe this is max power usage instead of actual.
If the 1k21 uses better (more efficient fans) this might might even the odds (1 vs 2 fans at 200% efficiency) but still would not explain the massive differences seen here.

Can you observe power consumption in your UPS maybe? Or buy a cheap meter at the local Obi If if total is off by a few watts the comparison should be off by more or less the same amount...

 

[_alex]

No UPS at the labs stuff, nothing important there have an apc 7920 but the accuracy is not worth to talk about.

What i remember from earlier years is that measuring AC, in special in lower range and on this type of PSU is definitely ,devils work'. So cheap Obi / Baumarkt is waste of time and money for this task.

Have bo'd on a capable device, as i somehow live w/a a multimeter for years now (lost/given to someone) and this is quite good reason to get smth. really useful.

 

[Rand__]

Ah I thought to remember otherwise
Unfortunately using a specific server will make it difficult to provide meaningful comparison points from other ppl...

 

[_alex]

Imho using a specific server (=consumer) on a psu is the only way to compare their efficiency. What exact server is behind the psu should not matter, as long as the same power is required from the psu.

Also a dummy (used often in rf) should do to get an idea.

Correct me if i'm wrong.

 

[Rand__]

No, in principal you are correct of course, keep as much constant as possible, eg stay in bios/boot prompt, same drives etc.
Just without having a distributed set of identical machines your's are the only boxes that we can use to measure, so data points from other ppl will not be comparable directly, thats what I meant

 

[_alex]

Back to the topic I agree, why we don't have some ~300w redundant options ?? I guess that the logic is redundant = datacenter = don't want to stuff around with many PSU types and the base say 500w should work for most configs except the largest ones.

For SM, there is already a quite hughe number of PSU's to deal with, lower specced options wouldn't make it worse.
If you look at the popular 826 / 825 there is nothing below 500w, and the most available PSU's are 720 / 920W.

I also feel that dual 500W PSU should be enough most of the time, often 300/350W could work.
Even with 500W in a dual PSU setup 10% would mean 100W and 20% - where they start to be within the 80+ specs - is already at 200W.
This is far away from idle / near idle draw, what means for this times the efficiency is terrible.

 

[Rand__]

Actually higher level 80+ spec's take 10% load into consideration. Titanium for sure, not sure about Platinum.
Unfortunately all Titanium PSUs are rated above 1k watts.

 

[_alex]

This are the verification sheets for the 500W and 740W Platinum.
There are numbers for 10% but not shown in the graph, and both are not within 80+ Platinum.
But especially on the 500W PSU it looks like this could bring down the power quite a bit.

Really need to get some and see how it comes out.

http://www.advancedhpc.com/rackmount_servers/datasheets/80PLUS_PWS-501P-1R.pdf

https://plugloadsolutions.com/psu_r...TER INC._PWS-741P-1R_740W_SO-420.1_Report.pdf

 

[michathe]

Thought i give some numbers since I've tested several PSUs with my 847 last summer:

Server was at a DOS prompt. (1 E5 v1, all fans at 100%, expander backplane)

All PSUs where measured at a redundant setup.

PWS 1K41P-1R 1400 Watt Gold -> 182W
PWS-1K21P-1R 1200 Watt Gold -> 184W
PWS-721P-1R 720 Watt Gold -> 166W
PWS-920P-1R 920 Watt Platinum -> 175W
PWS-740P-1R 740 Watt Platinum -> 159W

Power Metered on a Power Meter.

 

[Rand__]

According to Wikipedia Titanium needs 90% efficiency at 10% load, Platinum starts at 20% load

 

[_alex]

@michathe Thanks for that numbers, they look a bit different from those i got from IPMI when comparing the PSU's.
Do you remember what IPMI showed compared to the Power Meter ?
I really wonder why my 721P-1R showed that bad numbers in IPMI, could be that it's just in a too low percentage with the lower draw my system has and falls back in this range or it simply reported totally wrong via the internal measurement.

Somehow it makes sense that the 920W Platinum is worse than the 720W Gold at idle.

 

[michathe]

No unfortionatly not.

But i got my hands on some 500W Platinum PSUs and did the tests again with an E3 v3:

Redundand Setup

System at Uefi Shell

PWS-1K21P-1R 1200 Watt Gold -> 80W
PWS-721P-1R 720 Watt Gold -> 73W
PWS-920P-1R 920 Watt Platinum -> 80W
PWS-740P-1R 740 Watt Platinum -> 78W
PWS-501P-1R 500 Watt Platinum -> 56W

Single 500W Platinum 49W

Some random ATX Powersuply (OTZ 500W or something like that) -> 49W

System at Windows idle on the 500W Platinum:

Dual PSUs -> 40W

Single PSU -> 30W !!

So these things are Awesome for low power builds.

Hardware used:

E3-1241V3, Supermicro Board, 32GB Ram, and a 2U Supermicro Chassis (Only the PDB is Used No Backplane or Fans)

 

[_alex]

Hi, thanks, looks promising
Will get some 500w and 740w platinum soon and also managed to get a trms power meter ...
When everything arrived and i find some time i will measure the tested setup again to see differneces to ipmi and how much the lower platinum psu's matter at sub-20% load.