How much power savings moving from Xeon x5600 to E5

[azev]

I am considering revamping my home lab built this summer and I am trying to find out how much power will I save if I upgrade my servers with E5 series cpu from x5600.
Today I am paying about $300 for electricity every month, and I want to see if I can justify the upgrade from lower monthly electrical bill.

My systems are connected to a dedicated 40AMP 220V outlet that I purposely get just for the lab, and My old ups and PDU does not have any power meter,
My compute node is running on C6100 (only 2 are active) each configured with dual x5650 with 96GB ram (16 x6).
I am running 2 storage unit on 2 separate supermicro 836 chassis, both are running dual L5520 with 48gb ram. One of them running 16 x 450Gb 15K sas drive and the other running 16x 2TB hitachi. (IPMI shows on average power draw for both of these chassis are about 600W combined). I also have plenty of networking hardware, but I am not going to touch that.

My plan is to convert my (2) 836 chassis with dual E5 motherboard, and configure hyperconverged setup for storage. I am curious if anyone here ever measure their dual E5-2620 system for average power draw from the wall ? I want to see roughly how much much power saving will I gain from going with the new setup.

 

[T_Minus]

I can tell you my 8 E5s (SINGLE CPU) idle at 500 watts. 1, 6 core, the rest are 8core or more, 32gigs in most, 64gb in 2, all have NIC add-on cards.

2x E5-2620 v1, idle ~155 watts I think I posted that elsewhere too with more info.

2x 14Core v3, idle ~130 watts. With GOLD 1200w PSU. The 2nd PSU as you may be aware uses 11-15w idle. I got platinum to compare and haven't compared power draw yet, also testing with a 700w gold compared to the higher rated, but platinum too.

I don't think I have anymore x5650 but I'll soon be checking power on x5675 and x5680 systems I have, and can let you know power comparisons.

 

[Jeggs101]

2x 14Core v3, idle ~130 watts. With GOLD 1200w PSU. The 2nd PSU as you may be aware uses 11-15w idle. I got platinum to compare and haven't compared power draw yet, also testing with a 700w gold compared to the higher rated, but platinum too.

That is crazy high V3 idle. We have E5 V3 16 core machines with 16x16GB memory configs, 8x SSDs and are running 70-90w largely idle with 80+ gold. Do you have something else in that system like a full stack of PCIe cards, 3.5" spinners? What are you using to measure power?

 

[T_Minus]

That is crazy high V3 idle. We have E5 V3 16 core machines with 16x16GB memory configs, 8x SSDs and are running 70-90w largely idle with 80+ gold. Do you have something else in that system like a full stack of PCIe cards, 3.5" spinners? What are you using to measure power?

0 PCIE devices, 0 3.5" Spinners, 1 SSD. SM 846 chassis with 5 OEM fans.

** My Idle Description @ BOTTOM **

You're saying your 2P 16 Core, total of 32 cores E5 v3 machines idle at 70w with 8 SSD?
That seems crazy low. I'd love 100w or less idle with 24 cores.

OR, is that a single 16 Core that idles at 70-90w?

Also, what PSU are you using that's 80+ Gold?
(I notice 8-15w power saving using consumer Gold 500w vs the 1200w Gold SM PSUs, and the same 8-15w coming from v1 6 cores.)


I should probably clarify my definition of "idle" too for the test I ran on power.
Win7 + Samsung 830 128gb SSD
Idle on Desktop w/No Screensaver/Sleep/PowerSaving currently running at that time. DropBox, and a couple other small apps that didn't register much CPU usage, but might keep it from truly idling down to nothing ??? Not 100% as I don't use Windows for this usually so I'm not sure how it would compare to say CentOS 6.5 idle.

So it's "idle" but ready to go, not in a low power state or truly sitting idle.

 

[Marsh]

ASUS Z9PA-D8 dual E5-2650 , 8 x 8 gb mem , 1 single hard drive
WS12R2 idle at 70w , Prime 95 load at 320w
Power supply from Newegg $30 deal, not even 80+ bronze

 

[T_Minus]

Where are you guys reading power? (IPMI info, UPS, PDU, Kill-A-Watt Meter, or ?)

I'm starting to think I need to invest in something a bit more than the $15 Kill-A-Watt meter and/or check the reading from SM IPMI instead too. I just glanced @ KW Meter and that was it. I wasn't doing power testing in-depth at the time I put it together.

I think you can get much more accurate units for 150-250$ so I'm looking into those too now to make sure my readings are accurate.

From what it seems SM IPMI power readings are lower than KWM readings... anyone else notice?

 

[Marsh]

I didn't mean the reading as absolute power usage.
I use the Kill-a-Watt as a indicator of relative comparison between the systems that I owned.
I also considered the meter reading may be 20 to 50% off as well. just a ball park comparison.

 

[Marsh]

Although, in this month, I compared my Acer J1900 system and my newly AMD AM1i system to the published review.
The power usage reading are really close, 10-11w each.

 

[cheezehead]

A slightly different angle, but do you need the horsepower from the X5650's (130W TDP)? Can you pull a proc from the nodes and still function the way you want them to? Otherwise, there are some lower powered procs available.

4-core 40W TDP
L5630's $29/each
6-core variants 60W TDP
L5638's $70/each
L5640's $80/each
Above are all going rates on ebay.

The quad-core option may be an option at $58 for the pair but the 6-core variants still carry a premium in terms of small budget upgrades.

 

[cptbjorn]

FWIW I found no difference in power consumption between x5650 and l5639 both at idle and at my normal workload - peak usage was higher with the x5650 if I ran prime95 but there was also a lot more flops happening. Both were somewhat lower power than L5520, again at idle and similar workload.

 

[andrewbedia]

That is crazy high V3 idle. We have E5 V3 16 core machines with 16x16GB memory configs, 8x SSDs and are running 70-90w largely idle with 80+ gold. Do you have something else in that system like a full stack of PCIe cards, 3.5" spinners? What are you using to measure power?

Could be that he's running way out of the efficiency curve. 150W is way too low for a 1200W power supply. That's enough for an extremely hefty system--lots of drives/gpus. What else is in @T_Minus dual 14 core setup besides cpus?

For reference, I have a 1200W Corsair AX1200 in my desktop--it's still slightly overkill, but I'm running somewhere in the efficiency curve. i7-3770k @4.80GHz, R9 290, 16GB 2133MHz, 8 hard drives, 480GB SM843 SSD, cpu and gpu liquid cooled. Everything running full tilt, it's about 400-500W.

 

[T_Minus]

I already noted that I was running 2 1200w PSU, and 1 idles at 10-15w no matter what. These PSU are 10-15w less efficient than a desktop 500w gold, etc.. So you combine those two and you get 30w increase at max idel, I also said I wasn't runing anything else but it was on widnows desktop sitting there, so it wasn't 10000% idle.

I also used a Kill-a-Watt and want to get something better and more accurate.

 

[toomuchstuff]

I am considering revamping my home lab built this summer and I am trying to find out how much power will I save if I upgrade my servers with E5 series cpu from x5600....
My compute node is running on C6100 (only 2 are active) each configured with dual x5650 with 96GB ram (16 x6).
I am running 2 storage unit on 2 separate supermicro 836 chassis, both are running dual L5520 with 48gb ram. One of them running 16 x 450Gb 15K sas drive and the other running 16x 2TB ... I want to see roughly how much much power saving will I gain from going with the new setup.

For what it's worth, I just migrated from a dual W5580, 96GB, 16 x 3.5" SATA/SAS, 846TQ chassis to a dual E5-2620V2/128GB setup (rest is identical). Power difference on identical workload is 100W from what I recall checking last week. I/O is so much better on the new series, you could easily consolidate the storage to one 846 and bigger drives (like you mentioned). IMHO, the c6100 is pretty efficient, but I'm running dual L5520s on three nodes not X series. What battery unit are you running? That could be a large power sink since what you listed I could run with my APC 2200 off a 120V/15A plug. Btw, really jealous of 220V/40A....

 

[azev]

The UPS unit is super old (about 12yrs), its compaq R6000 and I've refreshed the battery twice, last one was about three years ago or so.
I think the biggest powerhog in my setup are probably the network gears. I have 3 switches, 1 router, 1 load balancer, 3 firewalls, 2 AP's, 1 WLC.
I honestly not sure how much power those network equipment draws, but it must be quite a bit, it's possible that it could be more than the storage/compute.
If spending a few grand for an upgrade would only save me about 100W (less than 10buks per month) then I know the big boss isn't going to let me spend the $$
I think the current setup is good, I might need more memory so I can turn up the other 2 nodes on my c6100 in case i need more compute. Storage is also fine, its only about 65% utilized.

Still I am itching for an upgrade

 

[toomuchstuff]

Ok, so this is very dependent on your networking gear. I'll just assume that you don't have a 6U core router and the rest of the configuration doesn't expand from standard L2/L3 switches or 1U networking. That would make most of the networking setup ballpark at around 1.2kW max and server side is around 1kW. Now that UPS is a 6kW model w/ 96% efficiency at 90% load (take load % with grain of salt since i could not find solid information). I couldn't find any efficiency graphs but chances are very good it is similar to older APCs (most of my experience is with them) of the time. So around max you are probably using ~33% of the UPS which is going to eat power.
From a quick scan of ebay I found 220V 1980W new style APC 2200s going for $350. You should be able to get a better deal than that though. Ones from the last 5 years tend to become efficient at low load. My suggestion is you consider replacing your UPS with two 2200s or something more recent.