Reducing power consumption of my homelab

[leonroy]

Looking to reduce the power consumption of my homelab which runs around 700-1000W per annum and costs about £900-1200 in electricity.

I have a pair of Supermicro Xeon E5540 SuperServers with 96GB DDR3, 4x 900GB SAS 2.5" drives. They run vSphere 6.0, and host Zimbra, Zabbix, Plex, Nakivo, etc. and consume 180W electricity.

I also have a pair of Supermicro FreeNAS servers which have 16x SATA drives. They're Ivy Bridge E3-1220L v2, and each have 32GB RAM. They consume 140-180W each.

Finally there's a Synology RS2416+ (80W electricity) and an assortment of HP Procurves (50W) and a pfSense gateway (16W).

Wondering what sort of cost savings I could make by switching to Xeon v5 or v6 based systems with DDR4, SSDs, etc.

 

[Patrick]

How big are your hard drives?

Are the E5540 servers single or dual CPU?

My first assumption is that you could replace the two E5540's with a Xeon E5 V3 dual CPU system with SSDs and try using fewer hard drives for FreeNAS storage.

 

[leonroy]

How big are your hard drives?

Aren't you in the Bay Area, must be 4am there!

10x 3TB Toshiba SAS drives in primary FreeNAS box.
16x 3TB SATA drives in secondary FreeNAS box.

Are the E5540 servers single or dual CPU?

Dual mainboards but each only has a single E5540.

My first assumption is that you could replace the two E5540's with a Xeon E5 V3 dual CPU system with SSDs and try using fewer hard drives for FreeNAS storage.

Good idea, but for redundancy and maintenance I like to be able to take one server down for maintenance whilst the other spins away. Since the performance of the E5540 is good enough for my needs wondering if maybe the C3000 series Atoms (as an example) or the newer E3 v6 Xeons are a more power efficient and equally performant solution. Whilst each box has 96GB RAM I use 30-40GB max per box so don't really need the new E5's ability to address more RAM.

Would a E3 v6 or C3000 box with a pair or quartet of 1TB SSDs be a reasonable replacement for ESXi?

Not a bad idea on the consolidation of disks. That's something that's easy enough to do just a matter of costing up the replacement of 16x 3TB SATA disks for 8x 6TB SATA drives. I wouldn't imagine the savings would be more than 5W per disk though?

 

[pyro_]

You would potentially also gain some power savings by not needing additional hba card for the additional drives. Could also consider the Xeon-d boards depending on how many pcie slots you need

Maybe try to consolidate the number of switches that you need depending on their location and use

 

[cheezehead]

Depending on your needs on the switching side some 24-port managed gig switches are under 12w.

 

[Patrick]

Aren't you in the Bay Area, must be 4am there!

I usually wake up at 4:30AM.

You are on the right track. My thought was that you could basically have a single E5 V3 server that has more performance and idles at ~70w. Cost would still be low for V3 and power would be significantly better.

The E3 might work, just thought you needed more RAM or would want it in the future.

The Xeon D-1541 would save you a ton of power immediately and still have more performance than you are used do. Actually, even a D-1521 would.

On the drives side, I was thinking 8TB drives. The answer may be finding how many fit on onboard SATA controllers.

C3000 still is not supported by OSes like FreeNAS, but I am told support is coming.

 

[Spartus]

does anything need upgrading, or is it purely a money saving approach? What kind of ROI do you need? 2 year, 3 year?

FYI 1 watt * 1 year is 8.77 kW*hours. I average around 12 cents so 1 watt * 1 year = 1 dollar for me (CAD). What is your average electricity price?

 

[Spartus]

Also, IMO, savings from new HDD are probably very minimal. I've done the calc myself many times as I also run 3TB drives.

say he consolidates from 16 x 3TB to 8 x 6TB, or even better 6 x 8TB. The 8x6 drive option is probably directly comparable but the 6x8 either comes with a loss of useable space or a loss of margin of safety.

Best case 6x8, save 5 watts * 10 drives + 10 more watts for the HBA. That might all be a little optimistic actually. Save 60 watts = $60 CAD / year

6 drives x 300 CAD = 1800 CAD + 13 % tax yay ontario = $2000 (probably even worst in england due to higher VAT.

ROI = $2000 / $60 per year = 34 years *cough*

Now that being said the salvage value on those $16 drives is probably about $1k CAD, maybe even upwards of $1200 if really lucky. But still, over 10 years to pay off.

focus on reducing the amount of equipment, preferably by consolidation into newer platforms. The old Xeons are first on the block.

 

[MiniKnight]

Many places in Europe have electricity costs that are 3-4x yours @Spartus

If you can remove a HBA and some fans that's another 20w too. You've also got cooling costs.

 

[Spartus]

Many places in Europe have electricity costs that are 3-4x yours @Spartus

If you can remove a HBA and some fans that's another 20w too. You've also got cooling costs.

I included the HBA, but fair enough I forgot the fans. Mine are 2 x 12v @ 0.1 amps, so another 2.5 watts for both (2 fans for 8 drives). Maybe another few watts due to power supply efficiency.

Bottom line is I wanted to demo how to approach this calculation. That is why my first two questions are are we upgrading or are we just trying to save money? and what is your electricity price?

 

[leonroy]

I usually wake up at 4:30AM.

Reminds me of a remark one of the New York finance guys made on an 11am (London time) conference call recently. Me: "Sorry about the early call for you guys in NY". His reply: "We're actually a couple of hours into our work day...". Finance guys are just animals when it comes to work ethic.

Xeon D-1541 would save you a ton of power immediately and still have more performance than you are used do. Actually, even a D-1521 would.

Good tip. I looked up Supermicro's offerings and they have a bunch of Xeon-D all in one boards with built in LSI HBAs, M.2 slots, SATA DOM ports and on-board Intel 10GbE. I think to reduce my power consumption on any significant level an all-in-one would be the best bet.

That would at a stroke eliminate the RAID controllers I have in every box and also remove the need to purchase 10GbE controllers in the future saving on power and complexity there. I also didn't realize how powerful Xeon-D systems (or how slow my E5540 Xeon is!) - about 20% faster in single threaded performance and 200% faster in multi.

does anything need upgrading, or is it purely a money saving approach? What kind of ROI do you need? 2 year, 3 year?

FYI 1 watt * 1 year is 8.77 kW*hours. I average around 12 cents so 1 watt * 1 year = 1 dollar for me (CAD). What is your average electricity price?

Thanks for all the calculations, very useful walking through those numbers. I came to similar conclusions myself since 3TB drives were until recently the sweet spot in price per TB.

The unit price for my electricity supplier here in the UK has just risen to £0.165 per kWh which is mad. It means 1W costs £1.447 to run 24/7 per annum. Thus my electricity bill for just the homelab is set to hit ~£1200.

If you can remove a HBA and some fans that's another 20w too. You've also got cooling costs.

Thanks for the tip on HBA power consumption. This table was very useful:
LSI Host Bus Adapter (HBA) Power Consumption Comparison - ServeTheHome

That is why my first two questions are are we upgrading or are we just trying to save money? and what is your electricity price?

Bit of both. Upgrading at a point where it makes economic sense to. This is a hobby/business for me, so I upgrade a little more often than a home labber might but less often than a business with 3 year rolling maintenance would

Thinking I'll upgrade my ESXi hosts first and replace them with Xeon-D based, cheap 1U boxes. Guess it's time to see what Dell, Supermicro and HP have!

Regarding network switches I actually switched recently from some noisy Procurve 2810s to Procurve 2530s which saved about 40W power per switch.

 

[james23]

also (IME), often a bigger power / $ issue with these home (or office, or even DC) setups, is your cooling cost per W of "lab/server".

in the next month im going to be able to get very good , exact data on this specifically (as im upgrading my home office setup with a whole house power meter + a small ac just for my rack area + per outlet ePDU W monitoring + 8x temperature monitoring probes, including outdoor temps).

So far my rough measurements (in a home type setup) is your AC adds 1x to 2x "watts" burned per watt of server/rack.
(ie you put 100w of power/heat into your home, your ac needs to remove that 100w , there is no magic heat destruction here or anywhere.)

ie:

assume you want to maintain the same average temperature in your home (lets say 74f):
AC systems (home central air) are ~20-45% "inefficient" (ie to remove 100w heat, ac will use 120 to 145w , or more).

ie if your rack is putting out, total, 300w. you will need/use 360-450w EXTRA of cooling capacity (to keep the same temperature 74f). (so total 660-750w)

So if you can reduce your rack by 50w of power, you are also saving 60-75w extra of cooling (so a total of 110w to 125w).

I have noticed a similar #s / effect myself, with months i was running a mining rig at my home that burned a constant 310w , 24/7. My monthly bill was NOT increasing anywhere near 310w worth (it was increasing by about double that, or more). then when i powered off the mining rig for a few months, the reverse happened.

(ofcourse there are external variables, like outdoor temp, daily direct sunlight, ect. thats why with my new setup, i will be able to measure this data much better)

One element im not sure of, is how do you calculate the fact that your AC does not run 24/7, like your servers though?
(i think answer is that even if your AC runs for 3mintues extra every 30min, due solely to your severs, it ends up averaging out to the 1.20 to 1.45 x i calculated above , bc a home compressor uses 2500-5000w when its running)

 

[kapone]

I'm not so sure about the AC part, but with caveats.

My "homelab" rack sits in the corner of the basement, that's the coolest part of the basement. It's not even actively heated or cooled. So, in theory, the homelab shouldn't be contributing to the overall AC bill.

But of course this assumes you have a basement.

 

[Evan]

Aircon is usually as we know it a heat pump, this means in a domestic situation it could be say 400% effective, ie 1w input power produces 4w cooling.

Anyway off topic but I would say Xeon-D with some SSD instead of 2.5” 15k spinners and drop your consumption by almost 4 times per server, but also consider do you need all those served, can you consolidate the servers and also use nice big 8+tb disks for bulk storage.

 

[leonroy]

Consolidation's always tempting me with the massive savings in cost of hardware and electricity. Only issue I have with consolidation is that it seems to become super fiddly, either using passthrough to allow HBA access (for ZFS setups), or other such trickery.

Keeping file servers and hypervisors separate and not mixing the two seems to make for more reliable setups, although I have been messing around with Synology's Docker feature which works really well.

After doing the monthly `apt update`, `apt upgrade` dance on every host Docker's becoming more and more tempting. My ESXi hosts were also EOL'd with vmware 6.5. It's a shame because vmware + NAKIVO works beautifully with nightly backups and perfect restores, but definitely exploring my options right now and it's going to be bye, bye vmware.

It's a huge shame Proxmox doesn't seem to support Docker properly. Not a big fan of being a system integrator in my spare time, so the thought of a custom Ubuntu + Portainer setup sounds like too big a time commitment for me.

 

[nerdalertdk]

If you are dining apt manually, you are doing it wrong

Automation, Automation, Automation

 

[leonroy]

heheh, I'd agree but when I had Ubuntu set to install security updates automatically it'd fill the /boot volume up and kill the server...never got round to seeing if there was a workaround. Don't suppose you've encountered that issue?

 

[nerdalertdk]

Apt autoremove
Apt clean

 

[Marsh]

Official doc : RemoveOldKernels - Community Help Wiki

other links
How to Easily Remove Old Kernels in Ubuntu 16.04 | UbuntuHandbook
Safest way to clean up boot partition - Ubuntu 14.04LTS-x64, Ubuntu 16.04LTS-x64