Upgrade advise to make my server more power efficient

[josiahrulez]

Hey all, so i have 2 file servers and they are all kind of old and not very power efficient (they idle at like 370watt tower and peak at 600watt during start up). I also have a bottleneck and i think its the motherboard/cpu (Not enough bandwidth) because it struggles at lans when 3 or more people leech from the same array (drops down to about 20-30mb/s user, using 10gbe, on an 11 disk raid 5 array).

But i also want to keep it power efficent, these servers run 24/7 at home and are mostly idling. I'm Also going to kill off the second file server and attach its HDD's to the main fileserver via external SAS SFF-8088. Any recommendations for a cheap 6GB/s External RAID card/SAS Expander?

So for this build i want to go for a server CPU/Motherboard, i wouldnt mind getting one with the LSI2008 Chip, or 10Gbe Ethernet. Like this > http://www.supermicro.com/products/motherboard/Xeon/C600/X9SRh-7TF.cfm

But a big problem is i dont know which CPU are more power effecient, while still having good performance, Intel Xeon or AMD Opteron? The AMD Opteron EE and HE CPU's look nice and seem to have a low TDP, and are cheap. I read somewhere that the newer AMD Opterons have a power saving feature where it cuts power to cores that aren't being used? Or are there any Intel CPUs that are good low power usage and still good performance? any recommendations?

My current File Servers

There both running Windows 7 at the moment, but i have a Windows Server 2008 R2 Enterprise key i can use when i upgrade, or i may be able to get a server 2012 key.
[U][B]First File Server[/B][/U]
Case - Norco 4224
PSU - Corsair HX1000Watt
CPU - AMD Phenom II 965 
OS HDD - Seagate Momentus XT (I have a Crucial M4 SSD to go in, but havent gotten around to reformating)
Motherboard - GA-890FXA-UD5
GPU - Nvidia 210 (Motherboard has no onboard graphics)
Memory - 16GB G.Skill 1333mhz
NIC - Dell 10GB Ethernet XR997
RAID Card - HighPoint RocketRAID 3560 24 Port
[U]Array 1[/U] - RAID 5
2x Hitachi 7200rpm 3TB
4x Hitachi 5400rpm 3TB
5x Seagate 7200rpm 3TB
[U]Array 2[/U] - RAID 5
8x Samsung 5400rpm 2TB
[U]Array 3[/U] - RAID 5
4x Samsung 5400rpm 2TB
1x Seagate 7200rpm 3TB


[B][/B][U]Second File Server[/U]
Case - Some no brand case
PSU - Some no brand PSU rated at 700watts
CPU - AMD Phenom II X2 555
OS HDD - Samsung 830 64GB
Motherboard - GA-990FXA-D3
GPU - AMD 6450 (Motherboard has no onboard graphics)
Memory - 4GB Corsair 1600mhz
RAID Card - SuperTrak EX16350 
[U]Array 1[/U] - RAID 5 
9x Samsung 5400rpm 1.5TB
2x Seagate 5400rpm 1.5TB

 

[mobilenvidia]

To me the biggest waste of power is the power supply its self.
Get your self a 'Platinum' or better certified PSU.

80% vs at least 91% efficiency
Get a PSU which when running at the power level the server runs at most of the time is at it's most efficient.
ie a 1000w PSU is inefficient at only outputting 100w most of the time, a 500w PSU would it much more efficiently.
PSU's run most efficiently at around 50%
But then also keep an eye on noise levels not all PSU's are the same, some ramp the fan speed up real quick = noise
Do some homework and it will pay off in the end.

 

[xnoodle]

What's your budget?

As far as raid cards go, you can go with something Areca or LSI based; they tend to work well with SAS expanders. (Does anyone know if Areca 1880ix/1882ix have problems with another SAS expander?) Depending on where you are and what you have access to, prices would vary.

CPU wise, do you do anything on those two machines apart from serving up files? Are you connected via 10gbe to your other machines? Didn't see any mention of it on your second machine. You can get a cheap(er) CPU and motherboard if you don't need 10gbe and can suffice with gigE. I would go Intel over AMD here. And I'd go socket 1155 over 2011.

You're going to see the largest reduction in power consumption when you get rid of those motherboard/CPUs, probably shave off ~200 W of usage easily. After that, you're going to want to get better power supplies. The HX1000 80plus is <80% efficient if you're only drawing <300W.

Also keep in mind you can't transfer raid volumes to and from dissimilar cards, so you'll need some sort of temp space for the 15 TB volume on Server #2 so you can initialize it on your new card.

 

[MiniKnight]

Tips:

More power efficient CPUs will save you a lot.
That motherboard is a good one and has the LSI sas2308 controller. Newer faster.
The board also has newer 10 gig Ethernet. Saves a good amount of power.
Wish this site would review Opteron 3000 chips but cant buy motherboards for them anyway

 

[dba]

Since the server will spend most of its time idling, the focus should be on achieving low power usage at idle. Just a few thoughts:

1) Make sure that your operating system and/or RAID/HBA card support idle drive spin down.
2) Your 10Gbe card is a bit old and probably draws 15-20 watts at idle.
3) Buy the smallest power supply you can - and make it an efficient one. A 1200 watt power supply that is rated as 95% efficient might be only 75% efficient when running a 150 watt server.
4) Probably most important For all of your components: make a spreadsheet that shows idle power usage specifications in watts. You'll probably be able to save 20-100 watts in places you'd never imagine.

Hey all, so i have 2 file servers and they are all kind of old and not very power efficient (they idle at like 370watt tower and peak at 600watt during start up). I also have a bottleneck and i think its the motherboard/cpu (Not enough bandwidth) because it struggles at lans when 3 or more people leech from the same array (drops down to about 20-30mb/s user, using 10gbe, on an 11 disk raid 5 array).

But i also want to keep it power efficent, these servers run 24/7 at home and are mostly idling. I'm Also going to kill off the second file server and attach its HDD's to the main fileserver via external SAS SFF-8088. Any recommendations for a cheap 6GB/s External RAID card/SAS Expander?

So for this build i want to go for a server CPU/Motherboard, i wouldnt mind getting one with the LSI2008 Chip, or 10Gbe Ethernet. Like this > http://www.supermicro.com/products/motherboard/Xeon/C600/X9SRh-7TF.cfm

But a big problem is i dont know which CPU are more power effecient, while still having good performance, Intel Xeon or AMD Opteron? The AMD Opteron EE and HE CPU's look nice and seem to have a low TDP, and are cheap. I read somewhere that the newer AMD Opterons have a power saving feature where it cuts power to cores that aren't being used? Or are there any Intel CPUs that are good low power usage and still good performance? any recommendations?

My current File Servers

There both running Windows 7 at the moment, but i have a Windows Server 2008 R2 Enterprise key i can use when i upgrade, or i may be able to get a server 2012 key.
[U][B]First File Server[/B][/U]
Case - Norco 4224
PSU - Corsair HX1000Watt
CPU - AMD Phenom II 965 
OS HDD - Seagate Momentus XT (I have a Crucial M4 SSD to go in, but havent gotten around to reformating)
Motherboard - GA-890FXA-UD5
GPU - Nvidia 210 (Motherboard has no onboard graphics)
Memory - 16GB G.Skill 1333mhz
NIC - Dell 10GB Ethernet XR997
RAID Card - HighPoint RocketRAID 3560 24 Port
[U]Array 1[/U] - RAID 5
2x Hitachi 7200rpm 3TB
4x Hitachi 5400rpm 3TB
5x Seagate 7200rpm 3TB
[U]Array 2[/U] - RAID 5
8x Samsung 5400rpm 2TB
[U]Array 3[/U] - RAID 5
4x Samsung 5400rpm 2TB
1x Seagate 7200rpm 3TB


[B][/B][U]Second File Server[/U]
Case - Some no brand case
PSU - Some no brand PSU rated at 700watts
CPU - AMD Phenom II X2 555
OS HDD - Samsung 830 64GB
Motherboard - GA-990FXA-D3
GPU - AMD 6450 (Motherboard has no onboard graphics)
Memory - 4GB Corsair 1600mhz
RAID Card - SuperTrak EX16350 
[U]Array 1[/U] - RAID 5 
9x Samsung 5400rpm 1.5TB
2x Seagate 5400rpm 1.5TB

 

[mobilenvidia]

At my house 1kwhr = $0.25

Running
1kw server 24/7 = 8760kwhr per year = $2190
500w server 24/7 = $1095
250w server 24/7 = $548
125w server 24/7 = $274
63w server 24/7 = $137

Basically every watt saved $2.19 savings per year.

But if your old 20w 10Gbe card = $43.80 per year
your shiney new 10w 10Gbe card = $21.90 per year
You save $21.90 per year, but how much did you spend to reduce the power ? how many years of savings will it take ?

 

[Patrick]

That's how I look at power also. In some places, as you use more power, that power costs more. Looking at somewhere like a datacenter, 1A at 120v = 120w runs around $50/ mo at many providers. So the difference between your 125w server and 250w server is more or less $50/mo or $600/ year.

 

[dba]

...
You save $21.90 per year, but how much did you spend to reduce the power ? how many years of savings will it take ?

Good point mobilenvidia - knowing your ROI is important. I didn't express it well, but I was wondering if 10Gbe was necessary at all. If josiahrulez could get by with one or two built-in (and otherwise unused) Gigabit ports, then maybe the entire 20 watts could be eliminated at no cost.

 

[Artzig]

I've got a new build on the bench at the moment where power consumption has been bourne in mind right from the outset. I'll gladly post some power consumption figures for it when it's fully up and running. Some preliminaries for now.

Design aim in order of priority:
1) Reliability
2) Ease of maintainence
3) Speed of backup
4) Quiet
5) Power efficient.

The machine is only aimed at efficient backup. It's not multiuser or streaming.

Motherboard: Supermicro X9SCM-F
RAM: 16GB of DDR3 ECC RAM Kingston KVR16E11/4. Designed to run at 1600Mz but currently only at 1066MHz with the G550T in it.
CPU: (temporary while I sort out the M1015 problem) is a SandyBridge G550T. Final will be a Xeon E3-1220Lv2 IvyBridge.
HDDs: 8x WD RE3 7200rpm 500GB HDDs eventually in a RAID Z2 config.
SATA Card: IBM M1015 probably flashed to 9211-8i. (Thankyou STH for a good read and info on this!)
PSU: Cooler Master GX550W. 80+ Bronze rated. Typical efficiency is about 85% apparently. Spare from an old build.
OS: FreeNAS 8.3.0 p1 running ZFS.

The PSU has its own fan. Despite the rest of the system only having a single large fan it's still overated for its needs. The CPU is passively cooled. The HDDs run fairly cool anyway and with a slight airflow over them are just warm to the touch. The RAM runs almost cold. The Cougar Point chip needs a small airflow to be happy but an upgrade to its heatsink will cure that. Some of the refinements here are aimed at reducing the noise level. Final aim is to leave the main system fan off but in reserve for if the temp does climb.

System plugged in but off, power consumed is 8.2W. Higher than I expected. With FreeNAS booted, ready, but idle and 7 of the 8 drives spun up, tickover is 86.2W. The system turn on surge (which is a guess due to unknown ability of the logger to read peaks) is about 250W. All drives spin up together.

The system power consumption will likely fall when the Xeon E3-1220Lv2 goes in with its 22nm lithography but will rise again as the 8th drive goes in. Also the system is truly idle at the moment whilst I sort the M1015 flashing out.

What I would say about PSUs is make sure that a smaller PSU can handle the turn-on surge which will vary from system to system, and with things like staggered drive spin up. A fault caused by that could be difficult to diagnose.

Like has already been said, each power supply will have a sweet spot for efficiency and unless the makers have graphed and published it you'll not find that out. Likely they'll print their best result in the range, unless the specs denote a minimum efficiency?

I'd go for a PSU that has the best efficiency at an average price and then concentrate on the rest. Don't chase finding a PSU that's mega efficient.

One thing I've noticed whilst reading up is that the power consumption of the CPU rises dramatically with cache size. The E3-1220Lv2 has only 3MB. Whether the system will bottleneck due to this I'm not sure but it's easily upgraded. Thought I'd buy one and see.

 

[MiniKnight]

Why down clock the memory?

 

[cactus]

G550T will only use DDR3-1066

 

[josiahrulez]

Thanks for all the feedback, i would of posted back sooner but i was busy over the last few weeks.

What's your budget?

There isn't really a budget, but i won't go crazy and spend heaps, i also like buying second hand.

To me the biggest waste of power is the power supply its self.
Get your self a 'Platinum' or better certified PSU.

80% vs at least 91% efficiency
Get a PSU which when running at the power level the server runs at most of the time is at it's most efficient.
ie a 1000w PSU is inefficient at only outputting 100w most of the time, a 500w PSU would it much more efficiently.
PSU's run most efficiently at around 50%
But then also keep an eye on noise levels not all PSU's are the same, some ramp the fan speed up real quick = noise
Do some homework and it will pay off in the end.

Seasonic 860W 80Plus Platinum - $276
I was thinking about getting this PSU, unless someone objects or can recommend something better?
The Anandtech review tests say it runs at 90.77% efficiency at 20% load and 93.03% efficiency at 50% load.

I also did some more testing on my current servers, Server 1 runs at about 200-250 Watts, and Server 2 runs at about 170 Watts.

And server 1 with Idle Disk Spin down, it uses about 140 Watts

As far as raid cards go, you can go with something Areca or LSI based; they tend to work well with SAS expanders. (Does anyone know if Areca 1880ix/1882ix have problems with another SAS expander?) Depending on where you are and what you have access to, prices would vary.

Yeh I'm really not looking to buy a new RAID card unless i can get something decent second hand and for a good price. If i get the motherboard with the LSI 2308 chip, i can do some tests to see how well it performs against my Highpoint card (The Highpoint 3560 is Hardware RAID, so it's not one of their shitty software RAID cards, although its not as good as LSI or Areca). If the onboard LSI chip performs better then the highpoint RAID card, i could easily get a SAS expander.

They are SAS/SATA ports, so will a Sata-Sas Breakout cable to the SAS Expander work?
Is SAS2 compatible with Sata3?
I also know someone selling an Adaptec 31605 with BBU for $180?

CPU wise, do you do anything on those two machines apart from serving up files? Are you connected via 10gbe to your other machines? Didn't see any mention of it on your second machine. You can get a cheap(er) CPU and motherboard if you don't need 10gbe and can suffice with gigE. I would go Intel over AMD here. And I'd go socket 1155 over 2011.

Not really, but at LANs the CPU is normally at about 50% load, im not sure if the 10Gbe nic is CPU Intensive or its the filesharing application we use? 10GBe is a really nice feature to the motherboard, I use it alot, i backup my server to friends servers etc and they backup some of their stuff to mine etc, and we have a 10Gbe switch at the lan, so everyone benefits if the main sharers use 10Gbe. (We shgare alot of those free linux ISOs)

Also keep in mind you can't transfer raid volumes to and from dissimilar cards, so you'll need some sort of temp space for the 15 TB volume on Server #2 so you can initialize it on your new card.

I know this much, that's why i have a spare Highpoint 2340 (in case the shit hits the fan, i can still access my data). And i could borrow a friends fileserver for a week or so while i do the upgrades.

Tips:

More power efficient CPUs will save you a lot.
That motherboard is a good one and has the LSI sas2308 controller. Newer faster.
The board also has newer 10 gig Ethernet. Saves a good amount of power.
Wish this site would review Opteron 3000 chips but cant buy motherboards for them anyway

When i was looking around, i couldnt find any 42XX EE chips, but anyways I've decided to go for Intel over AMD.
I wouldn't mind going 1155 if they had a motherboard with 10Gbe and LSI RAID onboard, those 17Watt i7's look nice.

Also when will 22nm socket 2011 Consumer or xeon CPUs be released?

4) Probably most important For all of your components: make a spreadsheet that shows idle power usage specifications in watts. You'll probably be able to save 20-100 watts in places you'd never imagine.

I might try that, thanks for the tip.

Good point mobilenvidia - knowing your ROI is important. I didn't express it well, but I was wondering if 10Gbe was necessary at all. If josiahrulez could get by with one or two built-in (and otherwise unused) Gigabit ports, then maybe the entire 20 watts could be eliminated at no cost.

I would use 10Gbe at home if i could find a cheap 10Gbe switch, and at home i normally take out the 10Gbe nic because the fan is so loud.


So i think i can get rid of the second fileserver and fit all the HDDs into the norco 4224,

If i got a normal sized PSU, that would give me alot more room and if i moved the Fan thing closer to the HDDs it would give me room to maybe put some HDD bays in, i would just have to mod them a bit to allow more airflow (I'm not using the HDD bays in the picture) and get better fans. Maybe the Corsair high performance fans? (The Noctua fans dont move much air)