Power protection for many servers, I am lost - any advice?

Discussion in 'DIY Server and Workstation Builds' started by DrStein99, Jun 2, 2018.

  1. DrStein99

    DrStein99 Member

    Joined:
    Feb 3, 2018
    Messages:
    86
    Likes Received:
    4
    I have 9 rack servers in the home lab, across (3) 15-amp breakers in a room with it's own air conditioner. So now the thunderstorms come in and lights flicker in house, and (2) servers just power off. So far I am lucky and have not discovered unusual malfunction.

    I started browsing around for surge-suppressor recepticals to install, but they are like $40+ each, and I might need like 10 of them is like $400+/-. Then I was looking at cheaper surge supression power-strips, and am lost since it seems some of them only output 1,500 watts with 8 recepticals. Just one of my servers runs at 650 watts with 2 power supplies.

    Does anyone have experience, can help guide me please? Should I just research a whole battery continuous UPS system? Can I get a module and just lace something together with a stack of lead batteries? It seems like building these power supplies can cost me as much as I pay for the used servers themselves :(
     
    #1
  2. aero

    aero Active Member

    Joined:
    Apr 27, 2016
    Messages:
    291
    Likes Received:
    50
    Have you actually measured the draw of your servers? I would be exceptionally surprised if they were anywhere close to 650watts each, unless you're running beefy gpus in them.
    If you haven't, get a killawatt device to measure, or probe to measure at the breaker panel.

    For loads like this you will need a bigger UPS.... Look for used APC 5000-6000 watt.
     
    #2
  3. Blinky 42

    Blinky 42 Active Member

    Joined:
    Aug 6, 2015
    Messages:
    465
    Likes Received:
    152
    Surge suppressors won't help you for the little brown outs that are causing your systems to drop offline. You will need to go with one or more UPS systems to power the equipment.
    As a starting point - what is the rated capacity of the power supplies in all the systems in watts? If they have redundant power then you only need to add one of them since the system should be able to run with a single supply only.
    Next to refine it down, if your servers have more modern PMBus support you can get an you may be able to get power usage from IPMI/iLO/IMM/DRAC/etc, otherwise you should try the kill-a-watt that @aero mentioned to get a handle on actual power usage.

    Sum that up across all oyur servers and see what the total is - then use that to scale the UPS you purchase to power everything.
     
    #3
  4. DrStein99

    DrStein99 Member

    Joined:
    Feb 3, 2018
    Messages:
    86
    Likes Received:
    4
    Yes, all servers are Dell or HP with idrac and Ilo. (4) 130 watt processors each server plus the fans and whatever else is a space-heate. Takes dedicated 10k BTU a/c to keep up with them all. 648 watts reported by idrac, so if that is at 12 volts, 24 volts, or 110 - I am not sure yet. I will get my external meters on the power lines and get a more specific number. I have 3 of those, and the rest are dual xeons. I split power supplies between breakers, so each server draw power from separate breakers. I wish my SWITCH had 2 power supplies, because when the breaker trips on that one, it has no redundant power.

    Ok, so battery backup on 1 power supply each server is good idea I did not think of. I was concerned in the event an electric brown out failure into the house like it did 2 years ago when I had to claim insurance that wiped out 1/2 all my appliances in the house. I would hope if that happened to my servers, I would only be replacing their power supplies - which is a lot cheaper than these surge suppressor costs.
     
    #4
  5. Blinky 42

    Blinky 42 Active Member

    Joined:
    Aug 6, 2015
    Messages:
    465
    Likes Received:
    152
    I would still put a UPS behind each redundant power supply, both to protect the power supply and to provide power conditioning/backup.
    I was thinking more if you have redundant 750W power supplies in your server, by design the server shouldn't be drawing a total of more than 750W at any time such that 1 supply can fail and the other keep your server running, so count it as 750W of load instead of 1500W when summing up your total power draw.
    If the server is drawing say 600W it will probably do ~300W + several % for overhead from each supply under normal circumstances.

    Take your total load in Watts and plug it into one of the UPS calculators:

    APC: UPS Selector
    Eaton: UPS Power Calculator, UPS Power Consumption Calculator - Eaton
    Tripp-Lite: Total Load Calculator | Tripp Lite

    Decide how long you want your UPS to keep things up and running, and be sure to pad the results up for future expansion. With many of the 3kVA and higher units you can get models that have expansion connectors to add extra battery packs for longer runtime.
    There are *lots* of used APC and Eaton UPSs out there, and companies that will refurbish them and replace the batteries and give you a UPS that is nearly as spiffy as new at a fraction of the price. Keep in mind that you will need to replace the batteries every 3-5 years depending on the unit/type of battery/how often it is used.

    If your power is crappy with frequent brownouts / voltage sagging I would go for an "on-line" UPS (a.k.a. double-conversion, constant conversion etc). Basically the voltage from the outlet comes in -> goes to DC and is converted back to AC again to run your devices. When the voltage sags or goes out totally the batteries fill in the gap and the output voltage is stable. The UPS can correct minor variations w/o needing to kick over to battery and that gets you through a lot of the issues you were probably experiencing that are not full outages.
    The trade-off is it is nowhere as efficient as the other style that passes through the line voltage as-is until voltage goes out of spec at which point it kicks over to the internal inverter driven by the batteries.

    You can get automatic transfer switches to connect single-supply devices (like your switch) to more than one power source. They can be picked up used as well.

    If your current setup is spread out across several 15A outlets - can you run a dedicated circuit from the breaker panel to where your rack is? Running everything @ 240V will save some power % overall. Running a 30A 240V circuit is 7200W before de-rating vs your 1800W with a 15A 120V. Most of the 208V UPSs you will find used can be set to run at 240V (check online).

    In case it isn't obvious - don't put the A/C on the UPS! If you *need* the A/C to keep the servers running happy, you are better off getting a generator and scaling your UPS runtime to cover the time needed to spin up the generator and then and ATS that swaps the input to the A/C and all the UPS units over to the generator until utility power returns for a configurable amount of time. Getting an ATS that can do a staggered power on/load shedding will help not beat up your generator as much with all loads kicking on and drawing max power on as soon as the generator is up to speed.
     
    #5
    leebo_28 likes this.
  6. nk215

    nk215 Active Member

    Joined:
    Oct 6, 2015
    Messages:
    285
    Likes Received:
    82
    large lead acid batteries are for run time. The UPS unit itself is for wattage.

    You can get powerful used APC UPS with external battery connectors (or just wire the internal connector to the outside) and hook those up to lead acid batteries. The 1500 watts units are cheap used. Get a few of them.

    Maybe it's time to consolidate and upgrade your old equipments. Older gen CPU is really not efficient to run 24/7 especially the X5600 cpus.
     
    #6
  7. dandanio

    dandanio Member

    Joined:
    Oct 10, 2017
    Messages:
    30
    Likes Received:
    8
    At your scale, maybe it is time to start thinking about a colo?
     
    #7
  8. DrStein99

    DrStein99 Member

    Joined:
    Feb 3, 2018
    Messages:
    86
    Likes Received:
    4
    Yes I am aware to not install battery backup line conditioning and / or generator on my air conditioning, cable tv, home stereo system, microwave oven, and hot tub. I will be using this for critical system purposes. Eventually I will run a generator in my shed so I can still play multi-player
    video games while my neighbors complain how long it going to take to turn electric back on.

    A colo is a good idea, I will have to shop rates on 1/2 stack space. I am near Philadelphia, there are several big ones I can shop out. I have to add up the numbers on the battery / power conditioning. I prefer to keep my monthly cost zero'ed out.

    I can change my lines over to 220v just by swapping breakers, and maybe swapping receptacles. I made certain to run 12-2 romex to all the new boxes in the data room. This is a good idea, I will start switching over to 220v now.

    I have only 2 x5600's, one of them I use as san/nas, that cost me an entire $80 shipped to me, and the other one was $138. Even if I bought a brand new system, I still need power conditioning. I would consider this option if someone else paid for the upgrades, or turned me on to some work that was profitable enough to pay to have so much left over for me to spend on thousands of dollars upgrade to save about 150 watts of electric each month.

    Yes this sounds like a great idea. I am interested in device(s), I can hook up in-line and use these salvaged batteries with 3/4 life left that cost me next to free.

    What is the math I consider for shopping out an UPS? Just for simple example; if I measured 900w drawn on the mains 120v line between 3 servers. Maybe each server is 300w each for the whole circuit if I plugged all power supplies on the same circuit. Under typical conditions would I shop a UPS unit of 1800 watts (2x)? 900 watts (about the same) ? Or 1/2 power 450 watts?
     
    #8
  9. DrStein99

    DrStein99 Member

    Joined:
    Feb 3, 2018
    Messages:
    86
    Likes Received:
    4
    I landed government auction for APC Smart-UPS units, from a school:

    SMT1500RM2U 1500VA LCD RM 2U
    DLA3000RM2U 3000VA Rackmount 2U

    There were like $35 each. Has anyone heard of them ? Ideally, I wanted just one big appliance to run the whole thing off that, but these seemed like I was getting a deal even if I have to replace batteries. I was just going to use some used car batteries and wire up externally, since it's cheaper than buying a bunch of the small replacement batteries.
     
    #9
  10. vl1969

    vl1969 Active Member

    Joined:
    Feb 5, 2014
    Messages:
    518
    Likes Received:
    53
    not sure if you can wire a car battery to them safely.
    I have 2 of them in my home rack. got them free when I won an ebay action for a half rack cabinet. the whole thing was $24 local pickup.
    came in and they where in the rack.
    just recently got a set of batteries for one of them. works good. had a power outage a few weeks back hold out my server, a pfSense Pc , cable modem and switch for 3 hours.
     
    #10
  11. nk215

    nk215 Active Member

    Joined:
    Oct 6, 2015
    Messages:
    285
    Likes Received:
    82
    Car batteries work fine with UPS. If you want super long life you can change the charging voltage but it's not really needed. The issue with car batteries is that they don't take complete discharge kindly. Because of very high battery capacity, you most likely won't run it down during a 1-2 hrs power outage.

    People recommend marine batteries but if you get free old car batteries, use them.

    My 300 watts 24/7 server run 6 hours on a set of 2 car batteries no problem (Smart UPS 1400).
     
    #11
    MikeWebb likes this.
  12. MikeWebb

    MikeWebb Member

    Joined:
    Jan 28, 2018
    Messages:
    63
    Likes Received:
    13
    "Car batteries" is rather generic. There are mainly two categories for car batteries "start" and "accessories". A car battery that is designed for starting the car all about the cold cranking power, it can handle a massive amp draw by the starter when cranking over the engine, it dies fast when constantly draining it and charging it (cycling), usually marketed with CCA (cold crank amps) figures.

    Accessory batteries are made for cycling, more importantly deep cycling, usually marketed with amp hour figures.

    Be sure to purchase deep cycle lead acid or better still a non-gassing valve-regulated lead–acid/gel cell/AGM battery to avoid the build up hydrogen if installed in an enclosed space.

    There are some other exotic battery chemistries (other then lithium) that are popular with caravaners/4wders that offer great amp hour performance but that comes with some sticker shock.
     
    #12
  13. rune-san

    rune-san Member

    Joined:
    Feb 7, 2014
    Messages:
    71
    Likes Received:
    14
    It's worth noting that hacking together extra run time with other batteries is especially caveat emptor. As other posters already mentioned, there are big unknowns with both the batteries and the unit that need evaluated.

    For instance, you list that you got an SMT and a DLA unit. The DLA unit is just a Dell rebrand of the SMT unit. That said, these are line-interactive UPS's, not Online. Unless the unit has its base in Online units with expandable runtime, you run the risk overheating the unit under high load for long durations if the battery capacity you add is too much higher than the stock capacity (you can obviously get away with some increase, there's safety margin after all).

    As far as the batteries are concerned, it's also worth noting that those old APC Units don't have float chargers, just trickle chargers. They'll cook their batteries eventually, so don't invest lots of money into the finest cells, just plan to proactively replace them every 2-3 years.
     
    #13
  14. Tom5051

    Tom5051 Active Member

    Joined:
    Jan 18, 2017
    Messages:
    230
    Likes Received:
    28
    I have a 15 year old APC Smart-UPS 1500 protecting my servers. I modified it so it has 10ga cables coming out the back that are attached to 2x 100amp 12v deep cycle SLA batteries in series. Runtime is a shade over 6 hours. Power draw is something like 15-17Ah @ 24vDC.
    Never had issues with it cooking batteries and lightning strikes haven't been an issue either, though a huge bolt did strike the back shed last year. Scared the $%$#% out of me. Got it on film. 15m away.
     
    #14
    Monoman and Tha_14 like this.
  15. Monoman

    Monoman Active Member

    Joined:
    Oct 16, 2013
    Messages:
    255
    Likes Received:
    64
    Could you possibly share the video?? ;)


     
    #15
  16. mstone

    mstone Active Member

    Joined:
    Mar 11, 2015
    Messages:
    449
    Likes Received:
    104
    15A circuit at 120V is 1800W, and you can't actually run it that high continuously; 80% derate is just under 1500W...

    In general, PSU ratings give people a very warped view of how much power they're actually using. But--there's a difference between running power and startup power: I've seen facilities where everything was on UPS but the first time power completely ran out and things tried to come back up when power was restored, the UPSs shut down because the startup load was over the current limit.
     
    #16
Similar Threads: Power protection
Forum Title Date
DIY Server and Workstation Builds Low Power ESXI build vs Supermicro E300 Oct 11, 2018
DIY Server and Workstation Builds Intel NUC Thunderbolt networking for super low-power cluster? Jul 30, 2018
DIY Server and Workstation Builds Q: Low idle power home server/NAS Jul 28, 2018
DIY Server and Workstation Builds The lowly Rack PDU / Power Strip / UPS Jun 28, 2018
DIY Server and Workstation Builds Power supply sizing for X10SDV build Jun 26, 2018

Share This Page