APC UPS management strategy and sequential shutdowns

[lpallard]

Hello,

I acquired a NMC AP9360 management card for one of my SMT750RM2UC which is protecting a few devices and servers, namely my Proxmox host, a pfsense router and a 48 port switch.....

Right now during a power outage apcupsd running on my PVE initiates a shutdown when the battery reaches a certain level. pfsense gets shutdown by "force" and so is the switch and other smaller devices....

for the devices not smart enough to run software for power control, so be it, but for the main machines (PVE, pfsense, etc) I'd like to achieve the following:

1. NMC on the UPS detect a power outage and monitors battery level
2. Battery level reaches a certain %
3. NMC issues a shutdown order to PVE then waits for the shutdown then kills the power on the outlet
4. NMC does the same for the outlet powering pfsense
5. NMC kills the power to the switch outlet
6. NMC shutdowns the UPS

Power comes back on mains:

1. NMC detects power is back
2. NMC charges the battery until a certain % is reached (this is to prevent re-powering everything then another outage occurs and the battery level is too low)
3. NMC powers up on outlet 1
4. NMC powers up on outlet 2
5. ....

The overall strategy may not be doable "as is" but I'm sure with the proper hardware (serial/usb/ethernet cables, NMC, etc...) this is feasible somehow. I cannot imagine datacenters manually managed all the time....

I tried fiddling with the UPS outlet groups, apcupsd, etc but ended up with ungraceful shutdowns which almost ruined PVE........

Anybody?

 

[oneplane]

opnsense (and presumably pfsense) supports apcuspd and nut, so those can totally use network based UPS monitoring and shutdown. Anything with SSH can be shut down using a script on any host that has apcupsd or nut, so the whole shutdown case is covered.

The real trick is powering everything up again, especially if you are not using a PDU with hardware that is set to auto-power-on when AC is switched on (assuming IPMI is not in place). Sometimes WoL can work, sometimes the only true option is to wire up some GPIO to the power button (i.e. using a relay or SSR or transistor if you have a common ground rail).

As for the NMC card: I generally only use those to read data from the UPS and send SNMP traps. The actual brains are somewhere else (i.e. using an SNMP to prometheus bridge via an exporter and trap receiver that re-exports as SNMP counters), so we can use common metrics to trigger on instead of using different protocols and data formats for every random piece of equipment.

 

[lpallard]

pfsense indeed supports NUT or apcupsd. I'd rather not rely on scripts via SSH to powerdown hosts and clients.

You totally lost me on the last paragraph....

Your UPS'es are not doing any logics per-se, only sending network broadcasts of the metrics and something else catches these and interpret/does logics?

Pardon my poor understanding. The best I did so far is run and config apcupsd on PVE to shutdown the server based on UPS battery level...

 

[heromode]

The real trick is powering everything up again, especially if you are not using a PDU with hardware that is set to auto-power-on when AC is switched on (assuming IPMI is not in place). Sometimes WoL can work, sometimes the only true option is to wire up some GPIO to the power button (i.e. using a relay or SSR or transistor if you have a common ground rail).

most BIOS have settings for this, specifically power off, last state, or power on. With last state enabled, and the UPS shuts down the server gracefully, it remains powered off once power returns to the outlet. But with power on, the server will always power on when the outlet power returns.

again with last state, if there is a sudden power cut, then last state was powered on, but if the server shuts down gracefully, and the motherboard cuts power to PSU, then last state will be power off, and thus server will remain off when AC power returns. So for UPS controlled graceful shutdown, set the BIOS setting to 'power on' or 'always power on' depending on your BIOS

edit. It's highly unlikely that your BIOS doesn't have this setting, i can't remember a single computer i've had in my lifetime that didn't have this option in BIOS. Sometimes it can be hard to find, but it's in there somewhere, and the reason for it is exactly the issue the OP describes.