ESXi edge device: shutdown & power on

[awedio]

ESXi v8 (running 2 vms), one of the vm's is the firewall.

This is an edge device in a medical xray trailer.

The trailer's generator gets powered on in the morning & turned off at night.

Any good idea's on how I deal with daily ESXi shutdown & power-on?

 

[Rand__]

What exactly are you looking for?
Shouldnt be an issue to turn the box on via bios setting when it gets power?
And shutting it down should be easily done via a cron job if the time is set?

If you have a UPS in play there are other options o/c, you could add a smart home power plug ....

 

[awedio]

What exactly are you looking for?
Shouldnt be an issue to turn the box on via bios setting when it gets power?
And shutting it down should be easily done via a cron job if the time is set?

If you have a UPS in play there are other options o/c, you could add a smart home power plug ....

I am not physically present at the server to be able to power it on/off.
There is no set time for the generator to be turned off!

 

[awedio]

Shouldn't be an issue to turn the box on via bios setting when it gets power?

Ok, how about shut down?

If you have a UPS in play there are other options o/c, you could add a smart home power plug ....

Not sure I follow you on this?

 

[Rand__]

Smart plug can have calendar entries with start up, shutdown times

So power could be cut at any time and you have no warning?
Can you get someone to press a button before that to trigger a shutdown (again, smart home parts)?

 

[awedio]

Smart plug can have calendar entries with start up, shutdown times

So power could be cut at any time and you have no warning?
Can you get someone to press a button before that to trigger a shutdown (again, smart home parts)?

Yes, power will be cut once they are done at the end of the day,

This is a job I'm doing for a client.
There are 2 xray technicians in the trailer - (these are the users) & they know nothing about hypervisors/vm's etc!

Yes, I could get them to press a button.

 

[Rand__]

If this were Europe I'd recommend a Shelly button then - that can trigger various activites (URLs, mqtt messages etc) (+ something that receives the press and triggers shutdown script). I am sure you have something similar in the US.
Edit - ok, should say this needs a Wireless connection, that might not be there?

O/c if the server is accessible they simply could press the power button to shut it down? ESXi standalone can be configured to power down vms on shutdown and up on powerup

 

[Sconk]

Put a UPS on that will power off esxi when power is cut.

 

[Rand__]

Put a UPS on that will power off esxi when power is cut.

And then power off itself? and on when power is back? Or run on battery till next work day [holidays]? Not so simple

 

[Sconk]

And then power off itself? and on when power is back? Or run on battery till next work day [holidays]? Not so simple

The ups will just holder power to the esxi server shutsdown and set bios to power on at the morning or when power is back.

 

[Rand__]

No problem for a day but maybe for longer periods. Also battery life and temperatures in the trailer...

 

[awedio]

O/c if the server is accessible they simply could press the power button to shut it down? ESXi standalone can be configured to power down vms on shutdown and up on powerup

Yeah, it has to be something simple like this!

Will ESXi "gracefully" shut down if you press the power button?

 

[awedio]

Put a UPS on that will power off esxi when power is cut.

What UPS do you have in mind?

 

[oneplane]

While there are a lot of technical integrations (of which many won't work with the likes of vmware unless you're an NDA/Integration partner) this sounds a lot like a thing that would be best served with a human process rather than a technical one.

Make sure of the following:

1. A clear indicator somewhere that tells people if it is safe to shut off the power
2. A clear input method (a momentary rotating button for example) to indicate if you are intending to turn the thing on, or turn the thing off
3. A UPS to prevent accidents having significant impact

Then a piece of paper that says:

STARTUP PROCEDURE:

1. Turn on generator
2. Set button to "turn on computer system"
3. Wait for indicator to say system is online before calling support if the system doesn't work

SHUTDOWN PROCEDURE:

1. Set button to "turn off computer system"
2. Wait for indicator to say system is offline
3. Turn off generator

 

[awedio]

While there are a lot of technical integrations (of which many won't work with the likes of vmware unless you're an NDA/Integration partner) this sounds a lot like a thing that would be best served with a human process rather than a technical one.

I agree with the human process (KISS principle)

I'd still like to hear more about the vmware integration side of things!

 

[oneplane]

On the vmware side you'd need something to talk to the UPS to report the upstream power status, and something to start/stop the virtual machines depending on the power status.

Since vmware mostly uses SuSE and Photon OS the software would need to be compatible with that and installed on the host (so not on a guest). Such custom work is error-prone and not supported and likely to break when applying vmware patches. As for what it needs to do, it's mostly just glue logic between standard components, shell scripts or python or javascript could do this.

1. Set VMs to not auto-start when the hypervisor is booted
2. Use a cronjob to launch a script that uses either apcupsd or nut to talk to the UPS
3. Ask the UPS if upstream power is on or off, if off, stop the virtual machines and shut down the hypervisor when power down is complete, tell UPS to go silent, if on, check if the virtual machines are running and if not, power them up; check if UPS alarms are silenced if they are, turn then back on
3. Either a mains-powered pulse relay or a mainboard that is capable of USB-based power-on. In case of the relay, put the pulse output on the on/off pins on the mainboard and only set it to pulse when the power is switched on

What happens when everything is off and the generator is started: pulse is sent to the mainboard, system powers on, cronjob starts running, VMs get powered on.

What happens when everything is on and the generator is turned off: UPS notices power is missing, starts beeping, cronjob asks the UPS, UPS says power is missing and is beeping, VMs are powered down, alarms are silenced, hypervisor is powered down.

You can make all of this more smarter with more work. For example, you could add a stand-alone power management system that is directly attached to all devices and has its own battery. It can turn all devices on or off as desired, communicate over the network to do it gracefully as well. Such a device could be an SBC like a Pi with a UPS HAT.

 

[TLN]

I would stick with simple linux box and some sort of button for turn off: Home assistant can connect to any wireless plug (tp-link kasa in my case) and can start action on that:
Turn off all the VMs, and schedule hypervisor poweroff in 10 minutes.

Alternatively you can track UPS power and turn of after power goes off. Again single button in that case

On the vmware side you'd need something to talk to the UPS to report the upstream power status, and something to start/stop the virtual machines depending on the power status.

You can track status of UPS with VM, and cron power off for VM right away and for ESXi in 5-10 minutes. That way you don't have to make ESXi talk to UPS.

 

[oneplane]

I would stick with simple linux box and some sort of button for turn off: Home assistant can connect to any wireless plug (tp-link kasa in my case) and can start action on that:
Turn off all the VMs, and schedule hypervisor poweroff in 10 minutes.

Alternatively you can track UPS power and turn of after power goes off. Again single button in that case




You can track status of UPS with VM, and cron power off for VM right away and for ESXi in 5-10 minutes. That way you don't have to make ESXi talk to UPS.

While you can do it in a VM, you get the issue where a depleted UPS and someone rapidly turning the generator on and then off causing datastore corruption (in the described long term shutoff scenario). If you can do it without mounting anything that's safer. Perhaps some sort of squashfs-based option would still be safe enough. Or just an oversized UPS.

Edit: come to think of it, when extra hardware isn't an issue, something OpenWRT-based might be a good fit since it's made to not be that easy to kill with just power interruptions. And since all of this stuff would preferably be out-of-band from the vmware devices anyway (like a BMC), you can do all the custom sequencing you want. Make sure the UPS is networked, put the WRT device on redundant power supplies (so it can even work when the UPS is dead but the generator is on), and make it networked to the edge BMC, edge host and UPS. No custom wiring required and as long as the UPS can self-start from a depleted state you can handle pretty much all scenarios using any scripting or programming languages you want. You can even stick on a basic 2x20 matrix LCD to let the user know what's up, and a few buttons to get out of odd scenarios (like, power down when the power is still up, or power up if the power is on but auto-power-up failed). Hell you can probably get a standard Matrix Orbital USB thing to do it. Or you could use a PI and a fancy HMI display.

 

[Rand__]

I think unless you need a UPS for other reasons you should leave it out, it complicates things

ESXi should power off everything fine when you press the button, the box should power on fine when you restore power (if bios is set to always power on) (just make sure to configure vms to start/stop with esxi, ensure order if its relevant).
It might take a minute to start/stop all vms, but f ppl are clearly instructed to wait till the box is off before they cut power then it should be fine.