ESXi / VCenter - New HomeLab Question

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

K D

Well-Known Member
Dec 24, 2016
1,439
320
83
30041
Quick Question :
Can I have a VCSA running on a Free ESXI server manage other licensed esxi hosts? The hosts themselves will not be managed by VCenter.

Background :
Scenario :
esxm1 - ESXi Free
esxm2 - ESXi Free
esx1 - Licensed
esx2 - Licensed
esx 3 (2 CPU)- Licensed
stor1 - Shared Storage (Virtualized NAS/SAN over ESXi)
stor2 - Shared Storage (Non ZFS/esxi)

The plan is to use esxm1 and esxm2 will be my management "cluster" hosting VCSA, Windows AD/DNS/DHCP/WDS/WSUS VMs. Will be parts from my old desktops/workstations to cobble these 2 together.

esx1-3 will be the lab cluster.

stor1-2 : Initially will provide storage for the lab. But eventually will be the backup for my home media server.

I do not have access to any NFR/ Non production licenses and plan to get a VMUG membership to get the EVALExperience license for my use. The number of licenses available in EVALEx is not sufficient based on this layout.

So can I just use the free ESXi hosts to host VCSA (Not join them to Vcenter) and use VCSA to manage esx1-3 and stor1?

If this is not possible, I can move to a windows based management cluster and use the VCenter for windows software instead of VCSA.

I have no experience in VMWare and the purpose of building this lab is to learn. So any suggestions/critiques are welcome.
 

IamSpartacus

Well-Known Member
Mar 14, 2016
2,515
650
113
Yea there's no reason it shouldn't work. Since the unlicensed hosts won't be added to vCenter it doesn't even know about them.
 

K D

Well-Known Member
Dec 24, 2016
1,439
320
83
30041
Thanks. I am currently trying to get things to a semi-functional state before I move them to the rack in the basement and since I am tearing down and rebuilding everyday due to one mess up or the other, I'm still running everything with the eval licences.
 

IamSpartacus

Well-Known Member
Mar 14, 2016
2,515
650
113
The one thing to keep in mind is that since vCenter will not be on licensed hows, you'll need to manage the backup of that VM through some other means since you won't have API access to the ESXi kernel like licensed only backup solutions such as Veeam.
 

K D

Well-Known Member
Dec 24, 2016
1,439
320
83
30041
The one thing to keep in mind is that since vCenter will not be on licensed hows, you'll need to manage the backup of that VM through some other means since you won't have API access to the ESXi kernel like licensed only backup solutions such as Veeam.
I understand. I havent gotten that far yet. I probably will just go with the vcenter installed on win 2012r2 to avoid the host licensing issue.
 

epicurean

Active Member
Sep 29, 2014
785
80
28
I don't mean to hijack this thread, but if you have several esxi hosts (all licensed), which is the "best" host for the VCSA if I intend to shut down all the hosts for a few hours every week?

should I create a very low powered, thin free hypervisor host with just the VCSA?
 

IamSpartacus

Well-Known Member
Mar 14, 2016
2,515
650
113
I don't mean to hijack this thread, but if you have several esxi hosts (all licensed), which is the "best" host for the VCSA if I intend to shut down all the hosts for a few hours every week?

should I create a very low powered, thin free hypervisor host with just the VCSA?
As long as you remember which host the VCSA is running on, all you'd have to do is turn that host on and access the vSphere client of it to power on the VCSA each time.
 

epicurean

Active Member
Sep 29, 2014
785
80
28
What I meant to ask was , how can I automate the shut down and restart of the hosts in the entire cluster if the host that the VCSA is on is also shut down?
 

cheezehead

Active Member
Sep 23, 2012
723
175
43
Midwest, US
Technically you could reload esxm1 & esxm2 at different days and then run them joined to vCenter. If your running the hosts without a license loaded, you have 60 days to play with all features enabled.
 

IamSpartacus

Well-Known Member
Mar 14, 2016
2,515
650
113
What I meant to ask was , how can I automate the shut down and restart of the hosts in the entire cluster if the host that the VCSA is on is also shut down?
Yes if you want to automate the process then the VCSA will obviously have to be on a host that is on. The host the VCSA is on doesn't have to be part of any cluster that it's managing so you could even have a nested ESXi host in VMware Workstation 12 for example running VCSA to manage your hosts.
 

NetWise

Active Member
Jun 29, 2012
596
133
43
Edmonton, AB, Canada
The best place for the VCSA is on the cluster. Put it on one of the hosts. When you shut down the system, your script would tell all the VM's to power off, then Host2, and Host3. Then tell Host1 to put VCSA VM into the automatic startup, configure the default power parameter to be shutdown (controlled) or suspend, and the VCSA which is the last one running will do so when the host triggers a shutdown.
 

epicurean

Active Member
Sep 29, 2014
785
80
28
The best place for the VCSA is on the cluster. Put it on one of the hosts. When you shut down the system, your script would tell all the VM's to power off, then Host2, and Host3. Then tell Host1 to put VCSA VM into the automatic startup, configure the default power parameter to be shutdown (controlled) or suspend, and the VCSA which is the last one running will do so when the host triggers a shutdown.
Then how do I get the whole setup to start up again?
Does that mean my pfsense router VM also needs to be on the same Host1? Which should be the one to start up first, pfSense or VCSA? All my hosts are also connect to a APC network UPS via Powerchute Network Shutdown VM. I was told THAT Vm should be the the first to start, and last to shutdown.
 

K D

Well-Known Member
Dec 24, 2016
1,439
320
83
30041
Yes if you want to automate the process then the VCSA will obviously have to be on a host that is on. The host the VCSA is on doesn't have to be part of any cluster that it's managing so you could even have a nested ESXi host in VMware Workstation 12 for example running VCSA to manage your hosts.
This.

I have VCSA on a stand alone host that is not part of my lab.
 

NetWise

Active Member
Jun 29, 2012
596
133
43
Edmonton, AB, Canada
Then how do I get the whole setup to start up again?
Does that mean my pfsense router VM also needs to be on the same Host1? Which should be the one to start up first, pfSense or VCSA? All my hosts are also connect to a APC network UPS via Powerchute Network Shutdown VM. I was told THAT Vm should be the the first to start, and last to shutdown.
If the host is in an HA cluster, setting the VM startup/shutdown order is disabled. So there's that to understand. But that's if you're configuring the host in vCenter. If you configure the host directly - via the Host Client (I'm on v6.5) you can use PowerShell or the GUI to set the host VM startup/shutdown options.

You want to set:
HOST -> Manage -> Settings -> System tab -> Autostart. Enabled=Yes. Start Delay 120 seconds or whatever you like. I'd enable "wait for heartbeat". STOP ACTION should be Shutdown or Suspend, according to your wishes. Once you do THAT, now when you tell the host to reboot or shutdown, it will/should attempt to shutdown/stop any VM's it can - or DRS vacate them.

So you need to make sure that DRS doesn't vacate them. You can:
- disabled HA and DRS if you like, this helps you have some control over what's happening.
- ensure that Host1 has your core VM's - your router if it's software, your VCSA, your Domain Controller. Maybe your SQL server or whatever if needed for other systems. Personally, I like to have a list/matrix of "homehost=" for each VM so when I do the shutdown, I can move the VM's to an appropriate host. That way, not only can I control the startup of Host1, but I can control the startup of VM's on Host2/Host3, which helps me get more VM's running if it takes 60-120 seconds for them to boot or "wait for heartbeat".
- set the VM startup order for those "known VM's" on each host to be what you want for a specific startup order. Set the rest to "any order" and they'll start after all the numbered ones, and keep going.
- stop all the VM's you don't need. They're going to take time to shutdown, especially at once. Use a loop to wait for them to shutdown on Host2/Host 3
- shutdown the non-primary hosts now that the VM's are off.
- now that only one host is up, HA and DRS can't move your VM's around anyway.
- shutdown the non-survivable host (eg: Host1) that will keep your core VM's on it. You've already set your startup order, so they'll startup on boot of the host. Because you also set the STOP ACTION to shutdown or suspend, those VM's will do that before the host powers off, as it does that in a controlled fashion.

How do you get it all to start up again?

- start Host1
- it will autostart the VM's in the order you specified. You can do your router before VCSA if you want, but until VCSA boots up some VM's, your router can't do much anyway - inbound or outbound. It's really personal preference. The only reason you need your Powerchute VM up, is if you are worried about a secondary power failure during your startup, and you need to emergency shutdown again. Personally, I feel that's a multiple cascade compound failure, and pretty hard to know where you're going with it. But it is a discussion to have, and design around. In your case, I'd start it up third. If you're a windows environment, you probably want your DC to come up before your router and before your VCSA, especially if your VCSA is AD integrated in any way.
- Power on Host2/Host3 - manually or via a script on the VCSA or another VM that sends a poweron signal to the IPMI/IDRAC, or the PDU ports to enable power to the host with the BIOS set to auto-power on, on power resume. Whatever will get your hosts to turn on.
- Host2/Host3 will start their specific startup VM's in order, then their any order - while Host1 continues to do it's own.

At this point you're started up.

Bonus points if you have to figure out how to auto start it ALL from scratch, with no humans, Switches, SAN, and all. In that case, I:
* Had a "building power PDU" that did NOT auto power on on power resume.
* Had a "UPS PDU" that powered on via the port on the UPS - ONLY if the UPS was sufficiently recharged to tolerate a second emergency shutdown. So not upon power resumption, but upon battery level reaching 60% or whatever the level is.
* The PDU then, upon having power resume, turned on the PDU power Primary power to the switches - which took 6 minutes to fully boot.
* The PDU then, turned on the PDU power Primary to the SAN - which took 4.2 minutes to start, but shouldn't/couldn't come up before the switches, or the network could go into a panic mode.
* The PDU then, turned on the PDU power Primary to Host1 - which had the BIOS set to Auto-Power On. This was set to be about 1 minute before the SAN was booted, so it had that time to get to the point that it needed the SAN.
* Host2/Host3 then get their PDU power Primary set to enabled, powering them up - about 2 minutes after Host1 PDU port comes on
* The VCSA/Windows vCenter runs a startup script to tell the Building Power PDU powering all the secondary PSU's that it can now turn on and restore secondary power to all devices.

Not saying it's perfect. But it works well enough for a 24U 3 Host/2 Switch/1 SAN environment, in a PLC/Gas Plant that has to be able to shutdown completely AND power on, with no trained or IT staff of ANY kind, or even humans period, as it might be in a remote location that no one can even get to.

I'm sure I can dig up my scripts for this, to help out. But the above is the basic process. There's absolutely no need for VCSA to be off cluster. You just need to look at each piece, its dependancy, and solve for it. Took me a lot of trial and error, and some stopwatch timers to figure out the startup order, and I had to retime things after firmware upgrades to ensure things still took the same time. But worked like a charm.

I hope that helps?
 

epicurean

Active Member
Sep 29, 2014
785
80
28
@NetWise, Thanks so much ! I hope I can continue to tap on your experience in this matter when I run into a brickwall trying this out
 

NetWise

Active Member
Jun 29, 2012
596
133
43
Edmonton, AB, Canada
I'll be playing all week. Tip your wait staff. Enjoy the show.

To be fair, when I was tasked with this a few years ago, it was a really interesting project and I have full time to do whatever I needed - the project had a near unlimited budget for time. So I stuck it out. If you're trying to do this just for a home lab, it doesn't need to be quite so fancy. But for most businesses, MANY do not have auto shutdown or UPS shutdown figured out at all, and certainly not startup. They sort of have a DR plan to bring things up in a secondary site, but it assumes the infrastructure is up, and infrastructure supporting VM's are up (eg: secondary DC, linked vCenter, backup server to do the orchestration of the VM failover/failback), and have NO idea how to do this from scratch.

You can do the stuff I'm talking about above with (pair of) AP7900 class managed 1U 15A PDU's and a SURT1500 15A UPS, so labbing it doesn't even need to cost a fortune. Teaches you a lot about IPMI/iDRAC/IMM/ILO as well, and your lower level scripting. Good "homework" sort of stuff.

Have to do much the same, but without some of the magic for SMB customers who have 1 host with a bunch of VM's on them that need a controlled shutdown/startup. With or without a secondary/older replication secondary host. Fun stuff!
 

K D

Well-Known Member
Dec 24, 2016
1,439
320
83
30041
You want to set:
HOST -> Manage -> Settings -> System tab -> Autostart. Enabled=Yes. Start Delay 120 seconds or whatever you like. I'd enable "wait for heartbeat". STOP ACTION should be Shutdown or Suspend, according to your wishes. Once you do THAT, now when you tell the host to reboot or shutdown, it will/should attempt to shutdown/stop any VM's it can - or DRS vacate them.
This is what I've done. Also order of host startup can be set. I've set it to AD Server, VCSA, FreeNAS. 5 minutes after i hit reboot host all 3 are shutdown in the order and restarted in the order very cleanly.
 

NetWise

Active Member
Jun 29, 2012
596
133
43
Edmonton, AB, Canada
This is what I've done. Also order of host startup can be set. I've set it to AD Server, VCSA, FreeNAS. 5 minutes after i hit reboot host all 3 are shutdown in the order and restarted in the order very cleanly.
Sorry I thought I put that there. Yes, you'd set the Startup Order for the ones you care about, and then the rest would be left as "automatic startup". It is very important, that you're not the kind of 'sloppy' person who leaves VM's powered off as a "control state" vs setting them to disabled, moving them offline, changing them to templates, etc.

Nothing drives me battier than inheriting someone else's junk, and doing a restart of the system (usually I'm called in after it went to hell, because it either failed and restarted or someone did things they shouldn't have), and having some old DC's power up or something stupid because the previous admin thought that "just powering them off" was the same as "properly decomissioning" them. Or having a "VM_restored" and "VM" both boot up together, that's always awesome too :(

Should be a PowerCLI command for that - something like "Beat-Admin -Sequence $Previous -Severity $Hard"
 
  • Like
Reactions: K D

K D

Well-Known Member
Dec 24, 2016
1,439
320
83
30041
Nothing drives me battier than inheriting someone else's junk, and doing a restart of the system (usually I'm called in after it went to hell, because it either failed and restarted or someone did things they shouldn't have), and having some old DC's power up or something stupid because the previous admin thought that "just powering them off" was the same as "properly decomissioning" them. Or having a "VM_restored" and "VM" both boot up together, that's always awesome too :(
My day job is not infra admin but I actually had this happen in one of my old jobs. After a Data Center move we were facing weird issues with one of our applications and after several hours the issue was finally identified to be due to an old VM that had an older version of a custom middleware component that had powered on automatically when the hosts were started up in the new DC. That VM was supposed to be backed up and archived and someone had just shut it down without changing the auto start settings and the host had not been rebooted for over a year.

Good Times :)
 
  • Like
Reactions: NetWise