vCenter and RAM

[charlespick]

I deployed my vCenter using the Tiny preset because I have 2 hosts and about 15 VMs. I've actually since edited the VM to double it's CPUs to 4 but I don't have alot of RAM in my homelab. 12GB already seems like a ton of ram to dedicate. Even still, I often get alerts like "Memory Exhaustion on vcenter01" in my vSphere. When I login to appliance management it also shows a warning on Memory. However, vSphere reports only 2.28GB is "active." What does this mean?

I am also getting a VMware vAPI Endpoint Service Health Alarm constantly. Could this be related or unrelated?

Charles

 

[DavidWJohnston]

I run vCenter 8 with 8G RAM, 2 vCPU, and thin disk - And I have 5 hosts, 42 VMs, 14 networks and 25 datastores, with guest users doing stuff, and it works perfectly fine.

I get the memory exhaustion error all the time, I think at one point I disabled the alarm. Even with the error present, there has never been any noticeable performance degradation. It does take a while to start up, but even with 16G allocated it still does.

I'd say just go ahead and run it and see what happens. If it's annoying, disable the alarm.

Also just as a tip, if your hosts are different or use non-standard CPU features (ex. ES XEONs) it's possibly to vMotion vCenter to a host and have it get stuck there. I found a way out by modifying the .vmx to trick the CPU features, but it's best to avoid this.

Good Luck!

 

[BoredSysadmin]

Also just as a tip, if your hosts are different or use non-standard CPU features (ex. ES XEONs) it's possibly to vMotion vCenter to a host and have it get stuck there. I found a way out by modifying the .vmx to trick the CPU features, but it's best to avoid this.

Why not simply enable EVC on the cluster? This is what it's for - in case different (CPU gen) hosts are in the same cluster.

Enhanced vMotion Compatibility (EVC) Explained | VMware

vSphere Enhanced vMotion Compatibility (EVC) ensures that workloads can be live migrated, using vMotion, between ESXi hosts in a cluster that are running different CPU generations.

core.vmware.com

 

[DavidWJohnston]

Why not simply enable EVC on the cluster? This is what it's for - in case different (CPU gen) hosts are in the same cluster.

Enhanced vMotion Compatibility (EVC) Explained | VMware

vSphere Enhanced vMotion Compatibility (EVC) ensures that workloads can be live migrated, using vMotion, between ESXi hosts in a cluster that are running different CPU generations.

core.vmware.com

Unfortunately some of my CPUs (ES) do not share a common standard feature set that fall under one of the EVC definitions like "Merom", "Sandy Bridge", etc. Even after loading all hosts into an EVC cluster and try to vMotion I'd get Ex: "XSAVES (save supervisor states) is unsupported."

Also with EVC it downgrades the CPU features on better hosts, so you lose the ability to make use of things like AVX2, on all VMs. This makes certain workloads very slow because you're not getting the full benefit of CPU-based features.

What I did was by trial & error figure out a method to do "per-VM EVC" by modifying the vmx files with CPU feature flags, to remove certain conflicting features, like this:

Start VM with this on lowest spec host and it can vmotion everywhere:

cpuid.80000001.ecx = "0000:0000:0000:0000:0000:0000:0000:0001"
cpuid.80000001.edx = "0010:1100:0001:0000:0000:1000:0000:0000"
cpuid.80000008.eax = "0000:0000:0000:0000:0011:0000:00--:----"
cpuid.80000008.ebx = "0000:0000:0000:0000:0000:0000:0000:0000"
cpuid.80000008.ecx = "0000:0000:0000:0000:0000:0000:0000:0000"
cpuid.7.ebx = "0000:0000:0000:0000:0000:0000:0000:0000"
cpuid.7.ecx = "0000:0000:0000:0000:0000:0000:0000:0000"
cpuid.7.edx = "0000:0000:0000:0000:0000:0000:0000:0000"

I attempted vMotion, copied out all the errors of missing features, then went on sites like this: x86 instruction listings - Wikipedia to figure out which CPUID bitmasks to apply.

It works but only for one VM startup. vCenter sees the cpuid entries and removes them, so if you shut a VM down and need to vmotion it across incompatible feature sets again, you need to login to esx host directly, remove from inventory, modify vmx on disk, re-add it to inventory, then start it. Once it's started it stays "good" and you can vmotion it everywhere.

 

[BoredSysadmin]

Unfortunately some of my CPUs (ES) do not share a common standard feature set that fall under one of the EVC definitions like "Merom", "Sandy Bridge", etc. Even after loading all hosts into an EVC cluster and try to vMotion I'd get Ex: "XSAVES (save supervisor states) is unsupported."

Also with EVC it downgrades the CPU features on better hosts, so you lose the ability to make use of things like AVX2, on all VMs. This makes certain workloads very slow because you're not getting the full benefit of CPU-based features.

What I did was by trial & error figure out a method to do "per-VM EVC" by modifying the vmx files with CPU feature flags, to remove certain conflicting features, like this:

Start VM with this on lowest spec host and it can vmotion everywhere:

cpuid.80000001.ecx = "0000:0000:0000:0000:0000:0000:0000:0001"
cpuid.80000001.edx = "0010:1100:0001:0000:0000:1000:0000:0000"
cpuid.80000008.eax = "0000:0000:0000:0000:0011:0000:00--:----"
cpuid.80000008.ebx = "0000:0000:0000:0000:0000:0000:0000:0000"
cpuid.80000008.ecx = "0000:0000:0000:0000:0000:0000:0000:0000"
cpuid.7.ebx = "0000:0000:0000:0000:0000:0000:0000:0000"
cpuid.7.ecx = "0000:0000:0000:0000:0000:0000:0000:0000"
cpuid.7.edx = "0000:0000:0000:0000:0000:0000:0000:0000"

I attempted vMotion, copied out all the errors of missing features, then went on sites like this: x86 instruction listings - Wikipedia to figure out which CPUID bitmasks to apply.

It works but only for one VM startup. vCenter sees the cpuid entries and removes them, so if you shut a VM down and need to vmotion it across incompatible feature sets again, you need to login to esx host directly, remove from inventory, modify vmx on disk, re-add it to inventory, then start it. Once it's started it stays "good" and you can vmotion it everywhere.

Correct. EVC is applied to the lowest possible CPU feature set, and you will lose access to newer CPU features. That's the price to pay if you want an easy way out. Per VM EVC is possible as you discovered, but it requires tweaking VMX files as you'd mentioned.

In general, it is good practice to start a cluster with nodes of the same CPU gen and enable EVC masking on the cluster to that CPU gen.
This would allow in the future to add newer CPU hosts without having to reboot all the VMs in the cluster.

 

[marv]

per VM EVC is supported officially. you just need VM with virtual hw version 14 or higher. its in VM configure tab under vmware EVC section

 

[DavidWJohnston]

@marv Thanks, yeah if I didn't have strange ES CPUs in my lab that's what I'd be using. Unfortunately, with my ES CPUs they have very unusual feature sets, and there is no "lowest common denominator" between all of my hosts. Even with the oldest generation, there are a couple of features missing.

If there was some way to add a custom feature set and use that for EVC that would be excellent.