Principles of virtualization

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

ullbeking

Active Member
Jul 28, 2017
506
70
28
45
London
I know virtualization is a dark art. But there are principles behind it.

I am still trying to understand _properly_ how these factor interact in their complex ways to form virt hosts.

Does anybody know of a book that explains this seriously and in depth? I guess you could call it A formalized approach to capacity planning for virtualization clusters.

Any help would be great. Thanks!!
 

EffrafaxOfWug

Radioactive Member
Feb 12, 2015
1,394
511
113
Whilst complicated (especially if you're one of the people who writes the stuff), there's very little black magic voodoo about virtualisation. It's been around since the 60s with the S/360 and happening on x86 since the 90s and all of it just follows the same basic principles; applications and operating systems do stuff by sending data and instructions of what to do with that data in to the CPU with the finished results coming back out again. It takes specialised hardware and software for CPUs to be able to do this security without one process being able to snoop what's going on in other process, and a specialised OS or software (the hypervisor) to keep the processes from one segregated from all the others.

But that's all academic really, it's basically just "CPU runs several things at once" as far as most people are concerned.

How much you're able to shunt through a CPU (and all the other associated data paths) depends on pretty much every conceivable variable, so there's no one-size-fits-all approach but a general rule of thumb for most performance characteristics is that the further away from the CPU it is, the more likely it it to become a bottleneck the more VMs you have sharing that resource but it's hugely dependent on workload.

All of the major hypervisors will typically give you a range of performance monitors to keep tabs on the performance of the hypervisor itself, and of course your regular performance monitoring tools running within the VMs themselves are also a huge help. IMHO it's better to learn those inside-out so that in the event of something running slow, you can find out what's running slow in the OS and then see if it's slow in the hypervisor. For instance, you might have a workload that uses little in the way of CPU or memory but swamps the available disc IO which ruins performance for all other VMs reliant on that storage (a very common scenario).

I've got most experience with VMware; they've published a wealth of literature regarding performance monitoring in their hypervisors and have a fair number of specialised tools for pinpointing performance bottlenecks. Back when I was penniless having a homelab that was cobbled together from scrap turned out to be amazingly useful to me as it meant I was forever running to performance problems and it gave me a great incentive to dig into why since I didn't have any money to spend on upgrades. I still think practical experience from dealing with limited resources is the best way to learn this sort of stuff.
 

Rand__

Well-Known Member
Mar 6, 2014
6,634
1,767
113
I've got most experience with VMware; they've published a wealth of literature regarding performance monitoring in their hypervisors and have a fair number of specialised tools for pinpointing performance bottlenecks.
Which tool will tell me why a vm is only using 10% of a disks IO capabilities ?:)

@ullbeking sorry for the off topic q.
 

EffrafaxOfWug

Radioactive Member
Feb 12, 2015
1,394
511
113
Which tool will tell me why a vm is only using 10% of a disks IO capabilities ?:)
I'm sure we both wish there was a simple answer to that, but there usually isn't - I don't think there's ever any one tool that can tell you everything that's going on. I know you're not one of them, but there's loads of people who don't understand the difference between random or sequential IO, or read vs write vs mixed workloads, or combinations of all the above. 90% of the time in my experience though it's a constraint somewhere within the application stack that's got nothing to do with the infrastructure. Especially when you have developers that assume that storage is infinite in both capacity and speed, and that workloads will scale linearly across cores.

Incidentally I'd have a load of of great anecdotes about VMs (physicals too) supposedly using only a fraction of their available resources when no other bottlenecks were readily visible but sadly I'm not able to repeat the details. The longest lasting one stumped microsoft (who were blamed by the application vendor), VMware (who were blamed by microsoft) and the application vendor (who were blamed by microsoft), as well as dragging the vendors for the servers and the network in as well, for a three year circus of blame before I got to the bottom of it.
 
  • Like
Reactions: Rand__

Evan

Well-Known Member
Jan 6, 2016
3,346
598
113
Ok my perspective for general enterprise mixed workload.

16-20 core / 256-512gb ram (e5 v1/2)
28 core / 512gb ram (e5)
40 cores / 768g ram (scalable)

See a trend here, 16+ gb/core

In terms of over provision, 4-5x virtual to physical ratio is also a good start
Keep the virtual cores low for VM (match requirements) if you go high you have cpu ready wait issues, more low core count VM’s can schedule more efficient than small number of large ones. Especially in ESX (less bad in KVM it seems)

WTS/VDI - choose higher click speed for response desktop

Storage you can’t go wrong with fiber channel SAN, pretty much assured to work

Hyper converged etc (eg vSAN) can be complex, let’s just say expectation does not often match reality in terms of storage performance and upgrades are a complex issue sometimes.
 

EffrafaxOfWug

Radioactive Member
Feb 12, 2015
1,394
511
113
Ok my perspective for general enterprise mixed workload.

16-20 core / 256-512gb ram (e5 v1/2)
28 core / 512gb ram (e5)
40 cores / 768g ram (scalable)

See a trend here, 16+ gb/core
Isn't this entirely dependant on workload though? Yes it's a starting point but I think it's a bad assumption to start with "system has X CPUs and therefore needs X*Y GB of RAM" - I've seen countless cases where people have taken this rule of thumb and applied it religiously even when the workload hasn't warranted it. I've seen this is the physical world as well, not just virtual.

"Oh, you need to go from 16GB of RAM to 32GB of RAM to fit the dataset in to memory? Well then you'll also need to upgrade from 4 to 8 CPUs"

Which brings me on to your second point...

Keep the virtual cores low for VM (match requirements) if you go high you have cpu ready wait issues, more low core count VM’s can schedule more efficient than small number of large ones. Especially in ESX (less bad in KVM it seems)
This is completely correct and it can't be repeated enough because so many people seemingly fail to understand it - assigning more vCPUs to a VM than it needs will frequently result in poorer performance. To execute, say, four cycles on a 4 vCPU VM simultaneously, the hypervisor scheduler needs to wait for four of the physical CPUs to have a free "slot" all at the same time. The more VMs you have and the more CPUs they all have, the busier the physical CPUs become and the more and more times the vCPUs will need to wait for a free slot across however many pCPUs need to be made available. The kicker is that even if your massively overpowered VM never needs more than 4 simultaneous threads, if you've given it 16 vCPUs it still needs 16 pCPU "slots" to be available every time any of them need to run. Hence it's possible to get into a situation where you've running on, say, 50 pCPUs, your maximum workload needs 40 pCPUs-worth of processing power, but because you've allocated 120 vCPUs you can only achieve 20 pCPUs-worth of power because your hypervisor isn't able to schedule effectively because the VMs are continually blocking each other.

This "wait" time is called "CPU ready" by VMware at least, there's another brief rundown here:
CPU Ready: The Silent Hypervisor Killer - Appuals.com

It's less of a threat to performance than it used to be, generally due to the huge increase in core density we've seen over the last 10-15 years, but still an important factor to consider when gauging CPU performance. "Right Sizing" as it's called as an important component in making VM consolidations as efficient as possible but then there are plenty of people that don't care about efficiency or the increased cost.

Don't get me started on the number of arguments I've had with people who assume More CPUs == More Performance (again, not just in the virtual world either) without actually doing any performance analysis.

"We need this application to be faster! Bump the server from 16 to 32 CPUs."
"...but if you look at these performance stats you'll see it never actually uses more than 2. It looks like this bit of the code doesn't scale across cores..."
"So, if we increase to 32 CPUs it'll use 4 instead of 2 and be twice as quick! Get to it young man!"

Incidentally, this is the reason why enabling SMT on your hypervisors almost always gives a fairly drastic amount of extra CPU power "for free", simply because it doubles the amount of queues available to the hypervisor and makes stalls waiting for available slots much less likely. It's also a big reason why a lof of people have been badly hit by Intel's performance bugs.
 

Evan

Well-Known Member
Jan 6, 2016
3,346
598
113
Isn't this entirely dependant on workload though?
Totally, that’s why I said a mixed general workload, dealing with say a 10 node cluster the big boxed mentioned 40 core / 769gb, allowing 2/3 nodes or failure / maintenance would run say 250-300 or so vm’s. Ok I realize no real DB’s on that as they are on dedicated clusters usually for license reasons but you get a mix of everything , essentially small applications of every kind it just works out fine.

Was just to give an idea, as you say it’s highly work load dependent.

A heavy example of going the other way is a heavy 3D VDI workload where would only run 32 systems on one of those 40 core machines, or Virtual DB’s similar, they may be only 4 virtual cores and 24gb ram each but they tend to use those cores so you can over provision much at all.

remember it’s it’s not always about over provision at all, it’s about easy migration, hardware independence and availability. It’s not uncommon at all to run some applications almost 1:1 ratio due to load but it’s just easier to do it virtual.

I am afraid you will never find a formula for sizing other than evaluating you own workload as you go.