Recommended Virtualization Solution?

[Kevin Maschke]

Hi!

We just got two new DELL servers in the office with the following specs:

- 2TB Disk Space (using RAID 5)
- 192GB RAM
- 2x 8 code Intel CPUs each (32 cores each)

And we're now discussing which virtualisation solution would be the best to host multiple different environments (a kubernetes environment, a dev environment, staging, and some single machines, etc) spread/shared on both hosts.

Some solutions are

- ESXi
- Proxmox
- oVirt
- Xen
- KVM

I thought about ESXi because it's with what I have most experience with, it doesn't use much resources in my opinion and doesn't run on top of any OS. Some colleagues say Xen would be good and others say KVM is the best because it runs directly on the CentOS Kernel.

What do you guys think?

Regards,
Kevin.

 

[nk215]

It really depends on what you want to do with GPU that differentiate one hypervisor vs another.

I looked at a few options not too long ago. At the end, I picked ESXi. The strongest advocate for ESXi is the fact that it can do hardware assisted software GPU virtualization well (vSGA with a supported Quadro card). A step above that is vGPU with Grid cards. The worst part about ESXi is its way of limiting hardware solution with drivers. For example Quadro 4000 vSGA works with ESXi 5.5 but removed from support list for newest ESXi versions. ESXi also continue to restrict vDGA (PCI pass-through) to a certain GPU cards.

Others hypervision has little to no limitation on PCI pass-through, but they do not support share GPU well.

Other than that, it comes down to experience and license costs.

 

[Kevin Maschke]

It really depends on what you want to do with GPU that differentiate one hypervisor vs another.

I looked at a few options not too long ago. At the end, I picked ESXi. The strongest advocate for ESXi is the fact that it can do hardware assisted software GPU virtualization well (vSGA with a supported Quadro card). A step above that is vGPU with Grid cards. The worst part about ESXi is its way of limiting hardware solution with drivers. For example Quadro 4000 vSGA works with ESXi 5.5 but removed from support list for newest ESXi versions. ESXi also continue to restrict vDGA (PCI pass-through) to a certain GPU cards.

Others hypervision has little to no limitation on PCI pass-through, but they do not support share GPU well.

Other than that, it comes down to experience and license costs.

Thanks for the response. In our case we do not use any GPU. Non of our servers uses any type of graphical interface apart from web interfaces.

As per license costs, if possible we're looking for Open Source or free license.

 

[markarr]

So you can run hyper-v core for free (windows vms have to have license), or you can do proxmox route.

 

[Marsh]

kubernetes environment, a dev environment, staging, and some single machines

Are these Linux based VM?
If VM OS is Linux , use Proxmox

AND

What is your budget?
If you have lots dollars , use vSphere ( you are already have experience ) , Windows server Hyper-V.
If you have no budget , push yourself outside of your comfort zone, go for free or cheap solutions.

 

[Kevin Maschke]

Are these Linux based VM?
If VM OS is Linux , use Proxmox

AND

What is your budget?
If you have lots dollars , use vSphere ( you are already have experience ) , Windows server Hyper-V.
If you have no budget , push yourself outside of your comfort zone, go for free or cheap solutions.

Hi Marsh,

All hosts and VMs are/will be Linux based.
Budget for software would be none, unless there's no other option. That's why I personally would also go for Free/Open Source solutions.

 

[Blinky 42]

We use a lot of Xen (with virsh on Centos) for work, mostly because we started years ago and it was the best tool for the job with our budget, but is also had more predictable resource usage and networking performance. The big limiting factor for some use cases is no (or poor) windows support and you can't oversubscribe physical memory if you have a host with a lot of lightly used VMs. If starting from scratch today I would do another round of performance testing with kvm and xen to choose.

I would also seriously consider openstack for some of our environments where we now have many vm servers and the potential to do larger shared storage setups with high performance networking at the site.
I do use ESXi in production but really only for windows support (as a heavy user of VMware Workstation since 1.0, familiar with much of it and like the just works well aspect).

 

[realtomatoes]

Hi Marsh,

All hosts and VMs are/will be Linux based.
Budget for software would be none, unless there's no other option. That's why I personally would also go for Free/Open Source solutions.

if this isn't for production, you can still go vmware if you subscribe to VMUG Advantage. they got EvalExperience that provide 1 year licenses to most vmware software.

 

[Marsh]

$200 per year , 5 years = $1000.
No thanks.

Disclaimer , I been retired over 20 years , no need to keep current with vSphere skill set.

 

[Rand__]

If you're looking long term - its $460 for 3 years if you pre-pay and use the discount code - but does not change your point I guess

 

[Chris Web]

New here to the forums, but thought I'd chime in. Your going to get a lot of people with different answers, but does cost become a factor? Also do you HA? VMware gets expensive if you want HA features, which Xen has for free. I also don't like the vsphere appliance, its resource heavy. I used to run a few Xen servers and they were my favorite. I also use Hyper-v for ease of use in Windows, but its linux support is not great. Vmware has the most support, but again, resource heavy when using vsphere and advanced features, and most expensive.

I've heard good things about Proxmox and many people are switching to KVM, but haven't had much experience with either.

 

[vl1969]

if I am reading your post you plan for HA setup? or you just want to do VM on 2 independent hosts?

as per other posts and your replies here are some thoughts.

but first
- ESXi , Xen / XenServer and - KVM are the hypervisors

- oVirt is not a hypervisor. it is a hypervisor manager interface for KVM. similar to Proxmox.
yes Proxmox is not a hypervisor either, it is just a set of utilities and GUI to manage KVM.

That said, if you want totally free setup, go with Xen/XenServer
as far as I know this is the only Hypervisor that is completely opensource and free to use in production setting.
yes you can buy support for it but if you can learn how to manage it on your own it is absolutely free.

word of caution though, this is not a product for fainthearted. a simple setups are fine but anything more involved will require a lot of time and help where ever you can find it. lots and lots of CLI scripts etc...

another totally free setup is KVM(QEMU.)
it is available in almost every Linux distro out there.
it works, obviously , as it is used by Proxmox and oVirt.
the only issue I see with KVM is that if you want to manage/administer pure KVM setup, you either stuck with CLI on a bare server install, or need a third party manager/GU like oVirt, Kimchi,virt-manager Cloudmin etc.
some of this you need to run on separate management workstation so setup is similar to VMWare iESX.
CLI / virsh available free on any distro supporting KVM.
Kimchi /Cloudmin works on most distros , deb and rpm based
virt-manager also available on most mainstream distros.
ovirt is an add one of the bunch as it only available on CentOs/RedHat but you can add KVM servers running on different distros and manage them from it. but main server needs to be CentOS/RedHat based.


next step up from above is Proxmox.
it is somewhat free, you can setup and run it with no subscription at all, providing you can deal with a Nag-screen (can be removed but...) and the fact that no-subscription repos are not stable (by their own claims) maybe somewhat troubling for you.
but other than that it is a very capable product, subscription or not, and it works.
also, with Proxmox you get a nice clustering and HA even without shared storage (SAN/NAS etc. )
it has support for distributed storage and Ceph (version 4.3 and up have best support for Ceph)

Next choice after Proxmox would be oVirt. available in free and Paid version, just like Proxmox it is a set of utilities and GUI to manage KVM. it is good and all, but very difficult and convoluted to setup.
similar to VMWare it needs a dedicated computer workstation for management. it is possible to run management module on the primary server as VM but that adds a lot more problems with setup.
install and setup is very involved and confusing process using CLI only.
if you know Linux, CentOS/redhat specifically you might be OK

VMWare iESX is a good choice for most uses, providing your setup is simple and the free webinterface can do everything you need for management . if not you are out of luck here. also iESX is very picky about the hardware it will run on.

 

[gigatexal]

Hi!

We just got two new DELL servers in the office with the following specs:

- 2TB Disk Space (using RAID 5)
- 192GB RAM
- 2x 8 code Intel CPUs each (32 cores each)

And we're now discussing which virtualisation solution would be the best to host multiple different environments (a kubernetes environment, a dev environment, staging, and some single machines, etc) spread/shared on both hosts.

Some solutions are

- ESXi
- Proxmox
- oVirt
- Xen
- KVM

I thought about ESXi because it's with what I have most experience with, it doesn't use much resources in my opinion and doesn't run on top of any OS. Some colleagues say Xen would be good and others say KVM is the best because it runs directly on the CentOS Kernel.

What do you guys think?

Regards,
Kevin.

The least turnkey but most flexible solution I think is KVM. That’s what I would use. Between virsh and libvirt and qemu and say virt-manager as a front end you can’t go wrong.

 

[Connorise]

That's all about the money/budget which you are ready to spend on this project.

If we are talking about the good amount of dollars I would recommend ESXi either Windows server Hyper-V; therefore you should be comfortable with ESXi ( you already obtain knowledge concerning this product) and Hyper-V is easy by design, few days of googling should set up your infrastructure.

If you are trying to stick with budget-friendly solutions, I would vote for Proxmox, or for KVM combined with some CentOS.

 

[K D]

If you're looking long term - its $460 for 3 years if you pre-pay and use the discount code - but does not change your point I guess

Is there a discount code still available? I couldn't find any.

 

[Rand__]

VMUG Discount Codes - $40 off! • r/homelab

ADVVMWORLD is gone, the others still work (just tested). Just one can be applied, each were 10% off

 

[Hank C]

VMUG Discount Codes - $40 off! • r/homelab

ADVVMWORLD is gone, the others still work (just tested). Just one can be applied, each were 10% off

wonder how they stacked the coupons

 

[Rand__]

I assume they added "Only one coupon" after that thread