What VM host do you use for FreeBSD guest?

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

fmatthew5876

Member
Mar 20, 2017
80
18
8
38
Hi everyone,


I've been running FreeBSD (currently 11.1) / ZFS on baremetal for nearly 10 years at home. I just upgraded the machine to a new Supermicro Xeon Scalable platform. I am now considering running something else on the host and run FreeBSD within a VM, along with other linux vms.

There are vmware platforms (I know nothing about these). I could also try running CentOS on baremetal and run FreeBSD inside of KVM.

What are your experiences with this and what would you recommend? I'm only interested in mature boring technologies, the calibur you'd really use in production in serious company. I have no interest wasting my time with bleeding edge bugs and incompatibilities.


--- TLDR ---


FreeBSD has worked well enough in the past but its getting to the point where I keep running into the limitations of FreeBSD support in software and hardware. Every time I read some performance comparison article FreeBSD is always way behind Linux. In terms of hardware support FreeBSD is and always will be way behind Linux. For hardware, linux is even better than windows nowadays, especially for older hardware.

As much I like FreeBSD, the professional world is Linux and therefore I think investing my time learning FreeBSD specific tools over Linux tools is not optimal. There are a lot of things about Linux that I don't like, but if I were running a tech company I would never take the huge risk of running the core business on a BSD platform.

The immediate use case is that currently I need to setup a linux VM to run a piece of software that only works on Linux. This software requires that I pci-pass my sound card. I tried to set it up using bhyve VM. First I ran into problems with some linux distros failing to boot. Finally when I got a debian image to work, FreeBSD host dmsg started spitting out errors on my disk controller which was seriously alarming. No way I'm going to run bhyve if it has any chance of messing with my data on the host. The fact that it can't even boot some linux images really kills my faith that bhyve is worth taking seriously in production.

I was using iohyve to try to setup and manage vms which is still beta quality software and has some issues of its own. Its pretty bad that bhyve doesn't have a sane command line interface to let you easily start and stop vms with config files. I'm not interested in spending time to write my own scripted wrapper over top of the raw bhyve command.

The 2 killer BSD features for me are ZFS and Jails.

I have no plans to migrate my storage array off of ZFS. I don't at all trust ZFS on linux, so at a minimum for a new setup I would need to continue running FreeBSD in a VM with direct pci-passthrough on my storage controller.

I really like using jails to create isolated environments for running different services. This is more about managing an environment, user accounts, networking/firewall control, and pkg dependencies more than it is about security. Although security is a nice benefit too. Its also great you can do this without the overhead of full hardware virtualization. I guess the linux alternative here is docker?

Anyway, I'd be really curious to know if anyone else is dealing with similar concerns and how your solutions played out for you?

Thanks!
 
Last edited:

StammesOpfer

Active Member
Mar 15, 2016
383
136
43
Well I know there are a ton of us running FreeNAS on top of ESXi also a number of people running pfSense on it. Nice lightweight hypervisor. Then run whatever OSs you want on top of it.

A basic ESXi setup isn't too complex to learn. Can be a little difficult to find the menu for PCIe pass through. Other than that fairly simple and decent hardware support. Not sure about the newest line of CPU's but support has to have been added by now.
 

fmatthew5876

Member
Mar 20, 2017
80
18
8
38
What is the advantage of using ESXi or a platform like that over Linux / KVM?

I don't care about clickable guis and webapps. I'd rather just be able to ssh in, edit a config file, and run a command to do administration.

I'm imagining the new regime where pretty much all services are linux containers and only 1 FreeBSD VM is using hardware virtualization for ZFS. In that world, it seems to me that using ESXi or one of those platforms just adds virtualization overhead on the linux side.

Or am I misunderstanding something?
 
Last edited:

Evan

Well-Known Member
Jan 6, 2016
3,346
598
113
I think your understanding is spot on, I Guess ESX gets a good mention is ease of use and comparability , passthrough of a HBA is Super easy.
Linux with KVM is also a great option but a bit harder to setup and manage. Should be slightly better resource wise but I don’t think you would really notice.
 

StammesOpfer

Active Member
Mar 15, 2016
383
136
43
More of a type 1 hypervisor vs type 2 hypervisor (though some argue KVM is type 1). If you plan is mostly containers with a FreeBSD VM then there is nothing wrong with Linux/KVM.
 

fmatthew5876

Member
Mar 20, 2017
80
18
8
38
Ubuntu server actually looks pretty compelling. They offer first class zfs support in their package manager and lxd (for containers) is integrated with it. That might be worthy of trying zfs on linux.

The idea of attempting to import my zpool from freebsd to linux is terrifying though..

Also all this fancy new stuff in linux is here today and maybe gone tomorrow. FreeBSD zfs and jails would probably still be working for me in 20 years with little maintenance or changes.
 
Last edited: