Bhyve vs proxmox

[NeverDie]

I'm fairly happy with ProxMox, but according to this youtube video:

FreeBSD's Bhyve hypervisor can clone (or snapshot) a VM in less than a second--even if running on a 10 year old laptop computer--whereas ProxMox cannot (even if, as I have done, I install ZFS as the file system everywhere used by ProxMox). Is it because the FreeBSD ZFS is either more complete or more tightly integrated? Anyone here done a compare/contrast of Bhyve vs ProxMox? I'm also intrigued because FreeBSD/Bhyve has "jails" for added security of VM's.

My use case is to frequently create new clones of VM's for browsing purposes, where any malware I might encounter while browsing should hopefully stay sandboxed inside the VM. After completing a browsing session, I would simply "throw away" the entire VM and then instantly clone a new browsing VM from a "golden VM" when I next want to browse again. If it's true that Bhyve can clone a VM in under a second, then that would fit my use case much better than Proxmox, which no matter what I've tried takes much longer than that to clone a VM.

 

[Marjan]

I would say this is due to underlying file system. On TrueNAS you have ZFS file system, on your Proxmox installation I guess you don't have it.
I don't know exact technical difference but I think this is the reason.
Similar thing is on Windows. In NTFS volume Hyper-V will create snapshots much slower than on ReFS.

Hopefully someone with more knowledge will give better answer.

 

[NeverDie]

Could it be that in FreeBSD the ZFS is somehow "native", whereas on ProxMox, the ZFS isn't "native"? AFAIK, ProxMox is running OpenZFS to get its ZFS. Maybe it's a limitation of the OpenZFS implementation of ZFS as compared to FreeBSD's implementation of ZFS?

 

[Marjan]

It might be the case. I would like to know the good answer.

 

[RTM]

Sorry, but I am not going to watch the video, but I do want to say that it doesn't sound right to compare the two.
One is a full virtualization platform, complete with OS, webinterface and all, the other is just the hypervisor itself (AFAIK anyway).

As for why it is slower, perhaps (and this is just a guess) the webinterface executes commands asymetrically.
Meaning that you submit a request, this pushes a job into a queue, a worker polls the queue (delayed of course) at certain intervals for new jobs, when it finds them executes the job and returns the result into the queue (or DB), from which the browser (which may also be using polling, while waiting for completion) can eventually be notified of completion.

 

[infidelkastro]

AFAIK, ProxMox is running OpenZFS to get its ZFS. Maybe it's a limitation of the OpenZFS implementation of ZFS as compared to FreeBSD's implementation of ZFS?

Isn't Freebsd also running on OpenZFS these days? I remember reading that somewhere.

It would be fair to assume that KVM, as mature and as feature-rich as it is, probably has more cruft than bhyve. But if I was asked to guess, the "cloning lag" referenced by OP's use-case is dependent on host system configuration such as the nature of storage (flash/spinners), amount of memory installed, or even type of clone (linked/full) effected.

 

[NeverDie]

As for why it is slower, perhaps (and this is just a guess) the webinterface executes commands asymetrically.
Meaning that you submit a request, this pushes a job into a queue, a worker polls the queue (delayed of course) at certain intervals for new jobs, when it finds them executes the job and returns the result into the queue (or DB), from which the browser (which may also be using polling, while waiting for completion) can eventually be notified of completion.

No, that can't be the reason: 1 second clone on FreeBSD Bhyve (as demonstrated in the video at time index 3:10) vs. about a 2 minute clone on proxmox. That's too great a disparity to be accounted for by a polling delay.

 

[NeverDie]

Isn't Freebsd also running on OpenZFS these days? I remember reading that somewhere.

I had thought they were different, but, yes, it looks as though FreeBSD and OpenZFS were merged together for the FreeBSD 13.0 release: OpenZFS Support Merged Into Mainline FreeBSD - Phoronix

But since version 13.0 was released only recently, the comparison may have to be redone using that as the new baseline. It's conceivable It might mean that bhyve now takes 2 minutes instead of 1 second to do clone.

 

[NeverDie]

OK, I think I'm closer now to having this figured out. I created a VM using the TruNAS Core (version TrueNAS-12.0-U3.1, which is the most current version) GUI, because TruNAS Core runs whatever ZFS that FreeBSD is using and because TruNAS Core uses bhyve as its hypervisor. I did it that way, because I'm not a bHyve savant, and because the GUI makes the process not too difficult for a BHyve beginner.

Doing that, I was able to confirm that "cloning" the VM indeed appears to be very fast.

However, the impression I'm getting is that TruNAS/Byhve is not actually creating a true clone of the VM itself, but instead containerizing the VM and creating a clone from that. The same can be done in proxmox by first containerizing a VM and then creating a clone from the containerized VM. Creating a clone from a container is also very fast in proxmox (well, compared to creating a true clone), though so far not quite as fast as in TruNAS/Bhyve: around 9 seconds in Proxmox vs around 1 second from the TruNAS GUI. These are imprecise measurements using my wristwatch, but they give some approximation of relative speed. That said, there's a world of difference in usability between a 9 second cloning and a 1 second cloning, especially with regard to my use case (above). Actually, come to think of it, the 9 second measurement was taken on a slower computer, so the time difference is likely to be less on a true apples-to-apples comparison. I'll try to do a better comparison along those lines sometime soon. If there's interest, I can report back on the results.

Maybe there's a better GUI for Byhve? The TruNAS VM GUI feels rather rudimentary (especially when it comes to editing properties of a VM that I previously created). At least in that regard, I'm preferring ProxMox. That's not a strike against bhyve per se, though, but just a commentary on the state of the trunas VM GUI. Hopefully the trunas core VM GUI will continue to improve. According to the official documentation (Basic Management |), the TruNAS VM manager is exclusively a part of the TruNAS Core release. On the other hand, TruNAS SCALE is advertised as also having both containers and VMs, and a KVM hypervisor, so presumably (?) SCALE will include a different kind of VM GUI (?). It might be worth spinning up SCALE just to have a look.

 

[Jeggs101]

Sorry, that is a dumb video. I'm not going to watch. Bhyve is a less complete VM solution whereas KVM is very mature. KVM is the basis for the modern cloud, Bhyve is so bad that TrueNAS Scale is abandoning Bhyve for KVM.

Silly video.

 

[NeverDie]

Sharpening my language a bit, the TL;DR of my earlier post (above) is that TruNAS core creates "linked clones" rather than "full clones" when you press the clone button on your TruNas vm, and so I presume the same is true for BHyve.

If you want to prove this to yourself, try creating a clone, then a clone of that clone, and then a clone of that (the most recent) clone, and so on, until you hit an error condition--which in my case happened fairly quickly. Looking at the error output, you can see that the name for each clone is concatenated to one before it, with some dashes in-between, and what causes the error condition is that the clones file name becomes too long. With full clones, I can't think of any reason why there would be a name chain of this kind. Apparently with linked-clones in BHyve, on the other hand, it appears that the chain of links are literally in the file name.

Offhand, I don't see a way to create a "full clone" using the TruNAS GUI.

Regardless, according to the SmartOS wiki, BHyve has these advantages over KBM:
"

• Better tracking of, and integration wth, upstream FreeBSD
• Higher performance for CPU, and I/O operations (including disk and network I/O).
• Lower overhead, resulting in lower host CPU utilization while guests are idle."

source: Bhyve - SmartOS Docs

SmartOS gives you the choice of running VM's using either a KVM hypervisor or a Bhyve hypervisor, so maybe they draw their conclusions from that. I have no skin in the game either way, and so I'm only interested in ferreting out what is objectively true.

 

[dswartz]

No, that can't be the reason: 1 second clone on FreeBSD Bhyve (as demonstrated in the video at time index 3:10) vs. about a 2 minute clone on proxmox. That's too great a disparity to be accounted for by a polling delay.

Dunno about bhyve, but proxmox does the VMDKs as zvols. Maybe that sucks, performance-wise? Although 2 minutes sounds like a lot. Is this done at the CLI? I haven't used proxmox for quite awhile, but I'm wondering if they are not doing zfs snapshots, but something else?

 

[i386]

Bhyve is so bad that TrueNAS Scale is abandoning Bhyve for KVM

Not really.
Truenas Scale (not to be confused with core which is still using freebsd!) is based on linux (I think it was debian?), linux doesn't have bhyve.

 

[NeverDie]

^^^ This.

Also,

KVM is the basis for the modern cloud

Is that really true? If anything, I thought VMware was the basis for the modern cloud. Just my perception though. Not sure as to what the relative market shares actually are, or their evolution over time. I'd be interest to know though. Surely somewhere those percentages are rigorously tracked by somebody.

 

[i386]

Vmware is huge with on premise datacenters, but cloud providers & hyperscalers operate on a totally different level and use other tools

Edit: Microsoft uses azure hyper visor which is based on hyper-v, other cloud providers use kvm based solutions

 

[NeverDie]

Reporting back: being careful to setup everything right with the most current ProxMox on new hardware with a Ryzen 5700G and ZFS on root, it takes less than 0.1 second to create a linked clone of a virtual machine, as reported by Proxmox's task log. Size of the VM doesn't seem to matter. Same for snapshots.

So, at least for me, that settles the debate. Due to greater familiarity, and a more complete GUI than bhyve on Trunas, I've decided to stick with ProxMox.

As to why creating full-clone VM's takes so much longer than it seems like it should, I did manage to learn something new: on this fresh hardware, which uses nvme for storage, it turns out that creating a full clone of a VM is very much CPU bound rather than storage speed bound. Using htop to gain insight, I found that creating a full-clone pushes all 8 of the ryzen cores pretty nearly to maximum, even with automatic ZFS file compression and encryption both turned off. Running some tests, I found that the time taken to create a full clone is a function of both virtual disk size as well as how "full" that disk image is with the files inside it. So, for whatever reason, creating a full clone apparently involves more than just making a straight bit-for-bit copy of the VM's virtual disk image, and apparently that's why it ends up taking so long. Given this, I'm doubtful there's a workaround tthat would speed up the creation of full clones, but if anyone knows of one, or a different approach, then please do post a comment. Apparently the one thing that probably would help is using a CPU with more cores, like a Ryzen 9 5950x or a threadripper, since it' the CPU that's the chokepoint and the htop insight suggests that even more cores could be used in parallel to speed up the full-clone process.

 

[Ofloo]

Sorry, but I am not going to watch the video, but I do want to say that it doesn't sound right to compare the two.
One is a full virtualization platform, complete with OS, webinterface and all, the other is just the hypervisor itself (AFAIK anyway).

As for why it is slower, perhaps (and this is just a guess) the webinterface executes commands asymetrically.
Meaning that you submit a request, this pushes a job into a queue, a worker polls the queue (delayed of course) at certain intervals for new jobs, when it finds them executes the job and returns the result into the queue (or DB), from which the browser (which may also be using polling, while waiting for completion) can eventually be notified of completion.

FreeBSD is a complete OS don't say things you don't know nothing about. More so then Proxmox, because it's not optimized for virtualisation, it can be a desktop or a server or your vm machine and jails/containers. So I would say the comparison is pretty accurate. I would expect a OS specialized in one thing to be better at it !

Also If you would use TrueNAS as your virtualisation machine, you can use both jails/containers and bhyve. And you do get a webinterface with that and clone is just as fast on those.

Could it be that in FreeBSD the ZFS is somehow "native", whereas on ProxMox, the ZFS isn't "native"? AFAIK, ProxMox is running OpenZFS to get its ZFS. Maybe it's a limitation of the OpenZFS implementation of ZFS as compared to FreeBSD's implementation of ZFS?

Yes, you can choose to use zfs or openzfs on FreeBSD. I think FreeBSD is moving away from zfs and moving towards openzfs but at the moment both are supported.

 

[RTM]

FreeBSD is a complete OS don't say things you don't know nothing about. More so then Proxmox, because it's not optimized for virtualisation, it can be a desktop or a server or your vm machine and jails/containers. So I would say the comparison is pretty accurate. I would expect a OS specialized in one thing to be better at it !

Also If you would use TrueNAS as your virtualisation machine, you can use both jails/containers and bhyve. And you do get a webinterface with that and clone is just as fast on those.

You are reading something into what I wrote that is not there
I did not at any point suggest that FreeBSD is not a complete OS, but Bhyve obviously is not...

My point was simply that comparing Bhyve to Proxmox is like comparing apples to oranges.
A more meaningful comparison would be KVM (or Xen or Hyper-v) vs Bhyve (or as you point out, TrueNAS vs Proxmox) because that is where most of the differences lie, if you compare Proxmox to Bhyve, you would also have to factor in all the extra features you get with Proxmox and the effect they have on the underlying hypervisor.

In any case I am not saying Bhyve is bad by any means, and I am sure that there are areas where it is better than KVM.

 

[Ofloo]

My mistake then.

 

[richardm]

May I bring this dead post back to life? This is my $.02 on what's happening here -- Perhaps it will clarify the discrepancy for someone else who wanders along with this same question.

There's different ways to clone a VM:

• Make a physical copy of every storage block.
  • Straight-up data copy that leaves you with two copies of the original data.
  • This is the slowest, least sophisticated method.
  • Limited by drive throughput.

• Create a "reflink" file.
  • Generally requires a copy-on-write or redirect-on-write capable filesystem.
  • One can create what is effectively a file copy but in reality is simply pointers to the original data blocks.
  • When something changes in the "copy" (i.e. the cloned VM writes to its disk), this write is redirected to a new block and the pointer updated accordingly.
  • Fast and efficient -- the clone happens almost instantaneously and common data blocks remain shared which is like deduplication without actually having dedupe-enabled storage.
  • btrfs and Windows ReFS do this natively. XFS somehow supports reflinks without being a true copy-on-write filesystem -- I haven't looked into this. OpenZFS is probably getting this soon as people have been banging the drum for 12ish years and there's finally some movement.

• Create a "linked VM clone."
  • Somewhat similar to reflink above but with different mechanics.
  • With a reflink the filesystem does the heavy lifting but a VM linked clone is filesystem agnostic.
  • The virtualization platform effects copy-on-write behavior by snapshotting the original VM and running the clone from the snapshot.
  • Both VM's writes go into into new and separate files -- the original VM disk as a provider of common blocks becomes read-only out of necessity.

• Clone the zvol.
  • Each VM lives on its own ZFS zvol.
  • Cloning a VM is just a matter of cloning the zvol which is a fast copy-on-write operation.

Given the lack of true reflink support in OpenZFS, I suspect TrueNAS was doing either #3 or #4 or some combination thereof. Proxmox was probably doing #1.