Three virtualization setups: best performance?

[RudeRubbish]

I want to use virtualization to separate my tasks into virtual machines, at least one Windows and one Linux. I like the idea of using ZFS to maximize IOPS of my VMs. I also like the idea of using ZFS through a Solaris derivative, on the theory that it will offer the best ZFS experience. I'd use SmartOS, but I really need PCI passthrough of GPU, USB controllers, and possibly other PCI devices at some future time. Now that Ubuntu officially supports ZFS, I'm thinking that using a stripped down Ubuntu as a KVM hypervisor sounds appealing. But if there is a good chance of real benefits to doing the slightly Rube-Goldberg setup of virtualized ZFS presented back to the hypervisor, I'm willing to put in some work to make it happen.

Anyway, I'm curious what people here think of the three schemes outlined in the attached picture.

 

[ttabbal]

If you want virtualization and ZFS, look at Proxmox. It's simpler than most all in one designs with the Hypervisor managing the ZFS as well.

 

[RudeRubbish]

If I did Proxmox, wouldn't it be pretty similar to Ubuntu + KVM + ZFS? My concerns about Proxmox would be the same as with any other Linux.. that the ZFS might not be as refined.

 

[ttabbal]

It's the same code for all the OpenZFS implementations. It's been stable for a lot of people for a couple years now. The tools all come from Solaris like all the other implementations. I can't see any difference in "refinement", performance, commands, compatibility... I've even moved pools between Solaris, BSD, and Linux a few times without issues.

The only big difference I've seen is that "zfs share" commands are not as well supported outside of Solaris distros.

 

[RudeRubbish]

So, the fact that ZFS is part of the kernel for Solaris-derived OSs but not Linux doesn't make any difference to performance or stability? I also think I read somewhere (although I can't find it now) that suggested that the speed benefits of mirroring and striping differ between ZFS implementations.

 

[ttabbal]

Who told you ZFS isn't in kernel space in Linux? It absolutely is. It's not compiled in by default in most distributions, but if you add it, the system builds it for you.

There is some debate about distribution of compiled modules/kernels with ZFS. I don't personally see a problem, but some do. Proxmox is about the only one I know of that comes with it ready to go and supports root filesystems on ZFS.

I haven't noticed a significant speed difference on any platform. Performance of raidz and mirror sets seems about the same, though I don't do much benchmarking of it directly.

To be clear, I have nothing against Solaris based systems. But if you are wanting to do more bleeding edge virtualization, or need better hardware support, Linux is a good option.

 

[kendrick]

i have had zfs onlinux for 5 or 6 years. I have had multiple power outages and drive failures. both the solaris based vm and the linux based one have fared just as good as the next. no noticable speed difference except for the method of raid. raid z1 vs mirror. i can max the disk system out before my network saturates. I only have a couple drives in each so its not scaled massive enough to make a big speed difference. It does how ever run as well or better than an internal drive speed wise.

I have the 3 drives passed through esx 5 to the solaris vm and then nfs the shares back out to esx to store the rest of the vm's on. occasionally i have to tell it where the nfs share is again due to things not booting up in time/right order.