With Xen - in all of its incarnations (Xenserver, xen hypervisor, XCP, XAPI on Debian/Ubuntu, etc):
- The hypervisor layer itself is dead stupid (on purpose). All it does is manage the scheduler and provide a path for "domains" (VMs) to send IO requests to the privileged domain - Dom0 - which handles all IO.
- You can't run any filesystem or IO subsystem on the hypervisor itself.
- There is no particular reason why you can't run ZFS or some other advanced filesystem (BTFS, whatever) on Dom0.
- However, the "packaged" versions of Xen - XenServer and XCP - only support a 32-bit Dom0 and ZFS doesn't work well on it because it because the ARC is too limited. Without a lot of work you can't just slap a 64-bit kernel onto XenSever/XCP (though it may be easier now that XenServer is all open source...it would still be a PITA).
- Nitrobase's idea - "Why not just install ZoL on CentOS, set it up using your controller and create your NFS/iSCSI shares/luns; then just install xenserver" - doesn't work exactly this way. You don't "install XenServer" onto an OS. XenServer is an entire package. What you can do is install they key packages that create the same function as XenServer, basically package XAPI and a couple of others. This actually much easier right now with Debian (Wheezy) than CentOS because of some packaging work they've done in the latest release. Its as easy as "aptitude install xen-xapi", do some simple configuration of networking, etc, and reboot. This actually install the Xen hypervisor under your system and when you reboot your OS is running as Dom0.
- Unfortunately, right now, package xen-xapi is not really stable. It is still pretty buggy. Most important bug is that it doesn't work right with the most common management interface, XenCenter, and the "open source" managers are largely inactive projects and just don't work. There is also a problem with access to the correct PV drivers for Windows guests - its a licensing issue with Citrix. But this last part may be solvable now that Citrix has open sourced the whole thing.
- This last method, what Nitrobase24 suggests, is pretty much exactly the same thing that both Danschwartz and I have tried. And abandoned, because the current state of XAPI is not quite ready. I would LOVE for this to work because I think it comes closest to the environment i want to use. Unfortunately, I don't have time to futz with it and make it work.
- So for now I'll stick with Proxmox VE w/ZoL. While it works and is stable (two MAJOR pluses!!!) I just can't stand its "clustering" environment and insistence on using a shared voting database. The shared DB idea is great for true HA environments, but it is a nightmare for simpler uses of clustering - like just being able to move a VM from machine to machine to allow maintenance.