Does anyone still use XenServer? And is it any good?

AveryFreeman

consummate homelabber
Mar 17, 2017
370
45
28
41
Near Seattle
averyfreeman.com
Do you use xen server? Why?

Is it any good anymore? It seems like it's dead, I don't ever hear anything about it.

I think there was some kind of all-in-one OS released fairly recently that uses it that got a little attention, but I already forgot what it's called.

Would like to hear from people who use this (seemingly) dead HV platform just to see if I'm missing anything... (?)

Edit: XCP-ng - that's the OS I was thinking of. Anyone using that? Did you even know it uses Xen?
 

dswartz

Active Member
Jul 14, 2011
589
73
28
I last dabbled with it 5 years ago. The big pro: it was free (IIRC). The big cons: I already paid for veeam license, but not supported on this platform. More concerning, it was not tolerant of the NFS server being down when it tries to mount the datastore. e.g. you have a power failure, and XCP comes up before the NFS server, and fails leaving the NFS datastore unmounted and requiring manual intervention. Maybe it doesn't work that way anymore, but not willing to take the chance. ESXi doesn't give me that problem.
 
  • Like
Reactions: AveryFreeman

AveryFreeman

consummate homelabber
Mar 17, 2017
370
45
28
41
Near Seattle
averyfreeman.com
I last dabbled with it 5 years ago. The big pro: it was free (IIRC). The big cons: I already paid for veeam license, but not supported on this platform. More concerning, it was not tolerant of the NFS server being down when it tries to mount the datastore. e.g. you have a power failure, and XCP comes up before the NFS server, and fails leaving the NFS datastore unmounted and requiring manual intervention. Maybe it doesn't work that way anymore, but not willing to take the chance. ESXi doesn't give me that problem.
My ESXi will complain on boot if it has a datastore from a VM on that host (hyperconverged). It doesn't fail, though.

I am trying to remedy that by using multipath/MPIO (connecting two hosts to the same NFS share or ISCSI volume) and am just starting out, trying to figure out what the issue is - it's acting kind of weird. I'm going to write a new post about it.

Do you use MPIO for your nfs v4.1 shares to avoid the hang-on-boot issue?
 

AveryFreeman

consummate homelabber
Mar 17, 2017
370
45
28
41
Near Seattle
averyfreeman.com
Put your time and effort into KVM, I personally think Xen is on life support. I think even Citrix agree's with that.
I had a feeling but was still curious. I am going to keep using vmware until they finally kick me off vCenter, then KVM is next (a revisit). Hopefully by then it'll surpass vmware's performance - was def lagging behind back in Ubuntu 16.04 days as per some benchmarks inside a VM I performed.

I actually was thinking of creating my own in-memory hypervisor using either Alpine + Webmin or Ubuntu + Cockpit. It'd be fairly trivial on Alpine, as all the tooling already exists and it's very small, but I like Cockpit better and there's too many compatibility issues for Cockpit to run on Alpine without a complete re-write. Conversely, musl-based OS + openrc + Webmin would require a lot less memory than glibc-based OS + systemd + Cockpit, so it might make more sense to go that route from the outset (at least, to start with).

Then, there's all sorts of cool stuff that can be done with ostree, etc. I haven't even looked into, it's possible that might be an even better idea now that its technology has matured a bit since its release in Fedora 30-31.

In any event, I'd like it to be set up so you can just burn an OS image to a USB flash drive, the whole thing loads into memory, and changes are committed to RAM disk, written to config file to be committed for next boot, that way you can create new hosts by flashing another USB key with the image and committing an exported config (or starting over from scratch). ESXi has me kind of spoiled in that way, I'm surprised I haven't seen another vendor create the same system.
 

dswartz

Active Member
Jul 14, 2011
589
73
28
My ESXi will complain on boot if it has a datastore from a VM on that host (hyperconverged). It doesn't fail, though.

I am trying to remedy that by using multipath/MPIO (connecting two hosts to the same NFS share or ISCSI volume) and am just starting out, trying to figure out what the issue is - it's acting kind of weird. I'm going to write a new post about it.

Do you use MPIO for your nfs v4.1 shares to avoid the hang-on-boot issue?
Hmmm, no I don't. I no longer use hyper-converged, but the issue is the same even with physical storage. e.g. with the power down, when it comes back, no guarantee NFS is up before ESXi. That said, I would see complaints, but ESXi wouldn't start anything until the datastore re-appears. I vaguely recall some long amount of time where it will fail. How does more than one path help if the NFS server is down?
 

AveryFreeman

consummate homelabber
Mar 17, 2017
370
45
28
41
Near Seattle
averyfreeman.com
Hmmm, no I don't. I no longer use hyper-converged, but the issue is the same even with physical storage. e.g. with the power down, when it comes back, no guarantee NFS is up before ESXi. That said, I would see complaints, but ESXi wouldn't start anything until the datastore re-appears. I vaguely recall some long amount of time where it will fail. How does more than one path help if the NFS server is down?
Lol, I misunderstood the VMware docs and was thinking it could access more than one server at once via NFS

It looks like it's possible with iscsi though: Linux Multipath - Can a Host connect to two different SANs?
 

AveryFreeman

consummate homelabber
Mar 17, 2017
370
45
28
41
Near Seattle
averyfreeman.com
How does more than one path help if the NFS server is down?
I was thinking you could set up two separate NFS servers in a redundant multipath setup like a SAS multipath setup, but I was wrong. Should have known it was too good to be true. The thing I was thinking would do that was really just link aggregation for increasing speed with multiple NICs to the same NAS.

I DID see a serverfault comment on how to do multipath iscsi with two different servers, but it looks very unsupported. I might still try something like it, though


It looks something like the HA strategy employed by setups like Proxmox and vSphere that manage replication and a "switch-over" policy based on host outage / heartbeat loss. I think there's a bunch of open source examples to draw from - CARP, HAST, corosync + pve-zsync & other related tools on Proxmox, the old OmniOS/Solaris granddaddy I'm forgetting the name of, etc.

Opensuse has a front-end for these sorts of tools called Hawk2 I'm going to experiment with to see if I can't rig something up, since I'm running my zpools on Leap ATM (it's a nice, stable OS with lots of resources and "training wheels" type applications). I'd rather have the two accessible simultaneously, but I think that's a fantasy without a clusterFS.
 
Last edited:

dswartz

Active Member
Jul 14, 2011
589
73
28
Yeah, problem is with NFS as the serving protocol, you need some kind of cluster-aware FS to truly get HA (or have a dual-head setup like the ewwhite setup - I used that for awhile but stopped because I had no real need for it, and you need linux as opposed to a solarish ZFS host - which I've found to be more reliable and more performant.) A near-zero replication method work, but I don't see how that can be seamless - the vmware guests would need to be restarted, else you'd have almost certain FS corruption on the guests, no?
 

AveryFreeman

consummate homelabber
Mar 17, 2017
370
45
28
41
Near Seattle
averyfreeman.com
Yeah, problem is with NFS as the serving protocol, you need some kind of cluster-aware FS to truly get HA (or have a dual-head setup like the ewwhite setup - I used that for awhile but stopped because I had no real need for it, and you need linux as opposed to a solarish ZFS host - which I've found to be more reliable and more performant.)
I'm not familiar with the "ewwhite" setup, but I did see this post from a about 6 years ago detailing using OmniOS for HA ISCSI: ZFS storage with OmniOS and iSCSI

TL;DR I've been using Linux lately.

I started out with Ubuntu because its been very favorable to ZFS, but actually switched to OpenSUSE Leap 15.3 recently because of some stability issues I was having with Ubuntu mounting my pools on boot. Even though ZFS isn't packaged in any of the official maintainer repos, I did find a community version that seems really promising: openSUSE Build Service

That, and I wanted something super easy to set up iSCSI targets with. targetcli is OK, but I keep having problems getting the datastores to format in vSphere (which I'm pretty sure now are an issue in vSphere and not with the iSCSI config). There's not much easier to deal with in Linux land than YAST.

I think this is what I was trying to think of, "the stmf-ha project": GitHub - skiselkov/stmf-ha: COMSTAR High-Availability tools for Illumos distros -- searching for stmf-ha brings up a few other sources / discussions, too, but my guess is like all other Illumos-related projects, it's probs pretty niche.

I was pretty wedded to OmniOS because of running an AD environment and the OpenSolaris's kernel SMB, but because of all the other advances in Linux (e.g. pvrdma, containerd, etc.) I gave using Samba another shot after several years, and it seems stable enough for basic file sharing / permissions. It's no kernel SMB, but I can deal with it for backups. I've even been able to publish the shares to AD and edit some very limited share permissions in Windows using MMC, which in the past was a pipe dream. I'm using Samba 4.14+

Right now the main issue I'm running into is getting my iSCSI target to be used by vSphere. If I use my VM Network it works fine, but if I try and use it with the distributed network I set up specifically for storage providers/consumers, I get these weird messages in vmkernel.log when it fails to make a datastore. I know it's a configuration issue in vsphere because I've been able to get iSCSI to work with COMSTAR, a couple linux distros, etc. fine as long as I don't use the vDS.

A near-zero replication method work, but I don't see how that can be seamless - the vmware guests would need to be restarted, else you'd have almost certain FS corruption on the guests, no?
Yeah, I'm really not sure how vsphere or proxmox do it, but in the case of proxmox, I think a lot of their repos are open source, so we could check it out (?)
 
Last edited:

dswartz

Active Member
Jul 14, 2011
589
73
28
Home · ewwhite/zfs-ha Wiki is the site i was referencing. vsphere and proxmox both assume shared data, I believe. AFAIK, there is no way to do zero-downtime HA storage without some kind of shared storage like this. The ewwhite site uses dual-head HBA/JBOD so that host #2 can fence host #1, forcibly import the pool, and export the NFS share(s). The only solarish HA storage I've seen is a commercial (and very $$$) product, and a very outdated (and likely no longer available) system using heartbeat.
 
Last edited:
  • Like
Reactions: AveryFreeman

AveryFreeman

consummate homelabber
Mar 17, 2017
370
45
28
41
Near Seattle
averyfreeman.com
Home · ewwhite/zfs-ha Wiki is the site i was referencing. vsphere and proxmox both assume shared data, I believe. AFAIK, there is no way to do zero-downtime HA storage without some kind of shared storage like this. The ewwhite site uses dual-head HBA/JBOD so that host #2 can fence host #1, forcibly import the pool, and export the NFS share(s). The only solarish HA storage I've seen is a commercial (and very $$$) product, and a very outdated (and likely no longer available) system using heartbeat.
Yeah totes, that's the kind of thing Gea recommended for OmniOS in a PDF docs he has on the napp-it.com site. <shrugs> Even the fancy vSphere "regular" HA says about a minute downtime or more for the nodes to replicate the VM in its last known state and spin it up on the survivor node. Somehow the dual path HDDs never really interested me that much, probably when I did a cursory search for them on eBay and saw how expensive they are.

There's "Fault Tolerant" which runs two copies of the same thing simultaneously, I think that's kind of like CARP or HAST. But none of that deals with shared storage. (wait, nvm HAST IS shared storage, my bad. But who wants to deal with HAST).

So in vSphere, it's stuff like vSAN, starwind vSAN, etc. you can do the old OCFS2, GFS, AFS, lizard, gluster, drbd, blah, buzzword, acronym. If I really wanted to be wedded to ZFS I saw a how-to on putting ZFS on DRBD, but I did some trials with EXT4 on it a while back and found it irritating to set up, immediately had split brain issues, and lost interest.

I think to keep things simple and power-conscious I'm just going to run two nodes and do zfs replications daily. If it goes down, it goes down, I'm not going to worry about it until I put together a kubernetes cluster I'm working on specifically for experimenting with cluster stuff so my "real" equipment I actually use is completely left out of it.

That's a fun project, I bought a stack of Dell 7050 micro motherboards for like $50 last year, and have slowly been gathering CPUs and memory for each one, duct taping and and stapling them together at a snails pace (twine, elmer's glue, a little glitter-covered macaroni threaded along with colored marshmallows).

In case you're interested in building a mini-cluster, the Dell Micros happen to fit perfectly sideways inside a 4U rack using just the bottom of the case to secure the motherboard (there's no CPU retention otherwise), I got a 3U grate for $20 from sweetwater to attach 3 120mm fans to which should deal with their awful stock cooling issues nicely, was just going to stick the case bottoms + mobos in there with some velcro top + bottom for the time being.

Some of the models have vPro so you can do AMT-based IPMI stuff with them, and if you're REALLY lucky you can find the 3-retention-screw 12v fan header ones that can accept REAL non-T version processors (those are bad af). Hint: their typical 5v header on the 4-screw heatsink model is what I am working on circumventing with my 3x 120mm fan plate, if you get my drift.

Interesting, I hadn't seen your message, but I randomly came across this post on ServerFault: Just installed LSI 9211; no drives showing up to Linux

saying ewwhite had helped them with their SAS controller User ewwhite

Re: fencing, I just got this crazy UPS thing running due to an electrician wiring me a new outlet and I haven't even hooked it up yet. It's an APC smx3000rmlv2u I got for $150 on offerup. Dude was practically begging me to take the enormous thing tf out of his place. Has a network card and everything. I've had it around for a year and a half and haven't even used it because it required a 30A socket (the kind of thing for plugging in a dryer or an electric stove).

I should probably try it out, I was reading about it, apparently it supports hardware fencing,, which I wasn't even aware was a thing until I was poking through the Proxmox docs and it said it required "hardware fencing" with a UPS for "real" HA - written about 6 years ago, so not sure if that's true anymore...

Damn, I am rambling, I should probably go to bed but I'm trying to fumble my way through my first iSER setup in LIO/targetcli to connect it to vSphere/vCenter for datastores... I've got a MacOS VM I was running locally on an NVMe I took an image of in Macrium Reflect and restored onto an iSCSI-shared zvol, now trying to get moar powar and turn it up to 11. Fun times.
 
Last edited: