Hi Itronin,
How has your xcpng lab been working out? I'm considering future move to either Proxmox VE or XCP-ng. I love esxi, but the vwmare compatibility means my existing hardware (specifically my 1st gen XEON Skylake-SP CPUs) will not be supported in the next major esxi 9+ release.
Hallo BennyT. - so glad your journey is continuing apace and you seem to be enjoying it (even the systems side of things) !
XCP-ng Lab has been fine for computes. hardware pass through, not as "easy" or reliable as vmware IME esp GPU's. I'm running & supporting XCP-ng at 3 sites (including my home lab) and about to flip a fourth from vmware to XCP-NG - I am NOT an expert on XCP-ng.
I have not messed with native ZFS storage on xcp-ng, but if you still have a single physical server I'd be looking at that on XEN or Prox where it is also supported.
I have not played with prox, just helped a couple of folks out with their network configurations. I am likely to stand up a prox cluster in the next 3-4 months. Prox seems a bit more bleeding edge (so is it sacrificing stability for new stuff? - IDK) and there is certainly a much larger community so its clear it has much more time in the saddle for things like pass through gpu's hba's etc. For prox, I'm looking at it to use shared GPU's with some of the 5-7 year old shared GPU hardware that has become much more inexpensive.
Although I could upgrade to CascadeLake-SP (2nd gen Intel Xeon Scalable Processors). I want to consider other hypervisors other than VMware.
I'd probably upgrade to Cascade lake - anyway...when prices are right... lol - for me I'm just starting to migrate off broadwell-ep to sky lake and -sp
If you were going to migrate to XCP-NG you might set up an XCP-ng "test server" and just see how that plays out. If it worked okay you could use that to do a full migration of the vm's as a holding area. By test server I'm thinking basically a desktop with a bit more memory, and maybe hardware raid storage, nothing horribly expensive and to be blunt might be throw-away or sell cheap when done kind of hardware. I'm not suggesting you build a second of your first build (the 846 chassis alone - shudder to think of costs today for that). Re-install OS on your original host and then migrate vm's back off the test server. don't see why that process would not also work for prox.
Did you evaluate proxmox before deciding on XCPng? Were there any specific strong points or draw backs with either hypervisor that you discovered?
No, see above.
Biggest limiter with XCP-ng was early going and trying to move fast - the reliance on Xen Orchestra for gui since each compute node is (was?) gui-less. My understanding is that limited gui is coming (if not here now) in a simpler form on every XCP-ng node install. Bootstrapping XO required a bit of thinking/planning depending on whether you used the pre-packaged crippled XO or compile your own...
One thing I think that people miss with Xen and XO though is that XO does not have to live on the cluster (or node) its managing... you can for instance, run a hypervisor on your daily driver and install XO there to boot strap the process and/or leave it there if homelab... LOL - IIRC Broadcom gave away the desktop version of vmware (fusion, workstation etc.)
XO for most things provides the gui, configurations live on the nodes. Backups though - different story. Lots of good docs on this and other XCP-ng topics (including by
@fohdeesha) over on the xcp-ng site. Really, their docs are very good, seem to get refreshed and while I primarily lurk on the forums - honestly I have not had a need to date to ask for help. For my needs it has been relatively simple and straightforward...
While XCP-ng does not seem to be too picky about hardware I have run into a case with some DELL NICS (Broadcom, virtual NIC hardware) that performs about 25% slower than generic intel 520/ConnectX3/4 at 10Gbe. doing some research I read that NIC family has performance issues on Linux distributions and that the drivers were really optimized for Windows server and VMware.
Observation: XCP-ng seems more api/scripting friendly than VMware. But as with everything, it takes time to learn and understand commands cli etc... so XO is much easier (IMO) to use than cli and gets you running faster.
XCP-ng - is *UID centric. I feel that can be a bit intimidating to folks when seeing those in a gui. the development effort has come a long way (again IMO) to making names easy to apply and therefore remember rather than long strings of characters. There are still scenarios where copying and pasting *UID is required. Then again in the esxi-cli you have to do a bit of that as well.
Subjectively XCP-ng feels like it might be a teeny bit slower than VMware on the same hardware. I have nothing objective to substantiate that statement. Just feels that way.
I use Veeam Backup and Replication to backup my existing VMware VMs. And Veeam says they can work with Proxmox and XCP-ng too.
Thanks,
benny
From what I've seen Veeam is nice. One site has that as their backup solution. I briefly looked at it in 2018//2019 - so long ago. But I have to say I have not played with it at all.
XCP-NG has what appears to be a very robust backup methodology baked in. I have not used it either. At all the sites I work on including my home have two baremetal TNE/C used for shared storage and that's leveraged for creating snapshots, backups, pushing storage offsite, etc. etc.
I have by and large moved away from AIO deployments even in my lab environments.
The recipe I'm using at sites tends to be 1 or 2 baremetal TNE(HA) or TNC (soon Scalable - still testing). When there are 2 then the second is a replication node which also pushes offsite (Azure CS or other cloud | third party appliance | TNC). I don't just rely on ZFS snaps : XCP-ng or VMware periodic snapshots, flat file storage in NFS shares, and DB exports, I do rely on ZFS for data integrity and transport as much as possible.
I realize I wrote a bit but probably conveyed much less. sorry 'bout that.
