XCP-ng installation and setup hell

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

Octopuss

Active Member
Jun 30, 2019
428
68
28
Czech republic
I am trying to move away from ESXi and thought I would give XCP-ng a go (I don't want Proxmox since it's not type 1 hypervisor), but the main problem seems to be having to download the Xen Orchestra management from the internet. I don't know how to configure all this stuff. Basically, I have no idea what am I doing.
With ESXi, this is easy: you download one iso, install it, and then the 2nd iso is the vCenter. You don't even need it, the server is configurable already without it.

My setup makes it even more complicated, because I have "all in one" virtualized server that runs pfSense. Basically I need to install the server first, and then I can add a VM that gets me DHCP and other network specifics. At least I have an old router with my ISP's configuration at hand, but I still fail miserably trying to configure the base server.

XCP-ng doesn't seem to be popular on this forum, so before I pointlessly get into detail, is there anyone who is educated on the subject who could help me?
 

acp

New Member
Mar 7, 2014
18
12
3
I am trying to move away from ESXi and thought I would give XCP-ng a go (I don't want Proxmox since it's not type 1 hypervisor), but the main problem seems to be having to download the Xen Orchestra management from the internet. I don't know how to configure all this stuff. Basically, I have no idea what am I doing.
With ESXi, this is easy: you download one iso, install it, and then the 2nd iso is the vCenter. You don't even need it, the server is configurable already without it.

My setup makes it even more complicated, because I have "all in one" virtualized server that runs pfSense. Basically I need to install the server first, and then I can add a VM that gets me DHCP and other network specifics. At least I have an old router with my ISP's configuration at hand, but I still fail miserably trying to configure the base server.

XCP-ng doesn't seem to be popular on this forum, so before I pointlessly get into detail, is there anyone who is educated on the subject who could help me?
Your desired configuration makes it difficult. To built xen orchestra you need Internet access. Which you don't appear to have. You probably need to build pfsense first, however it easier to config using xen orchestra. In the xcp-ng documentation it tells you how to configure the nics.


Now building a new vm from command line is difficult. If you have access to Windows, you can use xcp-ng center. Otherwise, build xen orchestra in virtual box where you have Internet access and then connect to xcp-ng. There are scripts to do the heavy lifting, but they require Internet access.
 
  • Like
Reactions: fohdeesha

Octopuss

Active Member
Jun 30, 2019
428
68
28
Czech republic
Well, what I didn't try was simply connecting the backup router to the server's another NIC, that should in theory give it internet access without having to configure any extra stuff?
 

acp

New Member
Mar 7, 2014
18
12
3
Is this a mandatory step when running pfSense in XCP-ng? In ESXi I installed it like anything else.
From the documentation:

Now is the most important step: we must disable TX checksum offload on the virtual xen interfaces of the VM. This is because network traffic between VMs in a hypervisor is not populated with a typical Ethernet checksum, since they only traverse server memory and never leave over a physical cable. The majority of operating systems know to expect this when virtualized and handle Ethernet frames with empty checksums without issue. However pf in FreeBSD does not handle them correctly and will drop them, leading to broken performance.
 
  • Like
Reactions: fohdeesha

fohdeesha

Kaini Industries
Nov 20, 2016
2,828
3,268
113
33
fohdeesha.com
I don't understand. pfSense isn't freebsd?
yes is it, it's freebsd with a GUI. all *bsds have a bug in the PF packet filter when running under xen, so the checksumming blurb must be done. Also note if you need an offline deploy for XOA, you can download an XOA vm in xva format, to import to a XCP-ng host offline. You can do that via cli, or I would recommend just using the old windows client, XCP-ng center. To get the offline xva image, just click "sign up" to make a free account on the XOA website, then log into the member area and there's a big alternative download button Xen-Orchestra ✦ Web Interface for XenServer

here's the old windows management client, that you can use for VM importing, creation management etc until you get XOA online: Releases · xcp-ng/xenadmin

XCP-ng + XOA is amazing when up and running. But having your internet access (even your whole network / dhcp?) relient on your current hypervisor is going to make switching to *anything* a huge pain. As soon as you even reboot that thing you'll lose internet, dhcp etc
 

Octopuss

Active Member
Jun 30, 2019
428
68
28
Czech republic
yes is it, it's freebsd with a GUI. all *bsds have a bug in the PF packet filter when running under xen, so the checksumming blurb must be done. Also note if you need an offline deploy for XOA, you can download an XOA vm in xva format, to import to a XCP-ng host offline. You can do that via cli, or I would recommend just using the old windows client, XCP-ng center. To get the offline xva image, just click "sign up" to make a free account on the XOA website, then log into the member area and there's a big alternative download button Xen-Orchestra ✦ Web Interface for XenServer

here's the old windows management client, that you can use for VM importing, creation management etc until you get XOA online: Releases · xcp-ng/xenadmin

XCP-ng + XOA is amazing when up and running. But having your internet access (even your whole network / dhcp?) relient on your current hypervisor is going to make switching to *anything* a huge pain. As soon as you even reboot that thing you'll lose internet, dhcp etc
Is the bug getting fixed anytime soon? How serious is it, or rather, the workaround has any negative implications?

The old client has some downsides to it then I guess?
 

Octopuss

Active Member
Jun 30, 2019
428
68
28
Czech republic
Ok so I tried the old client and got nowhere, because when I thought I'd try adding a VM, I noticed the storage was ext3. Isn't that supposed to be ext4? I'd swear the installer said so. And I read ext3 was really bad.
1706812880880.png
 
Last edited:

tsteine

Active Member
May 15, 2019
178
84
28
@Octopuss Type 1 and Type 2 hypervisor doesn't really apply that well to QEMU/KVM.

KVM is a type 1 hypervisor built into the linux kernel, QEMU is a type 2 hypervisor in user-land, but the way this is done is managing KVM virtual machines through KVM extensions built into QEMU, because thats easier and more userfriendly than using KVM directly, so Proxmox(which uses QEMU/KVM) is kind of a strange Type1/2 hybrid, and QEMU/KVM still has Type 1 level performance.
If it not being purely a Type 1 hypervisor is the reason why you don't want to use Proxmox, that would be very misguided.
 
Last edited:
  • Like
Reactions: Pakna

tsteine

Active Member
May 15, 2019
178
84
28
I did some googling and it seems like I do want type 1 hypervisor for passthrough. Proxmox is supposedly problematic in this regard.
In that case, there is not really any distinction between ESXi, XCP-NG, etc.
KVM is still a type 1 hypervisor using hardware extensions for virtualization, IOMMU, etc, as I said, qemu just manages the type 1 KVM hypervisor from userland when it runs KVM workloads.
I'm not aware of any problems with passthrough on Proxmox with anything other than graphics cards(this is not unique to Proxmox either), because you need to do extra steps with firmware, etc.

If what you are passing through is a network card, or a host bus adapter for storage, it will not be a problem.
 

tsteine

Active Member
May 15, 2019
178
84
28
I don't really know. The less overhead the better, no?
No. No. No. Let me stop you there. KVM is not a slow hypervisor and it's not bogged down by huge overhead.

Amazon Web Services literally replaced their Xen based hypervisor (which is what XCP-NG is based on) in favor of KVM.

AWS adopts home-brewed KVM as new hypervisor • The Register

Edit: Don't misunderstand me here, I'm not saying you shouldn't run XCP-NG, you should run whatever you want and are comfortable with, I'm just saying that you shouldn't discount other hypervisors based on preconceptions that are inaccurate about that hypervisor technology.
 
Last edited:

Octopuss

Active Member
Jun 30, 2019
428
68
28
Czech republic
I work with what I googled up, lol. That's always very conflicting.
I just want something rock stable that can do passthrough of a HBA without any problems.
What are the main differences between XCP-ng and Proxmox for a "homelab"-like environment?
 

tsteine

Active Member
May 15, 2019
178
84
28
I work with what I googled up, lol. That's always very conflicting.
I just want something rock stable that can do passthrough of a HBA without any problems.
What are the main differences between XCP-ng and Proxmox for a "homelab"-like environment?
Based on those critera, I would agree that XCP-ng is the more appropriate choice, since it has a free LTS release built for stability, while with Proxmox, the stable enterprise repo is paywalled and only the upstream repo is available for free.

If you wanted a more bleeding edge homelab, I would have recommended Proxmox with the upstream repo.

Lawrence Systems on youtube has multiple videos on XCP-ng.

This would be a good place to start:
Getting Started Tutorial: Building An Open Source XCP-NG & Xen Orchestra Virtualization Lab (youtube.com)