XCP-ng installation and setup hell

[Octopuss]

I am trying to move away from ESXi and thought I would give XCP-ng a go (I don't want Proxmox since it's not type 1 hypervisor), but the main problem seems to be having to download the Xen Orchestra management from the internet. I don't know how to configure all this stuff. Basically, I have no idea what am I doing.
With ESXi, this is easy: you download one iso, install it, and then the 2nd iso is the vCenter. You don't even need it, the server is configurable already without it.

My setup makes it even more complicated, because I have "all in one" virtualized server that runs pfSense. Basically I need to install the server first, and then I can add a VM that gets me DHCP and other network specifics. At least I have an old router with my ISP's configuration at hand, but I still fail miserably trying to configure the base server.

XCP-ng doesn't seem to be popular on this forum, so before I pointlessly get into detail, is there anyone who is educated on the subject who could help me?

 

[acp]

I am trying to move away from ESXi and thought I would give XCP-ng a go (I don't want Proxmox since it's not type 1 hypervisor), but the main problem seems to be having to download the Xen Orchestra management from the internet. I don't know how to configure all this stuff. Basically, I have no idea what am I doing.
With ESXi, this is easy: you download one iso, install it, and then the 2nd iso is the vCenter. You don't even need it, the server is configurable already without it.

My setup makes it even more complicated, because I have "all in one" virtualized server that runs pfSense. Basically I need to install the server first, and then I can add a VM that gets me DHCP and other network specifics. At least I have an old router with my ISP's configuration at hand, but I still fail miserably trying to configure the base server.

XCP-ng doesn't seem to be popular on this forum, so before I pointlessly get into detail, is there anyone who is educated on the subject who could help me?

Your desired configuration makes it difficult. To built xen orchestra you need Internet access. Which you don't appear to have. You probably need to build pfsense first, however it easier to config using xen orchestra. In the xcp-ng documentation it tells you how to configure the nics.

pfSense / OPNsense VM | XCP-ng Documentation

A guide to run pfSense in a VM.

docs.xcp-ng.org

Now building a new vm from command line is difficult. If you have access to Windows, you can use xcp-ng center. Otherwise, build xen orchestra in virtual box where you have Internet access and then connect to xcp-ng. There are scripts to do the heavy lifting, but they require Internet access.

 

[Octopuss]

Well, what I didn't try was simply connecting the backup router to the server's another NIC, that should in theory give it internet access without having to configure any extra stuff?

 

[Octopuss]

pfSense / OPNsense VM | XCP-ng Documentation

A guide to run pfSense in a VM.

docs.xcp-ng.org

Is this a mandatory step when running pfSense in XCP-ng? In ESXi I installed it like anything else.

 

[acp]

Is this a mandatory step when running pfSense in XCP-ng? In ESXi I installed it like anything else.

From the documentation:

Now is the most important step: we must disable TX checksum offload on the virtual xen interfaces of the VM. This is because network traffic between VMs in a hypervisor is not populated with a typical Ethernet checksum, since they only traverse server memory and never leave over a physical cable. The majority of operating systems know to expect this when virtualized and handle Ethernet frames with empty checksums without issue. However pf in FreeBSD does not handle them correctly and will drop them, leading to broken performance.

 

[Octopuss]

Weird. I didn't have to do that under ESXi, and everything works.

 

[acp]

Weird. I didn't have to do that under ESXi, and everything works.

It's only required for freebsd.

 

[Octopuss]

I don't understand. pfSense isn't freebsd?

 

[fohdeesha]

I don't understand. pfSense isn't freebsd?

yes is it, it's freebsd with a GUI. all *bsds have a bug in the PF packet filter when running under xen, so the checksumming blurb must be done. Also note if you need an offline deploy for XOA, you can download an XOA vm in xva format, to import to a XCP-ng host offline. You can do that via cli, or I would recommend just using the old windows client, XCP-ng center. To get the offline xva image, just click "sign up" to make a free account on the XOA website, then log into the member area and there's a big alternative download button Xen-Orchestra ✦ Web Interface for XenServer

here's the old windows management client, that you can use for VM importing, creation management etc until you get XOA online: Releases · xcp-ng/xenadmin

XCP-ng + XOA is amazing when up and running. But having your internet access (even your whole network / dhcp?) relient on your current hypervisor is going to make switching to *anything* a huge pain. As soon as you even reboot that thing you'll lose internet, dhcp etc

 

[acp]

I don't understand. pfSense isn't freebsd?

The underlying OS is freebsd.

 

[acp]

Here is a script to help you build Xen Orchestra.

XenOrchestraInstallerUpdater

I use this for my setup. Works great.

 

[Octopuss]

yes is it, it's freebsd with a GUI. all *bsds have a bug in the PF packet filter when running under xen, so the checksumming blurb must be done. Also note if you need an offline deploy for XOA, you can download an XOA vm in xva format, to import to a XCP-ng host offline. You can do that via cli, or I would recommend just using the old windows client, XCP-ng center. To get the offline xva image, just click "sign up" to make a free account on the XOA website, then log into the member area and there's a big alternative download button Xen-Orchestra ✦ Web Interface for XenServer

here's the old windows management client, that you can use for VM importing, creation management etc until you get XOA online: Releases · xcp-ng/xenadmin

XCP-ng + XOA is amazing when up and running. But having your internet access (even your whole network / dhcp?) relient on your current hypervisor is going to make switching to *anything* a huge pain. As soon as you even reboot that thing you'll lose internet, dhcp etc

Is the bug getting fixed anytime soon? How serious is it, or rather, the workaround has any negative implications?

The old client has some downsides to it then I guess?

 

[Octopuss]

Ok so I tried the old client and got nowhere, because when I thought I'd try adding a VM, I noticed the storage was ext3. Isn't that supposed to be ext4? I'd swear the installer said so. And I read ext3 was really bad.

 

[tsteine]

@Octopuss Type 1 and Type 2 hypervisor doesn't really apply that well to QEMU/KVM.

KVM is a type 1 hypervisor built into the linux kernel, QEMU is a type 2 hypervisor in user-land, but the way this is done is managing KVM virtual machines through KVM extensions built into QEMU, because thats easier and more userfriendly than using KVM directly, so Proxmox(which uses QEMU/KVM) is kind of a strange Type1/2 hybrid, and QEMU/KVM still has Type 1 level performance.
If it not being purely a Type 1 hypervisor is the reason why you don't want to use Proxmox, that would be very misguided.

 

[Octopuss]

I did some googling and it seems like I do want type 1 hypervisor for passthrough. Proxmox is supposedly problematic in this regard.

 

[tsteine]

I did some googling and it seems like I do want type 1 hypervisor for passthrough. Proxmox is supposedly problematic in this regard.

In that case, there is not really any distinction between ESXi, XCP-NG, etc.
KVM is still a type 1 hypervisor using hardware extensions for virtualization, IOMMU, etc, as I said, qemu just manages the type 1 KVM hypervisor from userland when it runs KVM workloads.
I'm not aware of any problems with passthrough on Proxmox with anything other than graphics cards(this is not unique to Proxmox either), because you need to do extra steps with firmware, etc.

If what you are passing through is a network card, or a host bus adapter for storage, it will not be a problem.

 

[Octopuss]

I don't really know. The less overhead the better, no?

 

[tsteine]

I don't really know. The less overhead the better, no?

No. No. No. Let me stop you there. KVM is not a slow hypervisor and it's not bogged down by huge overhead.

Amazon Web Services literally replaced their Xen based hypervisor (which is what XCP-NG is based on) in favor of KVM.

AWS adopts home-brewed KVM as new hypervisor • The Register

Edit: Don't misunderstand me here, I'm not saying you shouldn't run XCP-NG, you should run whatever you want and are comfortable with, I'm just saying that you shouldn't discount other hypervisors based on preconceptions that are inaccurate about that hypervisor technology.

 

[Octopuss]

I work with what I googled up, lol. That's always very conflicting.
I just want something rock stable that can do passthrough of a HBA without any problems.
What are the main differences between XCP-ng and Proxmox for a "homelab"-like environment?

 

[tsteine]

I work with what I googled up, lol. That's always very conflicting.
I just want something rock stable that can do passthrough of a HBA without any problems.
What are the main differences between XCP-ng and Proxmox for a "homelab"-like environment?

Based on those critera, I would agree that XCP-ng is the more appropriate choice, since it has a free LTS release built for stability, while with Proxmox, the stable enterprise repo is paywalled and only the upstream repo is available for free.

If you wanted a more bleeding edge homelab, I would have recommended Proxmox with the upstream repo.

Lawrence Systems on youtube has multiple videos on XCP-ng.

This would be a good place to start:
Getting Started Tutorial: Building An Open Source XCP-NG & Xen Orchestra Virtualization Lab (youtube.com)