Experience with 2 node Proxmox VE 5.1?

[Jeggs101]

I have about 20 VM's that use between 2-16GB of RAM each and not too much CPU. I was thinking of using a 2 node cluster with Proxmox VE 5.1. I then saw they took away 2 node HA cluster support in 4.0 Two-Node High Availability Cluster - Proxmox VE

All I really want is to be able to migrate VM's from host to host in the event that one has to get updated.

That says use a third node so maybe the scheme now is to use a cheap Atom node for the third node.

Everything fits in 256GB RAM and a single CPU but when I need to update it isn't helpful.

 

[Monoman]

I've had this battle for a long time. I've built many proxmox clusters in trying to find the right setup but in the end I'm at a single host setup with acceptable downtime.

2 node clusters are close... all that is missing currently is live migration with local storage. You can do it now but only via the cmd line.

I have a 3 node cluster right now, but without shared storage you cannot live migrate. I've powered down 2 of my 3 nodes as they are just not used. Waiting on said with local_storage as mentioned earlier.

If you like, I show you my setup overview.

 

[_alex]

if you have shared storage pvecm expected 1 should let you live migrate.
but a third (small or even virtual) quorum Node for sure is a good idea.

 

[Jeggs101]

You can live migrate with local storage now? It can move a disk from node 1 zpool to node 2 zpool in the process?

 

[Monoman]

You can live migrate with local storage now? It can move a disk from node 1 zpool to node 2 zpool in the process?

Yessir!!

 

[_alex]

Ah, ok ... not really HA but moving VM's around. This should work.

 

[msg7086]

With Proxmox I ended up dd local LVM volumes between nodes to clone or migrate. A bit painful but not impossible.

 

[audio catalyst]

so just be sure:

been meaning to replace my esx 2 node test setup ( with external zfs iscsi storage ) to a proxmox cluster.

this will allow live migration and ha ?

send from a mobile device, so typo's are to be expected

 

[Marsh]

this will allow live migration and ha ?

The title of this thread is "2 Nodes Proxmox cluster", answer is NO to HA and Live migration.

You will need min 3 nodes cluster to form HA cluster, although there is a hack to form a 2 nodes cluster , but no idea on possibility with HA.

 

[Monoman]

The title of this thread is "2 Nodes Proxmox cluster", answer is NO to HA and Live migration.

You will need min 3 nodes cluster to form HA cluster, although there is a hack to form a 2 nodes cluster , but no idea on possibility with HA.

This is correct. With the " pvecm expected 1" you can setup HA but it will VERY likely fail and run into a dual headed beast where both are running the VM, you get disk corruption and destroy the vm/cluster and have to basically reinstall. It's brutal. I've done lab tests to simulate it to see what all the talk was about.

There's a beta repo for arm on proxmox. You can technically install the cluster framework on rpi and have that as the quorum device. This will allow HA and live migration with shared storage.

 

[Marsh]

Regarding the 2 nodes hack, I tested it, it would work in the lab. With proper shutdown of the cluster, cluster works OK.

But pull a power cable or network cable, more than 50% chance that the cluster is not recoverable.
In other word, it was fun to experiment a 2 nodes cluster in the lab , not real world usage.

 

[Monoman]

Exactly my results as well. Sorry if it wasn't clear.

 

[audio catalyst]

so that stinks....
if there was no external storage, i guess it would be expected that you'd need 3 nodes.

there goes proxmox

send from a mobile device, so typo's are to be expected

 

[Monoman]

so that stinks....
if there was no external storage, i guess it would be expected that you'd need 3 nodes.

there goes proxmox

send from a mobile device, so typo's are to be expected

Once they add the --local_storage option to the GUI then you could get away with an unsafe 2 node. best option is to get a 3rd node via rPi or something similar

 

[Thomas H]

if you have shared storage pvecm expected 1 should let you live migrate.
but a third (small or even virtual) quorum Node for sure is a good idea.

Once they add the --local_storage option to the GUI then you could get away with an unsafe 2 node. best option is to get a 3rd node via rPi or something similar

I have not thought of or tried clustering due to additional power consumption. Can I virtualize or use RPi for nodes 2 & 3 or only third node? Is performance or functionality affected?

 

[Patrick]

@Jeggs101 - A2SDi-2C-HLN4F | Motherboards | Products - Super Micro Computer, Inc. / https://www.servethehome.com/supermicro-sys-5029a-2tn4-review-a-small-intel-atom-c3338-nas/

Costs a bit more but at least it makes managing that last node remotely easier. Low power consumption impact as well. Even if you use local storage that gives you a third node simply to ensure the cluster continues to run.

 

[Monoman]

I have not thought of or tried clustering due to additional power consumption. Can I virtualize or use RPi for nodes 2 & 3 or only third node? Is performance or functionality affected?

whats your intent? Do you want HA? Live migration? Shared storage separate from compute?

 

[Thomas H]

My main use is homelab. Clustering, HA, live migration, etc. are not critical just nice to have and play with. I am okay if I am down (on a single host) for a short period. Power saving is more important to me. The bulk of my bill is at tier 2 at $0.28 kWh.
Cost to fire up:

• two RPi is 2 * 6W * 8.76 * .28 = $29.43/yr or $2.45/mo
• two Atom C3338 low power servers is 2 * ~25W * 8.76 * .28 = $122.64/yr or $10.22/mo
• two Xeon servers (say double the power) is $245/yr or $20/mo

Per Patrick's suggestion, is the max I could live with, but then I have to cough up $500 for the server. :-/ I am set if I can use my RPi 3's.

 

[Monoman]

With a 2 node cluster, (no HA basically), here's the code I use to do a live migration from one host to another with local storage. set $vm-host to what ever server you're live migrating it to. I'll reply more later as to your specific setup.

time for vm in $(qm list | awk '{print $1}' | grep -Eo '[0-9]{1,3}'); do qm migrate $vm $vm-host --online --with-local-disks; done

 

[msg7086]

You can even plug in a cheap old laptop as the quorum node, I guess.