Had a bit of fun today setting up a 7 node all flash (SATA, SAS and NVMe) Promxox cluster in one of the racks today.
Here is the picture I took while just doing the ugly test run (you can even see the KVM cart attached to one of the Xeon D nodes.) Ended up working on cabling after the fact but forgot to snap a picture.
For those wondering here is the Proxmox VE 4.0 cluster control:
This is now the 4th Proxmox cluster (third VE 4.0) in the rack. Here is how the previous three fared:
Being that this is now V4 of the cluster (not including previous VE 3.x generation clusters elsewhere) here are a few tips:
Note - a better way to do this would have been to use 01 as the master so it got Nodeid 01 but I was not able to in this instance.
LXC works well, but I would have rather seen Docker used to be frank.
Here is the picture I took while just doing the ugly test run (you can even see the KVM cart attached to one of the Xeon D nodes.) Ended up working on cabling after the fact but forgot to snap a picture.
For those wondering here is the Proxmox VE 4.0 cluster control:
This is now the 4th Proxmox cluster (third VE 4.0) in the rack. Here is how the previous three fared:
- The VE 3.4 4-node cluster was destroyed to swap to 4.0. Seemed easier than doing the upgrade but it was running fine.
- One Proxmox cluster died due to 4x Kingston V200 drives not handling Ceph logging in three nodes.
- Had an issue with bringing up a NIC in one of the four nodes. Ended up just making a new cluster (#4) then migrating a few VMs using ZFS send/ receive. Significantly easier than I would have expected. Probably could have kept 3 alive but decided to just use the 3 new nodes to make cluster #4 and then re-join four nodes to the new cluster.
Being that this is now V4 of the cluster (not including previous VE 3.x generation clusters elsewhere) here are a few tips:
- As you can see each of the nodes are labeled with a number -0x. That 0x is the last two digits of the IP address for the NICs. So fmt-pve-01's main external NIC would be 10.0.1.201 and the internal Ceph 10Gb NIC is 10.0.2.201. That naming convention makes troubleshooting really easy.
- Running too few nodes e.g. 3-4 was an issue when I added Ceph to the cluster, specifically when a node or two would fail. Now have extra nodes just to buffer the quorum in the event a node or two is rebooting/ failing:
Code:
Votequorum information
----------------------
Expected votes: 7
Highest expected: 7
Total votes: 7
Quorum: 4
Flags: Quorate
Membership information
----------------------
Nodeid Votes Name
0x00000004 1 10.0.1.201
0x00000003 1 10.0.1.202
0x00000005 1 10.0.1.203
0x00000006 1 10.0.1.204
0x00000002 1 10.0.1.205
0x00000007 1 10.0.1.206 (local)
0x00000001 1 10.0.1.207
- Using NVMe backed ZFS mirrors has been awesome. If something goes wrong, it is very simple to re-import those nodes.
- Having a few nodes with larger root drives has worked well. I have two nodes now that I can use to create local VMs then migrate the VMs quickly to other nodes/ storage.
- The Proxmox VE 4.0 initial installation checklist is really nice to keep handy. It has greatly reduced my provisioning time for new nodes.
- The total setup (4x Xeon D and 3x 2P E5) nodes idles in the 800w range even with a fairly expansive set of disks.
- If things go wrong in your installation, troubleshoot first, then add everything to a cluster. Adding too few nodes due to issues or adding troubled nodes just to get the node count up is not a wise idea.
LXC works well, but I would have rather seen Docker used to be frank.