What is the best way to implement high capacity redundant and/or distributed storage in a cluster of miny PCs?

[kayson]

I recently picked up 4 mini PCs that I'm going to use for a proxmox HA cluster:

• i5-8500T
• 32GB RAM
• 10Gb NIC
• 1TB M.2 SSD (glusterfs or ceph for vm storage)
• 128GB M.2 SSD (boot)

My current plan is to move my bulk storage into the following zfs NAS:

• Ryzen 3 3100
• 64GB ECC RAM
• 4x 16TB SATA HDD (2x mirror vdevs)
• 2x 1TB M.2 SSD (special vdev)
• 10Gb NIC

My main concerns with this are the extra power of an additional machine, since my electricity is very expensive, and the lack of any redundancy besides the drives. With the cluster, I can handle any one or even two complete nodes going down. The advantage to this setup, though is the higher memory capacity and ECC, which I can't use at all on my mini PCs.

Then I got to thinking, I have 4 nodes and 4 SATA drives, and I could easily connect them with an external USB 3.1+ adapter. Why not distribute the NAS storage too? Then I don't have to worry about any of the NAS hardware dying, and I'ved saved the power of a whole extra machine. There's less memory available in a given node, though and no ECC.

I'm not really sure how I would get everything working together. I was thinking maybe zfs over iSCSI? I could set up one drive on each node's bare metal, and have a single NAS VM that sets up zfs on top. I could also upgrade to a total of 2x 1TB SSDs per node, using one for boot + distributed VM storage, and the other for the NAS special vdev (again over iSCSI?) but I'm concerned about saturating the 10Gb networking.

Am I on the right track? Are there better options? There's a lot of moving parts here, and I'm new to all of this distributed/HA anything, so any guidance would be greatly appreciated!

 

[Sean Ho]

For such a small cluster, stick with a single NAS and have the compute nodes mount over NFS or iSCSI. Your life will be happier.

Ceph is great, but overkill for this use case, very complex, and you will not get the performance you might be expecting. Gluster is dying out. Longhorn has come a long ways in usability, but is still a moving target, and is replication-only, no EC.

 

[kayson]

For such a small cluster, stick with a single NAS and have the compute nodes mount over NFS or iSCSI. Your life will be happier.

Ceph is great, but overkill for this use case, very complex, and you will not get the performance you might be expecting. Gluster is dying out. Longhorn has come a long ways in usability, but is still a moving target, and is replication-only, no EC.

Thanks. What about replication for the VM storage? If ceph is overkill and gluster is dying, what other options do I have?

 

[zer0sum]

I'd say go with glusterfs.

Buttt, there is a trick in Proxmox where you just name all your drives with the same name on each host and you can then do replications, and migrations etc. without any shared storage.

Host 1 - nvme1, sata1
Host 2 - nvme1, sata1
etc.

Then you set up replication between hosts, and it works just fine as it's looking for the same named storage on each node.
When you migrate it is just a diff and you will see Linux vm's move in just seconds

 

[kayson]

I'd say go with glusterfs.

Buttt, there is a trick in Proxmox where you just name all your drives with the same name on each host and you can then do replications, and migrations etc. without any shared storage.

Host 1 - nvme1, sata1
Host 2 - nvme1, sata1
etc.

Then you set up replication between hosts, and it works just fine as it's looking for the same named storage on each node.
When you migrate it is just a diff and you will see Linux vm's move in just seconds

You have to periodically do the replication/migration though, right? Like with a cron job?

 

[zer0sum]

You have to periodically do the replication/migration though, right? Like with a cron job?

You just set it to whatever you like in the dashboard for that VM