I recently picked up 4 mini PCs that I'm going to use for a proxmox HA cluster:
Then I got to thinking, I have 4 nodes and 4 SATA drives, and I could easily connect them with an external USB 3.1+ adapter. Why not distribute the NAS storage too? Then I don't have to worry about any of the NAS hardware dying, and I'ved saved the power of a whole extra machine. There's less memory available in a given node, though and no ECC.
I'm not really sure how I would get everything working together. I was thinking maybe zfs over iSCSI? I could set up one drive on each node's bare metal, and have a single NAS VM that sets up zfs on top. I could also upgrade to a total of 2x 1TB SSDs per node, using one for boot + distributed VM storage, and the other for the NAS special vdev (again over iSCSI?) but I'm concerned about saturating the 10Gb networking.
Am I on the right track? Are there better options? There's a lot of moving parts here, and I'm new to all of this distributed/HA anything, so any guidance would be greatly appreciated!
- i5-8500T
- 32GB RAM
- 10Gb NIC
- 1TB M.2 SSD (glusterfs or ceph for vm storage)
- 128GB M.2 SSD (boot)
- Ryzen 3 3100
- 64GB ECC RAM
- 4x 16TB SATA HDD (2x mirror vdevs)
- 2x 1TB M.2 SSD (special vdev)
- 10Gb NIC
Then I got to thinking, I have 4 nodes and 4 SATA drives, and I could easily connect them with an external USB 3.1+ adapter. Why not distribute the NAS storage too? Then I don't have to worry about any of the NAS hardware dying, and I'ved saved the power of a whole extra machine. There's less memory available in a given node, though and no ECC.
I'm not really sure how I would get everything working together. I was thinking maybe zfs over iSCSI? I could set up one drive on each node's bare metal, and have a single NAS VM that sets up zfs on top. I could also upgrade to a total of 2x 1TB SSDs per node, using one for boot + distributed VM storage, and the other for the NAS special vdev (again over iSCSI?) but I'm concerned about saturating the 10Gb networking.
Am I on the right track? Are there better options? There's a lot of moving parts here, and I'm new to all of this distributed/HA anything, so any guidance would be greatly appreciated!