Can this be done? Think I can do it?

[infinitevalence]

Hey All,

So I have been tinkering and planning and researching for a while and my hope was to get a sanity check from people smarter than me, who do this either for a living or because they are even more nuts than I am.

Goal:
Build an HA Cluster to self host cloud replacement applications. Namely get myself off Google, Box, Dropbox, Amazon, ect. SAS or "freemium" services.

Gear:
3x Supermicro X8 series w/ 2 L5630, 96gb ECC Ram, Mellonox X3 10gb NIC, 2 onboard intel 1g NIC, LSI SAS HBA.
1x NetApp DS2246 Shelf w/ 17x 4tb drives
8x 12TB WD white-lable/red's
1x HPE 2910-AL48-G

Plan:
Proxmox as host on all 3
LXC for all my different applications
ZFS RAIDz 3x5+2 shared between 2 nodes
ZFS RAIDz 2x4+0 as a warm backup in node 3

Now from my research I should be able to build ZFS as a HA pool for 2 nodes: ewwhite/zfs-ha

My other hope was that by using LXC over Docker I could build containers that attach to the shared storage pool and can use directories that can grow based on use rather than have to partition off fixed storage for each application I want to run.

No, I am techie, but more so on the hardware side of things, I do not have the same software chops as many other people. This seems like its both possible and something I can build out and administrate.

I appreciate the sanity check, and any help to make it possible.

TIA,
~inf

 

[mbosma]

Now from my research I should be able to build ZFS as a HA pool for 2 nodes: ewwhite/zfs-ha

Unfortunately this would require dual port sas disks, since your disks are sata based this HA setup won't work.
You could opt to use all disks together in a ceph setup by assigning disks per node or maybe just a couple of disks to each node and use zfs replication to achieve "sorta HA".

Mounting zfs in lxc will work perfectly for your use case, I use this setup for my nextcloud, plex, radarr/sonarr and a few other services.
You can mount a directory in multiple lxc containers too if you have to.
Using docker in lxc is not officially supported but definitely possible.

 

[infinitevalence]

Thanks! I am thinking Ceph may be the way forward, I have been playing with it the last few days.

 

[mbosma]

Great! Let me know how it works out for you.

You should be able to mount cephfs in your lxc containers but you'll have to enable fuse/nesting in the options tab for the container.