Hello all,
I live in a relatively modest flat in central London, so space is at a premium and I'm trying to keep my infrastructure power consumption down for both cost and environmental reasons. With Covid my work wasn't able to send our lab techs into our own labs for more than a year, so I built up a small environment at home to support my research, and it's been so useful that I plan to continue even as folks start to come back into the facilities.
In my little farm have a group of six servers :
HPE Microserver Gen10 Plus - i3-9100F, 32GB RAM, 4x 8TB in ZFS + 1TB NVME
Lenovo M720q Tiny - i3 9100T, 16GB RAM, 256GB NVME with 4 port NIC which runs Opnsense or Sophos XG depending on my needs
Lenovo M720q Tiny - i5 8500T, 32GB RAM, 1TB NVME, 1TB SSD
Lenovo M920q Tiny - i9 9900T, 64GB RAM, 1TB NVME, 1TB SSD
Lenovo M920q Tiny - i7 8700T, 64GB RAM, 2TB NVME
Lenovo M75-1q Tiny - Ryzen 3400GE, 32GB RAM, 1TB NVME, 1TB SSD
All have 1Gbit NIC ports. I run a Brocade 24 port ICX-7150 switch. I turn off two Tinys when they're not in use and thus in normal running mode the whole stack consumes 150-160W, under full load this can double.
Up until November I ran Proxmox across the estate, then two Debian VMs running Docker for my home environment (~25 containers). Then a total of 9 K3s VMs for k8s tasks with Longhorn handling storage on the SSDs and another 3-5 Debian VMs for lab work, and a Debian LXC on the HPE providing central file serving via NFS and SMB. My aim is to minimise time spent administering what I think of as my home's basic services and lab infrastructure so I can focus on work, lab-ing, and relaxing in the evenings.
In October I migrated to VSphere/ESXi because this is better aligned to what we run at work, I was having some niggling issues with NIC freezes, and I found it simpler to administer. I don't want to run vSAN because from threads here and elsewhere it looks like the performance is a bit ... lacking ... and my main use of my storage is for home purposes, since the nodes themselves mostly have enough on board. Running a pure Debian LXC on the HPE has a bit of an admin overhead so my plan was to move that to more of an appliance since I have only occasional needs to run VMs on it, but I do need to run quite a lot of containers there so they can write to the ZFS store directly and not be hauling files across the network all the time at 1Gbps. I tried TrueNAS Scale but hated (hated) the VM and k8s environment - yes I know it's only beta - and it would have made my life harder, so for now the HPE is running OMV, which works fine but it's not well integrated into the rest of the stack.
At the moment what I need help on is whether it is better to continue as I am or change. The two main options are :
- VMware and add Tanzu, ditch Docker altogether and use k8s even for the home containers. Perhaps continue to serve storage by NFS, perhaps using Starwinds if it supports ZFS? Storage on the nodes for k8s do I keep as Longhorn, but do I have to reformat the underlying drive as VMFS first? Will this work?
- Harvester HCI, which is in beta but seems functional from some quick experimenting with it, and pulls k8s, Longhorn, Kubevirt, and Rancher together into a single entity that looks very tidy and cohesive to administer. I am not a k8s guru either but I can generally get the job done eventually. It's not as helpful in a work context but I'd trade that against less admin overhead.
For my home stuff Docker really is a better choice than k8s, since it's well supported online for my uses with plenty of guides to help me out as needed; but do I really want to keep running two container environments? k8s would also simplify failover when I need to put a node into maintenance, but there's no doubt it's more effort for each service I need to stand up despite the efforts of k8s At Home's Helm charts etc.
Does anyone have any views on the ease of use or otherwise of VSphere + Tanzu? Any thoughts on the storage in this scenario? I am still new to VMware so I don't know the options.
Anyone have any experience of Harvester?
Any better ideas?
Many thanks in advance.
I live in a relatively modest flat in central London, so space is at a premium and I'm trying to keep my infrastructure power consumption down for both cost and environmental reasons. With Covid my work wasn't able to send our lab techs into our own labs for more than a year, so I built up a small environment at home to support my research, and it's been so useful that I plan to continue even as folks start to come back into the facilities.
In my little farm have a group of six servers :
HPE Microserver Gen10 Plus - i3-9100F, 32GB RAM, 4x 8TB in ZFS + 1TB NVME
Lenovo M720q Tiny - i3 9100T, 16GB RAM, 256GB NVME with 4 port NIC which runs Opnsense or Sophos XG depending on my needs
Lenovo M720q Tiny - i5 8500T, 32GB RAM, 1TB NVME, 1TB SSD
Lenovo M920q Tiny - i9 9900T, 64GB RAM, 1TB NVME, 1TB SSD
Lenovo M920q Tiny - i7 8700T, 64GB RAM, 2TB NVME
Lenovo M75-1q Tiny - Ryzen 3400GE, 32GB RAM, 1TB NVME, 1TB SSD
All have 1Gbit NIC ports. I run a Brocade 24 port ICX-7150 switch. I turn off two Tinys when they're not in use and thus in normal running mode the whole stack consumes 150-160W, under full load this can double.
Up until November I ran Proxmox across the estate, then two Debian VMs running Docker for my home environment (~25 containers). Then a total of 9 K3s VMs for k8s tasks with Longhorn handling storage on the SSDs and another 3-5 Debian VMs for lab work, and a Debian LXC on the HPE providing central file serving via NFS and SMB. My aim is to minimise time spent administering what I think of as my home's basic services and lab infrastructure so I can focus on work, lab-ing, and relaxing in the evenings.
In October I migrated to VSphere/ESXi because this is better aligned to what we run at work, I was having some niggling issues with NIC freezes, and I found it simpler to administer. I don't want to run vSAN because from threads here and elsewhere it looks like the performance is a bit ... lacking ... and my main use of my storage is for home purposes, since the nodes themselves mostly have enough on board. Running a pure Debian LXC on the HPE has a bit of an admin overhead so my plan was to move that to more of an appliance since I have only occasional needs to run VMs on it, but I do need to run quite a lot of containers there so they can write to the ZFS store directly and not be hauling files across the network all the time at 1Gbps. I tried TrueNAS Scale but hated (hated) the VM and k8s environment - yes I know it's only beta - and it would have made my life harder, so for now the HPE is running OMV, which works fine but it's not well integrated into the rest of the stack.
At the moment what I need help on is whether it is better to continue as I am or change. The two main options are :
- VMware and add Tanzu, ditch Docker altogether and use k8s even for the home containers. Perhaps continue to serve storage by NFS, perhaps using Starwinds if it supports ZFS? Storage on the nodes for k8s do I keep as Longhorn, but do I have to reformat the underlying drive as VMFS first? Will this work?
- Harvester HCI, which is in beta but seems functional from some quick experimenting with it, and pulls k8s, Longhorn, Kubevirt, and Rancher together into a single entity that looks very tidy and cohesive to administer. I am not a k8s guru either but I can generally get the job done eventually. It's not as helpful in a work context but I'd trade that against less admin overhead.
For my home stuff Docker really is a better choice than k8s, since it's well supported online for my uses with plenty of guides to help me out as needed; but do I really want to keep running two container environments? k8s would also simplify failover when I need to put a node into maintenance, but there's no doubt it's more effort for each service I need to stand up despite the efforts of k8s At Home's Helm charts etc.
Does anyone have any views on the ease of use or otherwise of VSphere + Tanzu? Any thoughts on the storage in this scenario? I am still new to VMware so I don't know the options.
Anyone have any experience of Harvester?
Any better ideas?
Many thanks in advance.