Build’s Name: Malina (meaning "Raspberry" in various Slavic languages)
Operating System/ Storage Platform: Plain Debian 11 plus LXD and Ceph
CPU: 6x ARM Cortex-A72
Motherboard: DeskPi Super6C
Chassis: SuperMicro SuperChassis 510T-203B
Drives: 6x WD Red SN700 (1000 GB, M.2 2280)
RAM: 6x 8GB
Add-in Cards: n/a
Power Supply: 100W DC (came with the motherboard)
Usage Profile: Low-power European Energy-Crisis-Winter homelab and basic services for the home.
"Winter Is Coming" in Europe, and with the geopolitical situation being as it is, I wanted to make sure that I could run a very basic set of homelab services on a low-power architecture such that I can have my APC surt1000xli bridge a rolling blackout, should those occur.
Encouraged by, among others, Jeff Geerling, I ordered the DeskPi Super6C, and then I paid what felt like 3 limbs, a firstborn, and a fortune, to acquire 6x Raspberry Pi CM4 8GB modules on eBay.
I was used to having Proxmox VE on my normal Intel cluster, but it was not to be: Pimox7 does exist, but it is not in a state suitable for what I expect. I had my third attempt, and then after a lot of toil my third surrender regarding Kubernetes, and so I went OG: Plain Debian 11 with Ceph RBD and CephFS, with LXD in multi-master mode. This actually works surprisingly well: I run my various small services (IPv6 routing, CA, caches), as well as a full-blown ARM64 VM (KVM, through LXD) running Debian 11 with ZFS as a fileserver.
The main bottleneck in this little cluster is bandwidth: The CM4s have just 1 lane of PCIe 2.0, and the motherboard has a GBe switch on it. I have noticed that running my full-blown InfluxDb setup on it leaves the nodes, containers and VMs fighting with Ceph over the bandwidth, and it all ends in tears. I therefore moved that service to a server that I have running in a community-based colo in Zurich.
I did not do power measurements yet, but I will at some point. At any rate, it will be lower than the 90W-a-pop idle power of my Intel cluster.
Malina sitting in its chassis.
Operating System/ Storage Platform: Plain Debian 11 plus LXD and Ceph
CPU: 6x ARM Cortex-A72
Motherboard: DeskPi Super6C
Chassis: SuperMicro SuperChassis 510T-203B
Drives: 6x WD Red SN700 (1000 GB, M.2 2280)
RAM: 6x 8GB
Add-in Cards: n/a
Power Supply: 100W DC (came with the motherboard)
Usage Profile: Low-power European Energy-Crisis-Winter homelab and basic services for the home.
"Winter Is Coming" in Europe, and with the geopolitical situation being as it is, I wanted to make sure that I could run a very basic set of homelab services on a low-power architecture such that I can have my APC surt1000xli bridge a rolling blackout, should those occur.
Encouraged by, among others, Jeff Geerling, I ordered the DeskPi Super6C, and then I paid what felt like 3 limbs, a firstborn, and a fortune, to acquire 6x Raspberry Pi CM4 8GB modules on eBay.
I was used to having Proxmox VE on my normal Intel cluster, but it was not to be: Pimox7 does exist, but it is not in a state suitable for what I expect. I had my third attempt, and then after a lot of toil my third surrender regarding Kubernetes, and so I went OG: Plain Debian 11 with Ceph RBD and CephFS, with LXD in multi-master mode. This actually works surprisingly well: I run my various small services (IPv6 routing, CA, caches), as well as a full-blown ARM64 VM (KVM, through LXD) running Debian 11 with ZFS as a fileserver.
The main bottleneck in this little cluster is bandwidth: The CM4s have just 1 lane of PCIe 2.0, and the motherboard has a GBe switch on it. I have noticed that running my full-blown InfluxDb setup on it leaves the nodes, containers and VMs fighting with Ceph over the bandwidth, and it all ends in tears. I therefore moved that service to a server that I have running in a community-based colo in Zurich.
I did not do power measurements yet, but I will at some point. At any rate, it will be lower than the 90W-a-pop idle power of my Intel cluster.
Malina sitting in its chassis.