Low power kubernetes build (compute/storage)

[lvsb]

Hi,

Im looking to build a low power cluster to replace my current r720. I had a look at pi's but they are underpowered.
Is there another option than intel nuc's and if not, which ones are in the sweet spot now regarding to price/performance.

For storage i'm looking to replace my current n40 with 4 odroid hc2's with gluster on top.

All feedback is welcome.

 

[Marsh]

List of my favorite , range from cheap to expensive

I am using all of following machines in my K8s cluster

Cheap , $40-$45 range, add more $$ for DDR3 ram
HP 8300 SFF or USDT , I5-3470 , 4gb ram , SFF has PCIe slot for extra NIC
only few % performance then HP 800 haswell series, just as power efficiency as the Haswell CPU
SFF version has 4 ram slots

$65-$100 range , add more $$ for DDR3 ram
HP 800 SFF or USDT , I5-4570 , 4gb-8gb ram , SFF has PCIe slot for extra NIC , has mini-pcie slot , msata slot for USDT version
SFF version has 4 ram slots

$85-$100 range , add more $$ for DDR3 ram
Smaller than NUC, tiny machine
HP 800 G1 dm or mini , I5-4570s , 4gb-8gb ram , only 2 ram slots , no PCIe slot , has mini-pcie slot, m2 slot

$100-$150 range,add more $$$$$ for DDR4 ram , require Ebay Best offer magic.
HP 800 G2 dm or mini , I5-6xxxx , 4gb ram , only 2 ram slots, DDR4 ram could goes up to 32GB, no PCie slot , has mini-pcie slot
m2 slot

$200 and up
Supermicro C2xxx , expensive RAM

$300 and up
Xeon-D , not cheap , expensive DDR4 RAM, high-end, 10gbe, many network ports , lots of expansion capabilities

 

[Patrick]

I think we have every Dell EMC Intel Xeon E-2100 series server inbound, and the HPE DL30 Gen10 arrives tomorrow (we already did the HPE DL20 Gen 10 review) and we did two Supermicro platforms.

I really like them but they are not ultra cheap and use DDR4. 6 high-speed cores is very useful. If you want DDR3, the Xeon E5 V2 would be my choice.

I know a low-power cluster sounds fun, but I would not discount a virtualized Xeon E-2100 series server. With their clock speed, you can get two or three Xeon D-1518 equivalents in a single system and skip the external networking for three systems.

For a single machine replacement, there are much less expensive options. For multiple low power system replacements, virtualizing a larger system is an option that is worth looking into.

 

[Marsh]

I agree with Patrick.

I have split personality , some day, I fire up the 40 cores and 256gb ram Proxmox server to run a virtualized K8s cluster.

then on another day, I am marvel with the computing power of a cluster consists of older generation of I5 4cores with 8gb baremetal machines. Each HP machines consumes around 25w , sitting next to my desk in complete silent.

 

[Evan]

@Patrick

HPE DL30 or ML30 ?

One other option could be the C3955 16-core based system, lots of cores to play with at low power and standard DDR4

 

[Patrick]

@Evan my bad, ML30, tower. Sorry, my brain is absolute mush from this week. We published a normal week's worth of content on STH just yesterday.

 

[PigLover]

Even though I’m just departing KubeCon today I have to agree - building a small Kubernetes cluster to replace a single (larger) server is not going to be cost effective or particularly easy. It’s just not the right use case. After you add in RAM and disks and network building even 3 nodes using the parts suggested by @Marsh will add up to well over $500 - and a low end single-host E-2100 or C3000 would run rings around it for performance.

If y0u are building it to learn how to do bare-metal K8s then its a good move. But if its just to learn the basics of K8s you’d actually be better off building a bigger singe-node system and virtualizing the “nodes” for K8s. Much easier then to build it up, tear it down, experiment with networking options, experiment with redundancy options for the controllers, etc., etc. You’ll learn more and learn it faster with nodes on VMs (all except the special challenges of K8s on bare metal, of course).

As I’ve advised before when people were gung-ho on building out Ceph for their home lab. As a learning tool its a good thing to do - but the target use-case for K8s, the ones that make it useful and cost-effective, is really about much larger deployments than you are likely to see in the “production” part of your home lab.

 

[Marsh]

I totally agree with you and Patrick.

I have to confess, I have a unhealthy obsession with "tiny machine".
Whenever I see a tiny machine, I have to buy it. I think it is because my dream is to live a tiny Class B van ( less than 140 sq feet ),
be a vagabond , I could still have homelab cluster in my Class B van.

An example of tiny cluster.
Mix of HP 800 G1 and HP 800 G2 machines. Each machine is 7" x 7".
HP 800 G1 mini is I5-4xxx , max 16gb ram , 1 m2 slot, 1 x 2.5" , vpro
HP 800 G2 mini is I5-6xxx , 1 m2 nvme slot, 1 x 2.5" , vpro

Supermicro Xeon-d with 64GB ram running Proxmox.
VMs are virtual K8s cluster , virtual router , pxe boot server , matchbox server for CoreOS.

I ordered 6 more HP 800 G2 mini yesterday.

 

[PigLover]

My comments weren't to suggest doing a small cluster is not worth it. Just that if you are doing it to replace workloads running on a small home server its probably not the best choice.

But just because its not cost effective does not mean its not fun. And you can learn a heck of a lot doing it. My cluster (sorry, no pics) is K8s ARM on 8 Odroids (4x HC1 with small SSDs attached and 1 MC1 - which is really 4 nodes). Its cute and I do have some real workloads running on it. But everything on it could run on the Xeon-D that its sits on top of and I'm not sure the bigger box would notice the extra work. My biggest gripe with it is that these are Arm32 and too many things are starting to require 64bit.

I still want to do a small cluster of Odroid H1 when they get back in stock. $110/each + DDR4 SO-DIMMs + M.2 SSDs won't be "cost effective". But it will be cool!

I have a small love affair with K8s. And in my "real" job we have it running on several clusters with >100 Skylake R640s each. It really is cool stuff.

Sorry - no pics.

 

[Evan]

I am with you guys and I have a collection of small machines and I enjoy playing with them, can be fun even if not practical.

 

[Evan]

@Evan my bad, ML30, tower. Sorry, my brain is absolute mush from this week. We published a normal week's worth of content on STH just yesterday.

Oh good I don’t need to be upset at my HPE guys for not telling me hahaha (of course they never would and we are only interested in the box boxes)

 

[lvsb]

Just to clear some things out :-D
I already have everything virtuaklized on a single machine at this moment. A r710 with 148GB ram with vmware and k8s on top.
Now this is completely overkill and burning trough my electricitybill.
Thats why i'm looking to replace it with something more power efficient. And as k8s is whole lotta fun i was thinking of buying multiple nucs to replace the virtualized cluster.
But im having troubles finding out which nucs are the most interesting buying at this moment. (price/performance ratio).
And as a homelab should be fun i want the n40 replaced by something smaller too, now that ive started the shift to smaller stuff i cant stop :-D
Therefore i was eyeballing the odroid hc2's. The seem really cool with their ability to enclose the disk.

Most definitely fun times ahead of me

 

[k8Dev]

@Ivsb, I feel you, just turn off my R710 and couple of hp z computers running k8s, the hydro bill going skyrocketed as the electricity price double these days. seems like a really good idea to run HP mini as I have a few used in other places now.