Bare-metal kubernetes build - thoughts?

[gigatexal]

Everything comes in threes, except for the nvme drives, I have a spare 950pro nvme ssd to add. Comes to about 1800 EUR. The thought process is 3 AMD 2400G's 12 real cores 24 threads, each node would have 32GB of DDR4 2666mhz ram ... networking would be the next thing to tackle. Maybe a cheap 10GBe setup or something -- I haven't bought it yet just have a sizable German tax return that I am thinking about and I need to get better at K8s for work -- and not just using it but the nitty gritty stuff like the networking and the other things.

So 12-cores, 24-threads Ryzen 2400G's
96GB DDR4 total ram
3x 512GB NVMe SSDs
...

 

[PigLover]

That would be a pretty decent k8s lab, but if I was building it I wouldn’t be going ‘new’. I’d be looking at things like the HP Elitedesk ‘mini’ 800 g2. You can get them on eBay regularly with i7-6700t for $350-399/each, usually including 16gb ram and decent ssd.

The mini is really small, quiet, well packaged. I like the g2 or g3 because it also has an m.2 nvme slot in case you want to play with higher performance disk for things like Ceph (rook). You don’t have a 10gbe option but really - high performance networking isn’t the ‘main thing’ you need to experiment with in k8s.

 

[Marsh]

My build BOM.
Purchased 12 x HP 800 G1 dm ( mini ) barebone $18 each
12 x I5-4590T $40 each
2 x 8 GB ddr3L ram $20 x 2 = $40
intel 180gb ssd , $22 each
Total $120 per node.

I also purchased 8 x HP 800 G2 mini barebone , $35 each , waiting for cheap I5-6500 and DDR4 ram.

HP 800 G1 , G2 comes with vpro , similar to ipmi , kvm but with limited function.

 

[Marsh]

Couple days ago.
There was a listing for HP 800 G1 , I5-4590 , 8GB ram for $75 each.
Seller sold over 25 units in matter of hours.

 

[gigatexal]

Ooh i do like the idea of having an ipmi like interface

 

[gigatexal]

I’ll look to see if I can source something g like that here in the EU

 

[Evan]

Do you really want or need 3 physical systems ?
Could just have 3 VM’s on a single server to do the same thing right ?

 

[gigatexal]

Do you really want or need 3 physical systems ?
Could just have 3 VM’s on a single server to do the same thing right ?

Yeah still need a beefy host

 

[Evan]

Yeah still need a beefy host

Don’t disagree at all, just figured it may be easier to just buy one used that fits the purpose.
Just throwing out the option in case it made sense for you.
I only know I have been stuck with a number or small systems that in the end made no sense but a big one would have worked.

 

[gigatexal]

No it's a super viable option. For some reason the idea of 3 physical machines is cool just for the fact that I get to build 3 machines! Totally impractical though.

A single 2700 Ryzen chip with say 32GB of ram would suffice and have enough cores left over for other useful things.

 

[TomUK]

I agree with what's been said above with regards to getting 1 larger machine.

With that budget I think you could get a Threadripper 1920x or 1950x, these will do upto 128gb or RAM and have loads of PCIE and NVME connectivity.

I use a 1950x to run a 3 node K8s cluster, performance is not an issue for me at all.

 

[gigatexal]

Yeah I think I’ll keep an eye out for when new hardware drops to buy or just keep an eye out for a good deal. I like the idea of getting a 2950x since it’s newer and would have a higher resale but again the 1950x is perfectly fine for what I’ll be doing since I’m not chasing marginal FPS.

 

[MikeWebb]

WOW man, that's one hell of a lab. My K8s lab is 4 rPI 3B's with NFS from a little QNAP NAS (I also send ZFS snaps to that via ISCSI)