Building a kubernetes and general system

davidm

New Member
Oct 3, 2014
23
3
3
55
Hello all. I'm a systems & software developer. My work usually consists of full-stack javascript. I use Ubuntu on a workstation in combination with my notebook, using Synergy, so the server and builds can run on the faster workstation, which I access from the notebook. That works very well. But now (a bit late to the game) I'm moving everything to Kubernetes, and it will be an important part of upcoming work, where I will be doing a lot of experimenting. I want to put together a workstation, with microk8s, that is as fast as is cost-effective. I currently have 32GB of RAM, and an SATA Samsung 850 EVO. Just to set things up, I am using a loaner i3-8100, and it's in generally pretty fast, noticeably and measurably faster than my notebook with an i7-8550u for typical tasks. But I'm wondering if it would be worth it to replace the CPU with an i7-8700 (I also thought about going "sideways" and getting a 2400G, so I could upgrade it if the upcoming Ryzens are spectacular, but its 4MB cache would probably be quite unhelpful). I'm specifically noticing that Kubernetes tasks can be slow. All the components of a deployment take a while, even when resources are local, though there isn't always a correlation with CPU usage. Since a lot of my days will involve this activity, will it make a difference to have a faster CPU with more cores, or is there some other part of the system that would be more important?

Eventually these deployments will run in the cloud, but for now I just want to have the best flow for experimentation and local development, without spending in the wrong areas.

I should also mention that I plan to port some other general services to run on this system so it's the only one running, but they don't have much overhead.

Thanks!
 
Last edited:

MiniKnight

Well-Known Member
Mar 30, 2012
3,027
932
113
NYC
My advice would be that there are some Ryzen 7's that are 8 core and inexpensive. There's also Threadripper 16 core options that are very reasonable these days and can take more RAM.

Personally, I'd go for more cores and RAM as you can. A used Ryzen 7 1700 non-X may be a cheap alternative now unless you need the integrated graphics.
 

Ivan Dimitrov

Member
Jul 10, 2016
33
4
8
38
Hi, I would recommend to improve your storage. Some NVME storage or a RAID can help. When you think of the layers of abstraction in Kubernetes storage subsystem it really can get messy quickly.
 

EffrafaxOfWug

Radioactive Member
Feb 12, 2015
1,395
504
113
Just to set things up, I am using a loaner i3-8100, and it's in generally pretty fast, noticeably and measurably faster than my notebook with an i7-8550u for typical tasks. But I'm wondering if it would be worth it to replace the CPU with an i7-8700 (I also thought about going "sideways" and getting a 2400G, so I could upgrade it if the upcoming Ryzens are spectacular, but its 4MB cache would probably be quite unhelpful). I'm specifically noticing that Kubernetes tasks can be slow. All the components of a deployment take a while, even when resources are local, though there isn't always a correlation with CPU usage.
The 2400G is a nice little chip but probably unsuited to the kind of work you mention. However, your comment above suggests that you might have bottlenecks other than the CPU, so you'd be in a much better position to pick a suitable CPU if you did some analysis on what was actually slowing you down. Potentially your storage is a bottleneck as the steady state performance of those consumer drives can get pretty mediocre if they're under constant load.

Following on from MiniKnight's comment, non-APU Ryzen's are a great way of getting lots of threads and ECC support for cheap - it's just a crying shame that no-one's brought out a workstation-lite motherboard with IPMI (thus negating the need for onboard graphics) for them; even more so for the threadripper variants or the epyc 3000.
 

Evan

Well-Known Member
Jan 6, 2016
3,346
584
113
A lot of concurrent work happening when you run a k8’s cluster on a single drive so better make sure it’s a nice fast NVMe one. 4 fast cores or an 8 core and if the systems works with it 64gb ram is always nice but 32 works (then again my install was used very little just to take a look see at what it can do).
I actually set mine up in a VM. But you do get better use of hardware if you do it native.
 

davidm

New Member
Oct 3, 2014
23
3
3
55
I really tried to go Ryzen, but it doesn't quite add up. I'd have to get a GPU, and I'd prefer not to get a cheap one (since I might want it later for ML experiments), so that adds a lot to the cost. But more importantly, for a mix that includes front end development, single thread speed is still really important and Intel is around 20% faster than the comparable AMD chip in a number of significant benchmarks (if I were building a server, I'd clearly go AMD). This is why I considered the 2400G to hold me over, but as others affirm it would be it would be too slow to use in the meantime. So now I am tending toward the i7-8700 for the best single core speed and number of cores/threads in its price bracket. I don't want to overclock.

I am trying to use some equipment I have lying around, so in the short term I will add a 950 Pro, then eventually add an NVMe drive when I can find 2TB+ at a good price. I thought about putting the 850 + 950 in RAID0 (with good backups), but I think it would be better to make the 950 the most written to drive with the 850 for storage. Of course a good NVMe drive would put that combination in the dust.

I'm curious as I had a conversation around sustained write ability recently. I'd think that for this kind of work, where large images are often being deployed (sometimes, for machine learning, gigabytes of highly compressed text in multiple containers concurrently), with a fast CPU, sustained write speed would be significant comparing consumer drives. Though the 970 Evo Plus has quite a lot of write cache, so significant cases are diminishing.
 

EffrafaxOfWug

Radioactive Member
Feb 12, 2015
1,395
504
113
The problem with the 950 will be similar to that of the 850 in that it's a consumer drive and, after sustained performance, will start to slow down. If storage is indeed your problem (and I should stress there's no evidence that it is yet) then you'd be better off spending money on enterprise drives that have consistent performance no matter how long they're stressed for.

Most TLC-based consumer drives have something they call a pseudo-SLC cache - basically a portion of cells that it treats like SLC, which allow faster IO to be performed. As the drive gets full, less of these cells become available as they're instead being used as regular TLC, so performance diminishes the less free space there is on the drive. Enterprise drives don't do the pseudo-SLC thing TTBOMK and have much larger flash reserves.

If you suspect IO might be a bottleneck as a quick test I suggest you monitor the system with iostat and/or iotop when it's doing one of its runs. Running regular top/htop/atop in another window will likewise give you an idea of what CPU and/or memory might be doing - e.g. high wait times coupled with lots of queued read/write requests might suggest your storage can't keep up with the IOPS load.
 
Last edited:

davidm

New Member
Oct 3, 2014
23
3
3
55
The problem with the 950 will be similar to that of the 850 in that it's a consumer drive and, after sustained performance, will start to slow down. If storage is indeed your problem (and I should stress there's no evidence that it is yet) then you'd be better off spending money on enterprise drives that have consistent performance no matter how long they're stressed for.

Most TLC-based consumer drives have something they call a pseudo-SLC cache - basically a portion of cells that it treats like SLC, which allow faster IO to be performed. As the drive gets full, less of these cells become available as they're instead being used as regular TLC, so performance diminishes the less free space there is on the drive. Enterprise drives don't do the pseudo-SLC thing TTBOMK and have much larger flash reserves.

If you suspect IO might be a bottleneck as a quick test I suggest you monitor the system with iostat and/or iotop when it's doing one of its runs. Running regular top/htop/atop in another window will likewise give you an idea of what CPU and/or memory might be doing - e.g. high wait times coupled with lots of queued read/write requests might suggest your storage can't keep up with the IOPS load.
Oh, I thought the 950 Pro had better sustained writes, I see it doesn't, the later generation drives (970 Pro, 970 Evo Plus) are better though.

I am going to do some benchmarking, but mostly plan to find an NVMe drive with good characteristics. I wouldn't go to enterprise though, since this is mostly for my own experimentation (I'll have access to cloud resources via work) and I want to keep my workstation small and quiet.
 

EffrafaxOfWug

Radioactive Member
Feb 12, 2015
1,395
504
113
I'd benchmark before buying a new NVME drive myself (because I'm a skinflint), it's entirely possible you'll be able to get away with a good-enough enterprise SAS/SATA drive rather than paying the NVME premium (although of course depends on what sort of kit you've got available).
 

davidm

New Member
Oct 3, 2014
23
3
3
55
I'd benchmark before buying a new NVME drive myself (because I'm a skinflint), it's entirely possible you'll be able to get away with a good-enough enterprise SAS/SATA drive rather than paying the NVME premium (although of course depends on what sort of kit you've got available).
My system board has two x4 NVMe slots (one is currently used by the SATA 850 Evo). I don't think there is that much of a premium for a decent consumer drive like the 970 Evo+, I could justify the 500GB or 1TB model if it makes a real difference. But I am going to do some benchmarking first.
 

posternutbag

Member
Jan 2, 2019
64
30
18
For what it's worth (literally!), you could sell the 950 Pro to subsidize a larger drive. They have decent resale values, probably because of the OPROM. I know because I've always got one eye on them ;)