K8s on CWWK X86-P5 & X86-P6 mini-pcs

[bhigh]

I recently purchased 4 systems in order to build out a home k8s cluster. The X86-P5 systems use either the N100 or N305 whereas the X86-P6 systems have either an N150 or N355. The N150 systems cost about $5/ea more than the N100 which seemed like a simple choice. The N355 systems were about $50/ea more so I opted for the N305. I believe there was an STH article about the differences, but it amounts to 200 MHz on the N100/N150 and 100 MHz on the N305/N355.

The systems I bought are:
1. N150 X86-P6 4*M2 NVMe
2. N150 X86-P6 "Super Mini Router"
3. N305 X86-P5 4*M2 NVMe
4. N305 X86-P5 "Super Mini Router"

I purchased 2 x Crucial 32 GiB DDR5 5600 and 2 x Crucial 48 GiB DDR5 5600. My initial plan was to install 32 GiB in the N150 systems and 48 GiB in the N305 systems.

For storage, I purchased 10 x TeamGroup MP33 1TB SSD because they were cheap, and I'm not expecting a lot out of them. The "Super Mini" systems have a PCIe 3.0x4 M.2. The NAS systems use the same motherboard with an adapter plugged into the M.2 slot. The breakout board plugs into this with some pins and provides 4 M.2 slots at PCIe 3.0x1. Nothing in this setup screams "high speed I/O".

I ran Memtest86+ for a few hours on the N150 systems and there were no errors. The fanless system was very hot - Memtest reported in the high 70s C. The NAS system has a small fan which kept temperatures down, but still in the 60s.

Both of the N305 systems failed in Memtest+. From the outset I wasn't certain if the 48 GiB SO-DIMM would work, so I wasn't terribly surprised and I had additional 32 GiB SO-DIMM to test with. The 4*M2 system has been running for ~ 2 hours at this point without errors. I'll test the fanless system after the current system hits 3 hours. I have contacted their support to see if a newer BIOS is available for the N305 systems.

Fit and finish on the systems is pretty typical, maybe better than expected at the price points. The NAS systems include thermal pads for the NVMe drives. All of the systems came with a USB-powered drive and protective shroud that can be mounted to the top. SATA power/data cables were included with the "Super Mini" systems.

The included power supplies are from Dajing, rated at 12V 5A. I'd like to find a single power supply with multiple 12V outputs instead of using the 4 separate units.

I have a few minor gripes however
1. There are only 2 USB Type-A and the I/O panel is very crowded. It's a tight fit when HDMI, ethernet, KVM USB and a USB thumb drive are all connected. At least they're both 10Gbps. There is an "upgraded" version of the Super Mini with 4 additional USB 2.0 Type-A.
2. The board has a SATA interface, but the plug is blocked when a 2280 drive is used. There is a screw standoff for a 2242 drive that would allow a SATA drive to be used.
3. The 4*M.2 board has to be removed to install memory. It's a tight fit and I was afraid of breaking something the first time. It would be nice if there was a pull tab or something.

I'm planning to run Plex, Jellyfin, Immich, some game servers, etc. as well as migrate some services that are running in VMs on an old server.

I can keep this thread updated if there's interest. If you have anything you'd like me to test, let me know and I'll give it a shot.

 

[kalpik]

I am interested. Where did you buy them from?

 

[bhigh]

I am interested. Where did you buy them from?

I purchased them directly from CWWK. My wife is Chinese and spoke to a salesperson for me. She was able to get a lower price by paying in RMB rather than buying through their website. It was delivered by DHL and arrived within a week.

I had previously purchased a 4-port router from the CWWK website and it came with a nice little screwdriver in the box. No such freebies this time. ¯\_(ツ)_/¯

Both of the N305 systems failed Memtest with the 32GiB modules. Removing and reseating the memory worked for the NAS system, but the "Super Mini" failed again. (Test 6). I'm going to swap the modules between the N150 and N305 to see if that makes any difference.

I'd like to get some recommendations for the bare-metal install. Flatcar Linux or Alpine Linux seem like good lightweight options, but NixOS might make it easier to keep the systems in sync.

 

[bhigh]

I've swapped the memory between an N150 system and an N305 system. It's still testing, but there are no errors after 2.5 passes in Memtest86+.

The N150 bios is v 2.22.1293, built on 11/29/2024. The N305 is v 2.22.1289, built on 6/26/2024. My suspicion is that the newer bios has better memory support.

I've asked for a newer bios for the N305. I'm going to try a 48 GiB module in the N150 to see if it works better.

 

[Chance]

I've been eyeing those exact models recently, so I'm really curious to find out if the bios helps.

 

[bruce1066]

I have a couple of the n305 models. My use case is as a multi-function travel router and server. One i have configured as a proxmox server with numerous VM's including openwrt which works well including wifi. The other machine i have confugured running Openwrt bare metal but with the rootfs expanded and running a some Openwrt apps such as samba, docker, adguard and a ton of apps in docker/dockge/portainer. On both machines I use both an internal and usb wifi - for wan and ap.

I've designed a small 3d panel to mount wifi sma connectors to replace the sticker on the rear of the unit which is the existing non-metal path for wifi. With 8 dBi external antenna I get great wifi performance using an Intel BE200 pcie card. There are only two cards that CWWK say work. The BE200 and the AX210. I tried a couple of MediaTek cards, but no joy. The problem with the Intel cards is that they don't support AP mode at 5gHz, only 2.4gHz. Hence the usb wifi.

I'm using crucial 32Gb for main memory. I'm using a mix of 1 and 4 TB nvme drives which work well. I haven't noticed the machines getting too hot. I monitor using glances. The top heatsink gets warm but not what I'd call hot.

I did also configure one of the machines as an unraid server, loaded with 4x4Tb. Worked very well. The n305 is definitely a worthwhile step up from the N100.

 

[NioBium]

Is it audible from 3 feet away when idling?

 

[bruce1066]

Sitting here with two switched on, side by side, 3ft away., silence

 

[NioBium]

Neat, thanks!

 

[bhigh]

I've been eyeing those exact models recently, so I'm really curious to find out if the bios helps.

The rep I chatted with on WhatsApp said there is not a new bios for the N305 system. I *might* get a bios chip reader / programmer and pull the bios from the N150 systems to try out, but I'm also lazy and might not.

MemTest86+ reported the memory speed on the N150 systems as about 5% slightly slower than the N305.

I'm using crucial 32Gb for main memory. I'm using a mix of 1 and 4 TB nvme drives which work well. I haven't noticed the machines getting too hot. I monitor using glances. The top heatsink gets warm but not what I'd call hot.

I did also configure one of the machines as an unraid server, loaded with 4x4Tb. Worked very well. The n305 is definitely a worthwhile step up from the N100.

Did you use thermal pads between the drives and the top? I am not, and I'm not using the USB fan on the top. `smartctl` is reporting ~ 40 C for all drives. Drives in the fanless systems are closer to 70 C. I have a small desk fan that I'm going to blow across all systems, as soon as I can find where I put it.

The package temp on the fanless systems is ~ 45 C, and 42 C on the 4*NVMe systems.

Is it audible from 3 feet away when idling?

You can hear them, but only if the room is otherwise completely quiet. There's a switching power supply somewhere in my room that has coil whine which is more audible.

 

[amalurk]

Meanwell makes a bunch of 12v power supplies....you can find them on Amazon.

 

[bhigh]

As far as the software stack, I'm using the following so far:
OS
I'm using Ubuntu 24.10 Server, with the 'minimized' option. I considered Alpine and Flatcar, but didn't want to create custom install media for each host.

The hosts with a single NVMe drive use LVM on the raw device. LVM defines a 100G root filesystem, with the rest of the space in a separate filesystem for cluster data.

The hosts with four NVMe drives use RAID1 for the boot partition and RAID0 on remaining space. LVM defines a 100G root filesystem, with the rest of the space in a separate filesystem for cluster data.

K3S
I chose K3S as the control plane.

• Full-fat k8s has more overhead than a minimal k3s deployment, such as etcd. I'm not going to be adding hundreds (or even tens) of nodes and don't think that I'll need it.
• microk8s would probably have worked, but I have used k3s in the past.

Storage
k3s (and microk8s) has a local path provider that works out of the box, but pins pods to that host. I opted to use Longhorn since it works well with k3s.

• I installed a few helm charts before installing Longhorn. If I knew what I'm doing, I'd have installed Longhorn or some other provider first. Migrating between storage classes is a pain.

Monitoring
I'm using Groundcover's free tier, because it was easy to set up. I'm planning to set up OTEL collectors and prometheus "eventually".

Ingress and Load Balancing
I opted not to use k3s's bundled version of Traefik. I'm using the Traefik helm chart instead.

I'm using MetalLB for the ingress controller.

Workload
So far, I have a Factorio server running for myself and friends. I've installed AdGuard home, but my network is not using it (yet).

I'm going to set up Plex, Jellyfin, and Immich eventually. There are a few BSD jails on my TrueNAS system I need to migrate as well.

 

[bhigh]

Meanwell makes a bunch of 12v power supplies....you can find them on Amazon.

I don't think it's one of these machines and it's faint enough that I don't usually notice it.

I would like to have a single power supply for the cluster instead of 4 separate adapters. I've thought about using an ATX power supply or something like this, but haven't made up my mind.

 

[Sean Ho]

Before you get too far on setting up apps/infra on k8s, consider flux or ArgoCD!

 

[bhigh]

Good point.

I'll probably use ArgoCD since it's used at my current job. My previous employers (Google, FB, Dropbox) had in-house tooling so I don't have any hands-on experience with ArgoCD or Flux.

 

[bruce1066]

The rep I chatted with on WhatsApp said there is not a new bios for the N305 system. I *might* get a bios chip reader / programmer and pull the bios from the N150 systems to try out, but I'm also lazy and might not.

MemTest86+ reported the memory speed on the N150 systems as about 5% slightly slower than the N305.




Did you use thermal pads between the drives and the top? I am not, and I'm not using the USB fan on the top. `smartctl` is reporting ~ 40 C for all drives. Drives in the fanless systems are closer to 70 C. I have a small desk fan that I'm going to blow across all systems, as soon as I can find where I put it.

The package temp on the fanless systems is ~ 45 C, and 42 C on the 4*NVMe systems.



You can hear them, but only if the room is otherwise completely quiet. There's a switching power supply somewhere in my room that has coil whine which is more audible.

Not using thermal pads, looking at Glances with a steady stream of data (about 500gb so far being written), the cores are all sitting at about 60C. The heat sink on top shows 35C.



I've also designed and 3d printed a small housing that replaces the rear sticker (unpeal and re-apply to the side). The housing has a 1TB sata drive and a couple of coax terminals for wifi. The sata drive is connected to one of the mini headers internally. Works at normal ssd sata speed (250MB/s). I took apart a new sata - the guts are about 50mmx40mm, retaining the std sata connector. I wanted to find a way to use the two internal sata ports without having cables dangling outside - this way I can have apps plus boot run off the sata and have the 4nvme's as a clean raid. The little box has 4nvme's, a wifi slot plus a couple of sata ports - and some internal usb headers - it's a bit snug, but you can actually use it all.

 

[NioBium]

I've also designed and 3d printed a small housing that replaces the rear sticker (unpeal and re-apply to the side). The housing has a 1TB sata drive and a couple of coax terminals for wifi. The sata drive is connected to one of the mini headers internally. Works at normal ssd sata speed (250MB/s). I took apart a new sata - the guts are about 50mmx40mm, retaining the std sata connector. I wanted to find a way to use the two internal sata ports without having cables dangling outside - this way I can have apps plus boot run off the sata and have the 4nvme's as a clean raid. The little box has 4nvme's, a wifi slot plus a couple of sata ports - and some internal usb headers - it's a bit snug, but you can actually use it all.

Brilliant! Would you be open to posting your files? I would like to modify it so that it has two vertical 2.5" drive slots so that the drives could be slid in from the top. That way, I can avoid destroying the enclosures of the SATA drives, make them easy to swap, and make sure nothing rattles inside. Always wanted to jump on this 3D printing bandwagon, but never had a pretext.

 

[bruce1066]

Here's a number of files, in STEP format - you should be able to pull them in to any cad program or 3d printer. There are various options. All have a common mounting plate that slips inside the case snug down against the SODIM socket with the rear sticker removed.

The sticker can be eased away without damage and then re-attached to the side - mine haven't pealed away. I push from the inside starting at one end and letting it peal itself off.

The inner mounting plate needs to have 4 m3 thermal bushes installed - available from amazon, easy to install with a soldering iron. On the outside you can then mount the simple antenna plate which first needs to have the antenna a leads attached. With just wifi mounted its a two part assembly. It will be all very obvious how it assembles once you either import to CAD or 3D print.

If you want to use the mounting for the sata drive its a three part assembly as shown in the earlier post. There are two versions, depending on the size of the sata guts you use. I found the 4tb drives need a larger housing which is a few mm taller than a 2tb drive, which is a more 'elegant' solution - if you can call it that.

I print with ABS since it has a higher resistance to heat. But I haven't found there to be any heat issues - the drives I have used stay pretty cool - they don't generate the type of heat you see with an nvme. I have seen ssd's getting to 50 degree, but then others sticking at a max of 35. So that is probably worthwhile thinking about. ABS is good up to close to boiling point.

The sata assembly is in three parts since I first wanted to get as good a mount to the computer case as I could and then build up from there. The sata cable and the antenna leads have holes to thread through. While you 'could' change the sata drive once everything has been assembled it makes for a neater assembly if you assemble from the outside-in. You will figure out what I mean once you put it together. All fasteners are m3 countersunk.

I Had to upload as a zip file as that's the only format that would work for the step files on this msg board.

Btw, you don't need to destroy any sata drive. All the ones I've played around with, with the exception of things like the WD external drives, all come apart pretty easily - either plastic cases that can be unclipped or metal cases that have either an exposed set of screws or screws under a label. If you are going to go the full drive route you should take a hard look at if the design is strong enough - it's a different proposition having a small, static circuit board (ssd) in a small enclosure hanging off of parts of a case that doesn't have a lot of metal exposed to mount to, vs a larger assembly where there will be the loads from insertions. Not saying it won't work, or it will fail, just I designed with different assumptions and application. If I wanted to build the application you are thinking of, I 'may' look for a beefier solution - the challenge is not doing actual surgery on the case.

 

[NioBium]

Fantastic! Thank you for the files and even more so for the write-up! It will take me some time to understand parts of it...

 

[bruce1066]

Maybe these help.