[WIP] Future-Proof, Low IDLE Consumption, Yet Powerful LGA1700 Mini Server

[NablaSquaredG]

Hey folks,

This project is WIP, and I will use this thread as a build log.

I will try to precisely describe what I've done and especially which choices I've made and why I've made them to help others if they're in a similar situation.

What's the goal?
I have let things slide for quite some time in my lab, and the time to clean up has come now. And now that I finally I take care of it, I'll do it properly.

I needed two machines that will run 24/7, so they need to have a low IDLE power consumption (< 10W for whole system with SSDs, excluding PCIe cards, no periphery connected). At the same time, I would like them to be powerful enough for the planned tasks and future-proof.

The machines will run the following applications (non-exhaustive, I've probably forgot stuff):
- DHCP (Kea DHCP)
- DNS (BIND9)
- Virtualised opnsense / pfsense as firewall and router
- Wireguard VPN endpoint
- Ruckus virtual SmartZone for Wifi AP(s)
- Print Server
- OpenSM (for Infiniband, as managed switches are far more expensive than unmanaged switches)
- syslog & stat server (rsyslog / zabbix / Prometheus / Grafana)
- Time Server (Stratum 1 via GPS), NTP and PTP master (yes, I only had this idea today...)

The search and research began... And after hours of weighing up pros and cons, I came to the conclusion, that the Intel LGA1700 platform is the way to go.

Why? Ryzen 5000 (AM4) cannot be upgraded, and it seems like it's very challenging to get Ryzen to have a low IDLE power consumption.
It's not really worth buying Rocket Lake (Intel 11000), and it also cannot be upraded.


Meanwhile, LGA1700 has all the latest features (PCIe5.0, DDR5) AND it's upgradable (Raptor Lake)!

Additionally, LGA1700 is unique because of Intel's hybrid architecture. Although I will NOT use CPUs with Hybrid technology (no E-cores, only P-cores) for the time being due to costs (I will explain later), I believe I will upgrade to do so at some point in the future.

Why is the hybrid architecture interesting for this use case? Well, as typical for such use cases, the server will have a constant, but low load all the time. I like to call it "Background Noise". It seems plausible that it would be more efficient to have more energy efficient, slower E-cores to handle this background noise, without having to wake up the high speed, less efficient P-Cores (Performance Cores)

Okay, so that was the justification for the platform itself. Let's get down to the specific hardware choices:


Build’s Name: "Well, that escalated quickly" MicroServer Gen 12 Plus
Operating System/ Storage Platform: Linux (Fedora)
CPU: Intel i5-12400
Motherboard: ASUS TUF GAMING B660M-PLUS WIFI (D5)
Chassis: Chieftec Mesh CS-12B
Drives: 2x 1TB Western Digital WD SN850 PCIe 4.0 NVMe
RAM: 2x16GB Samsung DDR5-4800 non-ECC UDIMM
Add-in Cards: 1x Intel i350-T4, 1x Mellanox NIC (CX354A or CB193A), 1x PCIe to RS232
Power Supply: be quiet! TFX Power 3 Gold 300W TFX
Other Bits: Navilock NL-8003P GPS Receiver, 2x Arctic P8 PWM PST, Arctic Alpine 17

Usage Profile: Homelab management HA Cluster (two machines), Kea DHCP, BIND9 DNS, opnsense/pfsense, Wireguard VPN endpoint, Virtual SmartZone, Print Server, OpenSM, Syslog & Stat Server, Stratum 1 GPS-based PTP & NTP time server


Let's speak about the component choice / justification.

Platform: Done. LGA1700 ticks all the boxes: Low IDLE power consumption, enough power, upgradable -> future-proof

Motherboard: ASUS TUF Gaming B660M-Plus WIFI (D5)
If we use LGA1700, we should make sure that we use all features of it (as we said, we want to be future-proof).

That means we want:
- PCIe 5.0
- DDR5
- 2x PCIe4.0 x4 NVMe
- 2 PCIe Slots
- µATX

Those requirements leave exactly 4 options for motherboards:
ASRock Z690M PG Riptide/D5 -> 200€
ASUS TUF Gaming B660M-Plus WIFI -> 200€
ASUS ROG Strix B660-G Gaming WIFI -> 230€
ASUS ROG Strix Z690-G Gaming WIFI -> 340,59

We're not going to overclock and Z690 doesn't provide anything we need for this build, except higher power consumption. With B660, the x4 slot on the board is 3.0, while for Z690 it would be 4.0.. But as the second slot is only for the Mellanox NIC which does OpenSM, it doesn't need a lot of bandwidth. If we ever need more networking bandwidth, we have a PCIe5.0 slot with an Intel i350-T4!
Two boards left: TUF and Strix. Strix is more expensive and doesn't offer any benefits, so we have a winner: The TUF B660!


CPU: Intel i5-12400

The i5-12400 is a 6C/12T CPU with 6 P-Cores and no E-Cores. Wait - We said that we want E-Cores as they would be very well suited for the tasks this server is supposed to do!
Well... Yes. But: Look at Intel's pricing structure.
I need an iGPU (let's be honest, everything else just sucks. there's a reason why the BMC in every server has a VGA output), so no F-models.
I estimated at least 6 P-Cores are needed for full bandwidth. That means we have 9 Alder Lake models left to choose from.

The cheapest and most basic one is the 12400 for ~190€ with 6C/6T. There are a couple of other i5 models with higher turbo rates (12500 for 220€, 12600 for 255€), but they also don't have E-Cores.
The cheapest model with E-Cores is the 12600K with 6P + 4E, but it costs 308€! That's a whopping +61% just for 4 E-cores and a some higher turbo clock rates...
There is the 12700 which would be more suitable than the 12600K, because it also has 65W TDP. 12700 has 8 P-Cores and 4 E-Cores. But it's 355€, that would be +87% compared to 12400.

I don't believe that the other CPUs are worth their premium, so for now we will use the 12400. BUT we can upgrade at a later point! B660 will be compatible with Raptor lake, so in about 1 to 2 years, once Raptor Lake hits the second hand market,I can just grab some cheap used Raptor Lake CPUs and replace the 12400 (and perhaps faster RAM, as Raptor Lake does 5600 as opposed to the 4800 of Alder Lake)

Chassis: Chieftec Mesh CS-12B
I did a fair amount of research, and that one was the only one that I liked. I wanted to it to be flat (so it can easily be stacked in the "rack") and also have sensible fan positions.. Most cases of this form factor have stupid fan mounting positions.

Drives: 2x 1TB Western Digital WD SN850 PCIe 4.0 NVMe
NVMe SSDs have very low IDLE power consumption, yet they have insane power when you need them.
The SN850 1TB is 110€ per piece and probably the best allrounder on the market right now. Some SSDs are more expensive and beat the SN850 in some disciplines, but then lose out in others (like FireCuda 530, 980 PRO, etc...)

RAM: 2x16GB Samsung DDR5-4800 non-ECC UDIMM
We won't do any memory OC, so we don't need expensive binned gamer DIMMs. 32GB should be enough for everything.
People have made very good experiences with Samsung DIMMs, better than Hynix. And they were only 5€ more per piece, so Samsung it is.

Add-in Cards: 1x Intel i350-T4, 1x Mellanox NIC (CX354A or CB193A), 1x PCIe to RS232
i350-T4 is a classic. I probably don't need to say anything. 1 port for WAN, 1 port for interconnect with the other machine (Inter Peer Link / Sync Link), 1 port for LAN, 1 spare port for whatever.
Mellanox NIC: As I said, I have an Infiniband network in my lab, and I want to upgrade it to 100G. Managed 100G IB switches are quite expensive, while unmanaged ones can be found for okayish prices (additionally, I have an engineering sample Mellanox switch from @Rand__ - it seems to have been a SB7700 ES, but at some point the Xeon-D management board just randomly died. I hope to flash the IB backplane to unmanaged and use it as an unmanaged IB switch)
Anyway: Infiniband requires a subnet manager and managed switches with integrated subnet manager are expensive. Solution: Pop in a 30$ Mellanox NIC into the server that's running 24/7 anyway. Much cheaper.

PCIe to RS232: Accurate Stratum 1 timeserver with GPS requires low Jitter PPS (Pulse Per Second) Signaling. USB is possible, but jitter is a bit of a problem always. Sadly none of the boards listed above has a PS/2 port (which you can connect a GPS receiver to), so serial communication is the best replacement. I'll use a SUNIX SUN2212 / Delock 89444 card. Perfect for the free x1 slot.


Power Supply: be quiet! TFX Power 3 Gold 300W TFX
Smallest good-quality and efficient power supply that can be found. Yes, there is HDPlex GaN PSU which is probably more efficient, but it costs three times as much. I considered using a Supermicro PWS-341P-1H (as it is platinum and I have a couple of spares), but I wouldn't fit properly into the case.

Other Bits:

Navilock NL-8003P GPS Receiver
The GPS Receiver I'll likely buy soon. Not certain yet, but very likely.

2x Arctic P8 PWM PST
One will be a front fan, the other one will be placed right behind the PCIe cards, mainly for the Mellanox card.

Arctic Alpine 17
Very good price/performance cooler similar to Intel stock coolers. Cheaper than boxed vs tray, quiet, good cooling performance.

Alright, so much for the introductory post. If you have any questions or comments, let me know!

 

[NablaSquaredG]

First update:

http://imgur.com/a/hBgG8nE

First parts (two mobos) arrived this morning. The rest is ordered and should hopefully arrive next week.

 

[Wasmachineman_NL]

Oh the irony of you saying you can't upgrade AM4 beyond Zen 3 and yet you go with a dead platform.

Should have waited for Zen 4.

 

[NablaSquaredG]

yet you go with a dead platform.

Should have waited for Zen 4.

The platform is not dead, as it can be upgraded to Raptor Lake, which is expected to be on par with Zen 4

And why not wait for Zen5? Or Meteor Lake?
Because I want the server now, and not in three months. You can always say "You should gave waited for <Next Gen CPU>", but in that case you'll never buy anything, because there's always a new architecture a couple of months away

 

[Stephan]

Accurate Stratum 1 timeserver with GPS requires low Jitter PPS (Pulse Per Second) Signaling. USB is possible, but jitter is a bit of a problem always. Sadly none of the boards listed above has a PS/2 port (which you can connect a GPS receiver to), so serial communication is the best replacement.

Do you really need sub-µs time accurracy on the network?

Take a look at hxxs://austinsnerdythings.com/2021/09/29/millisecond-accurate-chrony-ntp-with-a-usb-gps-for-12-usd/

I am using a Bus 004 Device 003: ID 1199:9071 Sierra Wireless, Inc. AirPrime MC7455 3G/4G LTE Modem with chrony and get around half a millisecond stddev with some chrony conf special sauce:



From chrony.conf, first line used to measure, second to activate:

#refclock SHM 2 refid GPS poll 8 filter 1000 noselect
refclock SHM 2 refid GPS offset 0.032 prefer

There is some udev+systemd trickery involved, /etc/udev/rules.d/61-wwan.rules:

ACTION!="add", GOTO="gpsd_end"
SUBSYSTEM!="tty", GOTO="gpsd_end"

ENV{ID_BUS}!="usb", GOTO="gpsd_end"
ENV{ID_VENDOR_ID}!="1199", GOTO="gpsd_end"
ENV{ID_MODEL_ID}!="9071", GOTO="gpsd_end"
ENV{DEVNAME}=="", GOTO="gpsd_end"

ENV{ID_USB_INTERFACE_NUM}=="02", RUN+="/usr/bin/setfacl -m g:wheel:rw -m g:gps:rw '$env{DEVNAME}'", ENV{SYSTEMD_WANTS}+="gpsd.service chronyd.service"

LABEL="gpsd_end"

Special /etc/systemd/system/gpsd.service:

[Unit]
Description=GPS (Global Positioning System) Daemon
BindsTo=dev-serial-by\x2did-usb\x2dSierra_Wireless__Incorporated_EM7455\x2dif02\x2dport0.device

[Service]
Type=simple
User=gps
Group=gps
AmbientCapabilities=CAP_SYS_TIME

Environment="DEV_GPS=/dev/serial/by-id/usb-Sierra_Wireless__Incorporated_EM7455-if02-port0"
Environment="DEV_CMD=/dev/serial/by-id/usb-Sierra_Wireless__Incorporated_EM7455-if03-port0"

ExecStartPre=sleep 10
ExecStartPre=sh -c "stty 9600 raw cs8 echo echok echonl echoe -F ${DEV_GPS}"
ExecStartPre=sh -c "stty 9600 raw cs8 echo echok echonl echoe -F ${DEV_CMD}"
ExecStartPre=sh -c "chat -V TIMEOUT 5 \"\" ATE1 OK AT\\!GPSEND OK AT\\!GPSTRACK=1,255,100,1000,1 OK \"\" >${DEV_CMD} <${DEV_CMD}"
ExecStartPre=sh -c "echo \'$GPS_START\' > ${DEV_GPS}"

ExecStart=gpsd -n -N -s 115200 ${DEV_GPS}
ExecReload=/usr/bin/kill -HUP $MAINPID
Restart=on-failure
RestartSec=10

What happens, using the powerful systemd facilities, is that system waits for hotplug event of the 4G modem, which has an active Mikrotik ACGPSA antenna attached. Using udev, the event triggers start of gpsd and chronyd. Also, through setfacl, r/w access is granted to the USB device for wheel and the gps-user. In the snippet above, before gpsd is started, the interface on the 4G modem is prepared so that it emits NMEA lines continually. By default this is off. I also use 115200 bps to push latency and jitter way down from 4800 bps.

Of course you'd need a 4G modem which I happened to have employed anyhow.

 

[NablaSquaredG]

Do you really need sub-µs time accurracy on the network?

Wait - When did we start asking that question in STH / Homelab?

But thanks a lot for the insights! Very interesting.

Jokes aside, there are some potential future projects on my list which might require sub-µs time accuracy and now is the opportunity to tackle everything in one go.

Of course you'd need a 4G modem which I happened to have employed anyhow.

Yeah, no 4G modem. So I'd need to add either a GPS or 4G modem, which somehow comes down to the same

 

[Rand__]

Yeah, no 4G modem. So I'd need to add either a GPS or 4G modem, which somehow comes down to the same

You could use the 4G (5G?) modem as backup internet connection though...
...unless that would reduce the precision on the time... hm might depend on the hw in the modem.

 

[Stephan]

there are some potential future projects on my list which might require sub-µs time accuracy

If I had that on my list, I'd shoot straight for IEEE1588-2019. Precise up to ~10ps, jitter about the same. That is light in vacuum travelling 3 millimeters. There are various implementations based around FPGAs, temperature-compensated oscillators, and gigabit ethernet, all sub-1ns.

 

[Stephan]

You could use the 4G (5G?) modem as backup internet connection though...
...unless that would reduce the precision on the time... hm might depend on the hw in the modem.

I use Dell 3P10Y exactly for that, because only 25 fiat on ebay, firmware upgradeable, very very well documented see hxxps://github.com/danielewood/sierra-wireless-modems. Changed it to generic Sierra Wireless, also you'd want QMI. GPS is separate from 4G, I can see no degradation in time measurement when 4G is up.

Why only 4G, because I live in museum-country Germany, where mobile phone tariffs are outrageously expensive, and nobody can use 5G anyway because their 3 GB tariff would be used up within minutes, and phone get dropped back to 64/16 kbps ISDN speed.

 

[NablaSquaredG]

I use Dell 3P10Y exactly for that, because only 25 fiat on ebay, firmware upgradeable, very very well documented see hxxps://github.com/danielewood/sierra-wireless-modems. Changed it to generic Sierra Wireless, also you'd want QMI. GPS is separate from 4G, I can see no degradation in time measurement when 4G is up.

Interesting - The ASUS Board has an NVMe Slot for Wifi, which I don't need. Maybe I can replace it with such a card at some point

Why only 4G, because I live in museum-country Germany, where mobile phone tariffs are outrageously expensive, and nobody can use 5G anyway because their 3 GB tariff would be used up within minutes, and phone get dropped back to 64/16 kbps ISDN speed.

Although I have unlimited mobile data in Germany - I doubt my provider Vodafone would be happy if I actually make use of it

 

[Rand__]

I use Dell 3P10Y exactly for that, because only 25 fiat on ebay, firmware upgradeable, very very well documented see h
[...]nobody can use 5G anyway because their 3 GB tariff would be used up within minutes, and phone get dropped back to 64/16 kbps ISDN speed.

Hm any comparable options for 5G? T-Com recently upgraded me to 20GB/month... and I wouldnt need it that often..

Maybe 10 times a month considering how reliable Cable is...kidding o/c, its pretty ok at the moment, but only a matter of time until Vodafone manages to break it again.

How does one use the nvme card with a sim? Can't see a slot for it?

 

[Stephan]

Dell 3P10Y is a M.2 "B Key" card. You can't plug that into a Wi-Fi "E Key" socket. There are adapters, see hxxps://thewirelesshaven.com/product-category/mini-pcie-m2-adapters/ which also have a SIM card slot, which let you go from mini PCIe to B key. For PCengines APU4D4 I only needed a simple adapter, because the board already has two SIM slots, with multiplexer controlled by the CPU. PCIe is ignored, this is purely a USB device. Not sure if there is such a thing as E key to B key, especially considering the module's size.

For 5G I presume EM9191, again by Sierra Wireless, would be the best and most compatible with QMI on Linux. But at 10 times the cost, like 400-500 per card. Uff heh, nope...

In addition here in Germany operators are using 700/1800/2100/3500 MHz so you need an antenna suitable for that. For stationary use like at home as fallback, I always recommend poynting.tech antennae, especially for directional. But those are €€€... For omnidirectional like their OMNI-85 you only get a small PCB in black antenna case anyhow, with almost no gain at 700 MHz and modest 2 dBi at above 2 GHz. Depends how far away your nearest base station is.

 

[Rand__]

For 5G I presume EM9191, again by Sierra Wireless, would be the best and most compatible with QMI on Linux. But at 10 times the cost, like 400-500 per card. Uff heh, nope...

Ouch. Maybe not then

 

[Gio]

What was your final idle power consumption in watts using this board?

 

[andrebrait]

What was your final idle power consumption in watts using this board?

Came here looking for this.

I needed two machines that will run 24/7, so they need to have a low IDLE power consumption (< 10W for whole system with SSDs, excluding PCIe cards, no periphery connected). At the same time, I would like them to be powerful enough for the planned tasks and future-proof.

I have the same requirements, but for a NAS. I assembled the following:

Core i5-7600
ASRock H110M-ITX/AC
2x G.Skill 8GB DDR4-2666 RAM
2x WD Blue SATA 500GB SSDs (Docker, VMs)
2x WD Red SATA 500GB SSDs (Cache)
5x WD HGST Ultrastar DC HUS726T6TALE604 6TB 7200RPM HDDs
Corsair SF450 80Plus Gold SFX PSU
1x ASM1166 PCIe 3.0 4x (2x electrical) 6-port SATA controller

With the HDDs spun down and the ASM1166 patched to enable PCIe ASPM, having the CPU package mostly on C8, I see around 19~22W from the wall. With the HDDs on, around 50~55W. While the HDDs are spinning up, around 130W. All measurements taken from the wall.

So, not bad, but I wanted a more power efficient build. Could you share the numbers you're getting with yours?

 

[NablaSquaredG]

What was your final idle power consumption in watts using this board?

Without any periphery and load (only network active) ~95% PC10, >98% C7 and ~9.8W from wall

Real power consumption about 15W with some light load

 

[andrebrait]

Without any periphery and load (only network active) ~95% PC10, >98% C7 and ~9.8W from wall

Real power consumption about 15W with some light load

Ok, so about 1/2 of my build's power draw, but with a 6 core CPU and more advanced video transcoding capabilities.

Is that with or without an UPS? My UPS (an APC Back-UPS BX500MI) adds 8~14W, so my actual idle power draw is around 33W.

 

[NablaSquaredG]

Is that with or without an UPS? My UPS (an APC Back-UPS BX500MI) adds 8~14W, so my actual idle power draw is around 33W.

Yes measured without UPS. I would recommend against APC Back-UP UPS, as modern and especially high efficiency power supplies with active PFC often have issues with non-sine waveforms

 

[andrebrait]

Yes measured without UPS. I would recommend against APC Back-UP UPS, as modern and especially high efficiency power supplies with active PFC often have issues with non-sine waveforms

What sort of equipment would you recommend instead? I don't know where to start looking, in this case.

I can't return the UPS anymore, but if it's really that bad, I can sell it.

 

[NablaSquaredG]

What sort of equipment would you recommend instead? I don't know where to start looking, in this case.

Pure Sine Wave UPS. I only use higher end UPSes (like Eaton 5PX / 9PX, APC SMX / SRT), but APC Back-UP UPS PRO is probably fine.