Hey folks,
This project is WIP, and I will use this thread as a build log.
I will try to precisely describe what I've done and especially which choices I've made and why I've made them to help others if they're in a similar situation.
What's the goal?
I have let things slide for quite some time in my lab, and the time to clean up has come now. And now that I finally I take care of it, I'll do it properly.
I needed two machines that will run 24/7, so they need to have a low IDLE power consumption (< 10W for whole system with SSDs, excluding PCIe cards, no periphery connected). At the same time, I would like them to be powerful enough for the planned tasks and future-proof.
The machines will run the following applications (non-exhaustive, I've probably forgot stuff):
- DHCP (Kea DHCP)
- DNS (BIND9)
- Virtualised opnsense / pfsense as firewall and router
- Wireguard VPN endpoint
- Ruckus virtual SmartZone for Wifi AP(s)
- Print Server
- OpenSM (for Infiniband, as managed switches are far more expensive than unmanaged switches)
- syslog & stat server (rsyslog / zabbix / Prometheus / Grafana)
- Time Server (Stratum 1 via GPS), NTP and PTP master (yes, I only had this idea today...)
The search and research began... And after hours of weighing up pros and cons, I came to the conclusion, that the Intel LGA1700 platform is the way to go.
Why? Ryzen 5000 (AM4) cannot be upgraded, and it seems like it's very challenging to get Ryzen to have a low IDLE power consumption.
It's not really worth buying Rocket Lake (Intel 11000), and it also cannot be upraded.
Meanwhile, LGA1700 has all the latest features (PCIe5.0, DDR5) AND it's upgradable (Raptor Lake)!
Additionally, LGA1700 is unique because of Intel's hybrid architecture. Although I will NOT use CPUs with Hybrid technology (no E-cores, only P-cores) for the time being due to costs (I will explain later), I believe I will upgrade to do so at some point in the future.
Why is the hybrid architecture interesting for this use case? Well, as typical for such use cases, the server will have a constant, but low load all the time. I like to call it "Background Noise". It seems plausible that it would be more efficient to have more energy efficient, slower E-cores to handle this background noise, without having to wake up the high speed, less efficient P-Cores (Performance Cores)
Okay, so that was the justification for the platform itself. Let's get down to the specific hardware choices:
Build’s Name: "Well, that escalated quickly" MicroServer Gen 12 Plus
Operating System/ Storage Platform: Linux (Fedora)
CPU: Intel i5-12400
Motherboard: ASUS TUF GAMING B660M-PLUS WIFI (D5)
Chassis: Chieftec Mesh CS-12B
Drives: 2x 1TB Western Digital WD SN850 PCIe 4.0 NVMe
RAM: 2x16GB Samsung DDR5-4800 non-ECC UDIMM
Add-in Cards: 1x Intel i350-T4, 1x Mellanox NIC (CX354A or CB193A), 1x PCIe to RS232
Power Supply: be quiet! TFX Power 3 Gold 300W TFX
Other Bits: Navilock NL-8003P GPS Receiver, 2x Arctic P8 PWM PST, Arctic Alpine 17
Usage Profile: Homelab management HA Cluster (two machines), Kea DHCP, BIND9 DNS, opnsense/pfsense, Wireguard VPN endpoint, Virtual SmartZone, Print Server, OpenSM, Syslog & Stat Server, Stratum 1 GPS-based PTP & NTP time server
Let's speak about the component choice / justification.
Platform: Done. LGA1700 ticks all the boxes: Low IDLE power consumption, enough power, upgradable -> future-proof
Motherboard: ASUS TUF Gaming B660M-Plus WIFI (D5)
If we use LGA1700, we should make sure that we use all features of it (as we said, we want to be future-proof).
That means we want:
- PCIe 5.0
- DDR5
- 2x PCIe4.0 x4 NVMe
- 2 PCIe Slots
- µATX
Those requirements leave exactly 4 options for motherboards:
ASRock Z690M PG Riptide/D5 -> 200€
ASUS TUF Gaming B660M-Plus WIFI -> 200€
ASUS ROG Strix B660-G Gaming WIFI -> 230€
ASUS ROG Strix Z690-G Gaming WIFI -> 340,59
We're not going to overclock and Z690 doesn't provide anything we need for this build, except higher power consumption. With B660, the x4 slot on the board is 3.0, while for Z690 it would be 4.0.. But as the second slot is only for the Mellanox NIC which does OpenSM, it doesn't need a lot of bandwidth. If we ever need more networking bandwidth, we have a PCIe5.0 slot with an Intel i350-T4!
Two boards left: TUF and Strix. Strix is more expensive and doesn't offer any benefits, so we have a winner: The TUF B660!
CPU: Intel i5-12400
The i5-12400 is a 6C/12T CPU with 6 P-Cores and no E-Cores. Wait - We said that we want E-Cores as they would be very well suited for the tasks this server is supposed to do!
Well... Yes. But: Look at Intel's pricing structure.
I need an iGPU (let's be honest, everything else just sucks. there's a reason why the BMC in every server has a VGA output), so no F-models.
I estimated at least 6 P-Cores are needed for full bandwidth. That means we have 9 Alder Lake models left to choose from.
The cheapest and most basic one is the 12400 for ~190€ with 6C/6T. There are a couple of other i5 models with higher turbo rates (12500 for 220€, 12600 for 255€), but they also don't have E-Cores.
The cheapest model with E-Cores is the 12600K with 6P + 4E, but it costs 308€! That's a whopping +61% just for 4 E-cores and a some higher turbo clock rates...
There is the 12700 which would be more suitable than the 12600K, because it also has 65W TDP. 12700 has 8 P-Cores and 4 E-Cores. But it's 355€, that would be +87% compared to 12400.
I don't believe that the other CPUs are worth their premium, so for now we will use the 12400. BUT we can upgrade at a later point! B660 will be compatible with Raptor lake, so in about 1 to 2 years, once Raptor Lake hits the second hand market,I can just grab some cheap used Raptor Lake CPUs and replace the 12400 (and perhaps faster RAM, as Raptor Lake does 5600 as opposed to the 4800 of Alder Lake)
Chassis: Chieftec Mesh CS-12B
I did a fair amount of research, and that one was the only one that I liked. I wanted to it to be flat (so it can easily be stacked in the "rack") and also have sensible fan positions.. Most cases of this form factor have stupid fan mounting positions.
Drives: 2x 1TB Western Digital WD SN850 PCIe 4.0 NVMe
NVMe SSDs have very low IDLE power consumption, yet they have insane power when you need them.
The SN850 1TB is 110€ per piece and probably the best allrounder on the market right now. Some SSDs are more expensive and beat the SN850 in some disciplines, but then lose out in others (like FireCuda 530, 980 PRO, etc...)
RAM: 2x16GB Samsung DDR5-4800 non-ECC UDIMM
We won't do any memory OC, so we don't need expensive binned gamer DIMMs. 32GB should be enough for everything.
People have made very good experiences with Samsung DIMMs, better than Hynix. And they were only 5€ more per piece, so Samsung it is.
Add-in Cards: 1x Intel i350-T4, 1x Mellanox NIC (CX354A or CB193A), 1x PCIe to RS232
i350-T4 is a classic. I probably don't need to say anything. 1 port for WAN, 1 port for interconnect with the other machine (Inter Peer Link / Sync Link), 1 port for LAN, 1 spare port for whatever.
Mellanox NIC: As I said, I have an Infiniband network in my lab, and I want to upgrade it to 100G. Managed 100G IB switches are quite expensive, while unmanaged ones can be found for okayish prices (additionally, I have an engineering sample Mellanox switch from @Rand__ - it seems to have been a SB7700 ES, but at some point the Xeon-D management board just randomly died. I hope to flash the IB backplane to unmanaged and use it as an unmanaged IB switch)
Anyway: Infiniband requires a subnet manager and managed switches with integrated subnet manager are expensive. Solution: Pop in a 30$ Mellanox NIC into the server that's running 24/7 anyway. Much cheaper.
PCIe to RS232: Accurate Stratum 1 timeserver with GPS requires low Jitter PPS (Pulse Per Second) Signaling. USB is possible, but jitter is a bit of a problem always. Sadly none of the boards listed above has a PS/2 port (which you can connect a GPS receiver to), so serial communication is the best replacement. I'll use a SUNIX SUN2212 / Delock 89444 card. Perfect for the free x1 slot.
Power Supply: be quiet! TFX Power 3 Gold 300W TFX
Smallest good-quality and efficient power supply that can be found. Yes, there is HDPlex GaN PSU which is probably more efficient, but it costs three times as much. I considered using a Supermicro PWS-341P-1H (as it is platinum and I have a couple of spares), but I wouldn't fit properly into the case.
Other Bits:
Navilock NL-8003P GPS Receiver
The GPS Receiver I'll likely buy soon. Not certain yet, but very likely.
2x Arctic P8 PWM PST
One will be a front fan, the other one will be placed right behind the PCIe cards, mainly for the Mellanox card.
Arctic Alpine 17
Very good price/performance cooler similar to Intel stock coolers. Cheaper than boxed vs tray, quiet, good cooling performance.
Alright, so much for the introductory post. If you have any questions or comments, let me know!
This project is WIP, and I will use this thread as a build log.
I will try to precisely describe what I've done and especially which choices I've made and why I've made them to help others if they're in a similar situation.
What's the goal?
I have let things slide for quite some time in my lab, and the time to clean up has come now. And now that I finally I take care of it, I'll do it properly.
I needed two machines that will run 24/7, so they need to have a low IDLE power consumption (< 10W for whole system with SSDs, excluding PCIe cards, no periphery connected). At the same time, I would like them to be powerful enough for the planned tasks and future-proof.
The machines will run the following applications (non-exhaustive, I've probably forgot stuff):
- DHCP (Kea DHCP)
- DNS (BIND9)
- Virtualised opnsense / pfsense as firewall and router
- Wireguard VPN endpoint
- Ruckus virtual SmartZone for Wifi AP(s)
- Print Server
- OpenSM (for Infiniband, as managed switches are far more expensive than unmanaged switches)
- syslog & stat server (rsyslog / zabbix / Prometheus / Grafana)
- Time Server (Stratum 1 via GPS), NTP and PTP master (yes, I only had this idea today...)
The search and research began... And after hours of weighing up pros and cons, I came to the conclusion, that the Intel LGA1700 platform is the way to go.
Why? Ryzen 5000 (AM4) cannot be upgraded, and it seems like it's very challenging to get Ryzen to have a low IDLE power consumption.
It's not really worth buying Rocket Lake (Intel 11000), and it also cannot be upraded.
Meanwhile, LGA1700 has all the latest features (PCIe5.0, DDR5) AND it's upgradable (Raptor Lake)!
Additionally, LGA1700 is unique because of Intel's hybrid architecture. Although I will NOT use CPUs with Hybrid technology (no E-cores, only P-cores) for the time being due to costs (I will explain later), I believe I will upgrade to do so at some point in the future.
Why is the hybrid architecture interesting for this use case? Well, as typical for such use cases, the server will have a constant, but low load all the time. I like to call it "Background Noise". It seems plausible that it would be more efficient to have more energy efficient, slower E-cores to handle this background noise, without having to wake up the high speed, less efficient P-Cores (Performance Cores)
Okay, so that was the justification for the platform itself. Let's get down to the specific hardware choices:
Build’s Name: "Well, that escalated quickly" MicroServer Gen 12 Plus
Operating System/ Storage Platform: Linux (Fedora)
CPU: Intel i5-12400
Motherboard: ASUS TUF GAMING B660M-PLUS WIFI (D5)
Chassis: Chieftec Mesh CS-12B
Drives: 2x 1TB Western Digital WD SN850 PCIe 4.0 NVMe
RAM: 2x16GB Samsung DDR5-4800 non-ECC UDIMM
Add-in Cards: 1x Intel i350-T4, 1x Mellanox NIC (CX354A or CB193A), 1x PCIe to RS232
Power Supply: be quiet! TFX Power 3 Gold 300W TFX
Other Bits: Navilock NL-8003P GPS Receiver, 2x Arctic P8 PWM PST, Arctic Alpine 17
Usage Profile: Homelab management HA Cluster (two machines), Kea DHCP, BIND9 DNS, opnsense/pfsense, Wireguard VPN endpoint, Virtual SmartZone, Print Server, OpenSM, Syslog & Stat Server, Stratum 1 GPS-based PTP & NTP time server
Let's speak about the component choice / justification.
Platform: Done. LGA1700 ticks all the boxes: Low IDLE power consumption, enough power, upgradable -> future-proof
Motherboard: ASUS TUF Gaming B660M-Plus WIFI (D5)
If we use LGA1700, we should make sure that we use all features of it (as we said, we want to be future-proof).
That means we want:
- PCIe 5.0
- DDR5
- 2x PCIe4.0 x4 NVMe
- 2 PCIe Slots
- µATX
Those requirements leave exactly 4 options for motherboards:
ASRock Z690M PG Riptide/D5 -> 200€
ASUS TUF Gaming B660M-Plus WIFI -> 200€
ASUS ROG Strix B660-G Gaming WIFI -> 230€
ASUS ROG Strix Z690-G Gaming WIFI -> 340,59
We're not going to overclock and Z690 doesn't provide anything we need for this build, except higher power consumption. With B660, the x4 slot on the board is 3.0, while for Z690 it would be 4.0.. But as the second slot is only for the Mellanox NIC which does OpenSM, it doesn't need a lot of bandwidth. If we ever need more networking bandwidth, we have a PCIe5.0 slot with an Intel i350-T4!
Two boards left: TUF and Strix. Strix is more expensive and doesn't offer any benefits, so we have a winner: The TUF B660!
CPU: Intel i5-12400
The i5-12400 is a 6C/12T CPU with 6 P-Cores and no E-Cores. Wait - We said that we want E-Cores as they would be very well suited for the tasks this server is supposed to do!
Well... Yes. But: Look at Intel's pricing structure.
I need an iGPU (let's be honest, everything else just sucks. there's a reason why the BMC in every server has a VGA output), so no F-models.
I estimated at least 6 P-Cores are needed for full bandwidth. That means we have 9 Alder Lake models left to choose from.
The cheapest and most basic one is the 12400 for ~190€ with 6C/6T. There are a couple of other i5 models with higher turbo rates (12500 for 220€, 12600 for 255€), but they also don't have E-Cores.
The cheapest model with E-Cores is the 12600K with 6P + 4E, but it costs 308€! That's a whopping +61% just for 4 E-cores and a some higher turbo clock rates...
There is the 12700 which would be more suitable than the 12600K, because it also has 65W TDP. 12700 has 8 P-Cores and 4 E-Cores. But it's 355€, that would be +87% compared to 12400.
I don't believe that the other CPUs are worth their premium, so for now we will use the 12400. BUT we can upgrade at a later point! B660 will be compatible with Raptor lake, so in about 1 to 2 years, once Raptor Lake hits the second hand market,I can just grab some cheap used Raptor Lake CPUs and replace the 12400 (and perhaps faster RAM, as Raptor Lake does 5600 as opposed to the 4800 of Alder Lake)
Chassis: Chieftec Mesh CS-12B
I did a fair amount of research, and that one was the only one that I liked. I wanted to it to be flat (so it can easily be stacked in the "rack") and also have sensible fan positions.. Most cases of this form factor have stupid fan mounting positions.
Drives: 2x 1TB Western Digital WD SN850 PCIe 4.0 NVMe
NVMe SSDs have very low IDLE power consumption, yet they have insane power when you need them.
The SN850 1TB is 110€ per piece and probably the best allrounder on the market right now. Some SSDs are more expensive and beat the SN850 in some disciplines, but then lose out in others (like FireCuda 530, 980 PRO, etc...)
RAM: 2x16GB Samsung DDR5-4800 non-ECC UDIMM
We won't do any memory OC, so we don't need expensive binned gamer DIMMs. 32GB should be enough for everything.
People have made very good experiences with Samsung DIMMs, better than Hynix. And they were only 5€ more per piece, so Samsung it is.
Add-in Cards: 1x Intel i350-T4, 1x Mellanox NIC (CX354A or CB193A), 1x PCIe to RS232
i350-T4 is a classic. I probably don't need to say anything. 1 port for WAN, 1 port for interconnect with the other machine (Inter Peer Link / Sync Link), 1 port for LAN, 1 spare port for whatever.
Mellanox NIC: As I said, I have an Infiniband network in my lab, and I want to upgrade it to 100G. Managed 100G IB switches are quite expensive, while unmanaged ones can be found for okayish prices (additionally, I have an engineering sample Mellanox switch from @Rand__ - it seems to have been a SB7700 ES, but at some point the Xeon-D management board just randomly died. I hope to flash the IB backplane to unmanaged and use it as an unmanaged IB switch)
Anyway: Infiniband requires a subnet manager and managed switches with integrated subnet manager are expensive. Solution: Pop in a 30$ Mellanox NIC into the server that's running 24/7 anyway. Much cheaper.
PCIe to RS232: Accurate Stratum 1 timeserver with GPS requires low Jitter PPS (Pulse Per Second) Signaling. USB is possible, but jitter is a bit of a problem always. Sadly none of the boards listed above has a PS/2 port (which you can connect a GPS receiver to), so serial communication is the best replacement. I'll use a SUNIX SUN2212 / Delock 89444 card. Perfect for the free x1 slot.
Power Supply: be quiet! TFX Power 3 Gold 300W TFX
Smallest good-quality and efficient power supply that can be found. Yes, there is HDPlex GaN PSU which is probably more efficient, but it costs three times as much. I considered using a Supermicro PWS-341P-1H (as it is platinum and I have a couple of spares), but I wouldn't fit properly into the case.
Other Bits:
Navilock NL-8003P GPS Receiver
The GPS Receiver I'll likely buy soon. Not certain yet, but very likely.
2x Arctic P8 PWM PST
One will be a front fan, the other one will be placed right behind the PCIe cards, mainly for the Mellanox card.
Arctic Alpine 17
Very good price/performance cooler similar to Intel stock coolers. Cheaper than boxed vs tray, quiet, good cooling performance.
Alright, so much for the introductory post. If you have any questions or comments, let me know!