Looking for an efficient CPU+MB

[UnknownPommes]

Hi,
so i am looking for a cpu/mb for over two weeks now, but there is just so much to choose from and i dont want to buy sth i regret when its too late.
I am currently planing a home server that will handle everything that needs to be on 24/7 (cctv, nextcloud, HA) to be able to switch my main server off during the night to safe power and be able to run those things off battery during power outages.

What should be running:
- Proxmox as OS
- docker - 15 low usage containers (pihole, unifi controller, nextcloud, HA......)
- TrueNas VM thats going to handle storage for said containers
- 3-4 Linux VMs also no heavy usage

Requirements:
- ~70w at avg (with ~25% usage) & not more than 50w idle, system from wall, but the lower the better
- cpu & mb around 400€/$
- ecc support
- min 2 pcle x8 slots
- min 6 sata
- sfp+ 10g Port (alternativ +1 pcle x4)

Components i already decided/bought:
Case: Intertech 40255 4u (should fit basically anything)
Storage: 5x Crucial mx500 1tb (i know consumer but with my avg TBW/year i should be good to go for 4-5 years)


If you were building the above, what would you go with?




What i have looked into so far:
- Supermicro/AsRock Xeon D Soc like the X10SDV-7TP4F
Pro: has basically all features i want
Neg: Cant upgrade cpu, super expensive for the performance (1000€+ with the 8c), 4c is probably to weak, not sure and still 600€

- Ryzen like 4650g or 5600
Pro: cheap, high performance, upgradable
Neg: ecc support only udimm one bit, power consumption

- Other Xeon (v4,v5,v6, scalable)
So much to choose from but regarding real power consumption i couldnt find much besides tdp which doesnt really say much about idle.
But flexibility so if you find a xeon platform that meets the requirements that would be my favorite.
thx for reading

 

[Wasmachineman_NL]

The Ryzen APUs have amazing power efficency because they're based on mobile silicon.

 

[joeribl]

In Europe the Fujitsu mainboards are well known for their power efficiency. They are made by Kontron now.

There is a board with a C246 board. There is nothing for socket 1200 as they skipped this generation. They are launching the socket 1700 boards soon, but unknown yet as how efficient they are. My suspicion is that it will be not that good due to the CPU consuming a lot.

 

[UnknownPommes]

The Ryzen APUs have amazing power efficency because they're based on mobile silicon.

yeah, can you recommend good mainboards for it, also could that with only single bit correction be a problem for a zfs pool?

 

[RolloZ170]

My suspicion is that it will be not that good due to the CPU consuming a lot.

only on high loads it can consum a lot, this is not a permanent state.
have heared on high loads any CPU rises power consumtion.

 

[Wasmachineman_NL]

yeah, can you recommend good mainboards for it, also could that with only single bit correction be a problem for a zfs pool?

Big bucks: any of the ASRock Rack boards.
If you can even find one: Asus Pro WS X570-ACE

ASUS seem to support ECC on all their 3rd gen AM4 boards: [Motherboard] List of AMD Ryzen™ Processors with support for ECC memory | Official Support | ASUS Global

So does the X570 Taichi: Planning a Ryzen 5900x workstation-style build with ECC memory

 

[mrpasc]

Hi,
so i am looking for a cpu/mb for over two weeks now, but there is just so much to choose from and i dont want to buy sth i regret when its too late.
I am currently planing a home server that will handle everything that needs to be on 24/7 (cctv, nextcloud, HA) to be able to switch my main server off during the night to safe power and be able to run those things off battery during power outages.

What should be running:
- Proxmox as OS
- docker - 15 low usage containers (pihole, unifi controller, nextcloud, HA......)
- TrueNas VM thats going to handle storage for said containers
- 3-4 Linux VMs also no heavy usage

Requirements:
- ~70w at avg (with ~25% usage) & not more than 50w idle, system from wall, but the lower the better
- cpu & mb around 400€/$
- ecc support
- min 2 pcle x8 slots
- min 6 sata
- sfp+ 10g Port (alternativ +1 pcle x4)

Components i already decided/bought:
Case: Intertech 40255 4u (should fit basically anything)
Storage: 5x Crucial mx500 1tb (i know consumer but with my avg TBW/year i should be good to go for 4-5 years)


If you were building the above, what would you go with?

Given your requirements especially for power consumption and your budget I‘d go the „used components“ way, at least for CPU.
Myself is happy with Xeon Scalable Gen2 Silver 4208 (got it for 150€ used a year ago) on a Asrock Rack EPC621D8A (got it used for 300€, new ones are around 425€). Enough PCI-E Gen3, plenty of SATA. DDR4 RDIMM 2400 ECC can be found cheap used.
With Intel X520 10G, Dell HBA 330, 4 Samsung PM883 SATA and 2 Samsung PM983 NVME it draws around 55W idle (Win Server 2022) from the wall, might be some more with Proxmox.

 

[UnknownPommes]

Given your requirements especially for power consumption and your budget I‘d go the „used components“ way, at least for CPU.
Myself is happy with Xeon Scalable Gen2 Silver 4208 (got it for 150€ used a year ago) on a Asrock Rack EPC621D8A (got it used for 300€, new ones are around 425€). Enough PCI-E Gen3, plenty of SATA. DDR4 RDIMM 2400 ECC can be found cheap used.
With Intel X520 10G, Dell HBA 330, 4 Samsung PM883 SATA and 2 Samsung PM983 NVME it draws around 55W idle (Win Server 2022) from the wall, might be some more with Proxmox.

thx, how much does it draw with a couple vms running like 10-15% usage?

 

[Stephan]

LGA 3647 seems overkill for what he wants. Look on ebay for a used SFF (small form factor) or tower system with 12xx E3 Xeon and C236 chipset. Max memory is limited to 64 GB if you put 16 GB UDIMMs in there and idle power should be 50-60 watts depending what is added in components. Edit: ECC is always better than nothing, harmonizes well with ZFS' paranoid way of storing and checking things. Make sure to run rasdaemon and have a regular check in place to get notified of corrected errors. 1-3 per month should be a normal baseline.

 

[UnknownPommes]

LGA 3647 seems overkill for what he wants. Look on ebay for a used SFF (small form factor) or tower system with 12xx E3 Xeon and C236 chipset. Max memory is limited to 64 GB if you put 16 GB UDIMMs in there and idle power should be 50-60 watts depending what is added in components. Edit: ECC is always better than nothing, harmonizes well with ZFS' paranoid way of storing and checking things. Make sure to run rasdaemon and have a regular check in place to get notified of corrected errors. 1-3 per month should be a normal baseline.

yeah it probably is, but as long as i stay around the power limit, i wouldnt say no to more performance and a newer platform with more upgradability and it seems to be like 400-500€ so also fine.
Edit: shouldnt be newer also mean more performance per used watt?

 

[mrpasc]

thx, how much does it draw with a couple vms running like 10-15% usage?

Sorry, can’t give any numbers as it is not running any Hyper-V. With medium load it goes up to around 90W.

 

[UnknownPommes]

Sorry, can’t give any numbers as it is not running any Hyper-V. With medium load it goes up to around 90W.

oh ok and full power?

 

[mrpasc]

Somewhat around 115W

 

[mrpasc]

yeah it probably is, but as long as i stay around the power limit, i wouldnt say no to more performance and a newer platform with more upgradability and it seems to be like 400-500€ so also fine.
Edit: shouldnt be newer also mean more performance per used watt?

Yes, it does. And, LGA3647 or LGA4189 does not mean high power consumption per default. Most people think of the Gold and Platinum monsters which do suck a lot of watts from the wall, but it’s called „scalable“ and you can scale down with Silvers. Bronze are useless, indeed.
And, even the new Xeon D-1700/2700 ice lake are based on the SP3 for that reason.

 

[UnknownPommes]

Yes, it does. And, LGA3647 or LGA4189 does not mean high power consumption per default. Most people think of the Gold and Platinum monsters which do suck a lot of watts from the wall, but it’s called „scalable“ and you can scale down with Silvers. Bronze are useless, indeed.
And, even the new Xeon D-1700/2700 ice lake are based on the SP3 for that reason.

yeah thx i am probably going with scalable
i just found a 4108 for 70€ (no ES) and i mean if i want to change sth in the future i can go and get sth overkill like gold or plat and use it as another high power vm host.
Now there is only the question regarding board...

 

[mrpasc]

I stay with my recommendation of the Asrock Rack EPC621D8A. If you want to save another some idle Watts you can search for workstation boards without IPMI… Maybe a board of a Dell Precision T7820 workstation, but then you should double check the pinouts as Dell does it’s own regarding power supply….

 

[UnknownPommes]

I stay with my recommendation of the Asrock Rack EPC621D8A. If you want to save another some idle Watts you can search for workstation boards without IPMI… Maybe a board of a Dell Precision T7820 workstation, but then you should double check the pinouts as Dell does it’s own regarding power supply….

yeah i am currently looking through different boards, is the ipmi/remotemanagement on your board html or still over the old java thing?
Also would i be good to go with a normal "consumer" atx psu for now?

 

[mrpasc]

It is AST2500, so HTML interface. See BMC user manual
Standard ATX PSU is an option. Do some math to not to “overprovision”, most people tend to use too “large” PSU and loose efficiency. Go for platinum rated ones.

 

[UnknownPommes]

It is AST2500, so HTML interface. See BMC user manual
Standard ATX PSU is an option. Do some math to not to “overprovision”, most people tend to use too “large” PSU and loose efficiency. Go for platinum rated ones.

thx, yeah i also did that it the past, the thing is i got a 750w corsair one laying around and my idea was to just use it for now until i get a redundant one in a couple months

 

[Stephan]

I run three EPC621D8A, with 8259CL CPU (210 W TDP, 24C/48T), ConnectX3 NIC (Oracle 7046442 reflashed to MCX354A-FCBT), 192 GB RAM (6x32GB Samsung B-Die DDR4-2666), two Lite-On SATA 120 GB SSDs in RAID1 for OS boot, one Micron Max SATA SSD 960 GB for data, Noctua NH-U12S DX-3647, Seasonic Prime Titanium Fanless 600W ATX 2.4, one more Noctua NF-A12x25 PWM 120mm to cool chipset, NIC and part of VRM heatsink. With IPMI set up and connected, system idles at C6 (CPU and package) drawing 50 W. Add the wrong PCIe NVME or PCIe card and that will shoot up to 80 W because then idle will be in C2 and no longer C6. Edit: When compiling e.g. a Linux kernel with 48 threads, power consumption of the entire system will jump to 300 W.

Some caveats... Bought 18 RAM sticks from German ebay seller, 3 defective as detected by rasdaemon/stress-ng and Memtest by Passmark. One with outright ECC errors, the other two with M2M ("mesh2mesh") errors within the CPU itself, but caused by the RAM module. These appeared to be errors between cores and the memory controllers. So deep within the chip. At least that is what I deducted from Linux kernel comments when RAS functionality for Intel Scalable was added. Good luck getting any help with such issues... ;-) Also no support from Samsung to exchange a module that is not even 3 years old yet. Well, their SSD firmwares suck, too. Would probably switch to SK Hynix or Micron next time. One board had a defective RAM slot. Asrock Rack service will be through merchant and only if that is no longer possible, will Asrock Rack Netherlands work directly with you via an RMA. They exchanged the board for a working one within 1 or 2 days. The board receives no more updates for BIOS and ASpeed BMC. So whatever bugs remain, will remain. This is different with Supermicro, but most Supermicro boards will require a patched BIOS and patched VRM to enable off-roadmap CPUs. Like X11SPL. Also cheaper than Asrock. That C621 chipset is a nice space heater. I guess 15 W are going up into warm air from this chip alone. On the EPC621D8A this chip will power 4x Intel X722 1 Gbps ethernet connections. Which really shouldn't produce so much heat, but I guess they just left the 10 Gbps circuits on which are active in higher tiers of the C62x.

Bought two CPUs from @tesla100 really the packaging king. ;-) Third CPU came from chinese ebay seller sell-cpu, also working and was nice enough to send picture of CPU beforehand. If you buy from ebay USA chances seem much higher to get half defective CPUs with dinged corners from a recyclers trash bin. So beware. Got two such CPUs, even if I explicitly told the seller what I wanted (flawless condition). I am still waiting for 150 EUR import tax refund from Finanzamt Köln-Bonn. You can read the full story elsewhere. I am just saying, careful with 3647.

Btw used Noctua NT-H1 thermal paste because it flows better on such large IHS like 3647 than H2. Watched their spot sizing video and applied it this way. CPU even during maximum stress will now only reach 55 deg C with the Noctua cooler, while being barely audible with fans at 1300 rpm.

Sorry for this much text, but guessing you'd be interested in some info. ;-)