Looking for a recommendation for a CPU and motherboard for a power-efficient (at idle) home NAS

[piranha32]

Hi!

I'm in the process of designing my home NAS server. Its primary function will be disk storage, but I also want to use it to run several services.
NAS will be used for file storage, hosting video projects for editing off the server (I don't need super smooth scrubbing, though), and backups. The disks will also be used by locally running lightly loaded databases. I'm planning to start with ZFS on mechanical drives (30-40TB) with a couple of SSDs for caching, which I want to upgrade to SSDs when the big drives will become more affordable. The databases and ZFS like to have a lot of memory, so I want to start with 32GB of RAM, with option to upgrade to 128GB or more.
I have fiber in the house for 10G local networking, installed mainly for use with the NAS.

As for services, the server will be used to consolidate services which currently are running on several other machines: PostgreSQL, InfluxDB, MongoDB, Home Assistant, mosquitto, nodered, Jellyfin, and several others.

My initial plan was to use ProxMox for the OS, however all the services that will be migrated run in docker containers, so I may not have a need for full-blown VMs.

Most of the time the server will do almost nothing, and I'd like to minimize power consumption when at idle or light load. At the same time, it should be able to push close to the full bandwidth of the 10G network.
It's not a big problem to build a server optimized for one of the requirements, but I'd like to find a compromise solution, with more weight put on power efficiency at idle.

For the enclosure I found such a case, which will fit in my rack: 4U 19" Short 15.25" 8 Hot-Swappable SATA / SAS Drive Trays IPC-4U08S (PLinkUSA / RackBuy)

What motherboard and CPU what you recommend for such build?

 

[kpfleming]

I bought a used (from thread on this site) H11SSL-i and EPYC 7232P combination, populated it with 128GB of DRAM, and its idle power consumption is relatively low (around 80W, including SSDs and one spinning rust drive).

 

[CyklonDX]

while i won't recommend any mobo or cpu; i'll mention this

sas hdd's by default do not spin down (which will result in higher power utilization) - There are some tools that allow you to do so; but its not automatic as with normal sata disks - you would potentially need to script something out for cheap sas 8+TB disks.
I would also recommend 800GB SAS SSD's for l2arc cache for zfs (they have endurance of 36PBW - so they work great for cache - while its going to be quite easy to use up your endurnace on normal ssds/nvmes as cache over a year or 2 depending on usage ~ like Jellyfin, transcode going to eat your ssds/nvmes - put your system on nvme tho much better.)

You might be able get away with 8-12-16c ryzen system, the P states, as well as high clock speeds on demand would help keeping the usage low. (5900x can go as low as 550MHz, and can support up to 128GB of ram - it works with even old ddr4 2133MHz ECC Regs - under linux you can control the clock turbo/boost limits too if you want to, and even schedule different tuned profiles by time, so it can downclock everything into lowest P states including memory.)


There isn't a lot space inside anyway. - i doubt any nice dual cpu board will fit. (unless thats just the perspective the pictures are made.)
I would also recommend A4000 for jellyfin transcode, gpu transcode will be faster - and use up less power overall. A4000 power states are quite nice. At idle it bearly draws 15W (if you swap cooling on it -its likely you can potentially get lower wattage under load.)

for your network, i'd recommend you get intel x520 (sfp) or x540 (rj45) -- dell branded ones come in with fan.

 

[unwind-protect]

Questions:

How likely are you to want more than 128 (maybe 192) GB of RAM? You'll have to move to registered RAM platforms.

Are you happy with 2.5 GB/s ethernet? If not you have to plan for PCIe lanes.

 

[piranha32]

sas hdd's by default do not spin down (which will result in higher power utilization) - There are some tools that allow you to do so; but its not automatic as with normal sata disks - you would potentially need to script something out for cheap sas 8+TB disks.

I've always been afraid of spinning disks down, to not make them fail prematurely.
In my case there will have databases receiving a slow but constant stream of data to log. Unless I will give them a dedicated disk drive, the array will not have

You might be able get away with 8-12-16c ryzen system, the P states, as well as high clock speeds on demand would help keeping the usage low. (5900x can go as low as 550MHz, and can support up to 128GB of ram - it works with even old ddr4 2133MHz ECC Regs - under linux you can control the clock turbo/boost limits too if you want to, and even schedule different tuned profiles by time, so it can downclock everything into lowest P states including memory.)

Do you have any good references to read about deeper dive into profiles tuning?

There isn't a lot space inside anyway. - i doubt any nice dual cpu board will fit. (unless thats just the perspective the pictures are made.)

The case is short, but that's max what I can fit in my rack. According to the specs max motherboard size is CEB (a bit bigger than standard ATX). It doesn't look like I can mount there a nice, large server MB.

And thanks for all the other advices.

 

[piranha32]

Questions:

How likely are you to want more than 128 (maybe 192) GB of RAM? You'll have to move to registered RAM platforms.

Are you happy with 2.5 GB/s ethernet? If not you have to plan for PCIe lanes.

It's hard to tell right now if I will need more than the 128GB of RAM. Time will tell. However, I don't expect to need more than 192GB.
I haven't used 2.5G ethernet, but judging by my experiences with 1GbE, it probably will not be the speed upgrade I'm looking for.

 

[CyklonDX]

Do you have any good references to read about deeper dive into profiles tuning?

CPU frequency scaling - ArchWiki

wiki.archlinux.org

I would advise creating 2-3 sh scripts inside

power-saving.sh
balanced.sh
high-performance.sh


within them you would want to have following

# Governor
cpupower frequency-set -g conservative
# Max Clock Freq
cpupower frequency-set -u 1200MHz
# Min Clock Freq
cpupower frequency-set -d 550MHz

*(and each one to match frequencies, and governor you would want them to have)

then you can use crontab to schedule them to run at specific time

0 22 * * * /scripts/power-saving.sh
0 8 * * * /scripts/balanced.sh
0 13* * * /scripts/high-performance.sh

// would recommend checking all governors to math your needs. Its important that you have all turbo/boosts, and P states turned on in bios, so you can control powerstates through ACPI driver.
(some governors do more than just freq scale on gpu - powersave governor changes power states of memory too as example - depending on acpi driver.)


It would also be worth to use nvidia-smi to put the gpu into P states when in power-saving mode.

NVIDIA GPU Power Capping

How to Lock Nvidia GPU Clock and Power Limit Under Windows with Nvidia-smi

The Nvidia System Management Interface (nvidia-smi) is a command line utility, based on top of the Nvidia Management Library (NVML), intended to aid in the management and monitoring

cryptonews.net

You can also throw those into tuned profiles (you would need to create new ones tho - or edit existing)
You can find the files typically in /usr/lib/tuned/ [ profile name ]


or if you plan on getting unraid, just use Tip&Tweaks plugin (and thats build in)



// Note: running lz4 compression on zfs will also give you better power efficacy on the drives.

It's hard to tell right now if I will need more than the 128GB of RAM. Time will tell. However, I don't expect to need more than 192GB.

in my exp once you get to 128GB of RAM you will always find yourself wanting more.
But as long as your L2ARC is fast enough you can rely on it for ZFS cache rather than on physical memory and save more memory due to that - 32G is mid point in my perspective if you have big, and fast enough l2arc cache for 1250MB/s goal.
2-4 sas ssd's do the job, or single nvme pcie4 -- in terms of nvme micron 3400 as mid point enterprise, decent endurance(you can under-provision it and get much better endurance), and most importantly really good temps.