NAS "Downgrade" Advice

[ptyork]

So not really a "downgrade", per se. But I'm OP at the moment. Literally. Idling at ~200w with drives spun down and BIOS power set to the most eco-possible modes. And I'd like to halve that if possible.

Here's the minimums:

12 x SATA HDD
4 x NVMe SSD
Extra SSD or NVMe for boot drive
1 x 10GBe SFP+ (prefer DAC, though I could maybe stomach a transceiver on the switch if needed)
GPU/iGPU for Plex Transcode

Currently I'm running a dual Ivy Bridge (SuperMicro X9DRI-F + 2x E5-2650 v2), 8x16GB RAM, 9207-8i-based HBA, ConnectX 3, Nvidia Tesla P4, and I've bifurcated (quadfurcated??) an x16 slot to work with the 4x NVMe SSD's. All runs Proxmox. Hosting 2 major VM's: TrueNAS Scale and an Ubuntu image running a bunch of Docker-based apps. I'm happy. It's quiet. Runs great. But I want to scratch my eco-itch, even if it's going to take half a century to pay off the savings.

Really I need advice on CPU/Mobo combo. It's obviously massive overkill for my needs. Certainly I don't need dual CPU's. I don't need 16 cores and 32 threads (though I won't lie and say I won't miss them). ECC and IPMI and other server-grade bits are probably just "nice-to-have" given my use case (largely a media server with some duty as the working drive for my video editing).

I keep wanting to buy a new consumer board. Zen 4 or Intel 13th gen. But I keep pausing because of the lack of PCIe. I can't seem to figure out the right combination of onboard + add-ons that will meet my needs. They're all about that game, no storage. I've considered a Epyc Rome-based combo (Supermicro H11SSL-i + 7282), but that's unlikely to get me below 100w once I add back in all of the cards. AliExpress stuff gives me the chills.

Anyway, I'm hoping someone here has some great advice. Thanks in advance!

 

[j_h_o]

Unfiltered thoughts:

• Use a power meter and quantify how much each component is contributing.
• Stay away from AMD if you want lower idle power usage. Use Intel iGPU for transcoding. (Do you need the Tesla P4?). Maybe you should split into 2 systems: do a small Lenovo Tiny (with Intel iGPU) for transcoding, and then a much newer Xeon system for your VMs.
• Reduce the number of sticks of RAM.
• Eliminate the HBA. Do you have a SAS backplane, requiring the HBA? Or can you connect everything to SATA ports on a motherboard?
• How many fans do you have running?
• A newer Mellanox card might lower power usage too.

 

[reasonsandreasons]

What's the capacity of your SATA drives? Depending on your storage needs and when you set up the system (X9 makes me think it was a few years ago) you can likely meaningfully consolidate the number of disks you have and eliminate the need for the HBA there.

No matter what you do, get rid of the P4 and go for a TMM node. I can't find numbers for the P4, but the 1080 (shares the same silicon) idles at 6W; a whole Alder Lake system will match that and provide better transcoding performance compared to the Tesla. You can probably shunt the majority of the docker images over there without too much fuss.

If you break out transcoding/Plex into another box, that gives you much more latitude for the storage server. Something like this Xeon D board would give you ECC, IPMI, 10G Ethernet, an integrated 16-port HBA, a m.2 slot, and 16 spare PCIe lanes for your M.2 storage at around ~50W idle and 100W flat out. You're cutting CPU cores down substantially, but if you can offload enough to the TMM node you won't miss them.

 

[ptyork]

Unfiltered thoughts:

• Use a power meter and quantify how much each component is contributing.
• Stay away from AMD if you want lower idle power usage. Use Intel iGPU for transcoding. (Do you need the Tesla P4?). Maybe you should split into 2 systems: do a small Lenovo Tiny (with Intel iGPU) for transcoding, and then a much newer Xeon system for your VMs.
• Reduce the number of sticks of RAM.
• Eliminate the HBA. Do you have a SAS backplane, requiring the HBA? Or can you connect everything to SATA ports on a motherboard?
• How many fans do you have running?
• A newer Mellanox card might lower power usage too.

Thanks for the thoughts!

* AMD v. Intel...thanks for the input. I wasn't aware that the idle power was significantly different between the two
* Fans...including the CPU coolers, 13 total. 9 are low RPM Noctua. Consumption is < 1.2w each (far less for the smaller ones). So maybe 10W. Hard to remove much of that.
* RAM...yeah, I'll definitely have fewer in a new system. In this one, I've maxed out the DIMM size at 16GB and it's finicky about RAM configuration, so I'd need to halve the RAM to reduce the number of sticks. But, yeah, that's another 12W or so that I can trim easily.
* HBA...this is harder. I've got 14 SATA connected drives ATM (a 2x for a mirrored boot). The mobo support 2x SATA 3 + 4 via a mini-SAS connector. I've got these filled. So the HBA handles the other 8. So another 10W there. I AM curious about the potential for an SATA card vs. a true HBA, though.
* Connect X...yeah, definitely a potential saving there, though I'm not sure what newer Mellanox would be better. May consider trying the Intel x710, though, as they seem to have dropped significantly in price since I last looked.
* P4...only for transcoding. I thought I'd use it for some ML training, but I have Jetson's for that, and really I do most of that on my desktop. Thus I'm hoping to drop it and go w/ iGPU.

Good thought on running a separate mini-pc for the Plex and transcoding. I got a little Beelink N100 for BF that I have stuck in my rack that could work well for that. Could open up additional possiblities for the NAS. Will have to think on it.

 

[ptyork]

What's the capacity of your SATA drives? Depending on your storage needs and when you set up the system (X9 makes me think it was a few years ago) you can likely meaningfully consolidate the number of disks you have and eliminate the need for the HBA there.

Those are pretty new. 8x14TB + 4x4TB. Going to upgrade the 4TB's soon. I keep a lot of media. Most is even legal.

No matter what you do, get rid of the P4 and go for a TMM node. I can't find numbers for the P4, but the 1080 (shares the same silicon) idles at 6W; a whole Alder Lake system will match that and provide better transcoding performance compared to the Tesla. You can probably shunt the majority of the docker images over there without too much fuss.

I've definitely been somewhat transfixed with the notion of an all-in-one for this, but, yeah, need to take off those blinders.

Something like this Xeon D board would give you ECC, IPMI, 10G Ethernet, an integrated 16-port HBA, a m.2 slot, and 16 spare PCIe lanes for your M.2 storage at around ~50W idle and 100W flat out. You're cutting CPU cores down substantially, but if you can offload enough to the TMM node you won't miss them.

What a funky board! Hmm. Yeah, if I get a couple of 32GB DIMMs and bare metal TNS on this, I'm likely not to really feel a difference. I rarely touch 15% CPU utilization as is unless I'm "playing" with other VMs. Interesting...

Thanks for the food for thought!

 

[homeserver78]

Not a direct response to your question, but since I just yesterday measured the power draw of my new NAS machine, I can give you a comparison point.

My machine:

• PSU: Be quiet! Pure Power 10 CM 400W (Model "L10-400W")
• CPU: AMD Ryzen 5 Pro 4650G
• MB: ASRock B550M-HDV (I don't recommend this board for a server: too few PCIe & M.2 slots and a single 1 Gbit/s Realtek NIC, but it's what I have)
• RAM: 2x Mushkin DDR4 - 16 GB - 3200-CL22 - Single Proline ECC (MPL4E320NF16G18)
• SSD: Crucial P3 4TB M.2 2280 PCI-E x4 Gen3 NVMe SSD (CT4000P3SSD8)

Idle power draw at the wall measured at 22-23 W using a cheap but power-factor-aware meter "Model EMT707CTL". If I add an Intel P4610 U.2 NVMe the measured idle power draw increases by 4-5 W, which seems about right.

Edit: Updating UEFI and setting AMD CBS -> CPU Common Options -> Global C-state Control to Enabled reduced idle power draw with the P4610 from 27 W to 23 W. In addition, setting AMD PBS -> PM L1 SS to <can't remember, but both sub-states active> reduced power by an additional watt to 22 in total, including the P4610 NVMe U.2 drive.