What CPU for LOW power ceph cluster

linuxguy

New Member
Jun 20, 2022
3
2
1
Hey guys!

I want to build a low power ceph cluster!

Rejected Xeon "X79" LGA2011 etc

I've considered getting some old Xeon LGA2011 or whatever servers from AliExpress which would make cost low, ECC RAM abundant and IO aplenty. But they'll probably idle at at least 60W each for just the motherboard and CPU. I'm targeting 10W per node (excl hard drives and switches) but I would still be satisfied if it's 30W per node.
I don't know for sure, but I sadly believe this will use too much power.

Craft Computing built a (Proxmox, not Ceph) cluster with LGA2011 parts... unfortunately he didn't give power consumption excl drives or down-clock the CPUs or under-volt or anything like that. But my sense is they weren't the kind of low idle power that I'm seeking.

Low idle power for UPS on South African power grid
I'm in South Africa and we have routine load shedding from our power grid, where we must keep our servers and workstations and screens and routers etc running for a minimum of 2.5 hours on battery. Additionally it's not possible for me to run a generator or install solar panels etc.

It's much easier to keep everything running if I select equipment with low idle power consumption.

I'm trying to decide between Aliexpress 4x 2.5GbE with one of these CPUs
* J4125
* N5095
* N5105

OR a reburbished i3 6th gen...

I know all of these CPUs idle with low power usage.
But I've not been able to find much info about whether any of these CPUs' performance will be sufficient to run Ceph OSD's. But from my research I get the feeling the N or i3 CPUs will be fine. I have no idea about the J4125 though...?

Hey @Patrick thanks for your TinyMiniMicro videos, they're awesome!!! Have you tried running a ceph cluster with J4125 yet?

Aside from my CPU performance concern, I'm concerned that these aluminum enclosures lack SATA ports and PCI-E ports. They generally have only 1 SATA port for a 3.5" HDD. Now I could get one of these https://www.amazon.com/Internal-Non-Raid-Adapter-Desktop-Support/dp/B07T3RMFFT/ And then connect up plenty of HDDs. But then I'd lose the ability to have a SSD for Ceph journalling. But is the M.2 NVME port even any good on these boards?

The only IO config of the Aluminum 4x 2.5GbE that fully makes sense, is if I run a NVMe for ceph journal and connect only 1 SATA 3.5" drive to each node, and then add more nodes if I need more space. (which I could do) But that's a high cost per SATA port!

The other IO config of the Aluminum 4x 2.5GbE that ALMOST makes sense in my head, is if I run a NVMe for ceph journal and connect 1 SATA 3.5 PLUS 2 (or more) SATA 3.5 via USB 3.0. The performance should be good. But will those USB ports be stable and good...?? (scared face)

Which makes me think I might be better served getting a refurbished 6th gen i3. It should come with at least 2 SATA ports (enough for me) and I should hopefully be able to connect a decent NIC AND a NVMe SSD via a PCI-E to M.2 riser (or even a SATA SSD if I have to)

Interesting links
This NIC looks interesting but probably requires PCI-E bifurcation and probably won't work on 6th gen i3?

Aliexpress (search for 2.5 J4125 or "2.5 N5095" etc)

No ECC RAM
I've decided to not worry about using ECC RAM.

Ceph doesn't mention ECC ANYWHERE in their documentation, and says that their software is designed to run on cheap commodity hardware.
So I believe it's safe to run RAM tests and then assume ceph will be fine without ECC.
 
Last edited:
  • Like
Reactions: Brian Puccio

bayleyw

Active Member
Jan 8, 2014
167
53
28
The tricky thing is you're looking at idle power consumption, which is going to be dependent on the firmware tuning and peripherals, not the chipset. What sort of configuration are you looking at per node? What will you be using the cluster for?
 

Stephan

Well-Known Member
Apr 21, 2017
532
330
63
Germany
No ECC is imho a mistake. Your assumptions will bite you in the when you want to open that Excel file from 5 years ago, which you no longer can't because a bit flip corrupted it when it was last saved. Of course this can be somewhat compensated with deep backup depths possible through borg backup or some tape solution.

Recently I was in the same boat as you (Proxmox, again, but also will be Ceph) and decided to use three 3647 boards for it. ~50 watts idle with ConnectX3 dual 56 Gbps, 192 GB RDIMM DDR4, active Aspeed OOB management, and a single Micron MAX 5300 SATA drive. The Xeon can idle nicely in C6, which it will not if certain PCIe devices are present. Which are not necessarily add-in cards but could also be sub-par onboard NICs and stuff as I understand from reports. I have found instrumentation for hardware errors in Linux superior on this mainstream enterprise platform.

If you are that power constrained, and go low-power (EDIT, ment: low-power and without ECC), I recommend to run Testmem5 with Universal-2 profile at least for 6 hours per node to check for problems. See MemTestHelper/DDR4 OC Guide.md at oc-guide · integralfx/MemTestHelper
 
Last edited:
  • Like
Reactions: Brian Puccio

Sean Ho

seanho.com
Nov 19, 2019
412
165
43
Vancouver, BC
seanho.com
ECC is a somewhat orthogonal issue to the fs (zfs, btrfs, ceph, etc.). It's great to have checksumming fs, but memory errors can corrupt the input data prior to calculating the checksums.
 

unwind-protect

Active Member
Mar 7, 2016
174
36
28
Boston
I would just use a low core count Ryzen 3 in a board with ECC (unbuffered). I don't see a reason to make this complicated.

I do not see how to make the step to more PCIe lanes for I/O without ending up with high power consumption or high price, or both.
 

Stephan

Well-Known Member
Apr 21, 2017
532
330
63
Germany
I know energy prices are starting to really itch people around the globe. For the sake of a comparison I just made a measurement using a "Delock 11827" Tasmota energy socket of the system I was talking about.

51 watts idle with SSH running on a modern Linux and not much else:

Capture.PNG

System is:
  • Asrock Rack EPC621D8A, one 1 Gbps ethernet connected
  • Xeon Platinum 8259CL, package and core idling at C6
  • Aspeed 2500 active and connected
  • Arch Linux booted from an old 4 GB Kingston USB thumb drive
  • ConnectX3 (Oracle 7046442 reflashed to Lenovo firmware, Mellanox stock config), nothing connected
  • Noctua NH-U12S DX-3647 and one extra Noctua NF-A12x25 PWM running at 700 rpm
  • Seasonic Prime Titanium Fanless 600W ATX 2.4
  • 192 GB RDIMM DDR4 2666 RAM 6x 32 GB Samsung M393A4K40BB2-CTD6Y
  • Two Micron 5300 MAX 960 GB, idle
  • Built on a Streacom BC1 V2 Open Benchtable
With stress testing I am able to reach 250-300 Watts total power consumption, depending on exact test. But at 51 watts its not THAT far from OP's 30 watts upper target. But here I get a ton of expandibility with PCIe slots, a bunch of ECC-RAM, 56 Gbps ethernet, and a 24 Core 48 Thread CPU. Most importantly for me, hardware error reporting works very well, thanks to the big cloud shops who implemented it in Linux.
 

vincococka

New Member
Sep 29, 2019
25
12
3
Slovakia
@Stephan How did you achieved so low idle power consumption? Any special OS configuration/drivers tuning? Fresh kernel from GIT ?
Got AMD Epyc Milan 7313p + H12SSL-i, and Intel 3rd 6326 + X12SPL-F - both systems idle at least at 100W.
 

vincococka

New Member
Sep 29, 2019
25
12
3
Slovakia
- no GPU
- i hope that RHEL 8.5 / 8.6 without any app. running can sleep fine
- idling at

I'll try to measure consumption on "barebone" setup without any PCIe device attached
 

Stephan

Well-Known Member
Apr 21, 2017
532
330
63
Germany
I attribute low idle power to: CPU making it into C6 for package and cores, highly efficient PSU, no NVME but somewhat efficient SATA SSDs, low-RPM coolers. Running latest linux kernel and microcode, not sure if that makes a difference. I'll probably get a 5-10 watt increase once the ConnectX3 has both links up and running.
 
  • Like
Reactions: RolloZ170

zer0sum

Well-Known Member
Mar 8, 2013
715
374
63
Maybe try a Ryzen build? :D

I just built a tiny server and it idles at ~38W

  • Operating System: Proxmox
  • Case: ZZAW C2 - 13.8L and 12.83 x 6.34 x 10.31 inches
  • Mainboard: Asrock Rack X470D4U (MATX)
  • CPU: Ryzen 5600X ( 6C / 12T)
  • Cooler: Artic Liquid Freezer II 120mm AIO, top mounted exhaust
  • Case Fan: Noctua 120mm as bottom intake
  • Memory: 2 x 32GB UDIMMS
  • PSU: Corsair SF450 SFX (80+ Gold certified)
  • Storage: AOC-SLG3-2M2 PCIe Add-On card, with an Intel 660p 2TB NVME and ADATA SX8200 1TB NVME
  • Network: AOC-STGN-I2S rev 2.00
 
Last edited:

RedX1

Active Member
Aug 11, 2017
120
117
43
A low power surprise.



I did not have any reason to construct a workstation PC - it just happened.


I had a number of components from previous projects and updates, but no Motherboard.

I got intrigued by a "For-Parts" offer of a Supermicro X11SCA-F that the vendor could not get to boot. I made a bid and succeeded in obtaining it at a very, very low price. I did not expect to have much success.

I did some research and discovered that this MB can be very tricky to configure, there are several STH and Reddit threads about this MB. After a few attempts with different Bios options and Jumper settings I got this board to boot with an i3-8350k and the stock intel cooler.

I completed the project in a Fractal Define 6 case (3 Stock Fans) with a Supermicro PWS 665SQ PSU, it has 32G of Hynix Ram and a 1Tb Toshiba XG6 Nvme drive. It uses the on-board DP Video output. Nothing else is installed.

I was not expecting too much and installed W-11 with all the Bios Settings at default.


Screenshot 2022-06-30 121734.png


I was extremely surprised to find the very low power consumption at idle. With W-11 running It seems to sit around 21-22.5 Watts, even with the BMC enabled, I would expect this to be somewhat lower with the BMC disabled.

IMG_2957.JPG

I know that this MB can be costly at retail prices and maybe this is just another data point with other combinations producing even better outcomes. But in today’s world of ever-increasing energy prices, sometimes you just get lucky - this has been a very pleasant surprise.


Have fun with you search.





RedX1
 
  • Like
Reactions: Sean Ho