Hey guys!
I want to build a low power ceph cluster!
Rejected Xeon "X79" LGA2011 etc
I've considered getting some old Xeon LGA2011 or whatever servers from AliExpress which would make cost low, ECC RAM abundant and IO aplenty. But they'll probably idle at at least 60W each for just the motherboard and CPU. I'm targeting 10W per node (excl hard drives and switches) but I would still be satisfied if it's 30W per node.
I don't know for sure, but I sadly believe this will use too much power.
Craft Computing built a (Proxmox, not Ceph) cluster with LGA2011 parts... unfortunately he didn't give power consumption excl drives or down-clock the CPUs or under-volt or anything like that. But my sense is they weren't the kind of low idle power that I'm seeking.
Low idle power for UPS on South African power grid
I'm in South Africa and we have routine load shedding from our power grid, where we must keep our servers and workstations and screens and routers etc running for a minimum of 2.5 hours on battery. Additionally it's not possible for me to run a generator or install solar panels etc.
It's much easier to keep everything running if I select equipment with low idle power consumption.
I'm trying to decide between Aliexpress 4x 2.5GbE with one of these CPUs
* J4125
* N5095
* N5105
OR a reburbished i3 6th gen...
I know all of these CPUs idle with low power usage.
But I've not been able to find much info about whether any of these CPUs' performance will be sufficient to run Ceph OSD's. But from my research I get the feeling the N or i3 CPUs will be fine. I have no idea about the J4125 though...?
Hey @Patrick thanks for your TinyMiniMicro videos, they're awesome!!! Have you tried running a ceph cluster with J4125 yet?
Aside from my CPU performance concern, I'm concerned that these aluminum enclosures lack SATA ports and PCI-E ports. They generally have only 1 SATA port for a 3.5" HDD. Now I could get one of these https://www.amazon.com/Internal-Non-Raid-Adapter-Desktop-Support/dp/B07T3RMFFT/ And then connect up plenty of HDDs. But then I'd lose the ability to have a SSD for Ceph journalling. But is the M.2 NVME port even any good on these boards?
The only IO config of the Aluminum 4x 2.5GbE that fully makes sense, is if I run a NVMe for ceph journal and connect only 1 SATA 3.5" drive to each node, and then add more nodes if I need more space. (which I could do) But that's a high cost per SATA port!
The other IO config of the Aluminum 4x 2.5GbE that ALMOST makes sense in my head, is if I run a NVMe for ceph journal and connect 1 SATA 3.5 PLUS 2 (or more) SATA 3.5 via USB 3.0. The performance should be good. But will those USB ports be stable and good...?? (scared face)
Which makes me think I might be better served getting a refurbished 6th gen i3. It should come with at least 2 SATA ports (enough for me) and I should hopefully be able to connect a decent NIC AND a NVMe SSD via a PCI-E to M.2 riser (or even a SATA SSD if I have to)
Interesting links
This NIC looks interesting but probably requires PCI-E bifurcation and probably won't work on 6th gen i3?
Aliexpress (search for 2.5 J4125 or "2.5 N5095" etc)
No ECC RAM
I've decided to not worry about using ECC RAM.
Ceph doesn't mention ECC ANYWHERE in their documentation, and says that their software is designed to run on cheap commodity hardware.
So I believe it's safe to run RAM tests and then assume ceph will be fine without ECC.
I want to build a low power ceph cluster!
Rejected Xeon "X79" LGA2011 etc
I've considered getting some old Xeon LGA2011 or whatever servers from AliExpress which would make cost low, ECC RAM abundant and IO aplenty. But they'll probably idle at at least 60W each for just the motherboard and CPU. I'm targeting 10W per node (excl hard drives and switches) but I would still be satisfied if it's 30W per node.
I don't know for sure, but I sadly believe this will use too much power.
Craft Computing built a (Proxmox, not Ceph) cluster with LGA2011 parts... unfortunately he didn't give power consumption excl drives or down-clock the CPUs or under-volt or anything like that. But my sense is they weren't the kind of low idle power that I'm seeking.
Low idle power for UPS on South African power grid
I'm in South Africa and we have routine load shedding from our power grid, where we must keep our servers and workstations and screens and routers etc running for a minimum of 2.5 hours on battery. Additionally it's not possible for me to run a generator or install solar panels etc.
It's much easier to keep everything running if I select equipment with low idle power consumption.
I'm trying to decide between Aliexpress 4x 2.5GbE with one of these CPUs
* J4125
* N5095
* N5105
OR a reburbished i3 6th gen...
I know all of these CPUs idle with low power usage.
But I've not been able to find much info about whether any of these CPUs' performance will be sufficient to run Ceph OSD's. But from my research I get the feeling the N or i3 CPUs will be fine. I have no idea about the J4125 though...?
Hey @Patrick thanks for your TinyMiniMicro videos, they're awesome!!! Have you tried running a ceph cluster with J4125 yet?
Aside from my CPU performance concern, I'm concerned that these aluminum enclosures lack SATA ports and PCI-E ports. They generally have only 1 SATA port for a 3.5" HDD. Now I could get one of these https://www.amazon.com/Internal-Non-Raid-Adapter-Desktop-Support/dp/B07T3RMFFT/ And then connect up plenty of HDDs. But then I'd lose the ability to have a SSD for Ceph journalling. But is the M.2 NVME port even any good on these boards?
The only IO config of the Aluminum 4x 2.5GbE that fully makes sense, is if I run a NVMe for ceph journal and connect only 1 SATA 3.5" drive to each node, and then add more nodes if I need more space. (which I could do) But that's a high cost per SATA port!
The other IO config of the Aluminum 4x 2.5GbE that ALMOST makes sense in my head, is if I run a NVMe for ceph journal and connect 1 SATA 3.5 PLUS 2 (or more) SATA 3.5 via USB 3.0. The performance should be good. But will those USB ports be stable and good...?? (scared face)
Which makes me think I might be better served getting a refurbished 6th gen i3. It should come with at least 2 SATA ports (enough for me) and I should hopefully be able to connect a decent NIC AND a NVMe SSD via a PCI-E to M.2 riser (or even a SATA SSD if I have to)
Interesting links
This NIC looks interesting but probably requires PCI-E bifurcation and probably won't work on 6th gen i3?
Aliexpress (search for 2.5 J4125 or "2.5 N5095" etc)
No ECC RAM
I've decided to not worry about using ECC RAM.
Ceph doesn't mention ECC ANYWHERE in their documentation, and says that their software is designed to run on cheap commodity hardware.
So I believe it's safe to run RAM tests and then assume ceph will be fine without ECC.
Last edited: