WTB: TMM USB3 16GB GbE Low Power for Ceph OSDs

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

rich0

New Member
Mar 17, 2023
10
1
3
I'm not sure if this is the best place to look for hardware recommendations, but I was drawn to the TMM concept being discussed here and I am thinking about buying several units for use as Ceph OSDs. I don't have a ton of storage so I just need a couple of lower-power nodes that can handle a few hard drives each - I need to expand horizontally before I consider going vertical since running Ceph on one giant node doesn't make sense.

I've been a bit frustrated by used product sites since they rarely have filters for the IO or power draw, which is what I'm mostly concerned with.

I'll be mostly using external storage and spinning disks, so my requirements are:
  • USB3 ports
  • At least gigabit LAN (more if possible)
  • 16GB of RAM (8GB might work, but OSDs can consume memory during rebuilds)
  • Low power draw
I don't really care about the CPU at all - ARM would even be fine if they made cheap boards with that much RAM and IO.

I'd be interested in pointers to cheap hardware in any form. I'd rather have more cheap nodes than a few expensive ones at least to start out.
 
  • Like
Reactions: Samir

amalurk

Active Member
Dec 16, 2016
311
116
43
102
I was looking for the same thing and I concluded I was better off with SFF size boxes i3/i5 7xxx to 9xxxx gen at <$200 each used. The specs say most of them idle around 9-12 watts as a base. With SFF size you get 1-2x m.2 + 1-2x 2.5 for bulk flash and 1-2x 3.5 for bulk spinners depending on brand/model choice. And the BIGGEST thing, a PCIe slot for good networking like a i350t4 or SFP+ (or both even in some of them with an x16 and x4 slot) without hassle or need to hack the form factor. Also 4+ USB3 ports on some models. Most of the boxes are roughly 10-13 inches width and depth and <4 inches high. So again depending on exact model/brand, but you can fit 4-5 of them in 7U if you turn them on their sides.
 
Last edited:
  • Like
Reactions: rich0

rich0

New Member
Mar 17, 2023
10
1
3
Yeah, unfortunately the RAM is a pretty hard requirement. I'm actually running MooseFS for storage right now on Pi4s and that works nicely, but Ceph OSDs can have issues if they have to rebuild without sufficient memory. It is a nasty case because it can work just fine but if you have something like back-to-back failures or nodes coming back in the middle of a rebuild the OSD ends up consuming a lot of memory keeping track of what needs to go where. I really love everything else about Ceph but the memory footprint of the OSD is a bit of an achillies heel since you need so many of those. (I'm very happy with MooseFS but the FOSS version doesn't support high availability and it is a bit annoying having one host that can't be rebooted without shutting down an entire k8s cluster. This is all home lab use - I'd never do it that way in production.)

Thanks for the suggestion all the same!