SBC with 10Gbe + M.2 with at least 2-lanes of Gen 3?

[ctrahey]

Hi all! I'm dreaming up a Ceph (or other distributed storage) cluster for homelab setting, and I find myself wanting to get as close as practical to being network-bound on a 10Gb link - I think this should be possible with just 2 lanes each of Gen 3.0 PCIe for the NIC and the M.2, and chips like the Rockchip RK3588 support 2x2 on their PCIe interface... I would also explore slightly pricier chips like from Layerscape with on-chip 10Gbe controller...

Does anyone know of such a board? Most of the ones I can find have either just 1Gbe (maybe 2.5) _or_ they only wire one lane of PCIe to the M.2 slot. In the ideal arrangement, this should come in at a price under $200 each, if it really is just the SOC, basic IO, SFP+ cage, and M.2 slot - which would be awesome. Bonus points for a management 1Gbe port, and extra bonus if that port has PoE... But I'd buy at least 3 units today if the main requirements existed - essentially if I can push 1.25 GB/s toward an NVMe drive over the network at SBC prices... that would be so useful!

 

[Monoman]

last couple tiny/mini/micro review items here had 2x 10g sfp+

 

[SnJ9MX]

Why not a SFF computer such as dell optiplex? If $200 is your limit, you can easily find something that would work. Only difference is size. In inches 11.4x11.5x3.6. Micro also exists but doesn't have PCIe expansion. However, Lenovo M720q tiny does have PCIe port and is very small (https://www.servethehome.com/lenovo-thinkcentre-m720q-tinyminimicro-feature/).

$130 with shipping included for Optiplex 3070 SFF - i5-9500, 500GB HDD (toss), 8GB memory, idles at 9W (I have one) - DELL OPTIPLEX 3070 Intel Core i5-9500 3.00 GHz 500GB HDD 8GB DDR4 *NO OS* | eBay

You get 2x PCIe ports (x1, x16), 1 M.2 slot (80mm max), 1x SSD drive (either 2.5" OR 3.5" officially but you can stuff more in if so inclined), a fast CPU (compared to SBC), 2 DIMM slots for up to 2x32GB DDR4.

 

[amalurk]

Why not a SFF computer such as dell optiplex? If $200 is your limit, you can easily find something that would work. Only difference is size. In inches 11.4x11.5x3.6. Micro also exists but doesn't have PCIe expansion. However, Lenovo M720q tiny does have PCIe port and is very small (https://www.servethehome.com/lenovo-thinkcentre-m720q-tinyminimicro-feature/).

$130 with shipping included for Optiplex 3070 SFF - i5-9500, 500GB HDD (toss), 8GB memory, idles at 9W (I have one) - DELL OPTIPLEX 3070 Intel Core i5-9500 3.00 GHz 500GB HDD 8GB DDR4 *NO OS* | eBay

You get 2x PCIe ports (x1, x16), 1 M.2 slot (80mm max), 1x SSD drive (either 2.5" OR 3.5" officially but you can stuff more in if so inclined), a fast CPU (compared to SBC), 2 DIMM slots for up to 2x32GB DDR4.

This is what you want OP or even better than 7 series optiplex is the similar HP Pro desk that have 2 x m.2 slots and can fit 3 SATA SSD or 2x and 1x HDD drives in SFF. So each node could have 1x m.2 boot, 1x m.2 118GB Optane for DB/Wall, 2 x bulk SATA SSD per node, 1 x large HDD for capacity (or 3x SATA SSD if don't need bulk capacity) Do that with 5 nodes and you have great home lab cluster. They have room for PCIe card too networking. Cheap 118GB Optane for DB/wall makes large iops difference, like NVME+ performance without all the heat that 2 or 3x NVME 2.5in drives would make in SFF factor and lead to cooling issue. Don't know if you need even better performance though might try multiple high perf NVME in there instead but going to need better airflow/cooling in tight SFF. But should be able to find the SFF for $200 or less each I would think. Plus your SBC CPU is not going to be enough for Ceph. i5-9500 would be better.

 

[seany]

I just did something kind of close to this, and ended up with the same conclusion as SnJ9MX. I went with 4x ThinkStation P340 SFF.

 

[bvd]

Might get some better traction over on this subforum, not sure:

https://forums.servethehome.com/index.php?forums/nas-systems-and-networked-home-and-smb-software.39/

Far as I'm aware, this one is typically relegated to posts specific to buying/selling by/from users here, so pretty different audience and may not have the same visibility

 

[ctrahey]

Thanks all - Hadn't thought about used SFF before, but it makes sense given the flooding of such a market with some solid Intel Core options. Thanks for the tips all, including pointer to the other forum. I've only been exposed to STH on YT before this, new to the forums here!