Tiny homelab Cluster as homelab

[Naninotama]

Hey guys,

after starting a new job and getting interested about Cluster, shared storage etc. I still can't decide how to start my homelab for testing and maybe some things like a private cloud, Bitwarden etc. and use one of the nodes as a firewall if possible.

What I want:

• low form factor
• low power usage (it's getting expensive…)
• still beefy enough to virtualize a Windows server with a client for test purposes (Not expecting high performance in here)

What I want to do with it:

• Proxmox Cluster with shared storage like CEPH
• Use 1 of the nodes with a OPNSense VM to use as a firewall (I hope this works because of the amount of NICs)
• Being able to virtualize a Windows Server with a client for testing but not 24/7
• Having a HA Cluster, so if one node says goodbye I don't have to worry

I've been looking for quite some time and can't decide what to buy because everyone is saying I need 10 GBit NICs for CEPH, but otherwise I am just using it as a homelab and don't need to be as performant as in a professional environment but here is my idea:

• 3x https://tinyurl.com/aliexpresshomelab which I would like to upgrade with a 40x10 noctua fan and put 32GB of Ram and 1TB of NVMe in each one with being able to bundle 2 NICs on each one to at least get 5 GBit for CEPH while using one of the nodes to connect with the modem to use with as a firewall.

Is there anything I am not seeing here, or any other advice how I could better start with my little homelab here?

Thanks in advance!

 

[bigfellasdad]

Agreed ro 10gb for a home lab, it's a nice to have but not necessary, 2.5gbe is fine.

 

[bigfellasdad]

For compute nodes, I'd want a little more cpu than the device you linked. The tiny pc boxes sip power and would suit your needs

 

[amalurk]

Proxmox Cluster with shared storage like CEPH

Will you be happy with 1 drive per node for CEPH on those boxes you linked? Also the specs say 1xmsata and 1x2.5drive, so no NVME? So msata boot and then 1 sata ssd per node as a ceph osd?

 

[Naninotama]

Will you be happy with 1 drive per node for CEPH on those boxes you linked? Also, the specs say 1xmsata and 1x2.5drive, so no NVME? So msata boot and then 1 sata ssd per node as a ceph osd?

1 drive per node will be enough for me and testing, but you got a point with the NVME. I thought they have one because I wanted the 1 drive per node to be at least faster. Will look into another box now. Thanks for noticing!

For compute nodes, I'd want a little more CPU than the device you linked. The tiny pc boxes sip power and would suit your needs

The thing is I still have some air budget wise, but I really don't know what to choose while staying at a low power usage because I am from Europe, and it's pretty expensive.

I am open for recommendations, guys! I am right now not able to find a good machine as a sweet spot. Maybe even just one of the 3 nodes as something close to above what I linked and 2 others with some more power?

https://tinyurl.com/HomelabMiniV2
This one with the N6005 uses nearly the same and is a bit more powerful while bringing 2 NVMe.

 

[amalurk]

1 drive per node will be enough for me and testing, but you got a point with the NVME. I thought they have one because I wanted the 1 drive per node to be at least faster. Will look into another box now. Thanks for noticing!


The thing is I still have some air budget wise, but I really don't know what to choose while staying at a low power usage because I am from Europe, and it's pretty expensive.

I am open for recommendations, guys! I am right now not able to find a good machine as a sweet spot. Maybe even just one of the 3 nodes as something close to above what I linked and 2 others with some more power?

https://tinyurl.com/HomelabMiniV2
This one with the N6005 uses nearly the same and is a bit more powerful while bringing 2 NVMe.

All these Celeron devices have very limited PCIe lanes so with that N6005 one for example, the NVME devices are only x1 speed not standard x4.

I am thinking about a similar cluster and and have just about concluded I am going to go used SFF size from ebay because there are just too many compromises for me on the Celeron devices like those or with the micro form factors. I want dual m.2 and not just x1 speed plus room for 1 sata if I want to grow and either 2x2.5g nic or PCIe slot that I could put a network card in. Have to go SFF size to get those requirements. I also will go 4 nodes because with only 3 and 1 down there is no quorum.

 

[adman_c]

All these Celeron devices have very limited PCIe lanes so with that N6005 one for example, the NVME devices are only x1 speed not standard x4.

I am thinking about a similar cluster and and have just about concluded I am going to go used SFF size from ebay because there are just too many compromises for me on the Celeron devices like those or with the micro form factors. I want dual m.2 and not just x1 speed plus room for 1 sata if I want to grow and either 2x2.5g nic or PCIe slot that I could put a network card in. Have to go SFF size to get those requirements. I also will go 4 nodes because with only 3 and 1 down there is no quorum.

FWIW, the Lenovo Tiny “P” workstations actually fit your requirements: I have a Lenovo P350 Tiny with 2x NVME and a SATA boot drive, along with a 10G NIC in the PCIE slot. It’s a pretty amazing amount of hardware in a tiny device, but you do pay a premium for USFF vs SFF.

Also, if you‘re talking about proxmox cluster, you have a quorum with 2 if your cluster size is 3. Not sure about ceph though because I don’t use that.

 

[ano]

1 drive per node is ideal for ceph just got enough nodes

 

[amalurk]

[

FWIW, the Lenovo Tiny “P” workstations actually fit your requirements: I have a Lenovo P350 Tiny with 2x NVME and a SATA boot drive, along with a 10G NIC in the PCIE slot. It’s a pretty amazing amount of hardware in a tiny device, but you do pay a premium for USFF vs SFF.

Nice, I didn't realize you could fit the NIC and the 2.5 ssd drive at the same time. I have looked at those so they are back on the table. I was also considering doing 1x3.5 14-18TB drive per node for a slow media pool pool and then really have everything I need in the cluster so now I have a harder decision. If it comes down to dollars and higher node count though SFF will probably win.

Also, if you‘re talking about proxmox cluster, you have a quorum with 2 if your cluster size is 3. Not sure about ceph though because I don’t use that.

1 drive per node is ideal for ceph just got enough nodes

Agreed theoretically but, not with $$ constraints and not at node count of 4.

 

[adman_c]

Nice, I didn't realize you could fit the NIC and the 2.5 ssd drive at the same time. I have looked at those so they are back on the table. I was also considering doing 1x3.5 14-18TB drive per node for a slow media pool pool and then really have everything I need in the cluster so now I have a harder decision. If it comes down to dollars and higher node count though SFF will probably win.

To be clear, if you want all of those things, you have to use a "half slim" SATA SSD or do what I and @Parallax did and just "shuck" a SSD from its plastic casing. See here for pictures:

https://forums.servethehome.com/index.php?threads/lenovo-thinkcentre-thinkstation-tiny-project-tinyminimicro-reference-thread.34925/post-363980