LAB Downscale

[Bjorn Smith]

Hi,

I decided to downscale my homelab a bit from 3 - down to just one machine.

I did the downscale mostly because I was too ambitious about what I wanted to use the lab for and also to save a little power, since not having two switches and two servers running does add up.


My previous setup was:

3 machines with:

SuperMicro X10SRi-F + 1 x Xeon e5-2673v3 + 256GB RDIMM/LRDIMM

With combined equipment spread across the machines:

FreeNAS:
2 x Intel P4600 2TB
2 x Intel P4510 2TB
1 x Mellanox ConnectX-3/-Pro
1 x SuperMicro NVME controller
1 x USB Boot stick

VM1:
2 x Intel P3700 400GB
1 x Mellanox ConnectX-3/-Pro
1 x USB boot stick

VM2:
1x Intel Optane 900P 280GB
1xIntel DC S4600 480GB
1xMellanox ConnectX-3/-Pro
1 x USB boot stick

Switches:
1xMellanox SX6018 Switch
1xBrocade ICX 6450-24P

To:
1 Machine with:

X10DRi-T + 2xXeon e5-2673v3, 512GB LRDIMM
2xIntel P4600 2TB
2xIntel P4510 2TB
1x Intel Optane 900P 280GB
1xIntel DC S4600 SATA disk 960GB
1xSuperMicro NVME controller

It now runs my FreeNAS with the NVME drives passed through and all my previous VM's.

And idles around 150-200w with all VM's powered up doing their stuff (which is not a lot mostly)

So my not so scientific guess is that I have probably saved around 200w/h, which with the crazy power prices we have in Denmark amounts to around 660 USD per year in power savings.

Conversion was mostly smooth, by just migrating the VMS to the new server, exporting the FreeNAS configuration, creating a new FreeNAS VM, pass through all the hardware to the new FreeNAS VM (with a PITA panic attack, when I saw nvme msi errors in FreeNAS - which took quite some googling to figure out, but got solved by: https://forums.servethehome.com/index.php?threads/optane-slog-passthrough-or-alternatives.27596/ (hw.pci.honor_msi_blacklist=0) in FreeNAS - everything else did not work for me.
When FreeNAS was up and running, I just imported the config from the old server, changed the network assignments and voila everything just worked.

Good news is that I will have 3 motherboards + other stuff to sell

https://forums.servethehome.com/index.php?threads/lab-downscale.30504 (FS thread)

tada

 

[Dreece]

Optimization, which leads to downscaling at the cost of increasing the number of 'single-point-of-failures', a reasonable acceptable risk for zero-sla home rollouts, indeed the benefit of power savings is highly attractive to us homelab'ers but this often comes at the risk of the old "putting too many eggs into the same basket syndrome".

Similarly, I too recently knocked off 300ish watts, swapped out a Cisco for a D-Link, was only using L2 as it is purely an access switch connected to a distribution switch, still the dlink does come with 4 x 10gbe ports of which only using one for now.

I am still a strong believer in dense servers, however I also rely on microboxes, one of which actually does the backups of all the monsters, a great teeny-weeny little lenovo black box which itself internally runs a few virtual machines such as openhab and pihole.

Crazy how efficient we can be if we put our minds to it with the reward being a smaller utility bill, though sometimes I do question the RoI of such endeavours.

 

[Bjorn Smith]

Yeah - I also have two other servers running still - my "backup/media" server, which backs up all the VM's + hosts my media + one other server that runs my pfsense box - which also will be converted to a esxi soon so it can run more than just the pfsense.