Homelab update options/advise

[nry]

Hey, been a while since I have posted here.

My current setup is serving me well, almost flawless for a good few years now. Theres a few issues with with it and would like to put some sort of roadmap to upgrading various aspects over the next year. Also I'm getting an itch to upgrade

Apologies for the long post! Probably should have broken it down into multiple posts as I upgrade each part. Basically I could do with some guidance on which direction to take to future proof the setup for 4-5 years.

Budget is around £4000 excluding VAT (UK) but spread out over multiple phases. Small potential to increase this depending on sale of old components, but ideally sales would fund this figure.

I'll start listing the spec/use cases then run though my current issues.

Current Hardware

• APC UPS - SMT1000RMI2U
• APC PDU ZeroU
• Dell PowerConnect 5525P
• Dell PowerConnect 5524 (stacked with above)
• MikroTik CRS317-1G-16S+RM - not used currently, this will be the core switch
• UniFi AP Pro
• Main Server running ESXi 6 - runs 24/7
  • Supermicro X9SRL-F
  • Xeon E5-1620v2
  • 128GB ECC DDR3
  • 2x M1015 HBA
  • Samsung Evo 840 120GB boot drive
  • Samsung SM961 1TB NVMe - primary datastore
    
  • Samsung Evo 850 1TB - local backup datastore
  • WD 6TB Red - Media storage due to storage constraints on ZFS pool
  • 12x Hitachi/HGST 3TB 5k3000 drives (might be 1 or 2 7K3000 drives mixed in)
  • Intel X520-DA2 SFP+ NIC
• Secondary server running ESXi 6 or what ever OS I need to experiment with
  • Asus Z9PE-D8 WS (with IPMI chip)
  • 2x Xeon E5-2660
  • 128GB ECC DDR3
  • Mixed old SSDs for testing
  • 3-4 Hitachi 3TB 7K3000 drives
  • Asus HD5450 (used to passthrough to VM)
  • Intel X520-DA2 SFP+ NIC

Use cases

• pfSense - running in a VM with NIC passed through
  • DNS
  • OpenVPN
  • InterVLAN routing
  • DynDNS
  • Handle primary internet connection 300/20
  • Handle secondary backup internet connection (low speed)
• Provide wireless networks for guests alongside my potentially less secure home automation/IoT devices network
• Storage on FreeNAS - M1015's passed through to VM for 3TB drives - 12x drives in Raidz2
  • Media storage
  • Backups
  • Some VM volumes hosted
  • Extended storage for workstations
  • Home CCTV footage (probably a few days worth) - not being installed until June 2018
  • Large MySQL databases used during development/client's staging sites
  • Elasticsearch storage
• Downloading services
• Running staging environments for clients - note these are publicly accessible, downtime isn't ideal but causes no financial losses
• Continuous integration server (images/software builds)
• Development environments - basically run my Docker stack on this server to reduce any load on my workstation/resolve issues with Docker for Mac
• Experimenting with new OS/environments
• Processing large data sets

Issues

• Idle power consumption currently sits around 270-300w, would love to get this lower but admittedly that's going to be incredibly difficult, some rough usages per device left on 24/7
  • Main server - 170-190w
  • Dell 5524P - 40w
  • Dell 5524 - 14w
  • APC UPS - 30w
• Currently renovating the house which means the server gets powered down quite often to be moved around to try protect it from the dust. This results in no internet access! Also due to the network setup with pfSense, it takes 5-10 minutes before network access is restored from server startup!
  • Also when ever I want to tweak/update the main server, there's no internet. Somewhat annoying when trying to read online documentation or if others are in the house.
• Hard drives are all dated from 2010/2011 (few warranty replacements so might be a 2012 drive in there) - getting concerned about potential failures.
• Storage configuration could be better
• Approaching low free storage space
  • Currently there's ~24TB after parity.
  • Add the 20% free space limit for ZFS and that leaves me with ~19TB
  • Leaving me with just under 1TB free!
• Noise - mainly lack of fan control on the main server with ESXi results in them running at full speed
• Secondary server takes 5-10 minutes to boot ubuntu (think it's due to the amount of memory)

Phase 1 - Router/Firewall
The most logical step here feels to be a box specifically for pfSense, budget ideally under £500 (ex VAT).

I'd like this to support 10GbE to maintain as some level of performance with InterVLAN routing. This is mainly clients (LAN only) accessing media, but there's some cases of test environments running across multiple VLANs. CPU could be somewhat problematic.
Any of my substantial network use is configured to avoid routing.

Obviously this is going to increase idle power consumption, but think it will be possible to keep that increase under 30w.

There's a few options which all come in around the £500 mark (already have a spare SFP+ NIC), IPMI is a must. Will require 8GB RAM and a 1U case/fans.

• Xeon D - previous gen
• Atom C3xxx - will require SFP+ NIC
• Xeon E3 1220v6 - will require SFP+ NIC
• Xeon Bronze - overkill and probably above £500!

Alternative is something much lower power + layer 3 switch, but they tend to be loud and power hungry!

Questions
1) Any suggestions on alternatives here? I'm leaning more towards the Atom board but unsure on performance
2) Is there any benefit to SATA DOM over a cheap M.2 drive?
3) Potential L3 switch suggestions?

Phase 2 - Storage upgrade
Currently have ~19TB usable. Data storage growth is around 2TB per year. Although this may increase with a potential move to 4K content soon. No deleting of content

My plan is to replace all existing 3TB drives with large capacity disks as part of a new ZFS zpool.

Drive options:

• Seagate IronWolf 12TB ~£305 per drive
• Seagate IronWolf Pro 12TB - ~£335 per drive - unsure if the extra £30 per drive for an additional two years warranty is worth it

Storage configuration options (calculated using ZFS Capacity Calculator - WintelGuy.com):

• 6x 12TB in RAIDZ2 (33.3TB usable) with potential to add another 6x 12TB vdev once at capacity (66.62TB usable), this might be the most cost effective approach if drive prices drop enough by the time additional storage is required.
• 8x 12TB in RAIDZ2 (47.56T usable)
• 10x12TB in RAIDZ2 (63.84TB usable)
• 12x 12TB in RAIDz2 (76.14TB usable)

Questions
4) Given the varied use cases above, perhaps slightly lower capacity with more drives would be beneficial to performance? Although power consumption is a concern.
5) Should I be looking at smaller vdevs with such high capacity drives?
6) Are there still advisable drive configurations with ZFS eg. 8 drives per vdev?
7) Assuming I have read the documentation correctly, it's possible to simply add vdevs to a pool to increase available storage space?
8) Is RAIDZ2 still suitable for such high capacity drives or should I be looking at RAIDZ3?

Phase 3 - Storage performance
Currently a lot of the VMs running are stored on the SM961 1TB drive, these are backed up nightly to the ZFS pool, then backed up off site a few hours later. This leaves me with potential for 24 hours worth of data loss if the drive fails! This 1TB of storage surprisingly doesn't go very far which means quite a bit of manual moving around of VMs.

The current FreeNAS configuration has little to no performance optimisations. Just 48GB of RAM which should be used by the ARC.

After doing some research into ZFS, it looks like there's quite a few options which I could utilise. Specifically the L2ARC and ZIL.

Questions
9) Is it possible/sensible to store these VMs on my primary data pool with the SM961 drive added as a L2ARC?
9a) Should I be considering a different NVMe drive given the increase in writes?
10) What drive would be best for a ZIL?

Looking here https://www.servethehome.com/exploring-best-zfs-zil-slog-ssd-intel-optane-nand/
Seems like one of the Intel Optane offerings will be best such as the 32GB M.2.
11) Can the L2ARC be configured for specific datasets only? Eg VM storage?
Seems incredibly pointless having the media (films) ever stored in cache.

Phase 4 - Server upgrade
I think the next phase would be to upgrade the primary server, then retire the secondary server (never need 16 cores of processing power!), replacing this with the Xeon E5-1620v2 board. This can be for another day!

--

Thanks to anyone who takes the time to read my ramblings, any input/suggestions would be appreciated.

 

[Patrick]

Maybe check STH main site for #10 in a few hours. Still working on the article for today.

 

[nry]

Done a fair bit of reading over the past week. Might have to change the phases up a little.

Phase 1 - Router/Firewall

Still have no solid plan here, base spec I have so far:

• Inwin RF100 - 1U case inc PSU - £95
• 8GB DDR4 2133Mhz - £50

I'll be going with a M.2 drive over a SATA DOM mainly due to cost.

Leaning more towards the Atom C3000 series but given the recent launch of the Xeon D-2000 series and announcement of AMD EPYC (https://www.servethehome.com/amd-epyc-embedded-3000-series-launch-watch-out-xeon-d/) I'm tempted to put this project on hold and see how these platforms pan out. Alternatively run it on a old box as a temporary work around.

Phase 2 - Storage upgrade

As I'm running out on storage space I'm going to prioritise this! Still unsure on what route to take here, 12TB drives are the most expensive per TB but given the potential electricity savings over the next 4-5 years with less drives it might make the most sense.

Forgot theres a 6TB (5.4TiB) drive running standalone in my server with 4.1TiB used, putting my total used space at 22.5TiB.

Using a figure of 3TiB per year to be safe, thats 15TiB over 5 years. So 37.5TiB usable should be my minimum goal.

So I put together a very exciting spreadsheet with some help of http://wintelguy.com/zfs-calc.pl Decided that I'm going to stick with RAIDZ2.



Filtering this down and sorting by cost per TiB and a single vdev for now.



Then remove anything over £2500/60TiB usable. Also removing 8TB due to drive count vs power consumption.



This has certainly help put it in perspective!

 

[Joel]

Router/Firewall:

1) Dell r210ii, with Xeon E3-1220v1 or v2. ~$150 all in @ ebay, has AES-NI for future proofing. Check that one off your list. ~40w draw. Has plenty of horsepower for up to gigabit WAN speeds.

Going newer (Xeon-D or Denverton) it'd take 10+ YEARS (depending on power cost) to recoup the more expensive buy in for the ~10-20w savings at the wall.

2) About the same. Main benefit of both is the integration with the motherboard (no cables/drives to mount). Once setup, any old SSD would be fine. For PF use, a cheap SSD like an 80gb Intel 320 ~$40 would be fine.

3) Don't know much about that. My one foray into bigger switches is a Cisco 4948. 80w with nothing plugged in, audible, but not what I'd call "loud."

Storage: Answers to questions:

4) The real performance gains come from increasing VDEVs (same concept as striping drives, just on a larger scale).
5) See 4. The main benefit would be performance. Keep in mind if you lose a single VDEV you lose the whole pool though.
6) Just the even number of effective drives theory, which is mostly for performance reasons also.
7) Yes. (Sounds like you know better than to add a single disk VDEV though)
8) Depends on how tolerant you are of failure/downtime and how good your backups are. I might start thinking about Z3 with 12tb drives for "important" home use type stuff. If I was running a production system at a DC, I would probably do something like a 3-way mirrored RAID 1 with hot spares.
9/a) Write endurance would be my main concern on a L2ARC drive. Not sure about the 961 in that metric.
10) For ZIL, main concern would be PLP (EDIT: and write speed/IOPS).

Not sure what VMs you run, but you might want to consider migrating to Proxmox if power is a big concern. Runs ZFS at the metal so you could ditch the HBAs and run all your drives off the 10 onboard SATA ports for significant power savings. Not quite as user friendly as FreeNAS (no GUI for ZFS stuff), but nothing a few cron jobs couldn't fix.

 

[nry]

Thats for the response Joel.

I'll take a look into the Dell R210ii, they seem to be £230+ here in the UK on eBay. Low power on this is just a checkbox on the list, purchasing newer gear in the hope of better performance/features.

8) It's looking more likely I'll only be running 8x12TB drives to begin with, my backups are run nightly to a server at my mums house (10-15 mins drive). I think all things considered I'm sticking with RAID Z2 here.

9a) After reading more into how ZFS uses the L2ARC, I'm not too sure the small, high IO VMs should ever run from the ZFS array as theres a good chance it will never be stored on the L2ARC. Probably better off with a pair of SSDs mirrored for these.

As mentioned above I'm going to order some drives soon and think I'll sneak a couple of Intel Optane 32GB drives in to run some tests with before fully migrating. I'd like to sneak a Optane 900P in but would hate to spend that much money unnecessarily.

10) PLP, given the lack of production level data here, not a huge concern. Server runs on a APC UPS which shuts the server down properly when the battery drops below 50%.

Currently have no idea how many VMs are running, maybe 50? All on ESXi. Some of which are simple docker containers.
I'll take a look into Proxmox, GUIs are pretty and all but CLI usually gets the job done faster if you know what your doing