Hollow Oak Ranch: New HomeLab Design/Deployment in progress

Nugget

Member
Jul 13, 2017
32
25
18
Tejas Hill Country
keybase.io
Earlier this year I bought some rural property that will become my home closer towards the end of the year. I'm upgrading from a small city center townhome on a postage stamp lot to a 35 acre ranch in the country with three different buildings I will be wiring up with networking, servers, and WiFi. I've got a long list of decisions to make about infrastructure and I'm just now diving into the deployment. I thought I should get in front of the work so I can get some feedback and critique about my plans while it's still easy to make changes. I hope this thread can generate some good discussion. I'm excited to set up a TIA-606 Class 3 HomeLab!

TL;DR I'm deploying a UniFi campus network in my new home, including buried fiber runs to four different locations on the property. This will be an upgrade/replacement to my existing home network which consists of a pair of Proxmox VE servers, a FreeNAS Mini XL+, and a variety of other sundry boxes and RPis. This thread will document my progress and any related shenanigans.

The Current Situation

My new home had an existing "server" closet which was home to a junky 2-post round-hole rack that held A/V gear (media room is on the other side of one wall) and an old-and-busted security camera system. I'm keeping the room, but throwing everything else away. There's also enough existing cat5 in the walls and attic to qualify the home as a copper mine. I've already ripped all of that out, as well as the shag carpet that was in the server closet. New tile is down, but still waiting for an electrician to come in and tidy everything up and then for the walls to be cleaned up and new drywall.

This is how things looked when I bought the place (the UDM Pro is mine):

IMG_5839.jpeg IMG_5841.jpeg

The closet is empty now, all the cabling worth keeping has been pulled up through the walls into the attic, and I set up the ISP modem, UDM Pro, and an access point up there as a temporary measure while construction takes place in the closet and main house. Thankfully the attic is insulated and everything seems happy enough up there for the short term even in the crazy Texas heat.

Rough Plans (layer 2)

I've already bought a bunch of UniFi access points, switches, and the UDM Pro to host layer 2. I have no interest in anything UniFi beyond the Network app. No plans for UniFi security or access control or whatever other nonsense they're cooking up. I've been happy enough with the switching and very happy with their WiFi since switching from Ruckus a few years back. I will be doing 10gbit fiber from the main house to a workshop which is about 400 meters from the main house. I'll also be doing fiber from the main house to a powered front gate. There's an existing RF point-to-point network link from the workshop to a guest home which I plan to re-use if I can get it working. Previous owner was using it and claims they got "decent" bandwidth from it. They were not very technical, but did manage to install it themselves. The fiber trenching will start next Monday with a rental rock saw and ambitious optimism. If the two necessary fiber runs go well it might be possible to trench fiber to the guest house also, but the routing for that will be challenging. Should know more next week, we've got the rock saw rented for two days.

Rough Plans (software)

I currently run a two node Proxmox VE cluster (with RPi quorum device) and rely on automated zfs snapshots to support VM live migration. I've been very happy with this approach, it got me enough resiliency with not too much complexity in my current home. I haven't decided if I'm going to keep going forward with this approach or if I'm going to instead try a different approach. I've been using FreeBSD since 2.2.5 and zfs since it was in -STABLE. I've been happy with FreeNAS and I'm leaning towards a TrueNAS Core or Scale solution at the new place. I'm open to a TrueNAS Scale cluster and shared storage for VM live migrations, but I don't have a very clear vision of how this would look. Still researching, I don't really have to have a plan for this until the server closet construction is complete and that's at least a month out right now. It feels like TrueNAS Scale and Proxmox VE are really starting to overlap, which gives me some pause.

My goals are to support a bunch of VMs that will keep running even if one of the servers croaks. I also will have some form of Kubernetes running, but my needs there are simple and I've been well-served so far with just a k3s VM.

How and how much I can support Arm containers is an open question. My primary workstations are all Apple Silicon, so this is on my radar, but I'm strictly amd64 for homelab infrastructure currently.

Rough Plans (gear and everything else)

There will be a 42U four post rack in the new server closet. A wall-mounted small rack in an air conditioned room in the workshop, and a small switch in an interior closet in the guest house. I've got the UniFi switching needed to connect all that together. My current home servers are Supermicro Xeon D-1541 and I've been really happy with them, so I'm naturally leaning towards the new Xeon D-2700 platform (rack mount instead of mini tower this time), but haven't made any selections yet. Looks like they're finally available now, which is encouraging. I figure once I've fully moved I can repurpose my old FreeNAS hardware by putting it in the workshop for an offsite backup/zfs snapshot target.

I'll keep this thread updated as I make progress and I'd love to hear any advice or snark (as appropriate), both now and as I progress.

I wasn't quite sure where to post this. Mods please move if I missed the target.
 
  • Like
Reactions: Patrick and Lix

T_Minus

Build. Break. Fix. Repeat
Feb 15, 2015
7,324
1,785
113
CA
Fun :)

To share some of my experience on acreage... my single unifi AP (LR) covers around 2 acres maybe 3 of nearly flat and moderately treed area... to say I was surprised is an understatement as I did buy a 3 pack to cover my home\office\outside area, and only ever setup 1.
 

T_Minus

Build. Break. Fix. Repeat
Feb 15, 2015
7,324
1,785
113
CA
That's fantastic range! My workshop is a metal building that is like a giant faraday cage. I had to buy a cellular booster just to get cell phone coverage inside it.
EEEH! Yeah, I did P2P for that for more bandwdith but like you I'm going to put down some fiber one day :D maybe soon, as I need to run more electrical.
 

Sean Ho

seanho.com
Nov 19, 2019
349
137
43
Vancouver, BC
seanho.com
Sound like fun, and congrats on your new property! I'd say the thing to focus on now, while the renovations are ongoing, is your structured cabling and HVAC.

Direct-burial fiber was a good choice. Presumably, the front gate will also want a small PoE switch with SFP uplink, to power cams or other IoT. Within the house, if the old cat5 was not stapled down, it can be useful for pulling new cabling. Cat5e in good shape can run 10GbE over short distances just fine, but if there's a time to pull new wire, it's now before you move in. Conduit is even better: smurf tube is easy to run, PVC is easier to pull wires through.

Are you planning on putting most of your servers in that little closet, where all your cabling home-runs? Do you have a circulation fan exchanging air between the closet and conditioned space in the rest of the home (using the home's central air), or are you planning a dedicated mini-split for that closet? If using the home's central air, pipe the closet's exhaust as close as you can into the furnace's intake or return air duct, that way during the cooling season the exhaust heat from the servers helps alleviate the load on your furnace.

As for server hardware, PVE vs TNS, etc., I'd just get a single, simple server up and running first with PVE (since it's what you're familiar with), and after you're settled in you can take your time shopping for hardware and tinkering with software.
 

Nugget

Member
Jul 13, 2017
32
25
18
Tejas Hill Country
keybase.io
Fun week last week. We rented the rock saw again in order to dig the trenches to run fiber optic cable from the main house to the stable and then on to the workshop buildings. We'd tried to get this done last month, but ran into some setbacks (broken conduit that we thought we'd be able to use). This week's attempt was a resounding success, though, and now I've got good connectivity, internet, and WiFi in all three buildings.

First up, we used a scissor lift to install a cellular booster into the hangar. Without this booster it was basically impossible to receive calls and texts when standing inside the metal building. Now with the booster we get 5 bars of 5g on all carriers. (It's a WeBoost Complete. No affiliation, just happy with the results. I've never used a product like this before and it exceeded my expectations). Then we got started with the rock saw (that's my step-dad driving in the picture, but we all took a turn)...

68002944673__40B79108-9D15-4F2C-95D7-DBC8534B754E.jpeg IMG_0361.jpeg IMG_6330.jpeg IMG_6337.jpeg

The trenching was mostly uneventful, although we had to cross pavement in two places which will mean some asphalt patching in the future. I'd rather have internet than indoor plumbing, so it was a no-brainer decision.

Fiber loop pulled into the hangar workshop room and a temporary switch is set up now. Everything's pinging, but there's a lot more still to do. Ultimately this will be a different switch and I plan to mount a 19" equipment rack on the wall to hold everything. It's enough for now though...

IMG_6351.jpeg IMG_6354.jpeg IMG_6348.jpeg

I tore down the Proxmox cluster from the old house and brought one of the Xeon D servers here to be a temporary PVE/docker/k8s host so that I can start to get Home Assistant and Pi-hole set up here. I've provisioned all my VLANs and got wifi going in all the buildings now.

I feel like I'm finally in a position to start working on the software side of the infrastructure here, although everything will be largely temporary until the server closet in the house is ready to take equipment.

Also, rock saws are fun.
 
  • Like
Reactions: T_Minus

T_Minus

Build. Break. Fix. Repeat
Feb 15, 2015
7,324
1,785
113
CA
Happy to see you making progress... jealous realy :D
Smoke rolled in here from fires far and weekend plans are on hold.

We did the DIY mini-split AC units and they sip power, run on generator easily... I suggest them for your shop and server room :) I have another unit here waiting for me to install, your first one takes 2x as long as the subsequent. Their ability to greatly reduce power consumption while still outputting cold air is absolutely amazing.
 

Nugget

Member
Jul 13, 2017
32
25
18
Tejas Hill Country
keybase.io
Great advice about the mini splits. I'm almost certain that's the way I'll go for the main closet in the house. The stable/guest house just has the single 8 port switch, so no special needs. Hangar/workshop is covered with overkill 4 HVAC systems and a BigAss Fan. It's kinda nuts.

The only smoke here is a 21lb brisket we cooked last week. And now that I've got WiFi out there, I'll be able to monitor the FireBoard in Home Assistant instead of having to walk over to the hangar to check on it every hour. I was thinking of using one of those Hue globe bulbs and having home assistant set the color and brightness to correspond to the temperatures monitored by the FireBoard in the smoker. Then I can just carry the Hue bulb around with me while I've got something going in the smoker and monitor it from anywhere.

It's on the list. I don't even have Home Assistant set up yet. Gotta install Proxmox first. This is going to be so fun.

67830230636__08B87445-0F37-4F0F-92B1-5C42C7BE072F.jpeg IMG_1212.jpeg