Musings on automating "everything", homelab or otherwise.

[kapone]

Part 1

Hi guys - As some of you may know based on my posts, that I run a rather large infra in my home as it is a backup/DR site for my production environment as well. For the last few months, I was in the "construction" phase, basically, racking, hardware, cabling etc etc. And as I started down the software configuration route...I keep thinking, this is not right, there has to be a better way.

So, I've decided to go back to first principles. Life in 2026 is pretty tech heavy, networked and complex. Let's start from scratch. I'm doing this mostly as an exercise for myself, but I decided to put it up, as more brains/eyeballs is always good, and it may be helpful to others.

Goals:

• Automate "everything" to the extent possible. If it doesn't look like it can be automated, keep looking for a way.
• It has to be 100% reproducible. No "I somehow got it to work" type of sentiment. If something works, we know why. If something doesn't work, we figure out why it doesn't work, instead of slap dash fixes.
• The above point has obvious implications. No (or very little) "click-ops". Everything has to be scripted and version controlled (so you can trace its history and see what you did and why you did, over time).
• The scripting has an added benefit that it is self-documenting. You know the networks you want/created, you know the IP address schemes, the VLANs, the gateways...etc etc.
• This includes bare metal provisioning. We don't want to install a server/appliance manually and 6 months later - wait..what did I do there?
• This also includes switch configurations, firewall rules, interfaces etc. This is not going to be easy, but that's the goal.
• Edit: And no cloud dependencies. None. No Github, no AWS etc. This is my stuff, it should stay within my control.

This should work on a small setup or somewhat larger (like mine).

Preamble: A basic view:



Let's start with an extremely basic configuration where you have a home (or wherever) setup with Internet access. We could simply plugin the ISP router and call it a day. That's for..well...less than basic. This is STH after all. What comes first?

• Firewall configuration?
• Switch configuration?
• ??

More to come.

 

[Patrick]

The big thing I would say is you want the LLM to write reusable scripts so that you can test and deterministically reuse. You do not want LLMs to keep trying to format things every time.

 

[MBastian]

Your plan to start with the basic environment is sound.
But what installs the installer? You need a bare metal install entity that is capable of dealing with pxe, tftp and what not. Going down the road your hardware will change and the ... let's call it master-installer needs to be capable of dealing with different requirements. As much as I like scripting I think these are far to brittle. The right thing would be to use Ansible and Terraform (insert your favorite alternatives here) and If these do not support some things in your enviroment but it has an API, write a module.

The big thing I would say is you want the LLM to write reusable scripts so that you can test and deterministically reuse. You do not want LLMs to keep trying to format things every time.

As much as I like LLMs for analysis and menial tasks, vibe engineering is a very dangerous thing to do. Claude, Codex and other frontline models can pump out pretty sophiticated things but if you look under the hood more often than not it's an incident waiting to happen.

 

[kapone]

The big thing I would say is you want the LLM to write reusable scripts so that you can test and deterministically reuse. You do not want LLMs to keep trying to format things every time.

This is not about LLMs as such. It's about...well, first principles. The first question that needs to be answered (as @MBastian points out) is:

what installs the installer?

or the way I think of it is: What came first? The chicken or the egg?

You need a bare metal install entity

That's exactly where this thought process is going.

As much as I like scripting I think these are far to brittle. The right thing would be to use Ansible and Terraform (insert your favorite alternatives here)

That's exactly...wait I just said that, didn't I? When I say "scripted" I don't necessarily mean bash scripts or the like (although that'll likely come in play as well), I mean, using the right tool(s) for the chicken/egg problem.

At it's core, the base problem is relatively simple. We're connecting two networks together. Most people (STHers are likely excluded from this) don't quite think of connecting to the internet this way, but that's what it essentially is. And then we want to control how data flows back and forth between these two networks.

This is not rocket science per se, like I said, plug in your ISP router, or follow a guide on deploying your favorite firewall/router, or whatever. The challenge is in rethinking of it as code. IaC (Infra as Code) is par for the course in enterprises, but in the home/hobbyist space, I believe it doesn't get enough attention and consequently leads to things like "S**t, I just had it working, what the hell did I do to break it?"

• Firewall configuration?
• Switch configuration?
• ??

And the answer to the above, I think is the third option - ?? - A plan. But...a plan that is in code.

 

[Patrick]

I had a great chat about OCP streaming boot today if you want to go further than an OS installer

 

[kapone]

I had a great chat about OCP streaming boot today if you want to go further than an OS installer

OS installing/Bare metal provisioning is only a small part of the overall picture. My ultimate goal with this is a scriptable (Ansible/Terraform/?) configuration, that can stand up a fully working environment from scratch (and do it repeatedly, consistently, everytime) .

It will likely involve multiple steps, and bootstrapping (e.g. a very minimal switch and firewall config) to get going.

 

[kapone]

My DevOps-Fu is very poor, but what you're describing should be possible with a combination of Ansible, Terraform, and at least a local instance of git. All of your runners could be local. You'd just need a hypervisor to obviously host the VMs or containers that would be doing the work and storing the data.

Interact with your local git repo using your favorite IDE, and you'd be off to the races.

It's not that it's not "possible"...of course it is, but the devil is in the details. e.g. Trying to solve the chicken/egg problem, let's say:

Case 1: Extremely simple/basic, no switch involved, just a router/firewall that connects to the internet and downstream.

- For a scriptable/automated deployment, you need a machine with (at a bare minimum) Ansible (as an e.g.) and git (let's say Gitea).
- But...how do you even get the scripts into this git in the first place? ok...some USB shenanigans...
- But...now Ansible needs to "somehow" (SSH/REST/??) connect to the router/firewall to configure it. But the router/firewall isn't configured for any networks yet...

Expand that to Case 2 (with a switch involved) or Case 3 (Managed switch involved) or Case 4 (HA/stack of switches) or Case 5 (HA/stack of switches and an HA pair of routers/firewalls) or...

Edit: And this is just the network part. We haven't even started on actual servers/containers/VMs yet.

 

[Patrick]

OS installing/Bare metal provisioning is only a small part of the overall picture.

Check out the streaming boot. Not just the OS installation.

 

[kapone]

Ok, I understand the situation a little more. It sounds like you're a SMB or small organization starting from zero -- no infrastructure. You just have a firewall and a switch.

*cough* - you misunderstood.

 

[kapone]

I think you have too many and should share.

Spin-up a local code repository of your choice on your chosen hypervisor.

Stop.

Interact with it using an IDE of your choice.

Stop.

You'd usually authenticate over SSH.

Stop.

You now have a means for managing your code.

Stop.

Use Packer to update and manage your VM or container templates.

Stop.

Use terraform to deploy those images

Stop.

you are running composable and clustered services

Kind of.

constantly deploying and destroying things?

Kind of.

Terraform will help you here.

umm...uhh...er...maybe?

Use ansible to configure or reconfigure what you want.

Stop.

You're already committed to equipment and platforms, so hopefully whatever you have has an API endpoint.

Stop.

Most of the prosumer stuff does as far as switching goes.

Not even close.

For those things that don't, you can still execute custome scripts against them if you can interact with them via some other means

Stop.

Use your existing hypervisor

Stop.

I think this will get you to where you need to go.

If it was that easy...

Your power bill is probably unwordly;

It's not that bad. ~$2K/mo. That's...pennies.

sending some of that my way will relieve you of this problem.

Lemme say this as kindly as I can. You're on step 112..or something like it. Keyword...First principles. You gotta get it off the ground first.

Edit: ps. This is my backup site/in my basement. My production infra is in a DC and is...much sexier.

 

[kapone]

So, I think this is going to come down to some bootstrapping. i.e.

Step 1: The Plan
I think this is going to likely be the most complex part of the process because there's many different permutations and combinations possible. But, as a baseline, I'd argue that every "home" (or whatever) needs at a bare minimum (A WAN is a given):

- A LAN
- An "IoT" LAN (IoT is somewhat of a misnomer, but suffice it to say, that this LAN will contain devices that you do not want on your main LAN, and this LAN will likely be locked down a fair bit.

Optionally:

- A "Guest" (WiFi) LAN - I'm not entirely convinced this is necessary in today's day and age. Any guests that are coming to your premises with a WiFi capable device...that device has mobile connectivity as well. What's the pressing need for them to connect to a WiFi in your premises?

From here on, things start getting complicated, if you're trying to setup multiple networks. I'd argue that an "STH" level setup requires at a bare minimum:

- A LAN - This is for the "home" side
- An "IoT" LAN (IoT is somewhat of a misnomer, but suffice it to say, that this LAN will contain devices that you do not want on your main LAN, and this LAN will likely be locked down a fair bit.
- An OOB (out-of-band) LAN for...well, OOB things.
- A "Management" LAN - This is where all management UIs, interfaces, access live.
- A "Server(s)" LAN - This is where all server(s), homelab, play area live.

This is probably a template for a small SMB/business as well, with minor tweaks. The OOB and Management LANs really do become critical from a longevity, "Oh S**t"" and maintenance perspective.

As an e.g. I consider the "LAN" port on my current firewalls as essentially an OOB/Management port, because there's switches and trunks involved and the actual "LAN" is trunked over those. When things go south, you should always be able to get to your firewall UI/shell, switch terminal or similar.

Step 1.1: Layout/Draw your network(s)
TBD

Step 2: Automation - A host with Ansible (or Terraform or choose your automation) and local git (Gitea, Gitlab, choose your git)
This can even be your workstation as such, but trying to keep things "clean", my head is revolving around a separate host.

to be continued...

 

[SnJ9MX]

Bootstrapping jogged my memory for what I've been doing to training 300+ engineers/analysts/LT on Claude (Code) at my dayjob. Step 1: powershell script to install Claude Code pre-reqs along with Claude Code itself + preset bedrock keys. Step 2: give Claude a big fat prompt so that it can go configure itself the rest of the way. Step 3: host an MCP server on intranet that has a handful of other "bootstrap" files that the user's Claude already knows about (see step 2) so it can continue adding to itself.

It is now handed off to IT (I work in Ops). They are a bit further behind the ball and wondering how to get a repeatable package going for new hires/late to the party people. They've already pushed managed settings to everyone's computer via Intune. But they don't have an installer on Company Portal (MS store-ish thing), nor a bootstrap script, nor an MCP server on intranet. So they need to get those parts fitting together.

At some point something has to be done, draw a line somewhere and go for it.

 

[kapone]

Been doing a lot of doodling/playing on this in the last few days. I think the "bootstrapping" part is going to come down to this.

1. Have your automation host with a static IP, with one "extra" port for bootstrapping switch(es) and firewall(s) and a USB/serial port for...well, serial/terminal stuff. Once the bootstrapping is complete and the network is up, these shouldn't be needed, but are there for the "Oh S**t" moments.
   
   
2. Configure switch (assuming you have/need/want one)
   1. Connect the serial and management ports to the automation host
   2. From the serial/terminal screen configure an IP for the management port, and configure a matching IP on the automation host's port.
   3. Run your switch configuration "playbook" over the management port.
   4. Disconnect switch(es) and rack'em.
      
      
3. Configure firewall - This is actually tricky. The likes of pfSense/Opnsense don't really play well with CLI automation, while OpenWRT (UCI) or something like VyOS does. For the longest time I've been a *sense user, but I may start thinking about alternatives at this point. In fact, moving to a Linux based firewall should in theory be more performant than BSD based one, given progress over the last few years.
   1. Process would be somewhat similar to switches, with minor caveats.
   2. Configure your "LAN" port on the firewall. This is usually done by default, but may need to be done.
   3. Connect the LAN port to the automation hosts's port, it should get an IP address from the firewall, assuming you are configuring DHCP on it as well, or static IP it is.
   4. Run your firewall "playbook" over the LAN port.
   5. Disconnect firewall(s) and rack'em/place them where they need to go.
      
      
4. Start doing the cabling.

This should get the base firewall/switch configuration going, based on a reproducible playbook/configuration. There's obviously a lot more that will need to be configured by the time we get to server(s) and applications, containers, VMs etc, but we need a starting point.

So, I think the answer to the chicken/egg question is...none. The automation host comes first.