Need some help simplifying my network

[Jonthewise]

Hello all, I was hoping that someone with some actual network expertise might be able to look at my setup and tell me how to make it less... challenging.

Basically, I want to remove the dependency on my home lab from accessing the internet, but I kind of have it baked in and can't really figure out how to get it working better. I also want to do away with the Cisco router *if* at all possible, just because it's the loudest part of my rack, and I've never been able to successfully get it to do what I really wanted, it's kind of an either or thing: I can either have it do NAT forwarding to a DMZ gateway OR I can have it do failover between the Comcast connection and the LTE connection, but I can't get it to do the NAT forwarding while on the Comcast connection and still failover when Comcast goes down. But that's another topic... Back to the topic at hand:

I have a kinda complicated setup and I'd like to make it less complicated. Right now, I also have a chicken and egg dilemma when booting everything. The TrueNAS box depends on the Windows Domain Controller for Authentication. The ESXi VMs depend on TrueNAS for their Datastores. So I have autostart with a delay booting windows as soon as the main ESXi on the gen9 comes on, and then waits a few minutes before booting the ESXi VMs. Theoretically if I turn both Gen9 systems on at the same time, TrueNAS will boot up after windows, but before the ESXi VMs. In practice that doesn't always work out. But I digress, this is about my networking config.

So, I have 10G fiber connecting TrueNAS to VLAN10 (my lab VLAN) and a pair of 10G fibers connecting the ESXi Server to trunk ports, with a 1G copper connection to VLAN10 for management. Both of the Virtual ESX systems have 3 NICs connected to VLAN10, and the two trunks, respectively, and the Windows DC is connected to VLAN10. There's another Windows DC running in the virtual cluster on VLAN10, but I wanted the primary DC to start up as soon as possible, which is why it's on the bare-metal host.

I also have a ZoneMinder system running on a VM on VLAN10. Then I have my MythTV master backend running on VLAN10 as well. And some other things that are less relevant to this discussion.

Have I lost you yet? Hope not. Anyway, so I have VLAN11 segregated off for security cameras, this is only on the Aruba, and they're not pictured in the diagram, but I wanted to keep their traffic separate from my lab traffic. That's all that lives on VLAN11.

And then there's VLAN13, which I had hoped to make my DMZ. This is where I have my web servers, and my DMZ gateway (currently just VMWare Horizon UAG with some forwarders setup to pass traffic to the web servers or my horizon connection server back on VLAN10).

And finally VLAN15, which is my 'public' network. This is the VLAN that the Aruba WiFi access points are on, as well as my Wife's computer, and my virtual desktops.

I also have s dumb switch plugged in to VLAN15 that lives in the living room and feeds the video game consoles, TV, and a MythTV frontend.

Oh, and there's one more, VLAN 1000, which is the Cisco router subnet. Nothing else on it, just a way to tell the Aruba to route traffic out to the internet.

So as it stands, everything works swimmingly (except the aforementioned choice between failover and NAT, which I can have one or the other, but I've got the configs ready to swap in notepad on a laptop if comcast goes down) but I can't take down my ESXi system for any reason without breaking the internet (no more DHCP for new connections, and more importantly, no DNS for any existing connections)

The problem is, that I have network drives mapped on the desktops in VLAN15 to the TrueNAS server on VLAN10, so if I revert back to using the comcast router as a router, then it also is acting as the gateway, and doesn't care at all about any of my other VLANs.

So, to try and figure this out on my own, I made sure I was using the failover config and unplugged the comcast gateway from everything. Internet is still up and Wife's still happy. Then I factory reset the comcast modem/router/wifi/combo piece of junk and assigned it an IP on VLAN1000 and plugged it into a port on the Aruba. Now I can connect to it via IP from my laptop, which is still getting DHCP from my windows server, and I started looking around. It has an advanced section where I can put in additional routes, but when I tried to add a route to VLAN10 it complained about not being able to reach the gateway. So that's where I'm stuck.

On the Aruba, I have the following set up:

VLAN10 Gateway 192.168.10.1
VLAN11 Gateway 192.168.11.1
VLAN13 Gateway 192.168.13.1
VLAN15 Gateway 192.168.15.1
VLAN1000 Gateway 192.168.0.1
And the default Gateway is 192.168.0.2

The Cisco port connected to the Aruba is 192.168.0.2
I assigned Comcast to 192.168.0.3

I tried adding a route to 192.168.0.0/255.255.255.0 with gateway 192.168.0.1 in the Comcast static routes, and that worked. But then I tried adding the route to 192.168.10.0/255.255.255.0 with gateway 192.168.10.1, and it can't reach it.

Eventually I was going to spin up a PFSense VM on the ESXi server, and pass a couple of the gig nic ports through (I have 3 unused) but that wouldn't remove the dependency on my home lab for internet access. So I'm not entirely sure what my options are. I'd really like to be able to shut down/restart/do whatever with my home lab whenever without it impacting the household network. I understand I'm pretty much locked in with the TrueNAS as long as we've got network shares mapped there, but I already removed the network profiles (yes, I used to have our computers domain-joined and my wife's profile folder stored on the TrueNAS, which was great for backing data up to backblaze, but had other ramifications...) so if the network shares go down from time to time, it isn't going to render her computer unusable.

As far as I can tell though, if I allow Comcast to hand out IPs, it's going to be the gateway. I was thinking about possibly adding a 1G copper line from TrueNAS to a VLAN1000 port on the switch as a workaround, but it seems like I still wouldn't be able to access my lab from my desktops/etc.

Thoughts?
Here's a basic diagram of the working network. I didn't include all the extra stuff plugged into the Aruba for simplicity.

Green = 1G copper
Blue = 10G fiber

 

[Jonthewise]

This might have been simpler than I thought. I've added a static route to 192.168.0.0/255.255.0.0 to the Comcast device, pointed to 192.168.0.1 (the gateway on VLAN1000, that it can reach directly), and it appears that may do the trick, at least initially. I'd still love feedback on how to make my stuff less complicated

Edit: No, that only seems to solve half the problem.

The other half, DNS resolution, isn't agreeing... What I tried was to manually configure DNS on the comcast device, using my internal DNS server as the primary DNS entry and then their primary DNS server as the secondary DNS entry... but when I try to lookup my internal DNS names (both short name, or fully qualified) it seems to still be going out to comcast for the answer.

 

[altmind]

cool. come here again for an extra advice

 

[Jonthewise]

LOL, I thought I fixed it, but alas, I did not. I at least got it working so that we're back on the Comcast connection for all the internet (though I'm impressed that my wife could stream hulu over the LTE without noticing that I did anything).

Basically, I reset the comcast device to factory defaults, gave it the 192.168.0.3 IP address, added the static route, Set my DNS server as Primary and Comcast's DNS server as secondary, and plugged it into a VLAN1000 port. I set up another VLAN1000 port for testing and plugged my laptop into that. DHCP assigned an address and I can get to the internet, I can ping my internal stuff, but when I try to connect by the hostnames, it can't find them, and it's clearly trying to resolve over comcast's DNS, because they all resolve to the external IP address that shows up on my dynamic dns service. I have all my internal servers named "servername.home.mydnsnamedotcom" but am getting the IP of *.mydnsnamedotcom when I do a nslookup.

So for now, I unchecked the manual DNS, updated the cisco to put the new IP of the comcast router (instead of the bridge IP) as track 1 and plugged it back in. So my network diagram above stands as is, except there's an additional copper line going from comcast to the aruba so that I can reach it directly without having to go through the cisco.

I have a feeling that there's not going to be any way to make the comcast do my main routing tasks... if that's the case, I have a spare RaspberryPI 3B+ and a USB Ethernet adapter that I can use to build a little router. But that just seems like I'm switching one complication for another. I guess at least the PI runs on POE from the Aruba, so considerably less than the Cisco.

I've got time, but I would really like to get the internet dependency on my lab removed before next summer, I have time of use, and I would simply love to be able to shut down my lab from 5-8PM rather than paying the insane $0.35/kwh they're charging at that time of day. Plan B is to get enough battery capacity into my UPS to run the lab for 3 hours so I can just unplug the whole thing from the wall (or more realistically have a timer) during the summer peak.

I guess another option would be to add the comcast DNS address to my windows DHCP server as a tertiary DNS and then when the lab is down the rest of the network would still be able to do lookups...

Looking forward to advice and suggestions, as I ramble on about my thoughts on resolving this. Any better ideas are appreciated, I'm just kinda tossing ideas around.

And if anybody wants to help me figure out my Cisco either/or issue, that would be cool too. It's all configured by googling this and that, but I've basically got a NAT config that works for passing traffic back to the gateway device, or it works with route map config to do failover from comcast to LTE. But the route map config breaks the NAT config - honestly I was just going to use PFSense to try and solve that one. But I that actually creates another dependency on the lab (was planning a virtual pfsense server) unless I want to put another system in my rack, which I'm trying to avoid as power reduction is the name of the game. Which brings me back to the RasPI option.

Sorry, that was a ramble. I'm just not really sure the best way to handle all the dependencies that I've created for myself