Hello all, I was hoping that someone with some actual network expertise might be able to look at my setup and tell me how to make it less... challenging.
Basically, I want to remove the dependency on my home lab from accessing the internet, but I kind of have it baked in and can't really figure out how to get it working better. I also want to do away with the Cisco router *if* at all possible, just because it's the loudest part of my rack, and I've never been able to successfully get it to do what I really wanted, it's kind of an either or thing: I can either have it do NAT forwarding to a DMZ gateway OR I can have it do failover between the Comcast connection and the LTE connection, but I can't get it to do the NAT forwarding while on the Comcast connection and still failover when Comcast goes down. But that's another topic... Back to the topic at hand:
I have a kinda complicated setup and I'd like to make it less complicated. Right now, I also have a chicken and egg dilemma when booting everything. The TrueNAS box depends on the Windows Domain Controller for Authentication. The ESXi VMs depend on TrueNAS for their Datastores. So I have autostart with a delay booting windows as soon as the main ESXi on the gen9 comes on, and then waits a few minutes before booting the ESXi VMs. Theoretically if I turn both Gen9 systems on at the same time, TrueNAS will boot up after windows, but before the ESXi VMs. In practice that doesn't always work out. But I digress, this is about my networking config.
So, I have 10G fiber connecting TrueNAS to VLAN10 (my lab VLAN) and a pair of 10G fibers connecting the ESXi Server to trunk ports, with a 1G copper connection to VLAN10 for management. Both of the Virtual ESX systems have 3 NICs connected to VLAN10, and the two trunks, respectively, and the Windows DC is connected to VLAN10. There's another Windows DC running in the virtual cluster on VLAN10, but I wanted the primary DC to start up as soon as possible, which is why it's on the bare-metal host.
I also have a ZoneMinder system running on a VM on VLAN10. Then I have my MythTV master backend running on VLAN10 as well. And some other things that are less relevant to this discussion.
Have I lost you yet? Hope not. Anyway, so I have VLAN11 segregated off for security cameras, this is only on the Aruba, and they're not pictured in the diagram, but I wanted to keep their traffic separate from my lab traffic. That's all that lives on VLAN11.
And then there's VLAN13, which I had hoped to make my DMZ. This is where I have my web servers, and my DMZ gateway (currently just VMWare Horizon UAG with some forwarders setup to pass traffic to the web servers or my horizon connection server back on VLAN10).
And finally VLAN15, which is my 'public' network. This is the VLAN that the Aruba WiFi access points are on, as well as my Wife's computer, and my virtual desktops.
I also have s dumb switch plugged in to VLAN15 that lives in the living room and feeds the video game consoles, TV, and a MythTV frontend.
Oh, and there's one more, VLAN 1000, which is the Cisco router subnet. Nothing else on it, just a way to tell the Aruba to route traffic out to the internet.
So as it stands, everything works swimmingly (except the aforementioned choice between failover and NAT, which I can have one or the other, but I've got the configs ready to swap in notepad on a laptop if comcast goes down) but I can't take down my ESXi system for any reason without breaking the internet (no more DHCP for new connections, and more importantly, no DNS for any existing connections)
The problem is, that I have network drives mapped on the desktops in VLAN15 to the TrueNAS server on VLAN10, so if I revert back to using the comcast router as a router, then it also is acting as the gateway, and doesn't care at all about any of my other VLANs.
So, to try and figure this out on my own, I made sure I was using the failover config and unplugged the comcast gateway from everything. Internet is still up and Wife's still happy. Then I factory reset the comcast modem/router/wifi/combo piece of junk and assigned it an IP on VLAN1000 and plugged it into a port on the Aruba. Now I can connect to it via IP from my laptop, which is still getting DHCP from my windows server, and I started looking around. It has an advanced section where I can put in additional routes, but when I tried to add a route to VLAN10 it complained about not being able to reach the gateway. So that's where I'm stuck.
On the Aruba, I have the following set up:
VLAN10 Gateway 192.168.10.1
VLAN11 Gateway 192.168.11.1
VLAN13 Gateway 192.168.13.1
VLAN15 Gateway 192.168.15.1
VLAN1000 Gateway 192.168.0.1
And the default Gateway is 192.168.0.2
The Cisco port connected to the Aruba is 192.168.0.2
I assigned Comcast to 192.168.0.3
I tried adding a route to 192.168.0.0/255.255.255.0 with gateway 192.168.0.1 in the Comcast static routes, and that worked. But then I tried adding the route to 192.168.10.0/255.255.255.0 with gateway 192.168.10.1, and it can't reach it.
Eventually I was going to spin up a PFSense VM on the ESXi server, and pass a couple of the gig nic ports through (I have 3 unused) but that wouldn't remove the dependency on my home lab for internet access. So I'm not entirely sure what my options are. I'd really like to be able to shut down/restart/do whatever with my home lab whenever without it impacting the household network. I understand I'm pretty much locked in with the TrueNAS as long as we've got network shares mapped there, but I already removed the network profiles (yes, I used to have our computers domain-joined and my wife's profile folder stored on the TrueNAS, which was great for backing data up to backblaze, but had other ramifications...) so if the network shares go down from time to time, it isn't going to render her computer unusable.
As far as I can tell though, if I allow Comcast to hand out IPs, it's going to be the gateway. I was thinking about possibly adding a 1G copper line from TrueNAS to a VLAN1000 port on the switch as a workaround, but it seems like I still wouldn't be able to access my lab from my desktops/etc.
Thoughts?
Here's a basic diagram of the working network. I didn't include all the extra stuff plugged into the Aruba for simplicity.
Green = 1G copper
Blue = 10G fiber
Basically, I want to remove the dependency on my home lab from accessing the internet, but I kind of have it baked in and can't really figure out how to get it working better. I also want to do away with the Cisco router *if* at all possible, just because it's the loudest part of my rack, and I've never been able to successfully get it to do what I really wanted, it's kind of an either or thing: I can either have it do NAT forwarding to a DMZ gateway OR I can have it do failover between the Comcast connection and the LTE connection, but I can't get it to do the NAT forwarding while on the Comcast connection and still failover when Comcast goes down. But that's another topic... Back to the topic at hand:
I have a kinda complicated setup and I'd like to make it less complicated. Right now, I also have a chicken and egg dilemma when booting everything. The TrueNAS box depends on the Windows Domain Controller for Authentication. The ESXi VMs depend on TrueNAS for their Datastores. So I have autostart with a delay booting windows as soon as the main ESXi on the gen9 comes on, and then waits a few minutes before booting the ESXi VMs. Theoretically if I turn both Gen9 systems on at the same time, TrueNAS will boot up after windows, but before the ESXi VMs. In practice that doesn't always work out. But I digress, this is about my networking config.
So, I have 10G fiber connecting TrueNAS to VLAN10 (my lab VLAN) and a pair of 10G fibers connecting the ESXi Server to trunk ports, with a 1G copper connection to VLAN10 for management. Both of the Virtual ESX systems have 3 NICs connected to VLAN10, and the two trunks, respectively, and the Windows DC is connected to VLAN10. There's another Windows DC running in the virtual cluster on VLAN10, but I wanted the primary DC to start up as soon as possible, which is why it's on the bare-metal host.
I also have a ZoneMinder system running on a VM on VLAN10. Then I have my MythTV master backend running on VLAN10 as well. And some other things that are less relevant to this discussion.
Have I lost you yet? Hope not. Anyway, so I have VLAN11 segregated off for security cameras, this is only on the Aruba, and they're not pictured in the diagram, but I wanted to keep their traffic separate from my lab traffic. That's all that lives on VLAN11.
And then there's VLAN13, which I had hoped to make my DMZ. This is where I have my web servers, and my DMZ gateway (currently just VMWare Horizon UAG with some forwarders setup to pass traffic to the web servers or my horizon connection server back on VLAN10).
And finally VLAN15, which is my 'public' network. This is the VLAN that the Aruba WiFi access points are on, as well as my Wife's computer, and my virtual desktops.
I also have s dumb switch plugged in to VLAN15 that lives in the living room and feeds the video game consoles, TV, and a MythTV frontend.
Oh, and there's one more, VLAN 1000, which is the Cisco router subnet. Nothing else on it, just a way to tell the Aruba to route traffic out to the internet.
So as it stands, everything works swimmingly (except the aforementioned choice between failover and NAT, which I can have one or the other, but I've got the configs ready to swap in notepad on a laptop if comcast goes down) but I can't take down my ESXi system for any reason without breaking the internet (no more DHCP for new connections, and more importantly, no DNS for any existing connections)
The problem is, that I have network drives mapped on the desktops in VLAN15 to the TrueNAS server on VLAN10, so if I revert back to using the comcast router as a router, then it also is acting as the gateway, and doesn't care at all about any of my other VLANs.
So, to try and figure this out on my own, I made sure I was using the failover config and unplugged the comcast gateway from everything. Internet is still up and Wife's still happy. Then I factory reset the comcast modem/router/wifi/combo piece of junk and assigned it an IP on VLAN1000 and plugged it into a port on the Aruba. Now I can connect to it via IP from my laptop, which is still getting DHCP from my windows server, and I started looking around. It has an advanced section where I can put in additional routes, but when I tried to add a route to VLAN10 it complained about not being able to reach the gateway. So that's where I'm stuck.
On the Aruba, I have the following set up:
VLAN10 Gateway 192.168.10.1
VLAN11 Gateway 192.168.11.1
VLAN13 Gateway 192.168.13.1
VLAN15 Gateway 192.168.15.1
VLAN1000 Gateway 192.168.0.1
And the default Gateway is 192.168.0.2
The Cisco port connected to the Aruba is 192.168.0.2
I assigned Comcast to 192.168.0.3
I tried adding a route to 192.168.0.0/255.255.255.0 with gateway 192.168.0.1 in the Comcast static routes, and that worked. But then I tried adding the route to 192.168.10.0/255.255.255.0 with gateway 192.168.10.1, and it can't reach it.
Eventually I was going to spin up a PFSense VM on the ESXi server, and pass a couple of the gig nic ports through (I have 3 unused) but that wouldn't remove the dependency on my home lab for internet access. So I'm not entirely sure what my options are. I'd really like to be able to shut down/restart/do whatever with my home lab whenever without it impacting the household network. I understand I'm pretty much locked in with the TrueNAS as long as we've got network shares mapped there, but I already removed the network profiles (yes, I used to have our computers domain-joined and my wife's profile folder stored on the TrueNAS, which was great for backing data up to backblaze, but had other ramifications...) so if the network shares go down from time to time, it isn't going to render her computer unusable.
As far as I can tell though, if I allow Comcast to hand out IPs, it's going to be the gateway. I was thinking about possibly adding a 1G copper line from TrueNAS to a VLAN1000 port on the switch as a workaround, but it seems like I still wouldn't be able to access my lab from my desktops/etc.
Thoughts?
Here's a basic diagram of the working network. I didn't include all the extra stuff plugged into the Aruba for simplicity.
Green = 1G copper
Blue = 10G fiber
Last edited: