Setting up routing between 2 different subnets - what are the basics?

AveryFreeman

consummate homelabber
Mar 17, 2017
254
23
18
39
Near Seattle
averyfreeman.com
Hey,

So this might be kind of complicated - so far it's a bit over my head, but I know it's absolutely possible.

I'm trying to set up routing between two subnets with an AD trust between the two of them, but let's put the AD/IPA trust aside for a minute, because none of it will work without well-functioning network config

Here's the big picture:

I'd like 2 subnets, 192.168.1.0/24 for AD domain, 192.168.2.0/24 for IPA domain. The two will be different forests + domains (or IPA could be a subdomain, either way).

I'd like all VMs on both 192.168.1.0/24 and 192.168.2.0/24 to be able to reach the outside internet, and each other, without any restrictions.

Gateway/firewall is 192.168.1.1, 192.168.1.0/24 exists and is operational, 192.168.2.0/24 is what I'm trying to set up.

How do I set up the routing for:

1) AD Domain network w/ Windows 2019 DNS servers authoritative at 192.168.1.0/24

2) FreeIPA network w/ Fedora 33 DNS server (in FreeIPA) authoritative at 192.168.2.0/24

I know I need to set up conditional forwarding to point at each other's subnet, but how should I do just the basic network setup for each server?

E.g. do I need Windows to see 192.168.2.2 (FreeIPA server) as a DNS server in its ipv4 network settings?

Do I need Fedora to see 192.168.1.2 (AD DNS server) as a DNS server in its ipv4 network settings?

Do I need masquerade or anything like that to accomplish these goals? If so, what's the Windows equivalent?

Do I need Fedora at 192.168.2.2 (FreeIPA server) to see 192.168.1.1 as a gateway for internet traffic, or could/should all internet traffic go through 192.168.1.2 (AD DC DNS) before to/from gateway?

Little stuff like this has been tripping me up. If anyone could help out, I'd really appreciate it!

Edit: Just thought of a couple other things:

Do I need to set up something on my switch to be able to transmit two subnets? BTW it's a powerconnect 7048P, it has VLAN and layer 3 capabilities.

Should I set up another vNIC on the Windows (192.168.1.2/24) AD DNS VM in order to communicate with 192.168.2.0/24 (FreeIPA network)?

In the same vein as the last question, should I set up another VM network for the Linux VMs (192.168.2.0/24), or is that not necessary?

Thanks
 
Last edited:

ttabbal

Active Member
Mar 10, 2016
767
209
43
44
Masquerade is NAT. You don't need it for this. Forget all the upper layer stuff for now, you need to get L3 working. Unless you are using L3 routing in the switch, you don't need to configure it for subnets.

The subnets need to be isolated, either physically or with VLANs. It is possible to reduce the subnet mask to make them all one subnet I suppose, but they aren't different subnets at that point. You could set the mask to /16 if you want that.

To route between them, you need a router connected to both. This can be your internet gateway, but it would need to be aware of the other subnet and have an interface on it. Then you would usually need to set that router as the gateway on the machines on both subnets, and add firewall rules to permit the traffic you wish to have.

The inter-vlan router can be the L3 switch as well. It has all the same requirements.
 
  • Like
Reactions: AveryFreeman

AveryFreeman

consummate homelabber
Mar 17, 2017
254
23
18
39
Near Seattle
averyfreeman.com
Masquerade is NAT. You don't need it for this. Forget all the upper layer stuff for now, you need to get L3 working. Unless you are using L3 routing in the switch, you don't need to configure it for subnets.

To route between them, you need a router connected to both. This can be your internet gateway, but it would need to be aware of the other subnet and have an interface on it. Then you would usually need to set that router as the gateway on the machines on both subnets, and add firewall rules to permit the traffic you wish to have.
Ah, OK, so you think I should be routing through the gateway? I hadn't actually thought of that. I thought if I had a conditional forwarder set to forward traffic it could forward it to a second subnet, regardless of it all being on the same physical network.

I could set up new virtual fabric, but all physical connections will be limited to using the same fabric + interfaces, so maybe that's where the disconnect is happening mentally. It's possible what I'm proposing isn't possible after all.

If there is a necessity for new virtual network, that would be easy enough to set up, as I am virtualizing the gateway ATM. Maybe that would help with being able to 'visualize" how the whole thing is supposed to work, too.

The subnets need to be isolated, either physically or with VLANs. It is possible to reduce the subnet mask to make them all one subnet I suppose, but they aren't different subnets at that point. You could set the mask to /16 if you want that.

The inter-vlan router can be the L3 switch as well. It has all the same requirements.
I'd like to avoid lowering the resolution of the netmask, if possible. VLANs would be something I'd like to implement eventually, but I'm trying to keep it as simple as possible for now. I don't particularly need VLANs for security, but they would be nice for reducing possibility of packet collisions.

The idea is:

One subnet is for Linux clients, and has its own DNS & identity management (FreeIPA)
One subnet is for Windows clients, and has its own DNS & identity management (AD)

The Windows and Linux clients should be able to see each other through a transitive trust relationship implemented in AD + IDM (FreeIPA), regardless if there are VLANs or not.

Ideally it will all have one gateway, but implementing a second one is entirely possible.

Does that sound crazy? Or do you think I should try and implement something different? What would you do if you had two sets of ID management frameworks that require having their own DNS?
 

ttabbal

Active Member
Mar 10, 2016
767
209
43
44
I think you might be missing some parts of how IP works. The stack compares the destination address to the netmask and local IP to see if it is on the same subnet. If it is not, it checks the routing table for a route to that subnet. This can be a specific route, or the default route. It sends the packet to whatever address that route specifies. To get between subnets you need a router of some kind. A router is a device with an interface on each subnet you want to route between and some rules about how to do so. The most basic is just routing table entries. On top of that you can get security rules and such. This can live on the internet gateway, so long as whatever you are using there is configurable enough to do it. The junk ISPs usually give out, likely won't be able to. Note that a gateway IS a router. To get packets between subnets you need a router. It can be almost anywhere, it doesn't have to be the default route, but if it isn't, every machine needs to know about it via a routing table rule.

Since they are on the same physical link, you could also create virtual interfaces on some machines that link to the other subnet. Those machines would then be able to communicate to both subnets. This is pretty much the same thing as lowering the netmask though, just a bit more complicated. And will break if you do VLANs later.

The whole thing might be easier if you do VLAN first, as it forces you to think of them as completely separate networks, which they are. If you find VLAN complex, just use different switches and physically isolate them.

I know you want to do directories and such, but right now the only tools you should be using to test things are ping and traceroute. Get IP and routing working, then start up on the other things.
 
  • Like
Reactions: AveryFreeman

AveryFreeman

consummate homelabber
Mar 17, 2017
254
23
18
39
Near Seattle
averyfreeman.com
This definitely made the most sense to me so far.

A question, then:

What would you do? Lower the resolution and assign a range from part of the subnet to each of the directory servers' DHCP? Or my cockamamie plan to have two subnets connected via static routes?

re: VLAN, I agree, I need to attack this in chunks. If you think VLANs are a good place to start, I'll start there. What's a good starting point for isolation - different types of devices (e.g. cameras, networking equipment, servers, clients, guests)? Windows vs Linux vs hypervisor infrastructure vs my girlfriend's iPhone?
 

ttabbal

Active Member
Mar 10, 2016
767
209
43
44
I would, and did, separate them out with VLANs and use my OpnSense firewall to route between them, the little traffic I allow between them. My setup is pretty simple.

Internal network. Reasonably secure/trusted devices that I want to generally allow to do most things. Main file server, printers, PCs and phones for the family. I don't have any reason to separate Windows/Linux stuff, so I don't. I also don't have Windows servers though, so perhaps there is reason to do it.

IoT/Guest network. Untrusted, no internal network access, internet allowed.

Security Cameras. No access to anything other than the DVR, which is a VM on the main server. no internet.

Management. This is for things like the IPMI ports on servers. It's a physically isolated network with a small switch.

I built this up over time. Start simple with internal/guest. The firewall is a separate machine with 2 Intel 1gig ports. One connects to the primary switch. One to the internet link. VLAN interfaces are created on the internal port and the switch port is set to trunk mode with the various VLAN tags I want it to be able to see.

The wifi APs support VLAN per SSID, so I have an internal and a guest SSID. You could also use separate APs and tell the switch which VLAN the port is on.



The easiest way to start, I think, is to get a managed switch and learn how to set up VLANs on it. It varies a little based on the manufacturer, but the basic idea is that there are 2 modes for a port. The default is usually just a host port on VLAN0. So the device on that port is on VLAN0, no options. That device doesn't have to be VLAN aware, or know anything about it. You can change which VLAN it's on, but all traffic on that port is always on that one VLAN. This is also called "untagged" sometimes.

Trunk ports can have many VLANs on them. Most switches allow you to allow any or to specify which IDs are allowed. The devices on the port need to know how to tag packets for VLANs, though there is usually a default VLAN you can specify. So the firewall port in my setup is a trunk port. So is the main VM/file server.


I started with static addresses and host ports. So now I have 2 machines, on different VLANs and subnets, with static addresses. They can't see each other. One was on the default VLAN0, and has internet via the firewall. The other is now by itself and can't do much of anything. Change the firewall port to trunk and allow the VLANs to it, say 0 and 1. Then create a virtual interface on the firewall for VLAN1 and assign it an address. Now you should be able to ping between the firewall and the other machine. You might have to add a firewall rule to allow it. From here you can add internet access if you like, or allow the VLANs to talk to each other, or whatever. The firewall in my case also handles DHCP and such for the various VLANs. My first goal was a guest network. So I also set the AP port for trunking and configured it for VLAN after that, and used my firewall interface for internet only. And then the guest network was done.


In my setup, this means all traffic between VLANs has to traverse a 1gig link to the firewall, and back. In practice, this is a non-issue for me, as I don't allow much traffic between them, but it's something to be aware of. So something like a file server might end up needing a larger link if you do 10G+. You might also want to explore having the switch handle VLAN routing. It's usually trickier to set up though.

Just remember that every VLAN is a separate network. It's just like using different switches for each one with no connections between them. Sort of like how VMs are isolated from each other.
 
  • Like
Reactions: AveryFreeman

AveryFreeman

consummate homelabber
Mar 17, 2017
254
23
18
39
Near Seattle
averyfreeman.com
I would, and did, separate them out with VLANs and use my OpnSense firewall to route between them, the little traffic I allow between them. My setup is pretty simple.

Internal network. Reasonably secure/trusted devices that I want to generally allow to do most things. Main file server, printers, PCs and phones for the family. I don't have any reason to separate Windows/Linux stuff, so I don't. I also don't have Windows servers though, so perhaps there is reason to do it.

In my setup . . . all traffic between VLANs has to traverse a 1gig link to the firewall, and back. In practice, this is a non-issue for me, as I don't allow much traffic between them, but it's something to be aware of. So something like a file server might end up needing a larger link if you do 10G+. You might also want to explore having the switch handle VLAN routing. It's usually trickier to set up though.
I'm thinking I'll put my main file server on a more permissive VLAN, I'd like to have access to it at least from the preponderance of both Linux and Windows devices.

Funny you mention OPNsense, I also use it. I'd rather not have VLANs have to route LAN through the firewall, as I am using some 10Gbe. I have a pretty nice switch with lots of features I've never even touched, it'll do layer 3 routing, so I'll have to see if it'll take the burden off the firewall. (it's a dell powerconnect 7048P) Although if you're using a LAN or OPT port in OPNsense, why would the VLAN traffic be going through the firewall ... (?)

FYI, this is only tangentially related, but OPNsense is notorious for being slow at LAN routing - the HardenedBSD kernel + default tuning was only yielding 1.9-2.8Gbps for me in iperf3 between LAN clients. To be fair, the default FreeBSD kernel + /boot/loader.conf and /etc/sysctl.conf defaults were also slower than line, around 6.4Gbps. I compiled a kernel on FreeBSD 12.2 with the Netflix RACK mod for OPNsense, and altered a few tuning variables, including enabling TSO, LRO, other offload features (one coincidentally is VLAN offload) and I am now able to get line speed on the LAN (9.4Gbps).

There's a great page about how to modify FreeBSD network settings and compile the modified kernel here: FreeBSD Network Performance Tuning @ Calomel.org

IoT/Guest network. Untrusted, no internal network access, internet allowed.

Security Cameras. No access to anything other than the DVR, which is a VM on the main server. no internet.

Management. This is for things like the IPMI ports on servers. It's a physically isolated network with a small switch.
Wow, that DOES sound very secure. But what do you do when you need to change settings on your IOT devices or security cameras? How many devices do you have on the isolated network for IPMI/other config?

I built this up over time. Start simple with internal/guest. The firewall is a separate machine with 2 Intel 1gig ports. One connects to the primary switch. One to the internet link. VLAN interfaces are created on the internal port and the switch port is set to trunk mode with the various VLAN tags I want it to be able to see.

The wifi APs support VLAN per SSID, so I have an internal and a guest SSID. You could also use separate APs and tell the switch which VLAN the port is on.
OK cool, that sounds good. I think I can do that with Unifi AP..

The easiest way to start, I think, is to get a managed switch and learn how to set up VLANs on it. It varies a little based on the manufacturer, but the basic idea is that there are 2 modes for a port. The default is usually just a host port on VLAN0. So the device on that port is on VLAN0, no options. That device doesn't have to be VLAN aware, or know anything about it. You can change which VLAN it's on, but all traffic on that port is always on that one VLAN. This is also called "untagged" sometimes.

Trunk ports can have many VLANs on them. Most switches allow you to allow any or to specify which IDs are allowed. The devices on the port need to know how to tag packets for VLANs, though there is usually a default VLAN you can specify. So the firewall port in my setup is a trunk port. So is the main VM/file server.
Awesome. Thanks for laying this all out for me. The examples and terminology are super helpful - I was aware of tagging, but the term trunk port is new. I can do certain clients on VLAN0 so they can see everything, right? And then certain VLANs that can see some but not others ... (?) I will probably make it a bit more permissive that your setup in some respects, but I also might separate Windows and Linux servers, as I have some 50 or so VMs, and am always making more.

A quick aside: I'd like to switch to DHCP ranges for blocks of specific devices, so I can get away from setting static IPs for everything. It would make the adoption of new clients easier, so I can stop having to check my DNS server or run an IP scanner looking for space to put a new device. Since each of the directory servers (AD + FreeIPA) have their own integrated (read: required) DHCP and DNS servers, it should work out really well if it all plays nice. Things like network devices (switch, gateway/firewall, file server, hypervisors, IPMI, etc. will definitely remain static.

If you were doing something similar (two DIR servers for different platforms) would you prefer to put them both on the same, say, 0/16 and just give each of them separate blocks of DHCP ranges? Or do you think you'd opt for separate subnets like I was first thinking? There's nothing wrong with putting multiple DNS servers on the same subnet, I take it? (outside of failover, of course)

I started with static addresses and host ports. So now I have 2 machines, on different VLANs and subnets, with static addresses. They can't see each other. One was on the default VLAN0, and has internet via the firewall. The other is now by itself and can't do much of anything. Change the firewall port to trunk and allow the VLANs to it, say 0 and 1. Then create a virtual interface on the firewall for VLAN1 and assign it an address. Now you should be able to ping between the firewall and the other machine. You might have to add a firewall rule to allow it. From here you can add internet access if you like, or allow the VLANs to talk to each other, or whatever. The firewall in my case also handles DHCP and such for the various VLANs. My first goal was a guest network. So I also set the AP port for trunking and configured it for VLAN after that, and used my firewall interface for internet only. And then the guest network was done.

Just remember that every VLAN is a separate network. It's just like using different switches for each one with no connections between them. Sort of like how VMs are isolated from each other.
Nice. I am going to make some charts and see if I can't visualize this... thanks again
 

ttabbal

Active Member
Mar 10, 2016
767
209
43
44
I'm thinking I'll put my main file server on a more permissive VLAN, I'd like to have access to it at least from the preponderance of both Linux and Windows devices.
You will need to set up a route to it then. Another option is to give the server an interface on each VLAN. They can be virtual interfaces on the same physical interface. Exactly how you do that depends on the distro, Proxmox has a web UI that includes support for it.

Funny you mention OPNsense, I also use it. I'd rather not have VLANs have to route LAN through the firewall, as I am using some 10Gbe. I have a pretty nice switch with lots of features I've never even touched, it'll do layer 3 routing, so I'll have to see if it'll take the burden off the firewall. (it's a dell powerconnect 7048P) Although if you're using a LAN or OPT port in OPNsense, why would the VLAN traffic be going through the firewall ... (?)

FYI, this is only tangentially related, but OPNsense is notorious for being slow at LAN routing - the HardenedBSD kernel + default tuning was only yielding 1.9-2.8Gbps for me in iperf3 between LAN clients. To be fair, the default FreeBSD kernel + /boot/loader.conf and /etc/sysctl.conf defaults were also slower than line, around 6.4Gbps. I compiled a kernel on FreeBSD 12.2 with the Netflix RACK mod for OPNsense, and altered a few tuning variables, including enabling TSO, LRO, other offload features (one coincidentally is VLAN offload) and I am now able to get line speed on the LAN (9.4Gbps).

There's a great page about how to modify FreeBSD network settings and compile the modified kernel here: FreeBSD Network Performance Tuning @ Calomel.org
The traffic has to traverse the firewall, but it doesn't have to go outside. You can route OPT->OPT just as easily. I'm not using 10Gb on the firewall right now, my traffic on that link is quite low. It's mostly used for internet traffic. I probably should upgrade it though, now that I think about it, as I recently got 1gb internet.

The reason the traffic has to go there is that the VLANs have no link to each other, so the IP stack sends to the default route, which is the firewall. It sees that the traffic destination is VLAN2, and forwards it over there, if allowed.

Wow, that DOES sound very secure. But what do you do when you need to change settings on your IOT devices or security cameras? How many devices do you have on the isolated network for IPMI/other config?
I have multiple lines to my office from the server rack. One of them is a trunk port for the VLANs, so I can add interfaces on my workstation or VMs on it. So if I want to configure the security cameras, I set up a virtual interface for that VLAN, then my workstation can communicate with that VLAN directly.

The IPMI network is on it's own switch, with another of those lines in my office that I can connect to. This is for the config ports on the switches, IPMI for the servers.. I think there are 6 connections on it right now.

OK cool, that sounds good. I think I can do that with Unifi AP..
Yes, Unifi can do it, I have one of their APs running with it now.

Awesome. Thanks for laying this all out for me. The examples and terminology are super helpful - I was aware of tagging, but the term trunk port is new. I can do certain clients on VLAN0 so they can see everything, right? And then certain VLANs that can see some but not others ... (?) I will probably make it a bit more permissive that your setup in some respects, but I also might separate Windows and Linux servers, as I have some 50 or so VMs, and am always making more.

A quick aside: I'd like to switch to DHCP ranges for blocks of specific devices, so I can get away from setting static IPs for everything. It would make the adoption of new clients easier, so I can stop having to check my DNS server or run an IP scanner looking for space to put a new device. Since each of the directory servers (AD + FreeIPA) have their own integrated (read: required) DHCP and DNS servers, it should work out really well if it all plays nice. Things like network devices (switch, gateway/firewall, file server, hypervisors, IPMI, etc. will definitely remain static.

If you were doing something similar (two DIR servers for different platforms) would you prefer to put them both on the same, say, 0/16 and just give each of them separate blocks of DHCP ranges? Or do you think you'd opt for separate subnets like I was first thinking? There's nothing wrong with putting multiple DNS servers on the same subnet, I take it? (outside of failover, of course)


Nice. I am going to make some charts and see if I can't visualize this... thanks again

Not exactly, VLAN0 is usually just the default VLAN. It doesn't let you talk to other VLANs. VLANs are isolated at Layer 2 and cannot see packets on the other VLANs. If you want a single client on multiple VLANS, you need to have a client that can deal with VLANs, and set up an interface for each VLAN on that client. You could also use multiple physical connections. It is possible with some switches to create monitor ports, but it usually only works for a port or two and puts a load on the switch. It's intended to be used for diagnostics I believe. Just think of them the same as completely isolated, as if you had multiple switches.

DNS you can have as many servers as you like. DHCP is another matter. There can only be one active DHCP server on a layer 2 link. This is because it uses broadcast packets to communicate. So you will probably need the directory servers on their own VLAN. You could have a router connect them and allow traffic. There are some setups for DHCP that allow failover and such, but those are more complex and not what you're talking about wanting. The directory platforms will need to be on different subnets as well.

Honestly, I think talking about it makes it sound more difficult than it really is. :) I suggest just setting up a test network or even just a few ports with VLAN settings and experiment. I think it's easier to do than to discuss really.