Noob question on L3 switch routing

[Think]

I am currently using my l3 capable switch only for l2 - tagged and untagged vlans. My network is running with a pfSense firewall which handles inter-VLAN (192.168.1.0/24, 192.168.2.0/24, ...) routing and internet access. The switch has an IP defined in each VLAN.

Now I'd like to change things such that inter-VLAN routing will be handled by a separate router, while pfSense continues to handle Internet NATing. I guess I could implement this by giving all local machines the router as gateway, and configuring pfSense as the gateway on the router. However, I wonder if I could also make this more efficient and avoid Internet traffic going through the router by configuring two routes on the switch in the middle:

1. 192.168.0.0/16 to the router
2. default route to pfSense

Could this work? And would the switch still send traffic correctly within the /24 subnets and only use the routes for traffic leaving the respective VLANs?

 

[kpfleming]

So you want to have the switch running in layer 3 mode (thus a router), and a router handling inter-VLAN routing, and a router (pfSense) handling extra-VLAN routing?

 

[DavidWJohnston]

The way I've done inter-VLAN routing is using my 100G L3 switch. You could accomplish the same thing with yours. In this config, each VLAN no longer has an interface in the pfSense; instead there is a transit VLAN between pfSense and L3-Switch, with static routes for return packets. Here is roughly what you have to do:

• Goal: Have the pfSense handle internet routing, but your L3 switch handle inter-VLAN routing
• Give your L3 switch an IP on each of your VLAN interfaces. I use 10.41.x.1 for my L3 switch in each VLAN x
• Create a "transit" VLAN that handles routing between the switch and pfSense (I use 10.41.255.1/30)
• Create an interface on pfSense for the transit VLAN, I assigned it 10.41.255.2/30
• Create a default route on the L3 switch pointing to the pfSense IP on the transit VLAN
• Delete/disable all the LAN interfaces on pfSense; the L3 switch will now service those (Keep the Transit, WANs, VPNs, etc)
• Create static routes for all your VLANs, pointing to the the L3 switch transit network's IP (In my case 10.41.255.1)
• Use the DHCP helper feature in the L3 switch to send DHCP requests to a subnet-aware DHCP server (Like Windows Server)

If you can't just break your current setup, you'll need to stage the new infra in parallel then cut it over to maintain service.

Here are some screenshots to help visualize it:

The purpose of the static routes in pfSense is to handle return traffic - When stuff on the internet responds to your local devices, the pfSense needs to know where to send the packets, as the pfSense no longer has interfaces for those subnets - So you put the IP of the L3 switch inside the transit VLAN, where pfSense does have an interface. The transit VLANs are only for routers to pass traffic between each other.


Any firewall rules for devices need to go inside the Transit VLAN, and you can use the Source IP to match which device you want the rule to affect. In my case, I have an alias called "aBrowseThruBoxPN" which specifies certain devices to browse over a VPN instead of my normal WAN. The "Allow All" rule at the bottom allows any device on any VLAN to access the internet. You may not want this. Create Aliases for each VLAN, then use those to control whole subnets.

admin@sonic:~$ show vlan brief
+-----------+----------------+-------------+----------------+-------------+-----------------------+
|   VLAN ID | IP Address     | Ports       | Port Tagging   | Proxy ARP   | DHCP Helper Address   |
+===========+================+=============+================+=============+=======================+
|        50 | 10.41.255.1/30 | Ethernet0   | tagged         | disabled    |                       |
|           |                | Ethernet4   | tagged         |             |                       |
|           |                | Ethernet8   | tagged         |             |                       |
|           |                | Ethernet16  | tagged         |             |                       |
|           |                | Ethernet36  | tagged         |             |                       |
+-----------+----------------+-------------+----------------+-------------+-----------------------+
|       102 |                | Ethernet0   | tagged         | disabled    |                       |
|           |                | Ethernet4   | tagged         |             |                       |
|           |                | Ethernet8   | tagged         |             |                       |
|           |                | Ethernet16  | tagged         |             |                       |
|           |                | Ethernet36  | tagged         |             |                       |
|           |                | Ethernet112 | untagged       |             |                       |
+-----------+----------------+-------------+----------------+-------------+-----------------------+
|       107 | 192.168.7.4/24 | Ethernet0   | tagged         | disabled    |                       |
|           |                | Ethernet4   | tagged         |             |                       |
|           |                | Ethernet8   | tagged         |             |                       |
|           |                | Ethernet16  | tagged         |             |                       |
|           |                | Ethernet36  | tagged         |             |                       |
+-----------+----------------+-------------+----------------+-------------+-----------------------+
|      1000 | 10.41.0.1/24   | Ethernet0   | tagged         | disabled    | 10.41.2.20            |
|           |                | Ethernet4   | tagged         |             | 10.41.2.21            |
|           |                | Ethernet8   | tagged         |             |                       |
|           |                | Ethernet16  | tagged         |             |                       |
|           |                | Ethernet36  | tagged         |             |                       |
+-----------+----------------+-------------+----------------+-------------+-----------------------+
|      1001 | 10.41.1.1/24   | Ethernet0   | tagged         | disabled    | 10.41.2.20            |
|           |                | Ethernet4   | tagged         |             | 10.41.2.21            |
|           |                | Ethernet8   | tagged         |             |                       |
|           |                | Ethernet16  | tagged         |             |                       |
|           |                | Ethernet32  | untagged       |             |                       |
|           |                | Ethernet36  | tagged         |             |                       |
|           |                | Ethernet88  | untagged       |             |                       |
|           |                | Ethernet116 | untagged       |             |                       |
|           |                | Ethernet120 | untagged       |             |                       |
|           |                | Ethernet124 | untagged       |             |                       |
+-----------+----------------+-------------+----------------+-------------+-----------------------+
|      1002 | 10.41.2.1/24   | Ethernet0   | tagged         | disabled    | 10.41.2.20            |
|           |                | Ethernet4   | tagged         |             | 10.41.2.21            |
|           |                | Ethernet8   | tagged         |             |                       |
|           |                | Ethernet16  | tagged         |             |                       |
|           |                | Ethernet36  | tagged         |             |                       |
|           |                | Ethernet38  | untagged       |             |                       |
+-----------+----------------+-------------+----------------+-------------+-----------------------+
|      1003 | 10.41.3.1/24   | Ethernet0   | tagged         | disabled    | 10.41.2.20            |
|           |                | Ethernet4   | tagged         |             | 10.41.2.21            |
|           |                | Ethernet8   | tagged         |             |                       |
|           |                | Ethernet16  | tagged         |             |                       |
|           |                | Ethernet36  | tagged         |             |                       |
+-----------+----------------+-------------+----------------+-------------+-----------------------+
|      1008 | 10.41.8.1/24   | Ethernet0   | tagged         | disabled    | 10.41.2.20            |
|           |                | Ethernet4   | tagged         |             | 10.41.2.21            |
|           |                | Ethernet8   | tagged         |             |                       |
|           |                | Ethernet16  | tagged         |             |                       |
|           |                | Ethernet35  | tagged         |             |                       |
|           |                | Ethernet36  | tagged         |             |                       |
+-----------+----------------+-------------+----------------+-------------+-----------------------+
|      1009 | 10.41.9.1/24   | Ethernet0   | tagged         | disabled    | 10.41.2.20            |
|           |                | Ethernet4   | tagged         |             | 10.41.2.21            |
|           |                | Ethernet8   | tagged         |             |                       |
|           |                | Ethernet16  | tagged         |             |                       |
|           |                | Ethernet36  | tagged         |             |                       |
+-----------+----------------+-------------+----------------+-------------+-----------------------+

The L3 switch holds IPs in the VLANs, usually as *.1.

admin@sonic:~$ show ip int
Interface    Master    IPv4 address/mask    Admin/Oper    BGP Neighbor    Neighbor IP
-----------  --------  -------------------  ------------  --------------  -------------
Loopback0              10.1.0.1/32          up/up         N/A             N/A
Vlan50       Vrf_Prod  10.41.255.1/30       up/up         N/A             N/A
Vlan107      Vrf_Prod  192.168.7.4/24       up/up         N/A             N/A
Vlan1000     Vrf_Prod  10.41.0.1/24         up/up         N/A             N/A
Vlan1001     Vrf_Prod  10.41.1.1/24         up/up         N/A             N/A
Vlan1002     Vrf_Prod  10.41.2.1/24         up/up         N/A             N/A
Vlan1003     Vrf_Prod  10.41.3.1/24         up/up         N/A             N/A
Vlan1008     Vrf_Prod  10.41.8.1/24         up/up         N/A             N/A
Vlan1009     Vrf_Prod  10.41.9.1/24         up/up         N/A             N/A
docker0                240.127.1.1/24       up/down       N/A             N/A
eth0                   192.168.7.2/24       up/down       N/A             N/A
lo                     127.0.0.1/16         up/up         N/A             N/A

Here you can see the static route on the L3 switch pointing to pfSense at the other end of the transit VLAN. The other routes permit inter-VLAN traffic to pass. The routes in VRF default are to allow the DHCP helper traffic to pass into the Servers VLAN.

admin@sonic:~$ show ip route vrf all
Codes: K - kernel route, C - connected, S - static, R - RIP,
       O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
       T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP,
       F - PBR, f - OpenFabric,
       > - selected route, * - FIB route, q - queued, r - rejected, b - backup

VRF Vrf_Prod:
S>* 0.0.0.0/0 [1/0] via 10.41.255.2, Vlan50, weight 1, 01w3d22h
C>* 10.41.0.0/24 is directly connected, Vlan1000, 01w3d22h
C>* 10.41.1.0/24 is directly connected, Vlan1001, 01w3d22h
C>* 10.41.2.0/24 is directly connected, Vlan1002, 01w3d22h
C>* 10.41.3.0/24 is directly connected, Vlan1003, 01w3d22h
C>* 10.41.8.0/24 is directly connected, Vlan1008, 01w3d22h
C>* 10.41.9.0/24 is directly connected, Vlan1009, 01w3d22h
C>* 10.41.255.0/30 is directly connected, Vlan50, 01w3d22h
C>* 192.168.7.0/24 is directly connected, Vlan107, 01w3d22h

VRF default:
C>* 10.1.0.1/32 is directly connected, Loopback0, 01w3d22h
S>* 10.41.2.0/24 [1/0] is directly connected, Vlan1002 (vrf Vrf_Prod), weight 1, 01w3d22h
S>* 10.41.3.0/24 [1/0] is directly connected, Vlan1003 (vrf Vrf_Prod), weight 1, 01w3d22h

You'll need a subnet-aware DHCP server. Since I run AD I use MS DHCP (and DNS as well). You setup a scope for each VLAN with the desired IP range, domain suffix, etc.


Hopefully that gives a starting point. A "Hybrid" approach is also possible where both the L3 switch and pfSense act as a router for all VLANs. This allows you to keep using pfSense for DHCP. I don't like this option, because then you have 2 routers per VLAN doing asymmetric routing, and it causes all sorts of undesirable issues.

It's way better to use the enterprise-ish method of a subnet-aware multi-scope DHCP service and a small transit vlan between routers.

 

[klui]

@DavidWJohnston, couldn't you reference each subnet to their dedicated DHCP server IP instead of a single IP if one wants to use their existing DHCP infrastructure? I don't want to tie my infrastructure to another server/VM for DHCP/DNS simply because when I'm not available and we have downtime my family members would be lost on how to fix things. It's challenging enough for them know about modem, firewall, and switch.

This is the reason why I have not migrated my production network to my L3-capable switch and prefer to have most things running on bare metal that my family relies on.

 

[DavidWJohnston]

I think the best way to handle that would be to separate your network into home and lab. Home is a flat simple home network where the only troubleshooting needed is restarting a modem.

Then create a separate branch for your homelab. That way when your L3 switch/complex experimental lab setup breaks, your family may not be able to access Plex, but WiFi still works. The only thing they ever need to do is reset the modem, because that's all you can do, there is no config at all for the home portion, it's just your ISP's out-of-box experience.

The problem with pfSense having an interface on each subnet and do L3 switching isn't really due to DHCP - It's because of asymmetric routing and the issues that can cause. Asymmetric routing means egress packets and ingress/return packets take a different path.

This is asymmetric routing when a PC requests a website and gets a response back:

Egress path: PC -> L3 Switch -> pfSense -> ISP -> Website
Return path: Website -> ISP -> pfSense -> PC

The return path will not pass through the L3 switch, because pfSense has an interface on the destination subnet, so that's where it forwards the traffic. It is impossible to setup a static route for return traffic to go back through the L3 switch, because the return subnet is already a directly-connected subnet in the pfSense.

This asymmetric routing is the real source of the problem when you have pfSense maintain an interface in every subnet. The intent is just to allow DHCP, but it has a side effect of return traffic not going back through the L3 switch. This messes up the stateful firewall rules.

Another possible thing you can do is use static routes for specific IPs between your devices that pass traffic through the L3 switch instead of pfSense. For instance, between Plex in your DMZ and a file server on your Servers VLAN. You would need to manually configure static routes on devices where you want routing via L3 switch - But their default gateway remains the pfSense. Then you wouldn't have asymmetric routing.

I hope this makes sense, if not feel free to ask a follow-up, these situations are sometimes more complex than it seems at first.

 

[klui]

I did not take into account asymmetric routing nor have I any experience with it. So I greatly appreciate your answers here at STH.

When I have some time I will lab it up. I do separate my lab from production so there's that. I run an SRX instead of pfSense/OPNSense, where if the interfaces are in the same zone the firewall should be able to handle things.

 

[Think]

@DavidWJohnston - THANK YOU. This is surely one of the most helpful replies I have ever received - across all forums. I've read your post a few times, and learned a lot (I hope!).

So I think I have asymmetric routing in my current setup already, for the VPNs and 192.168.2.0/24 behind a transit VPN. I've quickly drawn a diagram of the current network and attached it to this post. The VPN router has an IP in 192.168.1.0/24, which I believe causes this problem. So I think I should change something in any case.

I've tried to come up with a new setup (which doesn't scrap everything that is currently in place), that addresses

• the asymmetric routing I have currently already, and
• moving inter-VLAN routing from the firewall (a small, overloaded ARM box) onto a VM on the quite beefy server.

I've also attached a diagram for a new setup which I hope correctly takes into account your advice (basically moving all routers into a transit subnet). Did I miss anything?

I may have a follow-up question on L3 switch routing still. But would first like to make sure that I got things right so far.

Thanks again!

 

[DavidWJohnston]

I took a look at your diagrams, and assuming I've read them correctly:

Your first diagram exhibits asymmetric routing when your PCs on 192.168.1.x (bottom-right of diagram) talk to anything via the VPN router - The PCs on 192.168.1.x have their default gateway set as 192.168.1.1, yet return packets coming back through the VPN router will go directly to the PCs, instead of passing back through 192.168.1.1, because the VPN router is directly connected via 192.168.1.117. Yup that's asymmetric routing.

Now with that being said, asymmetric routing, especially involving VPNs is not always catastrophic, though it is usually a pain. If you don't have any firewall rules that need to statefully track egress/response, like in a mostly-open set of VLANs it will "work".

The second diagram does appear to fix the asymmetric routing, IIUC. One thing of note is transit VLANs (in "old school" enterprise IT) are usually small (/30s), because router pairs are typically connected between buildings/sites using point-to-point links; they're not like a fabric where lots of routers live - But for labs, having a single larger transit VLAN where all routers have an IP is good for simplicity.

With the second diagram, I think it would work, just keep in-mind the following:

- Moving a router from physical to virtual can add additional single-points-of-failure. Previously, if your beefy server went down, your devices would still be able to access the internet - With the new config, their default gateway will be completely down.

- Physical (non-VM) inter-VLAN routing load will be moved off your ARM firewall, but it will still be limited by the interface and CPU on the server (1 GbE?) so all physical inter-VLAN traffic may still be bottlenecked. Inter-VLAN traffic inside your server (VM-to-VM) will not be subject to this limit, because those packets will be memcpy'd through the router VM, not through the server's real NIC.

Doing most of your routing on the L3 switch instead will improve performance, because you have the whole X GBps / TBps of backplane on the switch's ASIC to do your packet forwarding, instead of it happening in a VM through a NIC on an x86 CPU. It will also eliminate a point-of-failure, as the core switch itself is already one.

I'm assuming there's a reason you don't want to do most routing on your L3 switch, and I think your proposed network would work, and make routing problems easier to debug.

Edit: There's something important that I forgot - If you route via your L3 switch instead of pfSense, you lose the ability to implement inter-VLAN firewall rules using the pfSense GUI - Instead you must do it using ACLs on the switch, which can be a pain. But in return for that pain you get a big performance (and perhaps reliability) boost - So if you don't have (m)any inter-VLAN firewall rules it's probably worth it.

 

[awedio]

What are some other choices for a "subnet-aware DHCP server"?
I'd prefer to not run Windows AD.

 

[986box]

Is DHCP and DNS on Linux using dnsmasq comparable?

 

[DavidWJohnston]

Yes, dhcpd and dnsmasq in Linux can handle multi-subnet, multi-scope - You just specify more than one "subnet" directive in your dhcpd.conf, or multiple dhcp-range directives in dnsmasq. In fact, FreeBSD's implementation handles it too, its just pfSense doesn't provide a facility to take advantage of it.

Like this: DNSMASQ DHCP configuration for multiple subnets - Data center networking and virtualization

Also for this to work you need to use a DHCP relay (also called DHCP helper), so the DHCP requests from clients to the subnet's broadcast IP can be relayed to the DHCP server in a different subnet, and the responses relayed back.

The DHCP relay changes the source IP to itself, sends the packet along to the DHCP server's IP in a unicast packet (which is routed wherever it needs to go) - Then the DHCP server sends the response back to the relay via unicast, and the relay responds back to the corresponding client on behalf of the DHCP server. (The relay maintains a state table of which responses need to go back to which clients)

This way, you can have a DHCP server somewhere else on your network behind a single IP, without it needing to hear broadcasts from every VLAN. L3 switches always have a DHCP relay/helper feature built-in.

 

[awedio]

Does anyone make a GUI for DNSMASQ?

 

[DavidWJohnston]

Not sure about dnsmasq, but a quick Google search reveals there is a dhcpd admin module in the webmin framework:

DHCP Server

About This page explains what DHCP is and how to use Webmin to set up a DHCP server on your network so that other systems can obtain IP addresses automatically. Intro DHCP is a protocol that allows hosts to request and be assigned an IP address on a local area network. It is used to simplify the...

doxfer.webmin.com

There's also this GTK package in Ubuntu:

The Gui Dhcp Server Configuration Tool - Ubuntu Server Administration

Ubuntu now has a GUI configuration tool available, courtesy of the gdhcpd package. As it's built with GTK+ libraries, it does not require many additional

www.halolinux.us

 

[awedio]

Nice. Thank you for the link.

 

[Think]

Thanks for taking the time to look at my diagrams!

- Moving a router from physical to virtual can add additional single-points-of-failure. Previously, if your beefy server went down, your devices would still be able to access the internet - With the new config, their default gateway will be completely down.

And here we get to my follow-up questions. Because I started this thread as I was looking into using routing capabilities of the switch. However, what I was initially thinking about was just a few "backup" things to make sure that internet access will still be working for devices, if the server is down. If I understand things correctly, however, the simple solution that I thought of (having the switch route inter-VLAN traffic to a VLAN router and internet traffic to the firewall) won't even work with a transit VLAN. Because that would be asymmetric routing again, with VLAN traffic to the internet going from the core switch directly to the firewall, but traffic from the internet going from the firewall to the VLAN router first (while it's up, I could adjust routing dynamically on the pfSense).

So with this option not working, I would have to look into really taking the plunge of doing all inter-VLAN routing in the switch.

- Physical (non-VM) inter-VLAN routing load will be moved off your ARM firewall, but it will still be limited by the interface and CPU on the server (1 GbE?) so all physical inter-VLAN traffic may still be bottlenecked. Inter-VLAN traffic inside your server (VM-to-VM) will not be subject to this limit, because those packets will be memcpy'd through the router VM, not through the server's real NIC.

The server has a 2x10 Gbit LAGG to the switch with a 32 core EPYC CPU. It does 10 gig file serving within VLAN10. This is why I was considering the move away from the ARM... My hope was that the little incremental traffic from the internet and other VLANs would hardly be noticeable in top of this.

I'm assuming there's a reason you don't want to do most routing on your L3 switch, and I think your proposed network would work, and make routing problems easier to debug.

Well, this is my first L3 switch. I haven't done this before, I'm new to this. And there are a few things I'll need to figure out first:

• Do rules on a switch (while looking not as neat as on pfSense's GUI) work the same way as on pfSense, or do I have to do "more", since the switch is not stateful? I.e., on pfSense, with a whitelist and default block for each VLAN, I have (a lot of) rules focusing on outgoing traffic by VLAN, i.e. what which device on the VLAN may access (using source and destination IPs and protocols mainly). There are no rules for answers to such permitted traffic, it "just works" (because pfSense is stateful?). Would this concept work for stateless ACEs on the switch as well, or would I need more rules to also deal with responses?
• I will probably need a different solution for mDNS reflection. My CBS350 appears to not have this capability (unlike other Catalyst switches). So I may need to put a different machine on the required subnets for this (and am worried again about not causing asymmetric routing by this - it should probably not just another router doing this, as it would create a different flow of packets from that router into the VLANs.
• Aliases don't appear to work for ACEs. This will be a pain, as the rules will look much more difficult to understand, and I will need many more of them. Also some rules which depend on name resolution won't work at all, and in case of IP changes of individual devices, I will not just have to change an alias, but possibly many rules.
• Cisco has great documentation. But is there a tutorial/howto somewhere that goes into practical examples of having a working ACE setup with selected by-IP/port whitelist? I found high level tutorials ("this is how to create one rule so that you can start") and Ciscos detailed documentation with no working examples at this level of detail.

I do have an isc-dhcpd running (two, actually - one outside of the server in failover mode). So the subnet-aware dhcp server should not be an issue.

Edit: There's something important that I forgot - If you route via your L3 switch instead of pfSense, you lose the ability to implement inter-VLAN firewall rules using the pfSense GUI - Instead you must do it using ACLs on the switch, which can be a pain. But in return for that pain you get a big performance (and perhaps reliability) boost

I'm willing to give this a try despite the ton of inter-VLAN firewall rules - once I have a sense of how it could work and what I'd need to do to preserve current functionality. And there appears to be quite some learning required!

 

[vangoose]

The way I've done inter-VLAN routing is using my 100G L3 switch. You could accomplish the same thing with yours. In this config, each VLAN no longer has an interface in the pfSense; instead there is a transit VLAN between pfSense and L3-Switch, with static routes for return packets. Here is roughly what you have to do:

• Goal: Have the pfSense handle internet routing, but your L3 switch handle inter-VLAN routing
• Give your L3 switch an IP on each of your VLAN interfaces. I use 10.41.x.1 for my L3 switch in each VLAN x
• Create a "transit" VLAN that handles routing between the switch and pfSense (I use 10.41.255.1/30)
• Create an interface on pfSense for the transit VLAN, I assigned it 10.41.255.2/30
• Create a default route on the L3 switch pointing to the pfSense IP on the transit VLAN
• Delete/disable all the LAN interfaces on pfSense; the L3 switch will now service those (Keep the Transit, WANs, VPNs, etc)
• Create static routes for all your VLANs, pointing to the the L3 switch transit network's IP (In my case 10.41.255.1)
• Use the DHCP helper feature in the L3 switch to send DHCP requests to a subnet-aware DHCP server (Like Windows Server)

If you can't just break your current setup, you'll need to stage the new infra in parallel then cut it over to maintain service.

Here are some screenshots to help visualize it:

The purpose of the static routes in pfSense is to handle return traffic - When stuff on the internet responds to your local devices, the pfSense needs to know where to send the packets, as the pfSense no longer has interfaces for those subnets - So you put the IP of the L3 switch inside the transit VLAN, where pfSense does have an interface. The transit VLANs are only for routers to pass traffic between each other.
View attachment 28594

Any firewall rules for devices need to go inside the Transit VLAN, and you can use the Source IP to match which device you want the rule to affect. In my case, I have an alias called "aBrowseThruBoxPN" which specifies certain devices to browse over a VPN instead of my normal WAN. The "Allow All" rule at the bottom allows any device on any VLAN to access the internet. You may not want this. Create Aliases for each VLAN, then use those to control whole subnets.
View attachment 28595

admin@sonic:~$ show vlan brief
+-----------+----------------+-------------+----------------+-------------+-----------------------+
|   VLAN ID | IP Address     | Ports       | Port Tagging   | Proxy ARP   | DHCP Helper Address   |
+===========+================+=============+================+=============+=======================+
|        50 | 10.41.255.1/30 | Ethernet0   | tagged         | disabled    |                       |
|           |                | Ethernet4   | tagged         |             |                       |
|           |                | Ethernet8   | tagged         |             |                       |
|           |                | Ethernet16  | tagged         |             |                       |
|           |                | Ethernet36  | tagged         |             |                       |
+-----------+----------------+-------------+----------------+-------------+-----------------------+
|       102 |                | Ethernet0   | tagged         | disabled    |                       |
|           |                | Ethernet4   | tagged         |             |                       |
|           |                | Ethernet8   | tagged         |             |                       |
|           |                | Ethernet16  | tagged         |             |                       |
|           |                | Ethernet36  | tagged         |             |                       |
|           |                | Ethernet112 | untagged       |             |                       |
+-----------+----------------+-------------+----------------+-------------+-----------------------+
|       107 | 192.168.7.4/24 | Ethernet0   | tagged         | disabled    |                       |
|           |                | Ethernet4   | tagged         |             |                       |
|           |                | Ethernet8   | tagged         |             |                       |
|           |                | Ethernet16  | tagged         |             |                       |
|           |                | Ethernet36  | tagged         |             |                       |
+-----------+----------------+-------------+----------------+-------------+-----------------------+
|      1000 | 10.41.0.1/24   | Ethernet0   | tagged         | disabled    | 10.41.2.20            |
|           |                | Ethernet4   | tagged         |             | 10.41.2.21            |
|           |                | Ethernet8   | tagged         |             |                       |
|           |                | Ethernet16  | tagged         |             |                       |
|           |                | Ethernet36  | tagged         |             |                       |
+-----------+----------------+-------------+----------------+-------------+-----------------------+
|      1001 | 10.41.1.1/24   | Ethernet0   | tagged         | disabled    | 10.41.2.20            |
|           |                | Ethernet4   | tagged         |             | 10.41.2.21            |
|           |                | Ethernet8   | tagged         |             |                       |
|           |                | Ethernet16  | tagged         |             |                       |
|           |                | Ethernet32  | untagged       |             |                       |
|           |                | Ethernet36  | tagged         |             |                       |
|           |                | Ethernet88  | untagged       |             |                       |
|           |                | Ethernet116 | untagged       |             |                       |
|           |                | Ethernet120 | untagged       |             |                       |
|           |                | Ethernet124 | untagged       |             |                       |
+-----------+----------------+-------------+----------------+-------------+-----------------------+
|      1002 | 10.41.2.1/24   | Ethernet0   | tagged         | disabled    | 10.41.2.20            |
|           |                | Ethernet4   | tagged         |             | 10.41.2.21            |
|           |                | Ethernet8   | tagged         |             |                       |
|           |                | Ethernet16  | tagged         |             |                       |
|           |                | Ethernet36  | tagged         |             |                       |
|           |                | Ethernet38  | untagged       |             |                       |
+-----------+----------------+-------------+----------------+-------------+-----------------------+
|      1003 | 10.41.3.1/24   | Ethernet0   | tagged         | disabled    | 10.41.2.20            |
|           |                | Ethernet4   | tagged         |             | 10.41.2.21            |
|           |                | Ethernet8   | tagged         |             |                       |
|           |                | Ethernet16  | tagged         |             |                       |
|           |                | Ethernet36  | tagged         |             |                       |
+-----------+----------------+-------------+----------------+-------------+-----------------------+
|      1008 | 10.41.8.1/24   | Ethernet0   | tagged         | disabled    | 10.41.2.20            |
|           |                | Ethernet4   | tagged         |             | 10.41.2.21            |
|           |                | Ethernet8   | tagged         |             |                       |
|           |                | Ethernet16  | tagged         |             |                       |
|           |                | Ethernet35  | tagged         |             |                       |
|           |                | Ethernet36  | tagged         |             |                       |
+-----------+----------------+-------------+----------------+-------------+-----------------------+
|      1009 | 10.41.9.1/24   | Ethernet0   | tagged         | disabled    | 10.41.2.20            |
|           |                | Ethernet4   | tagged         |             | 10.41.2.21            |
|           |                | Ethernet8   | tagged         |             |                       |
|           |                | Ethernet16  | tagged         |             |                       |
|           |                | Ethernet36  | tagged         |             |                       |
+-----------+----------------+-------------+----------------+-------------+-----------------------+

The L3 switch holds IPs in the VLANs, usually as *.1.

admin@sonic:~$ show ip int
Interface    Master    IPv4 address/mask    Admin/Oper    BGP Neighbor    Neighbor IP
-----------  --------  -------------------  ------------  --------------  -------------
Loopback0              10.1.0.1/32          up/up         N/A             N/A
Vlan50       Vrf_Prod  10.41.255.1/30       up/up         N/A             N/A
Vlan107      Vrf_Prod  192.168.7.4/24       up/up         N/A             N/A
Vlan1000     Vrf_Prod  10.41.0.1/24         up/up         N/A             N/A
Vlan1001     Vrf_Prod  10.41.1.1/24         up/up         N/A             N/A
Vlan1002     Vrf_Prod  10.41.2.1/24         up/up         N/A             N/A
Vlan1003     Vrf_Prod  10.41.3.1/24         up/up         N/A             N/A
Vlan1008     Vrf_Prod  10.41.8.1/24         up/up         N/A             N/A
Vlan1009     Vrf_Prod  10.41.9.1/24         up/up         N/A             N/A
docker0                240.127.1.1/24       up/down       N/A             N/A
eth0                   192.168.7.2/24       up/down       N/A             N/A
lo                     127.0.0.1/16         up/up         N/A             N/A

Here you can see the static route on the L3 switch pointing to pfSense at the other end of the transit VLAN. The other routes permit inter-VLAN traffic to pass. The routes in VRF default are to allow the DHCP helper traffic to pass into the Servers VLAN.

admin@sonic:~$ show ip route vrf all
Codes: K - kernel route, C - connected, S - static, R - RIP,
       O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
       T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP,
       F - PBR, f - OpenFabric,
       > - selected route, * - FIB route, q - queued, r - rejected, b - backup

VRF Vrf_Prod:
S>* 0.0.0.0/0 [1/0] via 10.41.255.2, Vlan50, weight 1, 01w3d22h
C>* 10.41.0.0/24 is directly connected, Vlan1000, 01w3d22h
C>* 10.41.1.0/24 is directly connected, Vlan1001, 01w3d22h
C>* 10.41.2.0/24 is directly connected, Vlan1002, 01w3d22h
C>* 10.41.3.0/24 is directly connected, Vlan1003, 01w3d22h
C>* 10.41.8.0/24 is directly connected, Vlan1008, 01w3d22h
C>* 10.41.9.0/24 is directly connected, Vlan1009, 01w3d22h
C>* 10.41.255.0/30 is directly connected, Vlan50, 01w3d22h
C>* 192.168.7.0/24 is directly connected, Vlan107, 01w3d22h

VRF default:
C>* 10.1.0.1/32 is directly connected, Loopback0, 01w3d22h
S>* 10.41.2.0/24 [1/0] is directly connected, Vlan1002 (vrf Vrf_Prod), weight 1, 01w3d22h
S>* 10.41.3.0/24 [1/0] is directly connected, Vlan1003 (vrf Vrf_Prod), weight 1, 01w3d22h

You'll need a subnet-aware DHCP server. Since I run AD I use MS DHCP (and DNS as well). You setup a scope for each VLAN with the desired IP range, domain suffix, etc.
View attachment 28596

Hopefully that gives a starting point. A "Hybrid" approach is also possible where both the L3 switch and pfSense act as a router for all VLANs. This allows you to keep using pfSense for DHCP. I don't like this option, because then you have 2 routers per VLAN doing asymmetric routing, and it causes all sorts of undesirable issues.

It's way better to use the enterprise-ish method of a subnet-aware multi-scope DHCP service and a small transit vlan between routers.

I have the same design as you, my DMZ/VPN are routed and segmented by SRX, internal networks are routed by Arista, there are vrf and bgp between Arista and SRX.

People need to think networks as zones and design from there.

 

[DavidWJohnston]

"... having the switch route inter-VLAN traffic to a VLAN router and internet traffic to the firewall"

OK I mis-read your second diagram, I thought the .222 default gateway for all the devices was the VM - But it's actually the L3 switch - So yes that would still have asymmetric routing. Return packets would not be routed back through the L3 switch's routing table. The router VM has a directly connected interface for each VLAN, so return packets would go out of those, not back across the transit.

ACLs on L3 switches for the data plane are not usually stateful - They specify ingress/egress, and sometimes there are limitations depending on the brand/model of switch. Sometimes you can specify tcp flags in the rules, to only allow the conversation to be initiated in one direction.

Control plane rules (like allowing SSH into the mgmt interface on the switch) are usually stateful - But you need to read your docs to know for sure.

In general, use as few rules as possible, and probably don't refer to specific devices in your ACLs. Instead, allow or deny entire VLANs from talking to each other.

In my case, my ISP's out-of-box residential (Bell Fibe FTTH) home network NAT, WiFi, etc is still there (192.168.2.x) - That's where I put outage-sensitive stuff or devices needing mDNS/UPnP or internet only, like smart switches, work laptops, guest phones, etc. My main personal workstation has a NIC on this network that I usually keep disabled. If I break something, I enable that NIC, and I have a jump box straddling my networks I can use to get things back up without going to the basement.

You've got the right thought process going, thinking of all the variables. Maybe try it with a test subnet first to get a feel for it and test out various applications, and see how it goes. Then you could cut-over other VLANs out of pfSense and into your L3 switch as you solve problems.

 

[klui]

ACLs on L3 switches for the data plane are not stateful

Definitely read the documentation. For example the old Aruba MAS switches have a stateful firewall (they call it session ACL).

 

[oldpenguin]

@Think - depending on your switch capabilities, why not use a dynamic routing protocol?
Interface between pfsense and switch - assign it a /30 (i.e. - 10.20.30.1/30 on pfsense and 10.20.30.2/30 on switch), run whichever dynamic routing protocol both switch and pfsense seem to agree upon and let them share whatever is needed for the other party. This way, networks behind switch should be aware of the VPN networks on pfsense, you also gain the benefit of firewall policies remaining on pfsense without too much bother and you'll only have to deal with ACL policies between connected segments on the switch (might be a b**ch on the long run for a home/lab net).

 

[awedio]

Not to threadjack but does anyone here use an SN2700? (PM me, so I can start a new thread)
I really need to set mine up in L3 mode.