Asymmetric routing with pfsense and L3 switch

[kevindd992002]

So I bought a Brocade ICX6610 L3 switch which I will use as the interVLAN router in my home network and just have pfsense act as a firewall and a DHCP/DNS server. I stumbled upon this old post of @kapone from the Brocade thread and to do this, I need to have these interfaces in pfsense:

Transit - untagged port
Main, IoT, etc. - tagged ports

The L3 switch's default route points to pfsense interface IP in the transit network. DHCP/DNS will work fine too. However, the routing will be asymmetric.

Forward packet from client is: client -> switch -> pfsense -> internet
Return packet to client is: internet -> pfsense -> client (because pfsense sees the VLANs as directly-connected, bypassing the L3 switch router)

From what I understand, the return packet will still reach the client and won't get rejected but do you guys see any caveats with this? I know asymmetric is bad if the outgoing traffic has a stateful firewall that's different from the stateful firewall that handles the incoming traffic but this is not the case in this setup. Both outgoing and incoming traffic go through the same pfsense firewall so no packet should be dropped.

 

[NablaSquaredG]

Wouldn't it be easier to just use a transit VLAN (separate VLAN where only the LAN interfaces of the firewalls reside) to get rid of the asymmetric routing?

Assuming you would like to have wirespeed Inter-VLAN routing on the ICX6610

 

[kevindd992002]

Wouldn't it be easier to just use a transit VLAN (separate VLAN where only the LAN interfaces of the firewalls reside) to get rid of the asymmetric routing?

Assuming you would like to have wirespeed Inter-VLAN routing on the ICX6610

Not sure I follow. What I'm trying to do, based on the post above, is basically have a trunk port on the switch that carries the tagged local VLANs AND untagged Internet traffic from the switch. Then create the necessary interfaces in pfsense. If you create an interface in pfsense, the routing table automatically assumes that that subnet is a directly-connected network so it doen't send the return traffic back to the l3 switch routet interface, hence thr asymmetric routing.

Without creating the interfaces, you do not get DHCP from pfsense. If you do DHCP in the L3 switch, you cannot register dhcp hostnames back to the pfsense DNS server.

 

[NablaSquaredG]

Then create the necessary interfaces in pfsense. If you create an interface in pfsense, the routing table automatically assumes that that subnet is a directly-connected network so it doen't send the return traffic back to the l3 switch routet interface, hence thr asymmetric routing.

In that case (Firewall has one Interface per VLAN), the clean solution would be to set this interface from the firewall as the default gateway for the clients in the VLAN (and not create a VE / VLAN Interface / Router Interface on the ICX6610 and set this interface as the default gateway for the clients in the VLAN)

Without creating the interfaces, you do not get DHCP from pfsense.

If the DHCP within pfsense is not totally stupid, it should be able to handle this easily (different default gateway, aka the interface of the firewall within the specific VLAN, per VLAN / subnet)

If you do DHCP in the L3 switch, you cannot register dhcp hostnames back to the pfsense DNS server.

I don't really get what exactly this means.

Another alternative would be to utilise the DHCP Relay functionality in the ICX6610


if you have many VLANs, creating a VLAN interface on the firewall per VLAN becomes very annoying and time consuming and many people then use transit VLAN + DHCP relay

 

[kevindd992002]

Sorry, I'm mobile so it's really hard to do partial quoting. What you're explaining is a router on a stick model which is very easy tp implement but is not my goal here. The goal is to do the interVLAN routing in the L3 switch which makes DHCP and DNS a bit of a challenge if you don't have a separate server to put them on and just rely on the built-in pfsense services.

I agree. The DHCP server in pfsense is stupod and rely on interfaces. It's been that way for a long time now and I know people are complaining.

So if you have a DHCP server set in the Brocade switch, you cannot register their dhcp lease hostnames to the pfsense DNS server. This means that you cannoy resolve any of your client's FQDN. You just reach them via their IP which sucks.

I understand that you can use ip helper in the switch but again, that won't work, because the DHCP server in pfsense is so basic that you can only server an address pool for the subnet where it is enabled. No VLAN interface means no DHCP from pfsense.

So really, my question is focused on the asymmetric routing part. Is there anything wrong with it?

 

[NablaSquaredG]

The goal is to do the interVLAN routing in the L3 switch

In that case I'd recommend the Transit-VLAN approach to keep things clean.

Is there anything wrong with it?

Well, it should work (until it doesn't).


BTW, is there any particular reason why you use pfsense (especially given the recent news)? Opnsense's ISC-DHCP should be capable of handling DHCP relay properly

 

[kevindd992002]

Yeah, I'm still open to different approaches. That workaround in the post just seems like a good one.

Regarding pfsense, I was actually made aware of their dumb move two days ago and am thinking real hard if I want to switch. This is all for a home network and I'm just super used to managing pfsense. I'm not sure if I want another new learning curve with OPNSense. Are therr any known gotchas eith OPNSense?

 

[Tech Junky]

@kevindd992002

I just use Ubuntu as a router by configuring all of the things needed by a "router" and skip the hype of all of the senses. Smooth operation and best possible speeds when pushed. There's only a handful of things to configure and once it's done just leave it alone other than updating the packages / kernel to keep prying eyes out.

Going with the vanilla approach opens the door for uses other than just a router and not being BSD it's more compatible with other hardware. No need to run containers or VMs if you don't want to since it's bare metal.

Really when it comes down to it there's just...
iptables / FW
NAT / masquerade
interfaces file to assign your WAN/LAN - IP management / gateway / loopback / route statements
dhcp.conf

There are other approaches you can use though that are a bit more hands on to configure if you're coding minded.

 

[kevindd992002]

@kevindd992002

I just use Ubuntu as a router by configuring all of the things needed by a "router" and skip the hype of all of the senses. Smooth operation and best possible speeds when pushed. There's only a handful of things to configure and once it's done just leave it alone other than updating the packages / kernel to keep prying eyes out.

Going with the vanilla approach opens the door for uses other than just a router and not being BSD it's more compatible with other hardware. No need to run containers or VMs if you don't want to since it's bare metal.

Really when it comes down to it there's just...
iptables / FW
NAT / masquerade
interfaces file to assign your WAN/LAN - IP management / gateway / loopback / route statements
dhcp.conf

There are other approaches you can use though that are a bit more hands on to configure if you're coding minded.

I understand where you're coming from. If I had the time in the world, I would've gone with everything Linux. I mean, I may do that in the future but not now. I've been using pfsense since 2010 for my home networks. At least opnsense is a bit similar to pfsense so the learning curve will not be too steep.

But I'm still not convinced at this point to switch if it makes to just stay with CE. Judging from how these stupid Netgate owners act though, it's more than likely that CE gets abandoned in the near future and would leave me no choice but to switch to opnsense. So is it better to switch now altogether?

 

[Tech Junky]

So is it better to switch now altogether?

Depends on what they do and if it leaves you vulnerable.

Nice thing about vanilla os is you can apply kernel updates directly when you want and roll in the patches all at once. I just compiled and applied 6.6 since they're dragging their feet on releasing the deb format.

If you're going to switch though might as well go full monty and convert. I think you should be able to stage it in a VM and then make an image of that to apply quickly for less downtime. I setup a lot of it from scratch while connected to a router and then tweaked it when I pulled the router. Another option would be setup as dual boot and then reboot back to pf and edit the files needed and then boot to Linux to test that it's working. Then if it's not you just boot back to pf and fix the configs and try again.

 

[TonyArrr]

I’m doing the same as you with a 7250, and when I found out about pfSense’s DHCP limitations I just dropped it then.
Now I’m 50/50 on whether to use vyOS or a plain Debian on a LTS configured like @Tech Junky is rigging their Ubuntu.

I’m liking vyOS a lot, because:
- focussed scope, just packs in what it needs for the job
- Linux based, and as such can run extra features in a KVM based VM if needed
- very similar terminal-based control to the ICX and Cisco switches, so helps keep me getting used to that sort of management and removes the temptation to configure random things “just cause they’re there”
- it versions your config when you update, so if something breaks, you just note the error and logs, then reboot to the previous version and if the update had had to change config for any reason, you don’t get stung by that. (I know, backups are important, it’s just nice to not need to go looking for it when it’s the home network)

Still liking Debian for:
- rock solid stability
- familiar platform
- large community that use it

Hardest thing about vyOS was compiling it, since I wanted to run LTS, but if you run the rolling release it would be a bunch easier. Once you’re booted, very easy syntax to control it and lots of very thorough documentation.

Maybe not for an immediate change but another option out there

 

[Rttg]

I’ll add another big endorsement of VyOS.

It’s Debian with a switch-like config interface, which is great if you don’t like maintaining individual config files across a Linux installation. Despite the CLI, VyOS hides *very* little, if anything, that would otherwise justify working directly with config files (not that it’s really a practical option with VyOS).

Beyond the similarities to enterprise switches, VyOS will look awfully familiar if you’ve ever used the CLI on Ubiquiti routers (at least pre-Dream Machine) because they share a common lineage.

 

[fta]

I just use Ubuntu as a router by configuring all of the things needed by a "router" and skip the hype of all of the senses.

I do the same, except I use Debian. I use ansible to deploy my router config, and my playbooks are stored in a git repo. This means I get excellent version control and offsite backups (git repo is pushed to a private github repo). I recently bought an 8505 so I could move my 1235u to media server duties, and I was able to create my router on the 8505 in a few minutes and swap it in.

 

[Tech Junky]

@fta

Awesome work. Media always takes a bit more HP than router duties. I added an a380 to my AMD rebuild for media and that little card flies for $100.

 

[slidermike]

@fta which 8505 did you pull the trigger on?

I noticed active on the openWRT forums recently, helping out in a discussion about some simple cake script i believe it was. Interesting read.
One of the other contributers there is a frequent snbforum contributor. Ran across that thread as i am refreshing on openWRT as a likely replacement for pfSense on my 1235u proxmox.

 

[fta]

@fta which 8505 did you pull the trigger on?

This one. It's the same as my 1235, just with the 8505 instead. I get to use my unlocked BIOS for it as well.

I noticed active on the openWRT forums recently, helping out in a discussion about some simple cake script i believe it was. Interesting read.
One of the other contributers there is a frequent snbforum contributor. Ran across that thread as i am refreshing on openWRT as a likely replacement for pfSense on my 1235u proxmox.

If you're interested in shaping with CAKE, the openwrt forum is the place with the best info. I adapted that CAKE script you're referencing to use on my bare metal Debian install. CAKE is awesome!

 

[kevindd992002]

Depends on what they do and if it leaves you vulnerable.

Nice thing about vanilla os is you can apply kernel updates directly when you want and roll in the patches all at once. I just compiled and applied 6.6 since they're dragging their feet on releasing the deb format.

If you're going to switch though might as well go full monty and convert. I think you should be able to stage it in a VM and then make an image of that to apply quickly for less downtime. I setup a lot of it from scratch while connected to a router and then tweaked it when I pulled the router. Another option would be setup as dual boot and then reboot back to pf and edit the files needed and then boot to Linux to test that it's working. Then if it's not you just boot back to pf and fix the configs and try again.

I have two networks to setup.

* Sister's new house - very basic: ICX6450, 8 Zyxel AP's, 14 IP cameras, some RJ45 wall jacks, an IP camera NVR, and a firewall (PCEngines APU2C4)
* Parent's house - this would be a revamp from using Ubiquiti switches to just one ICX6610. I have a couple of servers here and if I switch to a better firewall I can do whatever here. This is my main "project" and is where I can test all sorts of things.

So the initial question in this thread was mainly for my the network in my sister's house. No server to put anything on and I would probably need a firewall with some GUI on it. My initial plan was to use pfsense because it's what I am familiar with the most. But then the news of pfsense came. Which would be better between Ubuntu and Debian as a router? I'm a bit familiar with Debian since I have a server in my parent's house that's Debian.

I’m doing the same as you with a 7250, and when I found out about pfSense’s DHCP limitations I just dropped it then.
Now I’m 50/50 on whether to use vyOS or a plain Debian on a LTS configured like @Tech Junky is rigging their Ubuntu.

I’m liking vyOS a lot, because:
- focussed scope, just packs in what it needs for the job
- Linux based, and as such can run extra features in a KVM based VM if needed
- very similar terminal-based control to the ICX and Cisco switches, so helps keep me getting used to that sort of management and removes the temptation to configure random things “just cause they’re there”
- it versions your config when you update, so if something breaks, you just note the error and logs, then reboot to the previous version and if the update had had to change config for any reason, you don’t get stung by that. (I know, backups are important, it’s just nice to not need to go looking for it when it’s the home network)

Still liking Debian for:
- rock solid stability
- familiar platform
- large community that use it

Hardest thing about vyOS was compiling it, since I wanted to run LTS, but if you run the rolling release it would be a bunch easier. Once you’re booted, very easy syntax to control it and lots of very thorough documentation.

Maybe not for an immediate change but another option out there

So I'm assuming vyOS doesn't have any GUI too? Is there a good guide regardings setting up Debian as a router?

 

[Tech Junky]

@kevindd992002

Either OS is fine as they're both the same thing with just a different GUI look.

GUI isn't really needed if you KISS for the rules. If I extract the fluff from the file IPTables only needs ~15 rules to secure things.

# Generated by iptables-save v1.8.7 on Sun Sep 24 07:29:14 2023
*mangle
:PREROUTING ACCEPT [162268:94207940]
:INPUT ACCEPT [153371:85513762]
:FORWARD ACCEPT [8747:8648094]
:OUTPUT ACCEPT [54650:846118632]
:POSTROUTING ACCEPT [63753:854799212]
COMMIT
# Completed on Sun Sep 24 07:29:14 2023
# Generated by iptables-save v1.8.7 on Sun Sep 24 07:29:14 2023
*security
:INPUT ACCEPT [152994:85423817]
:FORWARD ACCEPT [8745:8648014]
:OUTPUT ACCEPT [54641:846118164]
COMMIT
# Completed on Sun Sep 24 07:29:14 2023
# Generated by iptables-save v1.8.7 on Sun Sep 24 07:29:14 2023
*raw
:PREROUTING ACCEPT [162268:94207940]
:OUTPUT ACCEPT [54650:846118632]
:FORWARD - [0:0]
COMMIT
# Completed on Sun Sep 24 07:29:14 2023
# Generated by iptables-save v1.8.7 on Sun Sep 24 07:29:14 2023
*filter
:INPUT DROP [0:0]
:FORWARD DROP [2:80]
:OUTPUT DROP [0:0]
:PERMIT-FWD - [0:0]
:PERMIT-IN - [0:0]
:PERMIT-OUT - [0:0]
-A INPUT -j PERMIT-IN
-A FORWARD -j PERMIT-FWD
-A OUTPUT -j PERMIT-OUT
-A PERMIT-FWD -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A PERMIT-FWD -m conntrack --ctstate NEW -j ACCEPT
-A PERMIT-IN -i lo -j ACCEPT
-A PERMIT-IN -i br0 -j ACCEPT
-A PERMIT-IN -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A PERMIT-IN -j DROP
-A PERMIT-OUT -o lo -j ACCEPT
-A PERMIT-OUT -o br0 -j ACCEPT
-A PERMIT-OUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A PERMIT-OUT -m conntrack --ctstate NEW -j ACCEPT
-A PERMIT-OUT -j DROP
COMMIT
# Completed on Sun Sep 24 07:29:14 2023
# Generated by iptables-save v1.8.7 on Sun Sep 24 07:29:14 2023
*nat
:PREROUTING ACCEPT [690:213753]
:INPUT ACCEPT [251:63030]
:OUTPUT ACCEPT [927:89688]
:POSTROUTING ACCEPT [283:22315]
-A POSTROUTING -o nordlynx -j MASQUERADE
-A POSTROUTING -o bo0 -j MASQUERADE
COMMIT
# Completed on Sun Sep 24 07:29:14 2023

 

[fta]

If I extract the fluff from the file IPTables only needs ~15 rules to secure things.

Agreed. This is one of the things I didn't like about OPNSense. Before I've even added any of my own rules there were a whole bunch of rules I didn't care about. I use nftables rather than iptables, though. I really like it.

 

[Tech Junky]

@fta

There's a bit of fluff to get you online but, the meat of it is where it counts. I went with iptables because it made sense to me with a Cisco background.

To make it easier I made containers for the three sections and that just makes things easier to keep track of when playing around with rules to try other things.

I edit them in notepad++ and then save them to the server and refresh them in a one liner script.