Layer 3 Switch w/ PFSense

[kapone]

I'm having a hard time finding my specific scenario anywhere. I currently have a segmented L2 network but it's overtaxing my pfSense firewall. I am switching my Cisco SG300 with an ICX-6610 and want to have my trusted high-speed VLANs routed via L3 on the switch while keeping some on the firewall.

DHCP/DNS is on Windows Server. I know I have to create router interfaces for my 4 trusted VLANs, change DHCP option 3 to the IP of that interface and then it will route on the switch. If I ping a device on VLAN 200-Storage from VLAN 20-Trusted, it works. However, what happens when a device on 20 wants to ping a device on VLAN 30-IoT? The rules on the firewall allow that, but how do I ensure what is NOT going to my L3 VLANs get routed to pfSense?

Get a CCNA (or equivalent) certification?

 

[adman_c]

I'm having a hard time finding my specific scenario anywhere. I currently have a segmented L2 network but it's overtaxing my pfSense firewall. I am switching my Cisco SG300 with an ICX-6610 and want to have my trusted high-speed VLANs routed via L3 on the switch while keeping some on the firewall.

DHCP/DNS is on Windows Server. I know I have to create router interfaces for my 4 trusted VLANs, change DHCP option 3 to the IP of that interface and then it will route on the switch. If I ping a device on VLAN 200-Storage from VLAN 20-Trusted, it works. However, what happens when a device on 20 wants to ping a device on VLAN 30-IoT? The rules on the firewall allow that, but how do I ensure what is NOT going to my L3 VLANs get routed to pfSense?

I have a similar setup (dhcp is on pfsense and primary dns is a debian vm running pihole/unbound). Multiple VLANs, some trusted (home, infrastructure), some not (guest, IOT). I have a transit VLAN between the ICX and pfsense. My trusted VLANs (and only my trusted VLANs) have router interfaces on the ICX. The default gateway on the DHCP server for the trusted VLANs is the switch, while for untrusted VLANs it's pfsense. The default route for the switch is to the IP of the pfsense box on the transit VLAN.

Because only my trusted VLANs have router interfaces, I don't have to deal with any ACLs on the switch--all of my trusted VLANs can access everything, and all access control to/from other VLANs is handled via firewall rules on pfsense. This of course means I only get line speed routing between my trusted VLANs, but I'm OK with that. I rarely move much traffic between trusted/untrusted VLANs, and when I do I get 4-5gb/s across VLANs on my virtualized pfsense box, which is good enough for me.

 

[coxhaus]

I'm having a hard time finding my specific scenario anywhere. I currently have a segmented L2 network but it's overtaxing my pfSense firewall. I am switching my Cisco SG300 with an ICX-6610 and want to have my trusted high-speed VLANs routed via L3 on the switch while keeping some on the firewall.

DHCP/DNS is on Windows Server. I know I have to create router interfaces for my 4 trusted VLANs, change DHCP option 3 to the IP of that interface and then it will route on the switch. If I ping a device on VLAN 200-Storage from VLAN 20-Trusted, it works. However, what happens when a device on 20 wants to ping a device on VLAN 30-IoT? The rules on the firewall allow that, but how do I ensure what is NOT going to my L3 VLANs get routed to pfSense?

The routing statement that feeds the pfsense is 0.0.0.0 0.0.0.0 If the other routes are not picked up then it will use this default route to pfsense.

 

[rmflint]

No trunk as that is layer 2. pfsense needs to have a gateway or gateways defined for your networks defined on your L3 switch. You also need to add ACLs for the gateways to allow internet access as pfsense blocks all traffic other than pfsense traffic. This is called directly connected networks and not. The gateway networks are defined on the L3 switch not pfsense. Do not define VLANs on pfsense that are defined to the switch. You only need 1 network defined on the L3 and pfsense. So the default route on the L3 switch will point to the pfsense. pfsense gateways will point to the IP address on the L3 switch which connects to pfsense.

To define pfsense and a L3 switch together use example, 192.168.10.1/30 pfsense and 192.168.10.2/30 L3 switch. The mask would be 255.255.255.252. On the L3 switch side create a VLAN 10 and assign an IP of 192.168.10.2 255.255.255.252. Then on the L3 switch assign the default route, some may have default gateway, 192.168.10.1 which is pfsense's LAN interface. Assign a port to VLAN 10 and connect a CAT5e cable from the assigned port to the LAN port on pfsense.

There is a thread on L3 switches on pfsense forums as I wrote all that I did to make pfsense work with a Cisco L3 switch.

Can you post a link to the thread you wrote? I am trying to do the same. Thank you

 

[coxhaus]

That works and as long as it fits your needs I don't think there is too much wrong with that. You are correct no trunk required.

If you use a trunk to pfsense then you are not doing layer 3 on your switch. Pfsense is doing the layer 3 and your switch is doing layer 2.

 

[coxhaus]

So, I installed pfsense 23.01 with my Cisco L3 switch. I wanted to try 23.01 with my 10 gig Broadcom NIC as FreeBSD14 has drivers. It took less than an hour to set it up. I am doing asymmetrical routing right now. No transient network. I may change that latter.

 

[rmflint]

So, I installed pfsense 23.01 with my Cisco L3 switch. I wanted to try 23.01 with my 10 gig Broadcom NIC as FreeBSD14 has drivers. It took less than an hour to set it up. I am doing asymmetrical routing right now. No transient network. I may change that latter.

Why did you decide to not use the transit network option? After reading through this thread numerous times that seems like the ideal solution for doing DHCP on the l3 switch and only using pfsense as a firewall.

 

[coxhaus]

It works either way. You just have to be aware of the routing. I may change it. I think it is easier to install.

I have upgraded to 23.05.

I don't think it makes any difference in using DHCP on the L3 switch. I use my Cisco L3 switch with DHCP for all the VLANs.
I use a static for pfsense on the LAN side. You don't want 2 DHCP servers in the same broadcast domain. Pfsense does not realize all my networks are VLANs. Only my Cisco L3 switch knows.

 

[rmflint]

It works either way. You just have to be aware of the routing. I may change it. I think it is easier to install.

I have upgraded to 23.05.

I don't think it makes any difference in using DHCP on the L3 switch. I use my Cisco L3 switch with DHCP for all the VLANs.
I use a static for pfsense on the LAN side. You don't want 2 DHCP servers in the same broadcast domain. Pfsense does not realize all my networks are VLANs. Only my Cisco L3 switch knows.

Thank you for the update.

I’m trying to learn as much as I can about networking and utilizing my Catalyst l3 switch. I am trying to set everything up in my homelab.

Can you please elaborate on how you setup pfsense?

1. what do you mean by ‘static for pfsense on the lan side’ ?

2. How do you setup the WAN side in pfsense?

3. If you’re not using a transit connection to pfsense, how do computers connected to the layer 3 switch connect to the WAN (ie - internet access)?

 

[coxhaus]

I am not using a transit setup right now. I have in the past. I assign a static IP to pfsense on LAN side with a gateway IP defined on pfsense using the IP on the L3 gateway for the network that connects to a port on my layer 3 switch's DHCP VLAN. No VLANs are defined on pfsense. The L3 switch's DHCP points to the L3 switch for its gateway and intervlan routing is turned on. The L3 switch has a default route statement for 0.0.0.0 0.0.0.0 pointing to the static IP on pfsense. It all works. The LAN side of pfsense firewall needs to allow for all networks on the L3 switch. I have the pfsense WAN interface set as the default gateway for pfsense.
So, all local traffic flows to the L3 switch using an L2 gateway for each network VLAN and the L3 switch routes to the static IP address on pfsense's LAN side for outbound traffic (internet) only. This means the L3 switch will route local traffic at wire speeds internal to the L3 switch without having to send the traffic to pfsense and back to the switch. It is much faster.

I think pfsense could use route statements like Cisco and it would be easier than using all gateways.

I use a Cisco L3 switch.
I am using 23.05.

 

[rmflint]

I am not using a transit setup right now. I have in the past. I assign a static IP to pfsense on LAN side with a gateway IP defined on pfsense using the IP on the L3 gateway for the network that connects to a port on my layer 3 switch's DHCP VLAN. No VLANs are defined on pfsense. The L3 switch's DHCP points to the L3 switch for its gateway and intervlan routing is turned on. The L3 switch has a default route statement for 0.0.0.0 0.0.0.0 pointing to the static IP on pfsense. It all works. The LAN side of pfsense firewall needs to allow for all networks on the L3 switch. I have the pfsense WAN interface set as the default gateway for pfsense.
So, all local traffic flows to the L3 switch using an L2 gateway for each network VLAN and the L3 switch routes to the static IP address on pfsense's LAN side.

I think pfsense could use route statements like Cisco and it would be easier than using all gateways.

I use a Cisco L3 switch.

Thank you.

You clarified what I was trying to understand.

 

[phil9878]

What is the 3rd link you mean ?

The transit route must be untagged, or it will not work properly in my tests, but maybe I missed something in the early times I tested it
The transit route can be untagged on a dedicated interface on both Switch and pfsense. However, it can also be tagged to the pfsense Trunk on the Switch. In this case, we add it in pfsense as a dedicated VLAN300 like we did for other VLANs on the Trunk. You then assign VLAN300 in pfsense to the interface holding all the other trunk VLANs
The management VLAN 1 was added on purpose. I can untag it to the trunk, but I had the spare interface
pfSense only does the DHCP and DNS services + WAN <-> LAN Firewall. It is not concerned by any inter VLAN setup
I need the DNS resolver to restrict some clients to openDNS services. The DNS resolver makes it possible to even limit the DNS over HTTPS traffic in addition to DNS-over-TLS

The rules in pfSense are easy to setup. Only Transit interface needs internet control rules. The other interfaces need simple redundant rules, except for the DNS restrictions. So it is much more friendly than the ACLs in teh switch. If it supported a good 10 Gb switching I would clearly use pfSense. But I do not have the needed switch in that location neither the 10 Gb interface on pfSense. And I doubt it can do any efficient routing at that bandwidth with my i3 CPU

Necromancing this post to update a few things and correct some wrong assumptions
In the last 4 years I migrated the L3 SG350 switch to a 10Gb CBS350 Switch with same config

Now, I decided to upgrade pfsense to my new 10Gb WAN connection. I however had only a LAN card with 2x 10 Gbps porrts to replace my current 4x 1Gbps card

I had to merge my interface in a single Trunk

Tagging the Transit route VLAN on the Switch into the main Trunk, and doing the same in pfsense:
- create a VLAN300 in pfsense and assign it to the trunk interface, igb1 in my case
- assign the current VLAN300 transit from current dedicated interface igb0 to the new created VLAN300 on igb1
- all firewall rules are properly migrated
- the transit VLAN300 is now tagged in the main Trunk and we spare one port

I edited my post to properly reflect the changes for anyone looking at this setup
It's working through 3 hardware upgrades and moving to 10gbps

 

[phil9878]

I have not seen problems with asymmetric routing. You need to watch your gateways and make sure you are using the right one. Routing to me is independent of firewalling. The firewall should process traffic accordingly. It would seem strange to me that firewalling would happen before routing but I have never used Netgate.

Updated to pfsense 2.8.0 yesterday
They changed the default state for rules from "floating" to "strict interface bound" states
The setup with a transit default route for internet traffic will be broken

There are 2 fixes:
1- pfsense System / Advanced / Firewall & NAT: Firewall State Policy set it to "floating"
2- another option is to set the internet rules on the Transit VLAN tab individually to "floating" while keeping the general behavior to new default "strict". However, this can be very complex for VPN access to servers as return traffic from servers will use the Transit VLAN while traffic from VPN client to the servers will use the server VLAN interface. You'll need to define return traffic rules in Transit VLAN with floating state. This can become complex to manage duplicate rules on both interfaces. You cannot sadly create openvpn interface as a VLAN in pfsense. This would permit using a didicated tagged VLAN for openVPN on both switch and firewall

I went with option 1

 

[bugacha]

There are 2 fixes:

There are 3 fixes actually.


3 - Uninstall and purge Pfsense CE. Install OPNsense and call it a day.

 

[phil9878]

There are 3 fixes actually.


3 - Uninstall and purge Pfsense CE. Install OPNsense and call it a day.

It's on my roadmap, but pf 2.8.0 gave it a later schedule

It's something that needs to be planned with a complex setup and a learning curve...