Layer 3 Switch w/ PFSense

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

kapone

Well-Known Member
May 23, 2015
1,095
642
113
I'm having a hard time finding my specific scenario anywhere. I currently have a segmented L2 network but it's overtaxing my pfSense firewall. I am switching my Cisco SG300 with an ICX-6610 and want to have my trusted high-speed VLANs routed via L3 on the switch while keeping some on the firewall.

DHCP/DNS is on Windows Server. I know I have to create router interfaces for my 4 trusted VLANs, change DHCP option 3 to the IP of that interface and then it will route on the switch. If I ping a device on VLAN 200-Storage from VLAN 20-Trusted, it works. However, what happens when a device on 20 wants to ping a device on VLAN 30-IoT? The rules on the firewall allow that, but how do I ensure what is NOT going to my L3 VLANs get routed to pfSense?
Get a CCNA (or equivalent) certification?
 

adman_c

Active Member
Feb 14, 2016
257
135
43
Chicago
I'm having a hard time finding my specific scenario anywhere. I currently have a segmented L2 network but it's overtaxing my pfSense firewall. I am switching my Cisco SG300 with an ICX-6610 and want to have my trusted high-speed VLANs routed via L3 on the switch while keeping some on the firewall.

DHCP/DNS is on Windows Server. I know I have to create router interfaces for my 4 trusted VLANs, change DHCP option 3 to the IP of that interface and then it will route on the switch. If I ping a device on VLAN 200-Storage from VLAN 20-Trusted, it works. However, what happens when a device on 20 wants to ping a device on VLAN 30-IoT? The rules on the firewall allow that, but how do I ensure what is NOT going to my L3 VLANs get routed to pfSense?
I have a similar setup (dhcp is on pfsense and primary dns is a debian vm running pihole/unbound). Multiple VLANs, some trusted (home, infrastructure), some not (guest, IOT). I have a transit VLAN between the ICX and pfsense. My trusted VLANs (and only my trusted VLANs) have router interfaces on the ICX. The default gateway on the DHCP server for the trusted VLANs is the switch, while for untrusted VLANs it's pfsense. The default route for the switch is to the IP of the pfsense box on the transit VLAN.

Because only my trusted VLANs have router interfaces, I don't have to deal with any ACLs on the switch--all of my trusted VLANs can access everything, and all access control to/from other VLANs is handled via firewall rules on pfsense. This of course means I only get line speed routing between my trusted VLANs, but I'm OK with that. I rarely move much traffic between trusted/untrusted VLANs, and when I do I get 4-5gb/s across VLANs on my virtualized pfsense box, which is good enough for me.
 

coxhaus

Active Member
Jul 7, 2020
109
36
28
I'm having a hard time finding my specific scenario anywhere. I currently have a segmented L2 network but it's overtaxing my pfSense firewall. I am switching my Cisco SG300 with an ICX-6610 and want to have my trusted high-speed VLANs routed via L3 on the switch while keeping some on the firewall.

DHCP/DNS is on Windows Server. I know I have to create router interfaces for my 4 trusted VLANs, change DHCP option 3 to the IP of that interface and then it will route on the switch. If I ping a device on VLAN 200-Storage from VLAN 20-Trusted, it works. However, what happens when a device on 20 wants to ping a device on VLAN 30-IoT? The rules on the firewall allow that, but how do I ensure what is NOT going to my L3 VLANs get routed to pfSense?
The routing statement that feeds the pfsense is 0.0.0.0 0.0.0.0 If the other routes are not picked up then it will use this default route to pfsense.
 

rmflint

New Member
Dec 12, 2017
14
1
3
61
No trunk as that is layer 2. pfsense needs to have a gateway or gateways defined for your networks defined on your L3 switch. You also need to add ACLs for the gateways to allow internet access as pfsense blocks all traffic other than pfsense traffic. This is called directly connected networks and not. The gateway networks are defined on the L3 switch not pfsense. Do not define VLANs on pfsense that are defined to the switch. You only need 1 network defined on the L3 and pfsense. So the default route on the L3 switch will point to the pfsense. pfsense gateways will point to the IP address on the L3 switch which connects to pfsense.

To define pfsense and a L3 switch together use example, 192.168.10.1/30 pfsense and 192.168.10.2/30 L3 switch. The mask would be 255.255.255.252. On the L3 switch side create a VLAN 10 and assign an IP of 192.168.10.2 255.255.255.252. Then on the L3 switch assign the default route, some may have default gateway, 192.168.10.1 which is pfsense's LAN interface. Assign a port to VLAN 10 and connect a CAT5e cable from the assigned port to the LAN port on pfsense.

There is a thread on L3 switches on pfsense forums as I wrote all that I did to make pfsense work with a Cisco L3 switch.
Can you post a link to the thread you wrote? I am trying to do the same. Thank you
 

coxhaus

Active Member
Jul 7, 2020
109
36
28
That works and as long as it fits your needs I don't think there is too much wrong with that. You are correct no trunk required.
If you use a trunk to pfsense then you are not doing layer 3 on your switch. Pfsense is doing the layer 3 and your switch is doing layer 2.
 

coxhaus

Active Member
Jul 7, 2020
109
36
28
So, I installed pfsense 23.01 with my Cisco L3 switch. I wanted to try 23.01 with my 10 gig Broadcom NIC as FreeBSD14 has drivers. It took less than an hour to set it up. I am doing asymmetrical routing right now. No transient network. I may change that latter.
 

rmflint

New Member
Dec 12, 2017
14
1
3
61
So, I installed pfsense 23.01 with my Cisco L3 switch. I wanted to try 23.01 with my 10 gig Broadcom NIC as FreeBSD14 has drivers. It took less than an hour to set it up. I am doing asymmetrical routing right now. No transient network. I may change that latter.
Why did you decide to not use the transit network option? After reading through this thread numerous times that seems like the ideal solution for doing DHCP on the l3 switch and only using pfsense as a firewall.
 

coxhaus

Active Member
Jul 7, 2020
109
36
28
It works either way. You just have to be aware of the routing. I may change it. I think it is easier to install.

I have upgraded to 23.05.

I don't think it makes any difference in using DHCP on the L3 switch. I use my Cisco L3 switch with DHCP for all the VLANs.
I use a static for pfsense on the LAN side. You don't want 2 DHCP servers in the same broadcast domain. Pfsense does not realize all my networks are VLANs. Only my Cisco L3 switch knows.
 
Last edited:

rmflint

New Member
Dec 12, 2017
14
1
3
61
It works either way. You just have to be aware of the routing. I may change it. I think it is easier to install.

I have upgraded to 23.05.

I don't think it makes any difference in using DHCP on the L3 switch. I use my Cisco L3 switch with DHCP for all the VLANs.
I use a static for pfsense on the LAN side. You don't want 2 DHCP servers in the same broadcast domain. Pfsense does not realize all my networks are VLANs. Only my Cisco L3 switch knows.
Thank you for the update.

I’m trying to learn as much as I can about networking and utilizing my Catalyst l3 switch. I am trying to set everything up in my homelab.

Can you please elaborate on how you setup pfsense?

1. what do you mean by ‘static for pfsense on the lan side’ ?

2. How do you setup the WAN side in pfsense?

3. If you’re not using a transit connection to pfsense, how do computers connected to the layer 3 switch connect to the WAN (ie - internet access)?
 

coxhaus

Active Member
Jul 7, 2020
109
36
28
I am not using a transit setup right now. I have in the past. I assign a static IP to pfsense on LAN side with a gateway IP defined on pfsense using the IP on the L3 gateway for the network that connects to a port on my layer 3 switch's DHCP VLAN. No VLANs are defined on pfsense. The L3 switch's DHCP points to the L3 switch for its gateway and intervlan routing is turned on. The L3 switch has a default route statement for 0.0.0.0 0.0.0.0 pointing to the static IP on pfsense. It all works. The LAN side of pfsense firewall needs to allow for all networks on the L3 switch. I have the pfsense WAN interface set as the default gateway for pfsense.
So, all local traffic flows to the L3 switch using an L2 gateway for each network VLAN and the L3 switch routes to the static IP address on pfsense's LAN side for outbound traffic (internet) only. This means the L3 switch will route local traffic at wire speeds internal to the L3 switch without having to send the traffic to pfsense and back to the switch. It is much faster.

I think pfsense could use route statements like Cisco and it would be easier than using all gateways.

I use a Cisco L3 switch.
I am using 23.05.
 
Last edited:

rmflint

New Member
Dec 12, 2017
14
1
3
61
I am not using a transit setup right now. I have in the past. I assign a static IP to pfsense on LAN side with a gateway IP defined on pfsense using the IP on the L3 gateway for the network that connects to a port on my layer 3 switch's DHCP VLAN. No VLANs are defined on pfsense. The L3 switch's DHCP points to the L3 switch for its gateway and intervlan routing is turned on. The L3 switch has a default route statement for 0.0.0.0 0.0.0.0 pointing to the static IP on pfsense. It all works. The LAN side of pfsense firewall needs to allow for all networks on the L3 switch. I have the pfsense WAN interface set as the default gateway for pfsense.
So, all local traffic flows to the L3 switch using an L2 gateway for each network VLAN and the L3 switch routes to the static IP address on pfsense's LAN side.

I think pfsense could use route statements like Cisco and it would be easier than using all gateways.

I use a Cisco L3 switch.
Thank you.

You clarified what I was trying to understand.