Layer 3 Switch w/ PFSense

PGlover

Active Member
Nov 8, 2014
477
62
28
54
This has a lot more permutations/combinations than you realize... :) Just the first step...



What vlan interface?? The TRANSIT interface doesn't have to be a VLAN on the PFSense side. How's pfSense connected to the TRANSIT interface? physical NIC, virtual NIC? How's the WAN terminated? To pfSense or to the layer 3 switch?
Just trying to make sure I have the setup correct on my pfSense box.

I was under the impression that you needed to create the Transit Vlan on the pfSense box as well (see below)
upload_2020-3-12_22-4-42.png

In my setup, the Transit VLAN is created via 2 physical nics forming a LAGG interface.
upload_2020-3-12_22-9-12.png

In my setup, WAN is terminated on a physical nic on the pfSense box I think.
upload_2020-3-12_22-12-26.png
 

tommybackeast

Active Member
Jun 10, 2018
285
103
43
Just trying to make sure I have the setup correct on my pfSense box.

I was under the impression that you needed to create the Transit Vlan on the pfSense box as well (see below)
View attachment 13254

In my setup, the Transit VLAN is created via 2 physical nics forming a LAGG interface.
View attachment 13255

In my setup, WAN is terminated on a physical nic on the pfSense box I think.
View attachment 13256
excuse off topic question: on a dedicated pfsense box; what advantages are there in using a 4port NIC card vs a 2port NIC Rj45 card?
 

kapone

Well-Known Member
May 23, 2015
1,006
573
113
Just trying to make sure I have the setup correct on my pfSense box.

I was under the impression that you needed to create the Transit Vlan on the pfSense box as well (see below)
View attachment 13254

In my setup, the Transit VLAN is created via 2 physical nics forming a LAGG interface.
View attachment 13255

In my setup, WAN is terminated on a physical nic on the pfSense box I think.
View attachment 13256
Keep it simple to test.

- Connect a single PFSense NIC to the TRANSIT vlan as an untagged port.
- Don't use LAGGS for now, while testing.

As far as pfSense is concerned, there is no vlan for the TRANSIT network, the switch will strip away the respective VLAN header.
 

NYCone

Member
Jun 23, 2017
35
8
8
58
I just found this thread.

I was new to pfSense and Vlans a few weeks ago when I tried to set up a pfSense based multi Vlan network. I kept failing at DHCP and some DNS issues. I thought I was at fault for not configuring pfSense correctly (in a way still true, I didn't know pfSense couldn't handle DHCP on Vlans).

Time to reassess...
 
  • Like
Reactions: tommybackeast

ViciousXUSMC

Active Member
Nov 27, 2016
202
105
43
38
Just keying back in on this old thread.
I recently decided to flatten out my network to a single Layer 2 domain because I was having some IOT devices and other such devices not really like crossing a Layer 3 domain.

I also decided Jumbo Frames were no longer something I really needed and that was one of the main reasons I wanted to have a L3 network for all my server/storage traffic.

So I reduced my administrative overhead greatly.

I did think about trying to simplify my routing and use PFSense as the router instead of the switch (and allows me to use a L2 switch in a pinch due to a failure) should I ever decide to go back to mutli VLAN, and sure enough my worry is verified here.

I have say my desktop with a 10gb interface and my NAS with a 10gb interface and if I use PFSense to do the routing I may run into some performance issues.

Here is the one big thing I wanted to say though and the entire reason for this post.

I do not know why, how, etc but I had PFSense crash on me once in a while (especially with an influx of torrent traffic) if Surricata was running. I had to do every tweak you can imagine to the interfaces and metrics, and even went so far as to install an entirelly new NIC on my PFSense box.

I still had those random crashes now and then where the interface would just have a kernal panic.

Now that I have flattened everything out, it has been rock solid for weeks now and I am almost positive I would have seen a crash by now. So I don't know why/what/how but it seems that the way I had routing setup between the swtich and firewall maybe was inducing that crash?

I mean everything worked, all my routing was sound, every device coudl talk to each other device. IP's configured correctly, nothing I googled pointed to configuration as the issue but rather other issues. But it seems like that my issue was my configuration between the switch and PFSense, perhaps because I let the switch do all the routing and only let PFSense know a static route back to the switch for each vlan with a default route to the internet.

For my next adventure, if I install 10gb interfaces on my PFSense box and let it handle the routing for the vlans, anybody think that would show an appreciable difference compared to letting the switch handle it?
 
  • Like
Reactions: phil9878

kapone

Well-Known Member
May 23, 2015
1,006
573
113
For my next adventure, if I install 10gb interfaces on my PFSense box and let it handle the routing for the vlans, anybody think that would show an appreciable difference compared to letting the switch handle it?
I wouldn't do that, unless you need to firewall these VLANs. There's a reason Layer 3 switches (routers) exist. Line speed routing WITH all ACLs applied. To get that level of performance in pfSense....not going to happen.
 
  • Like
Reactions: phil9878

ViciousXUSMC

Active Member
Nov 27, 2016
202
105
43
38
Yeah I would like to have the ability to do simple FW rules to say block IOT devices from the network but allow access to the NVR and internet, allow access from X to Y etc.

Most if not all of it can be done by ACL's but not quite as clean and easy.

Its nice to have aliases and such all setup in PFSense.

What I have done in the past is actually have multiple interfaces on my Desktop on different VLANS that worked fine but felt kind of dirty.

Also what is everybodys take on a trunk port to PFSense and a Interface for each VLAN but set the Default Gateway to the Virtual IP for that VLAN for the switch as compared to a single non trunked routed port to PFSense with a default gateway set to the PFSense single interface?

I think it should work both ways, but if I have a VLAN interface on PFSense even if the routing happens from the switch I should be able to use PFSense for DHCP and such.
 

blinkenlights

Active Member
May 24, 2019
143
64
28
I recently decided to flatten out my network to a single Layer 2 domain because I was having some IOT devices and other such devices not really like crossing a Layer 3 domain.
It is possible to do that with multiple Layer 2 domains - I have three internal segments at home (VoIP, wired, wireless) and devices can see services offered across the network using Avahi/Bonjour/Zeroconf/etc. relaying on the firewall. Wife not being able to cast content from the smartphone to smart TV forced me to "get smart" quickly....

I do not know why, how, etc but I had PFSense crash on me once in a while (especially with an influx of torrent traffic) if Surricata was running. I had to do every tweak you can imagine to the interfaces and metrics, and even went so far as to install an entirelly new NIC on my PFSense box.

I still had those random crashes now and then where the interface would just have a kernal panic.
See, now I am curious and want to know more about these crashes. My home pfSense box is running the development branch (2.5.x), updated once a month or so. Except for some wonky PHP errors that force me to reinstall (*sigh*) the operating system has been rock solid, even with Squid + Snort. I've been doing this since the first iteration of my pfSense firewall (Atom D525 with dreadful onboard NICs) to present day (E5-1650 v4 with 2x Chelsio T520-BT).

For my next adventure, if I install 10gb interfaces on my PFSense box and let it handle the routing for the vlans, anybody think that would show an appreciable difference compared to letting the switch handle it?
Sorry if I missed it in the thread, but what type of hardware do you have in that box? Within reasonable limits, allowing the firewall to manage VLAN segments should have minimal impact on performance. I imagine this is especially true if your network cards can properly handle offloading (VLAN checksums, tagging, etc).
 

laserpaddy

Active Member
Jul 17, 2017
171
40
28
out there
I use the router on a stick , and 6 vlans dhcp being done on pfsense, all ubiquiti devices behind it, unifi 10g switch 24port, 48port, nanohd ap and another edge 24p switch which can do the routing if needed.
Lawrence systems on YouTube has an incredible step by step videos on all of this and much more.
 

ViciousXUSMC

Active Member
Nov 27, 2016
202
105
43
38
It is possible to do that with multiple Layer 2 domains - I have three internal segments at home (VoIP, wired, wireless) and devices can see services offered across the network using Avahi/Bonjour/Zeroconf/etc. relaying on the firewall. Wife not being able to cast content from the smartphone to smart TV forced me to "get smart" quickly....



See, now I am curious and want to know more about these crashes. My home pfSense box is running the development branch (2.5.x), updated once a month or so. Except for some wonky PHP errors that force me to reinstall (*sigh*) the operating system has been rock solid, even with Squid + Snort. I've been doing this since the first iteration of my pfSense firewall (Atom D525 with dreadful onboard NICs) to present day (E5-1650 v4 with 2x Chelsio T520-BT).



Sorry if I missed it in the thread, but what type of hardware do you have in that box? Within reasonable limits, allowing the firewall to manage VLAN segments should have minimal impact on performance. I imagine this is especially true if your network cards can properly handle offloading (VLAN checksums, tagging, etc).
I never had any crashes with Squad + Snort, and I do not think with Surricata in the legacy mode but the new in-line mode is what causes me the issues. If I disable Surricata I get no crashes.

The box is a Dell R210ii with a Xeon E3-1230 V2 w/ 8GB RAM and 2x 200GB SSD in Raid 1 running PFSense 2.4.5 Release
 

klui

Active Member
Feb 3, 2019
295
124
43
I thought I was at fault for not configuring pfSense correctly (in a way still true, I didn't know pfSense couldn't handle DHCP on Vlans).
How so? DHCP works with VLANs under a trunk interface.
 

ViciousXUSMC

Active Member
Nov 27, 2016
202
105
43
38
As long as there is an interface on the subnet you want you get a DHCP server entry, what it wont let you do is create an arbitrary DHCP scope that does not have an interface and use IPHelper or something to relay it across different subnets.
 

blinkenlights

Active Member
May 24, 2019
143
64
28
I never had any crashes with Squad + Snort, and I do not think with Surricata in the legacy mode but the new in-line mode is what causes me the issues. If I disable Surricata I get no crashes.

The box is a Dell R210ii with a Xeon E3-1230 V2 w/ 8GB RAM and 2x 200GB SSD in Raid 1 running PFSense 2.4.5 Release
Ah, yeah, inline mode probably stresses different parts of the system. As @laserpaddy alludes to, there should not be any problems with pfSense managing the VLANs itself.
 

coxhaus

Member
Jul 7, 2020
74
27
18
When I ran pfsense with my Cisco SG300-28 L3 switch before the SG350 came out several years ago all the VLANs, DHCP were defined to the L3 switch. pfsense only did firewall duties. I used a 30 bit mask between pfsense and my layer 3 switch. All local network functions were handled by the switch and pfsense just opened and closed the door to the internet.

I also setup my daughter's small business up with a Cisco SG500X-24 in L3 mode the same way plus using VOIP for IP phones but I used a router instead of pfsense.
 

phil9878

Member
Apr 2, 2021
36
5
8
Trunk VLAN 10, 20, 40, 50
So basically if you want pfSense to handle DHCP for an IP range pfSense must have an interface with an IP configured in that subnet. If the only way for pfSense to get to a subnet is via a L3 route handled by another device you can not setup a DHCP range for that subnet. It is just an interface limitation.

What this means is if you have your L3 switch handling the routing then pfSense probably isn't the right way to do DHCP.
Hi,

I really need some advice about my setup. I need to migrate my Local network to VLAN mainly to add VLAN port security and to simplify the isolation. Currently, I use a flat network with IP ranges based on MAC and assigned by DHCP. Isolation is done by a bunch of ACL rules in the switch to lock traffic based on IP ranges.

I have a pfsense box and after reading this thread, I plan to order a Quad LAN I305-T4 Intel card. The Pfsense box will have 2 built in LANs (WAN + 1 interface) and an additional 4 interfaces to accomodate a total of 5 subnetworks and a WAN.

I do not plan to run a dedicated DHCP server right now, even if I understand it would be the simplest approach.

Do you think I can have DHCP working on Pfsense with the below layout and the Cisco switch doing the inter-VLAN routing and VACL isolation ?

Only VLAN 5 and VLAN 40 won't have internet access or DHCP, but I do not need both.

I guess I no longer need to setup VLANs on Pfsense or use a Trunk

Internet Network Diagram Template 2.jpg

Is my network diagram ok you think ?
Note: The layout could be simpler, but I am limited by the devices grouping in the specified physical places

Best regards
 
Last edited:

coxhaus

Member
Jul 7, 2020
74
27
18
By default, if you pass a trunk to pfsense then you will be using pfsense for layer 3 routing. The easy way for L3 switching and pfsense is not to use VLANs on pfsense if you want the L3 switch doing the routing. This way only internet traffic is routed from pfsense to the L3 switch and L3 internet traffic is routed to pfsense. Doing this keeps all local traffic inside the L3 switch away from pfsense your internet gateway. Switching is faster than routing and will make for a faster network.

With your pfsense diagram there is no reason to have separate VLAN NICs. Use 1 NIC without a VLAN. If you need more bandwidth then you can go to a faster NIC or bond NICs for more bandwidth. Use 1 gig NIC and you should be fine as there is very little faster internet than a gig. You do not need 10 gig in pfsense as the switch is handling all the local high-speed traffic if you have 10 gig.

Use DHCP in the L3 switch or you can use Microsoft DHCP as they have a very nice one that handles multiple DHCP scopes for a large network if you need something bigger than a L3 switch.

You scale your internet firewall router for your internet speed and you scale your L3 switch for your local traffic speed. So, if you are doing 10 gig local you do not need necessarily need 10 gig on your internet router with 1 gig internet connection. If you want to get the full gig of internet then you will need a 2.5 gig NIC with a supporting modem or maybe bonded NICs. This is all new and not fully defined. I think when fully done bonded NICs will drop off and not be supported sometime in the future.

What I gathered off the pfsense forums back when I ran pfsense is they think you should use a 30 bit mask or a point-to-point connection from pfsense to a L3 switch. It does work either way as I have run a full class C and point to point connection.
 
Last edited:
  • Like
Reactions: phil9878

Vesalius

Member
Nov 25, 2019
95
63
18
@coxhaus If he is going to have pfSense be the dhcp server for a vlan he has to have a dedicated interface for that vlan on pfSense. One of the deficiencies in pfSense. So if no vlans on pfSense then either the Cisco switch or a new separate dhcp server, which is not handicapped like pfsenses, needs to be added.

@phil9878 this looks like it can be made to work as long as vlan 5 and 40 do not need dhcp or internet access. You could as time permits explore setting up a transit link on the sg350 and pfSense to give those 2 vlans internet access as a learning process on how that setup would work.
 
  • Like
Reactions: phil9878

phil9878

Member
Apr 2, 2021
36
5
8
@coxhaus : Thank you. However, maybe you missed the post I quoted from @StammesOpfer. Like @Vesalius answered, pfSensecannot be a DHCP server for a subnet that it has no interface on it (either by defining the VLAN interfaces or by adding dedicated NICs for the VLANs).
Also, as I said in my post, for the immediate time, I cannot afford running a dedicated DHCP server.

@Vesalius : VLAN 5 and 40 do not need internet access. The backup server has a second interface that will be unplugged and that I will set to VLAN 10 whenever I need it to be online for maintenance and updates.
The limitation I have is the SG-350 because it only has 10 ports. On pfSense, with my setup, I still have one spare port on pFsense, eventually two if I take VLAN50 out of internet. I can also gain 1 port on the SG350 by adding a simple VLAN aware switch to join the two VLAN 50 TVs.

I still have a few questions:
1- Having a trunk from SG350 Switch to Pfsense with the VLANs is a no go for downstream traffic routing issues as I understood. However, no one mentioned this: With the VLANs defined on pfSense + Firewall rules to disable any routing between the interfaces in pfSense, will the downstream traffic still cause routing issues at the pfSenseLevel ? That way, L3 switch can do the interVLAN routing and pfSense runs on-a-stick with a trunk to the switch !

2- With my diagram, does pfSense block by default any routing between the physical interfaces because they are on different subnets ?

3- With my diagram, do I still need to define the VLANs in pfSense and assign them to the physical interfaces ? If not, will pfSense remove the VLAN tags causing downstream traffic issues at the Switch ingress level ?

4- In my diagram, I put pfSense in the VLAN 20 IP range, same as the Switch it is connected to. Is it a a good choice or should it be on its own VLAN or IP subnet ?

I am new to VLANs and some concepts are not clear, despite the huge reading I did in the last months.

Thank you for your help setting this up.
 
Last edited:

coxhaus

Member
Jul 7, 2020
74
27
18
@coxhaus If he is going to have pfSense be the dhcp server for a vlan he has to have a dedicated interface for that vlan on pfSense. One of the deficiencies in pfSense. So if no vlans on pfSense then either the Cisco switch or a new separate dhcp server, which is not handicapped like pfsenses, needs to be added.

@phil9878 this looks like it can be made to work as long as vlan 5 and 40 do not need dhcp or internet access. You could as time permits explore setting up a transit link on the sg350 and pfSense to give those 2 vlans internet access as a learning process on how that setup would work.
pfsense does not need a VLAN for DHCP to work. In the past you could define DHCP for 1 network without a VLAN. I think the less information in your firewall device the better. I use static IPs in my router LAN network to my L3 switch.
 

coxhaus

Member
Jul 7, 2020
74
27
18
@coxhaus : Thank you. However, maybe you missed the post I quoted from @StammesOpfer. Like @Vesalius answered, pfSensecannot be a DHCP server for a subnet that it has no interface on it (either by defining the VLAN interfaces or by adding dedicated NICs for the VLANs).
Also, as I said in my post, for the immediate time, I cannot afford running a dedicated DHCP server.

@Vesalius : VLAN 5 and 40 do not need internet access. The backup server has a second interface that will be unplugged and that I will set to VLAN 10 whenever I need it to be online for maintenance and updates.
The limitation I have is the SG-350 because it only has 10 ports. On pfSense, with my setup, I still have one spare port on pFsense, eventually two if I take VLAN50 out of internet. I can also gain 1 port on the SG350 by adding a simple VLAN aware switch to join the two VLAN 50 TVs.

I still have a few questions:
1- Having a trunk from SG350 Switch to Pfsense with the VLANs is a no go for downstream traffic routing issues as I understood. However, no one mentioned this: With the VLANs defined on pfSense + Firewall rules to disable any routing between the interfaces in pfSense, will the downstream traffic still cause routing issues at the pfSenseLevel ? That way, L3 switch can do the interVLAN routing and pfSense runs on-a-stick with a trunk to the switch !

2- With my diagram, does pfSense block by default any routing between the physical interfaces because they are on different subnets ?

3- With my diagram, do I still need to define the VLANs in pfSense and assign them to the physical interfaces ? If not, will pfSense remove the VLAN tags causing downstream traffic issues at the Switch ingress level ?

4- In my diagram, I put pfSense in the VLAN 20 IP range, same as the Switch it is connected to. Is it a a good choice or should it be on its own VLAN or IP subnet ?

I am new to VLANs and some concepts are not clear, despite the huge reading I did in the last months.

Thank you for your help setting this up.
No trunk as that is layer 2. pfsense needs to have a gateway or gateways defined for your networks defined on your L3 switch. You also need to add ACLs for the gateways to allow internet access as pfsense blocks all traffic other than pfsense traffic. This is called directly connected networks and not. The gateway networks are defined on the L3 switch not pfsense. Do not define VLANs on pfsense that are defined to the switch. You only need 1 network defined on the L3 and pfsense. So the default route on the L3 switch will point to the pfsense. pfsense gateways will point to the IP address on the L3 switch which connects to pfsense.

To define pfsense and a L3 switch together use example, 192.168.10.1/30 pfsense and 192.168.10.2/30 L3 switch. The mask would be 255.255.255.252. On the L3 switch side create a VLAN 10 and assign an IP of 192.168.10.2 255.255.255.252. Then on the L3 switch assign the default route, some may have default gateway, 192.168.10.1 which is pfsense's LAN interface. Assign a port to VLAN 10 and connect a CAT5e cable from the assigned port to the LAN port on pfsense.

There is a thread on L3 switches on pfsense forums as I wrote all that I did to make pfsense work with a Cisco L3 switch.
 
Last edited:
  • Like
Reactions: phil9878