Layer 3 Switch w/ PFSense

PGlover

Active Member
Nov 8, 2014
468
55
28
54
This has a lot more permutations/combinations than you realize... :) Just the first step...



What vlan interface?? The TRANSIT interface doesn't have to be a VLAN on the PFSense side. How's pfSense connected to the TRANSIT interface? physical NIC, virtual NIC? How's the WAN terminated? To pfSense or to the layer 3 switch?
Just trying to make sure I have the setup correct on my pfSense box.

I was under the impression that you needed to create the Transit Vlan on the pfSense box as well (see below)
upload_2020-3-12_22-4-42.png

In my setup, the Transit VLAN is created via 2 physical nics forming a LAGG interface.
upload_2020-3-12_22-9-12.png

In my setup, WAN is terminated on a physical nic on the pfSense box I think.
upload_2020-3-12_22-12-26.png
 

tommybackeast

Active Member
Jun 10, 2018
251
82
28
Just trying to make sure I have the setup correct on my pfSense box.

I was under the impression that you needed to create the Transit Vlan on the pfSense box as well (see below)
View attachment 13254

In my setup, the Transit VLAN is created via 2 physical nics forming a LAGG interface.
View attachment 13255

In my setup, WAN is terminated on a physical nic on the pfSense box I think.
View attachment 13256
excuse off topic question: on a dedicated pfsense box; what advantages are there in using a 4port NIC card vs a 2port NIC Rj45 card?
 

kapone

Well-Known Member
May 23, 2015
796
388
63
Just trying to make sure I have the setup correct on my pfSense box.

I was under the impression that you needed to create the Transit Vlan on the pfSense box as well (see below)
View attachment 13254

In my setup, the Transit VLAN is created via 2 physical nics forming a LAGG interface.
View attachment 13255

In my setup, WAN is terminated on a physical nic on the pfSense box I think.
View attachment 13256
Keep it simple to test.

- Connect a single PFSense NIC to the TRANSIT vlan as an untagged port.
- Don't use LAGGS for now, while testing.

As far as pfSense is concerned, there is no vlan for the TRANSIT network, the switch will strip away the respective VLAN header.
 

NYCone

Member
Jun 23, 2017
35
8
8
57
I just found this thread.

I was new to pfSense and Vlans a few weeks ago when I tried to set up a pfSense based multi Vlan network. I kept failing at DHCP and some DNS issues. I thought I was at fault for not configuring pfSense correctly (in a way still true, I didn't know pfSense couldn't handle DHCP on Vlans).

Time to reassess...
 
  • Like
Reactions: tommybackeast

ViciousXUSMC

Active Member
Nov 27, 2016
201
98
28
37
Just keying back in on this old thread.
I recently decided to flatten out my network to a single Layer 2 domain because I was having some IOT devices and other such devices not really like crossing a Layer 3 domain.

I also decided Jumbo Frames were no longer something I really needed and that was one of the main reasons I wanted to have a L3 network for all my server/storage traffic.

So I reduced my administrative overhead greatly.

I did think about trying to simplify my routing and use PFSense as the router instead of the switch (and allows me to use a L2 switch in a pinch due to a failure) should I ever decide to go back to mutli VLAN, and sure enough my worry is verified here.

I have say my desktop with a 10gb interface and my NAS with a 10gb interface and if I use PFSense to do the routing I may run into some performance issues.

Here is the one big thing I wanted to say though and the entire reason for this post.

I do not know why, how, etc but I had PFSense crash on me once in a while (especially with an influx of torrent traffic) if Surricata was running. I had to do every tweak you can imagine to the interfaces and metrics, and even went so far as to install an entirelly new NIC on my PFSense box.

I still had those random crashes now and then where the interface would just have a kernal panic.

Now that I have flattened everything out, it has been rock solid for weeks now and I am almost positive I would have seen a crash by now. So I don't know why/what/how but it seems that the way I had routing setup between the swtich and firewall maybe was inducing that crash?

I mean everything worked, all my routing was sound, every device coudl talk to each other device. IP's configured correctly, nothing I googled pointed to configuration as the issue but rather other issues. But it seems like that my issue was my configuration between the switch and PFSense, perhaps because I let the switch do all the routing and only let PFSense know a static route back to the switch for each vlan with a default route to the internet.

For my next adventure, if I install 10gb interfaces on my PFSense box and let it handle the routing for the vlans, anybody think that would show an appreciable difference compared to letting the switch handle it?
 

kapone

Well-Known Member
May 23, 2015
796
388
63
For my next adventure, if I install 10gb interfaces on my PFSense box and let it handle the routing for the vlans, anybody think that would show an appreciable difference compared to letting the switch handle it?
I wouldn't do that, unless you need to firewall these VLANs. There's a reason Layer 3 switches (routers) exist. Line speed routing WITH all ACLs applied. To get that level of performance in pfSense....not going to happen.
 

ViciousXUSMC

Active Member
Nov 27, 2016
201
98
28
37
Yeah I would like to have the ability to do simple FW rules to say block IOT devices from the network but allow access to the NVR and internet, allow access from X to Y etc.

Most if not all of it can be done by ACL's but not quite as clean and easy.

Its nice to have aliases and such all setup in PFSense.

What I have done in the past is actually have multiple interfaces on my Desktop on different VLANS that worked fine but felt kind of dirty.

Also what is everybodys take on a trunk port to PFSense and a Interface for each VLAN but set the Default Gateway to the Virtual IP for that VLAN for the switch as compared to a single non trunked routed port to PFSense with a default gateway set to the PFSense single interface?

I think it should work both ways, but if I have a VLAN interface on PFSense even if the routing happens from the switch I should be able to use PFSense for DHCP and such.
 

blinkenlights

Active Member
May 24, 2019
143
64
28
I recently decided to flatten out my network to a single Layer 2 domain because I was having some IOT devices and other such devices not really like crossing a Layer 3 domain.
It is possible to do that with multiple Layer 2 domains - I have three internal segments at home (VoIP, wired, wireless) and devices can see services offered across the network using Avahi/Bonjour/Zeroconf/etc. relaying on the firewall. Wife not being able to cast content from the smartphone to smart TV forced me to "get smart" quickly....

I do not know why, how, etc but I had PFSense crash on me once in a while (especially with an influx of torrent traffic) if Surricata was running. I had to do every tweak you can imagine to the interfaces and metrics, and even went so far as to install an entirelly new NIC on my PFSense box.

I still had those random crashes now and then where the interface would just have a kernal panic.
See, now I am curious and want to know more about these crashes. My home pfSense box is running the development branch (2.5.x), updated once a month or so. Except for some wonky PHP errors that force me to reinstall (*sigh*) the operating system has been rock solid, even with Squid + Snort. I've been doing this since the first iteration of my pfSense firewall (Atom D525 with dreadful onboard NICs) to present day (E5-1650 v4 with 2x Chelsio T520-BT).

For my next adventure, if I install 10gb interfaces on my PFSense box and let it handle the routing for the vlans, anybody think that would show an appreciable difference compared to letting the switch handle it?
Sorry if I missed it in the thread, but what type of hardware do you have in that box? Within reasonable limits, allowing the firewall to manage VLAN segments should have minimal impact on performance. I imagine this is especially true if your network cards can properly handle offloading (VLAN checksums, tagging, etc).
 

laserpaddy

Active Member
Jul 17, 2017
166
39
28
out there
I use the router on a stick , and 6 vlans dhcp being done on pfsense, all ubiquiti devices behind it, unifi 10g switch 24port, 48port, nanohd ap and another edge 24p switch which can do the routing if needed.
Lawrence systems on YouTube has an incredible step by step videos on all of this and much more.
 

ViciousXUSMC

Active Member
Nov 27, 2016
201
98
28
37
It is possible to do that with multiple Layer 2 domains - I have three internal segments at home (VoIP, wired, wireless) and devices can see services offered across the network using Avahi/Bonjour/Zeroconf/etc. relaying on the firewall. Wife not being able to cast content from the smartphone to smart TV forced me to "get smart" quickly....



See, now I am curious and want to know more about these crashes. My home pfSense box is running the development branch (2.5.x), updated once a month or so. Except for some wonky PHP errors that force me to reinstall (*sigh*) the operating system has been rock solid, even with Squid + Snort. I've been doing this since the first iteration of my pfSense firewall (Atom D525 with dreadful onboard NICs) to present day (E5-1650 v4 with 2x Chelsio T520-BT).



Sorry if I missed it in the thread, but what type of hardware do you have in that box? Within reasonable limits, allowing the firewall to manage VLAN segments should have minimal impact on performance. I imagine this is especially true if your network cards can properly handle offloading (VLAN checksums, tagging, etc).
I never had any crashes with Squad + Snort, and I do not think with Surricata in the legacy mode but the new in-line mode is what causes me the issues. If I disable Surricata I get no crashes.

The box is a Dell R210ii with a Xeon E3-1230 V2 w/ 8GB RAM and 2x 200GB SSD in Raid 1 running PFSense 2.4.5 Release
 

klui

Active Member
Feb 3, 2019
176
73
28
I thought I was at fault for not configuring pfSense correctly (in a way still true, I didn't know pfSense couldn't handle DHCP on Vlans).
How so? DHCP works with VLANs under a trunk interface.
 

ViciousXUSMC

Active Member
Nov 27, 2016
201
98
28
37
As long as there is an interface on the subnet you want you get a DHCP server entry, what it wont let you do is create an arbitrary DHCP scope that does not have an interface and use IPHelper or something to relay it across different subnets.
 

blinkenlights

Active Member
May 24, 2019
143
64
28
I never had any crashes with Squad + Snort, and I do not think with Surricata in the legacy mode but the new in-line mode is what causes me the issues. If I disable Surricata I get no crashes.

The box is a Dell R210ii with a Xeon E3-1230 V2 w/ 8GB RAM and 2x 200GB SSD in Raid 1 running PFSense 2.4.5 Release
Ah, yeah, inline mode probably stresses different parts of the system. As @laserpaddy alludes to, there should not be any problems with pfSense managing the VLANs itself.
 

coxhaus

New Member
Jul 7, 2020
8
0
1
When I ran pfsense with my Cisco SG300-28 L3 switch before the SG350 came out several years ago all the VLANs, DHCP were defined to the L3 switch. pfsense only did firewall duties. I used a 30 bit mask between pfsense and my layer 3 switch. All local network functions were handled by the switch and pfsense just opened and closed the door to the internet.

I also setup my daughter's small business up with a Cisco SG500X-24 in L3 mode the same way plus using VOIP for IP phones but I used a router instead of pfsense.