I'll draw a diagram in a bit (don't have my Visio machine handy, it's at the office).
The problem I was trying to solve (in a somewhat elegant manner) was multi-fold:
- pfSense won't do DNS/DHCP for anything that's not an interface in it. But, if we do that with a layer 3 switch, the question becomes, which device is going to be the gateway for the VLAN? The switch or pfSense?
- Preserve line rate performance between VLANs, at the switch.
- Why do we need to have anything more than a single NIC in a firewall/pfSense device *IF* we're using a layer 3 switch?? The L3 switch IS a router, all we need to do is pass traffic back and forth between the firewall, depending on the rules you setup. If you want to setup rules between VLANs, do it at the switch (it IS a router after all), if you want to setup rules between your internal network and the WAN, that's when you go to your firewall.
- How do I do packet analysis/capture, traffic analysis, IDS etc etc on the WAN, without limiting myself to the hardware/software that runs the firewall?
So, in trying to solve these issues, the lightbulb moment actually came from someone here...
@Blue)(Fusion - you're the man! (or woman)
Let's take a layer 3 switch and create 3 VLANS on it.
View attachment 10644
1. The WAN VLAN - This is just a VLAN, add untagged ports to it (at least two, so you can mirror your WAN port and do any additional stuff with that traffic). There is no IP address on this VLAN, no SVI, nothing. Make sure you add the TRANSIT port as
tagged to this VLAN.
On the pfSense side, create a VLAN that has the matching ID on the same Nic that is used for the TRANSIT pipe. Don't add an interface based on this VLAN, go into interface assignment, and for the WAN, just choose it from the drop down.
View attachment 10641
Nothing special needs to be done beyond this, other than when we're finally done, reboot the firewall, and do release/renew on the WAN (assuming it's DHCP based).
2. The transit VLAN - This is the VLAN that connects the switch to the firewall. In my case, all it has is a single 10g port,
dual-mode (the native VLAN is the transit VLAN), but there's more than one way to skin this cat. The IP address for this VLAN on the switch side is
172.16.0.1/30 (in my case, we only need two IP addresses).
On the pfSense side, there is *no* VLAN for this (remember we added an untagged port on the switch side). The IP address on the pfSense side is static IPv4,
172.16.0.2/30. I named this interface
TRANSIT in pfSense. It does not require any additional gateway or anything like that.
View attachment 10642
Go to the firewall rules and allow ICMP on this interface.
View attachment 10637
This will allow you to ping the switch IP (172.16.0.1) from pfSense and the interface IP on the pfSense side (172.16.0.2) from the switch console. These two guys are happy as clams at this point.
On your switch add a default route (
0.0.0.0/0) to point to pfSense (
172.16.0.2). With that the switch will send all unknown traffic on the transit VLAN to pfSense. If you try to do a nslookup or anything like that on the switch console at this point, it won't work, because we haven't set up the rules yet.
3. Your home/whatever VLAN where the (some of the) devices are - Add whatever untagged ports you need on the switch but also add the transit port as
tagged. This will ensure that any unknown traffic from this VLAN will be tagged by the time it hits pfSense over the transit pipe.
In my case, on the switch side, I added a few untagged ports to this, added the tagged transit port and gave it an IP of
10.10.10.1/24. That IP address is important...
On the pfSense side, I created a VLAN with the same ID on the same NIC that the transit interface runs on. Then I added an interface based on that VLAN, and gave it a static IP v4 address of
10.10.10.254/24. No gateway or anything needed. But, we're not done yet.
Enable the pfSense DHCP server on this interface, and give it the following settings.
View attachment 10638
The important parts are:
- Make sure the DHCP range does not include .1 or .254, as we're using those IP addresses, but other than that, nothing special.
- The gateway is set to the
IP of the switch for this VLAN. That way, when a device connects, gets an IP address from DHCP, they get a gateway that is on the switch itself. If they access anything else on the switch itself, all L3 routing will be at the switch. The switch will only send traffic to the TRANSIT pipe if it does not know how to route it.
But we're not done yet...
Your device gets an IP address, DHCP is working, the DNS server (you'll be getting the x.x.x.254 IP as the DNS server) is ... inaccessible. That's because we haven't set up the firewall rules yet. So... for this interface in pfSense:
View attachment 10639
I added the ICMP rule for testing, it's not required. Essentially allow DNS traffic as
tagged traffic over the TRANSIT pipe.
Now if you do a nslookup on the device, it'll work and resolve the IP, but still no Internet... We're not done yet.
View attachment 10640
Allow the interface
net traffic on the
TRANSIT interface to whatever you choose, right now it's set to anything in my case. When you're setting up the rule in pfSense, the drop down for source will have the "xxxxinterface net" (which is essentially the IP range). And the switch knows how to route this traffic.
Assuming we've done everything right, you should be up and running with Internet access on that device VLAN. From this point on, knock yourself out adding more VLANS and/or firewall rules.
p.s. (I hope I got everything down accurately. This is actually good, as it's documenting it for me as well.