Layer 3 Switch w/ PFSense

phil9878

Member
Apr 2, 2021
39
5
8
I wonder if pfsense is pushing the return traffic back through the trunk port since there are no route statements. Can you disconnect the trunk port and add route statements and ping out from a client? I don't care if DHCP is broken.

Never mind. You can't add route statements with DHCP defined with those networks.
It is routing from the switch so it is returning right.
Yes, static routes cannot be added if the interface is defined any way, because pfSense knows about the interface

I doubt downstream traffic is going through the Transit route. I am not sure how to check it in pfsense, maybe an allow rule...
However, I am not seeing any issues in logs about specific TCP:S and TCP:FA blocks related to my use
 

coxhaus

Member
Jul 7, 2020
85
31
18
I have been thinking about this diagram and I think it will not work with your pfsense L3 structure. If you use only new networks on the SG350X L3 switch I think you can make it work Pfsense knows all the networks by DHCP and it would try to use your L2 connection as you can't point to the second L3 switch with a routing statement since the network is already defined on pfsense. Think about it that is the way multiple routers would work.
 
  • Like
Reactions: phil9878

phil9878

Member
Apr 2, 2021
39
5
8
I have been thinking about this diagram and I think it will not work with your pfsense L3 structure. If you use only new networks on the SG350X L3 switch I think you can make it work Pfsense knows all the networks by DHCP and it would try to use your L2 connection as you can't point to the second L3 switch with a routing statement since the network is already defined on pfsense. Think about it that is the way multiple routers would work.
I dropped that idea of the L2 switch in between pfSense and the L3, so I did not test it. Also, pfSense cannot do DHCP in any case with that diagram because it doesn't have the downstream VLAN interfaces. It has static routes.

With my last diagram configured however, everything works properly. pfSense doesn't care about the asymmetric path for downstream traffic coming from the wan! Other firewalls won't allow it I guess!

Still looking how to multicast between vlans. It seems complicated despite the switches should support it in docs
 

coxhaus

Member
Jul 7, 2020
85
31
18
Why do you think you need multicast? It is a 1 to many relationship. Unless you are going to put all your TVs on the same channel it probably won't help you.
 

phil9878

Member
Apr 2, 2021
39
5
8
The DLNA Media server is now running in a jail inside the Data Server, on the same VLAN as the TVs. The DLNA server and the Data server are on different physical interfaces and on different VLANs. TVs are fine with that as they get the DLNA brodcasting on the same subnet

However, I wanted to receive the DLNA stream on another VLAN. I can live without it. But, since the Cisco SG350 series support it, I want to make it work, also for learning purposes.

I am still progressively setting up the network, segment by segment. I just managed today to isolate the Media Server Jail in a dedicated VLAN different from the Host Server. Needed some work in TrueNAS...

Once I am completely done, I will post the config of the different devices. The nice part is that it is very simple to change the layout to a single transit, no interfaces in pfSense and only static routes, no DHCP in pfSense if someone has another firewall that wont accept the asymmetric part in the proposed config.

Meanwhile, if someone has an idea how to properly setup the switch for the DLNA across VLANs, please drop a comment :)

Best regards
 

phil9878

Member
Apr 2, 2021
39
5
8
I don't know as it has been many years since I setup a multicast and I think back then it was all in 1 VLAN. Here is something that might relate across VLANs. From what I remember it was a manual process for each port to assign multicast or not.

Configure Access Port Multicast TV Virtual Local Area Network (VLAN) Membership on a Switch - Cisco
I read that article
But it was not clear for me during testing:
- how to add the Media Server from a Trunk to the Multicast Group (Multi Cast group members are only Access ports on the receiver side)
- what IP is the Multi Cast Group

I will try it later once I finished configuring the network. I have to try the 239.255.255.250 IP:
 

phil9878

Member
Apr 2, 2021
39
5
8
Here's the final layout sample with pfSense properly working as a DHCP and DNS Server:


Internet Network Diagram Template 7.vpd.jpg

  • pfSense is the WAN <-> LAN Firewall
  • VLANs isolation must be done at the L3 routing Switch with ACL rules
  • VLAN 1 dedicated interface for the management LAN is optional and can be untagged with the main Trunk if there is no room for physical interfaces.
  • In pfSense, the only interface having internet access is the Transit interface: this simplifies internet control
  • In pfSense, define all VLAN interfaces (10.0.x.1) and set these rules for each interface
    • block access to management ports (80, 443, 22) from "any" to "This Firewall"
    • allow access from "VLAN net" to "VLAN address" for DNS ports (53, 853 as needed)
    • optionally allow UDP access from "VLAN net" to broadcast addresses (10.0.x.255 and 255.255.255.255)
    • the remaining is default deny
  • In pfSense, for the Transit interface:
    • block access to management ports (80, 443, 22) from "any" to "This Firewall"
    • add any specific internet rules you need (DNS resolver NAT rules are done on this interface if DNS restrictions are needed)
    • Allow IPv4+IPv6 access from "any" to "non RFC1918 Private Networks"
  • pfSense DHCP server for each VLAN must set the gateway of the clients to the Routing Switch VLAN interface (10.0.x.2)
  • Clients with a manually set IP, must point to the Routing Switch VLAN interface (10.0.x.2) as a gateway and pfSense VLAN interface (10.0.x.1) as DNS Server
  • L3 routing switch (SG350):
    • define all VLAN interfaces with static IP 10.0.x.2
    • enable inter VLAN routing
    • define the static route interface or VLAN 172.26.1.2
    • define the default static route for internet and non-VLAN traffic: 0.0.0.0/0 -> 172.26.1.1
    • define all ACL rules needed to isolate VLANs
  • L3 / L2 switch (SG350X)
    • the SG350X cannot be set to L2, so just disable inter VLAN routing
    • define the management interface VLAN 1 (10.0.1.3)
    • Optional: I added the VLAN 10 interface (10.0.10.3 ) to be able to manage the switch from VLAN 10 workstation without needing the VLAN 1 interface
    • No need to define other VLAN interfaces (just like an L2 switch) neither static routes
    • The trunk linking the 2 switches must have tagged all the VLANs specified in the L2/L3 downstream switch. The same VLANs must be defined on both ends of the Trunk obviously
  • Unifi AP :
    • trunk is untagged to the default management interface of the AP. Can be tagged also depending on the controller software versions
    • define in the trunk all the VLANs used in the AP for SSID broadcast

Hope this can help other people trying to get pfSense to work as DHCP server with a downstream L3 routing switch

And many thanks for all your help

I am continuing to tweak the ACLs on the switch and I will look at the Multicast later. Currently I can just add a virtual media Server and tag it to the VLAN for which I need DLNA access
 
  • Like
Reactions: cesmith9999

coxhaus

Member
Jul 7, 2020
85
31
18
You know you can use both your L3 switches if you don't try to control the network with pfsense.

Why did you add the third link?

When I ran pfsense with my Cisco L3 switch I used 1 30-bit link. So much simpler.
 

phil9878

Member
Apr 2, 2021
39
5
8
You know you can use both your L3 switches if you don't try to control the network with pfsense.

Why did you add the third link?

When I ran pfsense with my Cisco L3 switch I used 1 30-bit link. So much simpler.
What is the 3rd link you mean ?

The transit route must be untagged, or it will not work properly in my tests, but maybe I missed something in the early times I tested it
The management VLAN 1 was added on purpose. I can untag it to the trunk, but I had the spare interface
pfSense only does the DHCP and DNS services + WAN <-> LAN Firewall. It is not concerned by any inter VLAN setup
I need the DNS resolver to restrict some clients to openDNS services. The DNS resolver makes it possible to even limit the DNS over HTTPS traffic in addition to DNS-over-TLS

The rules in pfSense are easy to setup. Only Transit interface needs internet control rules. The other interfaces need simple redundant rules, except for the DNS restrictions. So it is much more friendly than the ACLs in teh switch. If it supported a good 10 Gb switching I would clearly use pfSense. But I do not have the needed switch in that location neither the 10 Gb interface on pfSense. And I doubt it can do any efficient routing at that bandwidth with my i3 CPU
 

coxhaus

Member
Jul 7, 2020
85
31
18
It looks like you have 3 cat5e cables from pfsense to your L3 switch.

I only used 1 CAT5e cable from pfsense to my L3 switch.

Yes, the traffic needs to be untagged for routing as you are using layer 3. Your L3 switch will know from where the traffic routed from. A header trailer is added to the IP packet.
 
  • Like
Reactions: phil9878

phil9878

Member
Apr 2, 2021
39
5
8
It looks like you have 3 cat5e cables from pfsense to your L3 switch.

I only used 1 CAT5e cable from pfsense to my L3 switch.

Yes, the traffic needs to be untagged for routing as you are using layer 3. Your L3 switch will know from where the traffic routed from. A header trailer is added to the IP packet.
I see. I can reduce it to 2 interfaces and spare one switch port. I need both vlan 1 and transit interface to be untagged. Or you think it is possible / recommended to tag Vlan 1 in the trunk?
 

coxhaus

Member
Jul 7, 2020
85
31
18
I have no idea with your setup as far as tagging. I for a while ran VLAN1 tagged in my Cisco network. I then switched a few years ago. My routing needs to be untagged from my L3 switch to my router. That is why I use an access port. If you are doing trunking then you are using layer 2.
 
Last edited:
  • Like
Reactions: phil9878

phil9878

Member
Apr 2, 2021
39
5
8
I have no idea with your setup as far as tagging. I for a while ran VLAN1 tagged in my Cisco network. I then switched a few years ago. My routing needs to be untagged from my L3 switch to my router. That is why I use an access port. If you are doing trunking then you are using layer 2.
Yes, probably it can be tagged on the trunk between pfSense and the Switch, no reason it cannot indeed
I can also untag it on the trunk. Having it untagged is useful to debug the pfSense box by a directly connected device to the untagged VLAN 1 port
 

phil9878

Member
Apr 2, 2021
39
5
8
@kapone and @coxhaus

Do you think there is an issue with that asymmetric WAN route: outbound from Transit and inbound from VLAN interface on pfSense ?
I do not see any dropped traffic. The mod at Netgate is claiming I am bypassing teh firewall states checks, but pfSense is not bothering at all. I can define Internet and LAN rules and they properly work, so it seems that pfSense does't track that sort of traffic (bug ?)

Is it fien security wise or am I missing something ? Outside tests from grc.com are fine also
 

coxhaus

Member
Jul 7, 2020
85
31
18
I have not seen problems with asymmetric routing. You need to watch your gateways and make sure you are using the right one. Routing to me is independent of firewalling. The firewall should process traffic accordingly. It would seem strange to me that firewalling would happen before routing but I have never used Netgate.
 
  • Like
Reactions: phil9878

phil9878

Member
Apr 2, 2021
39
5
8
It would seem strange to me that firewalling would happen before routing but I have never used Netgate.
pfSense will first route the traffic to the corresponding interface, then it will check the firewall rules and drop/pass traffic accordingly.
So, am I bypassing firewall ? I do not think so since the rules will be processed on the interface anyway
 

Pakna

New Member
May 7, 2019
16
2
3
Apologies if I am hijacking the thread but in trying to set this machinery up on my LAN and after going through this thread (any many others elsewhere) and having tried everything I could think of but am still partially successful at best. I suspect I am either misconfiguring the default route or have some fundamental misunderstanding about transit network and/or VLANs but after trying most things I'd regard sensible, I am still unable to ping the firewall from a machine in a VLAN outside of transit network.

I would really appreciate some insights here as I am thoroughly confused and I can't understand what part of setup/assumption is wrong.
Whatever additional info is needed I am more than happy to add.

Note that the Workstation B has a static IP whereas the rest is DHCP - this is temporary but intentional: in fullness, I intend to fully leverage the Dell 8024F on-switch DHCP server but wanted to keep the test scenarios tight and not introduce complexity unless this setup works (which is invariant from DHCP, as it is).

The firewall is a virtualised instance running in Proxmox VE 6.3.1 (full disclosure: no network bridge is made VLAN aware, but based on my understanding, it should not be anyway). If need be, I can spin up a bare metal firewall instance and retry (but I'd rather not go through the hassle, unless really needed).

Here are my test scenarios and results:
ActionExpected ResultObserved Result
telnet to switch via OOB and ping 172.16.0.1OK (no loss)OK (no loss)
ping from firewall to 172.16.0.2OK (no loss)OK (no loss)
8024F pfSense gateway statusOnlineOnline (loss -> 0.0%)
ping from Workstation A to FIBRE0OK (no loss)OK (no loss)
ping from Workstation A to VLAN 15 SVIOK (no loss)OK (no loss)
ping from Workstation A to VLAN 300 SVIOK (no loss)OK (no loss)
ping from Workstation A to Workstation BOK (no loss)TIMEOUT
ping from Workstation B to FIBRE1 (transit net other end)OK (no loss)TIMEOUT
ping from Workstation B to VLAN 15 SVIOK (no loss)OK (no loss)
ping from Workstation B to VLAN 300 SVIOK (no loss)OK (no loss)
ping from Workstation B to Workstation AOK (no loss)TIMEOUT

Note - test results above are the same for Port 4 configuration in ACCESS or TRUNK mode. Port 2 is always ACCESS mode.

Here is a picture of my setup:
Network Home.png

Here is the switch routing table output :

1623019199466.png

My current understanding is:
- inter-VLAN routing works OK
- pfSense gateway + static routing works OK

The most suspect configuration here is the default route - but I have followed the switch docs and this seems the correct way to configure this.

Questions:
- why can't I ping pfSense end of transit network from Workstation B despite it being ping-able from the switch?
- why can't I ping Workstation B from Workstation A (and vice-versa)?
 
Last edited:

coxhaus

Member
Jul 7, 2020
85
31
18
Usually what I have seen is it is related to the gateway and routing. In a layer 3 switch network you want the L3 switch to be the gateway for all clients. There will be a gateway for each network on the L3 switch. Then you want the default route on the switch pointing to the router. Then there should be routing statements on the router pointing to the L3 switch for all disconnected networks. More than likely, you need to open all disconnected networks in the firewall so they have internet access. I would check the gateway for each of the clients not able to ping. There should be a L3 path to the devices.

Adding multiple links between the router and the L3 switch can cause spanning tree problems. I only use 1 link but that will require the L3 switch to do DHCP or a device other than the router. The easy fix is to use 1 connection from pfsense to the L3 switch.
My guess is your routing statements do not match the connected links between devices. You are making this way too hard when you could use 1 link. The link can be 4 ports if you want but stay with 1 link between devices, much easier.
 
Last edited: