Layer 3 Switch w/ PFSense

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

coxhaus

Active Member
Jul 7, 2020
109
36
28
  • Like
Reactions: phil9878

phil9878

Member
Apr 2, 2021
39
8
8
Yes, I saw, and he was stuck at L2

I'd love to help you, but am swamped right now.
No problem, really, thank you. I am going to do the safest / easier way for a non network expert guy like me

Oops, found a workstation in the bottom right corner. He has the potential to saturate his backbone if it is fast enough.
For, now, it is 1 Gb only at that side. However, during backups, it can saturate the link between the L2 and L3 switches. Sadly, I am limited by these factors:
  • The room where the L3 switch and servers are located only have 2 wall LAN plugs reaching the L2 Switch room (One for the 10 Gb to desktop and one at 1 Gb to L2 Switch)
  • I cannot move the L3 Switch to the room with the workstation because of the switch noise. I even had to silent it the hard way because of the non acceptable noise through that room door. It needs to be cable plugs downside / fans upside for silent 40x40 fans. The link is here: https://forums.servethehome.com/ind...ries-fan-replacement.18880/page-3#post-300923
  • I need a 10 Gb connection between the Workstation and the servers
  • Once I can afford one of the new Cisco fanless 10 Gb switches, I will replace the SG350. I could do the L3 switching at 10 Gbps in the office room where pfSense resides and I can link the two switches with 2x 10 Gb cables and LAG

Below is the new diagram I plan. I will use a transit route to pfSense and the DHCP server on the SG350X. I tested interVLAN routing on one switch and I could properly create the VLANs, the transit route, the static routes in pfSense and the default next hop router in Switch. I could do the inter VLAN routing in the Switch when pfSense is down. I could properly ping IP/addresses between VLANs and to the cloud. So I feel confident configuring it this way.


Internet Network Diagram Template 4.jpg


I am just not sure how to setup the links between pfSense to the L2 Switch to the L3 switch:
  • pfSense to L2: a dedicated transit VLAN on the L2 switch and pfSense ?
  • L2 to L3: a trunk with all VLANs ?
  • pfSense: static routes for 10.0.* pointing to L3 transit interface: 172.26.1.2
  • L3 Switch next hop router 0.0.0.0/0 to pfSense 172.26.1.1

Does this sound good ?
I will have time in the next days to start testing hopefully

Best regards
 

Attachments

coxhaus

Active Member
Jul 7, 2020
109
36
28
If you are going to leave all the hardware as is then I would make the SG350 switch L3 to pfsense and on the SG350X switch I would plug all the 10 gig connections in 1 isolated network so they do L2 to each other. Connect the SG350X to the SG350 using a trunk port so they connect at L2. This is the way I would do it. Connect the wireless AP on a trunk port to the SG350. But you will need to use DHCP on SG350 L3 switch for this to work.

Your local network speed is going to be faster than your internet speed so I would keep pfsense separate. There is lots of latency in internet traffic that local LAN traffic using a local server will not have.
 
Last edited:
  • Like
Reactions: phil9878

phil9878

Member
Apr 2, 2021
39
8
8
If you are going to leave all the hardware as is then I would make the SG350 switch L3 to pfsense and on the SG350X switch I would plug all the 10 gig connections in 1 isolated network so they do L2 to each other. Connect the SG350X to the SG350 using a trunk port so they connect at L2. This is the way I would do it. Connect the wireless AP on a trunk port to the SG350. But you will need to use DHCP on SG350 L3 switch for this to work.
That's an option I didn't think at in fact. Since in any case the SG350 is wired at 1 Gbps to the L3 switch, I will not loose performance. Once I replace it by a fanless 10 Gb switch, I won't have to do my setup again !
The DHCP server is the same on both switches, so not a big deal where it runs.

Why do you think it is a bad idea to have the inter VLAN routing on the L3 as downstream while the L2 is connected to the Firewall ? Because of the bottleneck at the L2-L3 connection ? Or are there other issues ?

In fact, I do not need much inter VLAN interactions except to the data/media server and the printer. Not between other devices (will be blocked by ACLs). That's why I did not think using the SG350 as L3
 

coxhaus

Active Member
Jul 7, 2020
109
36
28
Both the SG350X and the SG350 are L3 switches and are much faster than any router.

There will not be any bottle necks if you keep all the 10-gig stuff together. Since all your 1 gig devices are coming across the 1 gig backbones the SG350 switch will be able to keep up just as well as the SG350X because the uplinks from the SG350X are 1 gig.

You can keep all your VLANs as the SG350 will be plenty fast enough since your backbone links are 1 gig. Just don't try to route between 10 gig devices across the 1 gig backbone that is why I say keep all the 10 gig in 1 VLAN network contained in the SG350X switch. If you route or pull data L2 across your backbone from a 10-gig device then you will be limited by the speed of the backbone.
 
Last edited:
  • Like
Reactions: phil9878

phil9878

Member
Apr 2, 2021
39
8
8
@coxhaus
I am definitely going that way, you are right. Thank you. Makes the setup simpler. Furthermore, the setup is working since all my tests were done on the SG350.

However, for my instruction, can you answer this:
Why do you think it is a bad idea to have the inter VLAN routing on the SG350X as downstream while the L2 is connected to the Firewall ? Because of the bottleneck at the L2-L3 connection ? Or are there other issues ?
 

coxhaus

Active Member
Jul 7, 2020
109
36
28
You can implement interVLAN routing if you want to. I don't think that it will buy you anything. You only have 2 servers and a workstation on 10 gig. Why do you need different networks?
 
  • Like
Reactions: phil9878

phil9878

Member
Apr 2, 2021
39
8
8
You can implement interVLAN routing if you want to. I don't think that it will buy you anything. You only have 2 servers and a workstation on 10 gig. Why do you need different networks?
I wanted to isolate the backup server (VLAN 5) so that it can only be accessed in VLAN 10. If I put it on VLAN 10, it will be accessible by other VLANs. I would need IP based ACLs which I find unsecure. I will have to verify if the switches can do ACLs by port or VLAN to port. Really not sure and I do not have access to the switch right now
 

coxhaus

Active Member
Jul 7, 2020
109
36
28
I wanted to isolate the backup server (VLAN 5) so that it can only be accessed in VLAN 10. If I put it on VLAN 10, it will be accessible by other VLANs. I would need IP based ACLs which I find unsecure. I will have to verify if the switches can do ACLs by port or VLAN to port. Really not sure and I do not have access to the switch right now
If you want different networks then I would turn on intervlan routing.

The ACLs on the Cisco SG350 and SG350X are very good. They do both port and VLAN. You can even do protocol I think, I am old and I may be mixing it up with Cisco IOS. ACLs are written once and then forget about them. So I don't write them real often.
 
  • Like
Reactions: phil9878

phil9878

Member
Apr 2, 2021
39
8
8
If you want different networks then I would turn on intervlan routing.

The ACLs on the Cisco SG350 and SG350X are very good. They do both port and VLAN. You can even do protocol I think, I am old and I may be mixing it up with Cisco IOS. ACLs are written once and then forget about them. So I don't write them real often.
I looked at the switch book. IPv4 ACL can bind to vlan or ports. So sounds good.

Enabling intervlan routing in both switches connected with a dedicated transit would cause asymmetric routing breaking rdp/htpps... Right?

Discussion on two L3 switches: Connecting multiple layer 3 switches together each with their own svi

Seems much more complex...
 

coxhaus

Active Member
Jul 7, 2020
109
36
28
I looked at the switch book. IPv4 ACL can bind to vlan or ports. So sounds good.

Enabling intervlan routing in both switches connected with a dedicated transit would cause asymmetric routing breaking rdp/htpps... Right?

Discussion on two L3 switches: Connecting multiple layer 3 switches together each with their own svi

Seems much more complex...
You have to do all the same stuff that we are playing with. It is just like running multiple routers on the same LAN. I would not try to route multiple 10 gig devices across the smaller switch. That is why I recommend using L2 to keep it all within the 10-gig switch.

If you could pull a second CAT6 cable you could use your SG350X as the core switch connected to pfsense. Then you could use your 10-gig switch for your 10 gig networks.
 
Last edited:
  • Like
Reactions: phil9878

phil9878

Member
Apr 2, 2021
39
8
8
I pretty sure protocol can be used on the SG350 and SG350X switches with ACLs.
Yes, it does. I am using it in my current single VLAN setup to restrict unsecure traffic between TV and servers to DLNA only. However, since it is an IP range filter, secured only by MAC assignment on pfSense DHCP, I wanted to improve security on my servers going with VLANs


If you could pull a second CAT6 cable you could use your SG350X as the core switch connected to pfsense. Then you could use your 10-gig switch for your 10 gig networks.
Would be a later improvement, yes, but I cannot afford it right now.

Having one switch to route drops my wish to isolate the backup server on a different VLAN. However, 2 L3 switches is more complex to manage the ACL and avoid asymmetrical routing.

I will go with only the SG350 as L3. Once I csn afford a 4x 10Gb fanless switch, i could again have my back server on a dedicated VLAN

Sound good, security wise?

I will post my whole config and switch/pfSense rules once done so that it cab help others

Thank you again
 

phil9878

Member
Apr 2, 2021
39
8
8
So, looking at your diagram to change to layer 3 switching you would need to change pfsense to not use VLANs on pfsense and define DHCP on the SG350 switch or like a Microsoft DHCP server.
@coxhaus and @kapone

I managed to cascade the L3 switch and pfSense + use the DHCP on pfSense:
  • tracert shows all interVLAN traffic not reaching pfSense
  • internet works for all VLANs
  • pfSense DNS resolver and DHCP work for each VLAN and I can assign specific DNS servers (openDNS) to given clients
  • I properly can RDP between VLANs
Internet Network Diagram Template 6.vpd.jpg

Below are the setup keys:

Clients config:
  • DHCP: 10.0.x.1
  • DNS: 10.0.x.1
  • Gateway: 10.0.x.2

L3 Switch:
  • Static route: 0.0.0.0/0 > 172.26.1.1
pfSense:
  • No static routes since the VLAN interfaces are defined
  • DHCP for each interface points to L3 Switch interface as the gateway
  • Firewall rules on the Transit interface to allow WAN traffic
  • Firewall rules on the VLAN interface to allow/block traffic if needed

However, I clearly see an asymmetric routing issue for WAN traffic:
PC 10.0.20.10 -> Switch 10.0.20.2 -> pfSense 172.26.1.1 -> Cloud Host
Cloud Host -> pfSense 10.0.20.1 -> Switch 10.0.20.2 -> PC 10.0.20.10

Can this cause any concrete issue for some applications:
- Remote cloud RDP (I don't need it)
- OpenVPN server on pfSense !
- Other examples so that I know and avoid them ?

Many thanks for all your help that made it work ! But is it a broken setup ?
Having pfSense DHCP to control openDNS for some machines is great addition for me as I need that feature
 
  • Like
Reactions: kapone

coxhaus

Active Member
Jul 7, 2020
109
36
28
I would not use a trunk from pfsense to the Cisco SG350 L3 switch. Using a trunk if your router gets slow then it will slow your switch for local routing for all the 10 networks. You realize your 10 networks are being routed by pfsense for foreign networks

Post a trace route from workstation 10.0.10.10 to 9.9.9.9 or workstation 10.0.20.10 either will work.

If you cross the 172 network it will be in the trace route. I think it will show L2 in the trace route for your switch.

PS
If you use networks closer together it is much easier to superscope ranges for less ACL statements.
 
Last edited:
  • Like
Reactions: phil9878

phil9878

Member
Apr 2, 2021
39
8
8
I would not use a trunk from pfsense to the Cisco SG350 L3 switch. Using a trunk if your router gets slow then it will slow your switch for local routing for all the 10 networks. You realize your 10 networks are being routed by pfsense for foreign networks

Post a trace route from workstation 10.0.10.10 to 9.9.9.9 or workstation 10.0.20.10 either will work.

If you cross the 172 network it will be in the trace route. I think it will show L2 in the trace route for your switch.

PS
If you use networks closer together it is much easier to superscope ranges for less ACL statements.
I have a 4x LAN intel PCI-E card added and I will use it for a total of 4 LAN + Transit + WAN

The tracert result:
Code:
tracert 9.9.9.9

Tracing route to dns9.quad9.net [9.9.9.9]
over a maximum of 30 hops:

  1     1 ms     1 ms     1 ms  10.0.20.2
  2    <1 ms    <1 ms    <1 ms  pfSense.intranet [172.26.1.1]
  3     9 ms    11 ms     9 ms  77-56-224-1.dclient.hispeed.ch [77.56.224.1]
  4    11 ms    11 ms     9 ms  217-168-54-9.static.cablecom.ch [217.168.54.9]
  5    14 ms    11 ms    15 ms  ch-nax01a-rc1-ae-55-0.aorta.net [84.116.204.249]
  6    14 ms    10 ms     9 ms  ch-gva01a-ri1-ae-0-0.aorta.net [84.116.133.154]
  7    10 ms    10 ms    15 ms  cixp-packet-clearing-house-as42.cern.ch [192.65.185.182]
  8     9 ms     9 ms    12 ms  dns9.quad9.net [9.9.9.9]

Trace complete.
Obviously it passes by the transit since it is the only interface with a WAN access
tracert from one VLAN to another doesn't pass through pfsense at all
 
  • Like
Reactions: kapone

coxhaus

Active Member
Jul 7, 2020
109
36
28
Looks right. But no static routes makes me wonder whether you are really doing L3.

PS
Back when I tried pfsense I used NTP on pfsense for the switches.
 
Last edited:
  • Like
Reactions: phil9878

phil9878

Member
Apr 2, 2021
39
8
8
Looks right. But no static routes makes me wonder whether you are really doing L3.
The trick I think is the gateway set to the L3 Switch
In pfSense, the interfaces are directly attached, so no need to static routes, just the gateway to the L3 switch and proper firewall rules to allow needed traffic.

tracert from 10.0.10.10 -> 10.0.20.10
Code:
tracert 10.0.20.10

Tracing route to 10.0.20.10 over a maximum of 30 hops

  1     1 ms     1 ms     1 ms  sg350.intranet [10.0.20.2]
  2    <1 ms    <1 ms    <1 ms  10.0.20.10

Trace complete.
InterVLAN traffic doesn't reach pfSense.

I tested RDP from WAN to VLAN 20 and it works also. RDP between VLAN 10 and VLAN 20 works both sides
 
  • Like
Reactions: kapone

coxhaus

Active Member
Jul 7, 2020
109
36
28
I wonder if pfsense is pushing the return traffic back through the trunk port since there are no route statements. Can you disconnect the trunk port and add route statements and ping out from a client? I don't care if DHCP is broken.

Never mind. You can't add route statements with DHCP defined with those networks.
It is routing from the switch so it is returning right.
 
  • Like
Reactions: phil9878