Layer 3 Switch w/ PFSense

[ViciousXUSMC]

Playing with that Aruba S2500 figured I would try to create some VLANs so that I can create a VLAN with MTU 9000 on my 10gb network and a MTU 1500 on my standard 1gb network and let the fast switch handle the inter-vlan routing and the packet segmentation.

It too me a bit to get that working but I did.

One issue I ran into was the PC firewall needed a rule for the other network segment for the ping to work, and I needed the default gateway for each machine to be the Switches layer 3 VLAN IP and let the switches default route be the PFSense IP.

I have not messed with MTU yet, just one thing at a time.

So I have
VLAN1 192.168.1.0/24
VLAN1 Switch IP 192.168.1.234
VLAN10 192.168.10.0/24
VLAN10 Switch IP 192.168.10.234
PFSense LAN IP 192.168.1.1
Uplink from the S2500 to PFSense is a Trunk and both VLANS are setup to be tagged from the access ports.

PC1
192.168.1.4/24
GW 192.168.1.234

PC2
192.168.10.5/24
GW 192.168.10.234

I can ping from one PC to another across the vlans, so I have the inter-vlan routing working.
From VLAN1 I can reach PFSense and the internet.

VLAN10 however cant ping the PFSense interface or reach the internet.

I have been messing with it for a few hours. Surely my traffic is getting there, it just cant get back.
I think 99% of the articles are focused on using a trunk port and using PFSense to do the routing, so you would create the VLAN interfaces, and then assign them an IP and then assign them to your LAN interface.

That means what ever my gateway for each PC is now (the switches VLAN IP) would change to be PFSenses VLAN IP.

That would not be what I want to do since I am sure the Layer 3 switch will be much faster for handling those jumbo frame fragmentations.

I tried to create a LAN Gateway, I tried a static route that said send 192.168.10.0/24 to LAN
I tried doing part of the above with creating VLAN interfaces
I created the outboud NAT rule for 192.168.10.0/24
I created an allow all Firewall Rule for 192.168.10.0/24

I feel like I am close, but just missing something, or the right combination of somethings.

I wonder who here has done this before and knows the magic fix?

I think IamSpartacus has or was going to do it a long time ago - https://forums.servethehome.com/index.php?threads/move-routing-off-pfsense-to-l3-switch.9608/

 

[StammesOpfer]

I am doing this with pfSense and a Brocade ICX6610

You are doing this in a non-standard way however it should work and your static route should be:
Destination Network = 192.168.10.0/24
LAN Gateway = 192.168.1.234

aka any traffic from pfSense destined for 192.168.10.0/24 should be sent to the switch interface on vlan1 for it to route to vlan10

That gives you the proper return route assuming that is the problem. If that is how it is setup then some more troubleshooting is probably in order.


In a more normal network layout your external router would not be in a user vlan but in a separate one that only the switch and pfsense is connected to then your 2 users vlans would route to pfsense and again out to the internet.

More like this:

     Internet
        |
      pfSense
        |-----------vlan A
      Switch
     /     \
vlan b      vlan c
1gbe         10gbe

DHCP can become a problem with the above situation if you are trying to use pfSense for it since it doesn't seem to support creating DHCP ranges for networks that are not directly connected (using the switch as a relay/helper). If I am wrong about this I would love to know so I can go back to pfSense for me DHCP server.

 

[ViciousXUSMC]

I was already thinking of doing a 3rd vlan for the more standard layout, so that its a point to point link. Just was doing what I could today to play around without interrupting any traffic on the network.

I see what I did wrong I for some reason though the LAN gateway had to be the actual IP of my LAN interface (192.168.1.1) I had no idea it was ok to create one with the IP of my VLAN on the switch.

If you use what is in Cisco a IP Helper I think you should be able to pass the DHCP.

So far the S2500 is not as easy to configure as the Cisco stuff I am used to. I hate having to create a switching-profile and apply it to a port instead of just configuring a port directly for example.

So if I do stand up that 3rd VLAN what would the rest of the config look like at that point?
I assume the switches default gateway (route of last resort) would still be the PFSense LAN IP, but what would PFSense look like for the return traffic (Two static routes and two LAN gateways?)

 

[StammesOpfer]

Was the static route set correctly?

Yeah the problem with the DHCP is that pfSense only lets you configure DHCP ranges for networks that are directly connected. For now my Windows Server is doing DHCP for me with ip-helper.

 

[ViciousXUSMC]

Isn't that what the DHCP Relay is for on PFSense? pfSense ip helper [DHCP relay] with multiple subnets : homelab

As for my problem, not sure my laptop is dead and its time to turn in so will try some more tomorrow, but I am really sure your correct. It didn't make sense to me how it could route back and it would not let me create a true static route where I could simply say Network X to Next Hop Y like I am used to with Cisco devices. It only let me choose a "gateway" and I didn't know I could create a gateway with any IP I wanted I thought it had to match an actual IP used on PFSense.

 

[StammesOpfer]

That link gets pfSense to act as the helper. I would like to use my switch as the helper and pfSense as the Server.

Yeah sounds like you have it now. You just have to add a gateway for x.x.1.234, then set up the static route as indicated.

Good luck!

 

[ViciousXUSMC]

I am not willing to give up my static DHCP leases setup in PFSense so I think I'll stick with my "odd" architecture of having my main VLAN on the same subnet as PFSense so it can do DHCP, and just have that server vlan separate. I always put static addresses manually on the servers anyways so loss of DHCP in that subnet is not a big deal, and if somebody somehow managed to get a machine into that subnet I would not want them to get a IP anyways.

I think it's going to work out really well.

Now with the Switch doing the routing and no actual vlan interfaces on PFSense I shouldn't even need a trunk port I think? I'll have to test that in more detail after I get the return routing working.

 

[kapone]

With a layer 3 switch, the general recommendation is to either let the switch do DHCP duties (it can), or have a dedicated DHCP server that can handle VLANs (pfSense cannot, at this time...).

With a layer 3 switch, I prefer to let pfSense be a pure firewall appliance. Have a dedicated "transit" VLAN in your layer 3 switch connected to pfSense. As an e.g.

- You've created a VLAN on your layer 3 switch called TRANSIT, added a single port to it, enabled the virtual interface on it, and the IP is set to 192.168.2.2/xx
- The physical port from above is connected to a port on your pfSense box on let's say the OPT1 interface.
- In pfSense this interface (OPT1) is set to a static IP = 192.168.2.1/xx

At this point, from your console on the switch, you should be able to ping 192.168.2.1 and get a response back. The switch is talking to your pfSense box, but your pfSense box only understands traffic on the 192.168.2.x network.

Now, in pfSense you create a Gateway (System-->Routing-->Gateways). In the settings for the gateway:
- Interface should be OPT1
- Address Family - whatever, v4/v6
- Name - Something meaningful like ICX6610-1 etc
- The Gateway IP address should be what the layer 3 switch TRANSIT IP is. In my example, that would be 192.168.2.2

Now, we have a gateway defined, but pfSense still doesn't know what type of traffic to expect/handle. So, go into System-->Routing-->Static Routes and define one or more static routes. As an e.g.

- Click Add
- The Destination Network should be one or more of the VLAN IP ranges on the layer 3 switch. For e.g. 192.168.20.0/24 (The zero is important...)
- The Gateway should be set to the gateway we just defined, as an e.g. ICX6610-1.
- Give it some meaningful description.
- Save.

At this point, the pfSense box, knows where the 192.168.20.x/xx traffic is coming from and how to talk back to it. But wait...things are probably not working just yet.

That's because your layer 3 switch is not configured with a "default" route that is pointing to the pfSense box. So, add a default route of 0.0.0.0/xx and point it to 192.168.2.1 which is the pfSense box.

At this point, all your VLANS on the layer 3 switch should be able to talk to the switch, the pfSense box, the internet (if it's set up in pfSense) and vice versa. What's not working is DHCP and DNS...To solve that, there's many different strategies, and I'm not going to go into the details of them. In my case, I use a dedicated Windows server for both DHCP/DNS and in the layer 3 switch, for EACH virtual interface, an "ip-helper" is setup to point to this Windows box (There's more configuration to be done for DNS to work efficiently/correctly, but that's a different topic).

What this does, is that all routing is local to the layer 3 switch, and only external networks are routed to the pfSense box. All DHCP is handled at the switch level/dedicated server and not by pfSense. If you've setup DNS correctly, all DNS queries by any clients on any of the VLANS, go to the local DNS server and if the entry is not in the DNS cache, only then does it go outside the network.

My .02

 

[StammesOpfer]

I am not willing to give up my static DHCP leases setup in PFSense so I think I'll stick with my "odd" architecture of having my main VLAN on the same subnet as PFSense so it can do DHCP, and just have that server vlan separate. I always put static addresses manually on the servers anyways so loss of DHCP in that subnet is not a big deal, and if somebody somehow managed to get a machine into that subnet I would not want them to get a IP anyways.

I think it's going to work out really well.

Now with the Switch doing the routing and no actual vlan interfaces on PFSense I shouldn't even need a trunk port I think? I'll have to test that in more detail after I get the return routing working.

That works and as long as it fits your needs I don't think there is too much wrong with that. You are correct no trunk required.

 

[ViciousXUSMC]

Props to both responses. I definitely was just one step away frmo having it working, needed that LAN Gateway set to the IP of the switch on the same subnet. In Cisco you can do routing not just by next IP but by interface, so when I was assigning my gateway the IP of 192.168.1.1 I figured it was just saying "your next hop is out this interface" but it obviously does not work that way with PFSense.

I am surprised it has DHCP and DNS restrictions. I really do like to manage my DNS from PFSense so i can give all my MAC addresses an alias and set them a static lease so I do not think I will want to convert that over to the switch. I will atleast take a look at the switches options however and see.

 

[fohdeesha]

dns should work fine, it just has an ancient implementation of DHCP that refuses to hand out leases to non-directly connected subnets (which is how an l3 switch should be set up with a firewall, using a dedicated transit link like the picture below). I and several others have been begging pfsense and related forks to fix this for a while, but no response. Most of us just moved to running isc-dhcp in a VM or similar and have it serve all the seperate vlans you want.

Feature: DHCP server able to handle non-interface configured subnets · Issue #1105 · opnsense/plugins

https://i.imgur.com/hl7UPjS.png

 

[ViciousXUSMC]

Well the PFSense gateway was the issue, only took a few seconds to fix and setup the static route. Got me up and going.
I then needed to go into DNS Access Lists, it seems once you have multiple subnets it becomes a requirement as my DNS requests from VLAN10 were denied until I explicitly added them in the ACL.

Then the next challenge, it was time to take on Jumbo Frames!

I took my laptop connected to VLAN10, took its port that it was connected to and set the MTU to 9000 and later tried the max (9216 or something) I also set the network card to 9000 MTU on the OS.

I was using a article how to test jumbo frames and it said to use Ping -f -l 9000 x.x.x.x I could ping normal but this failed... no wonder that -f flag is do not fragment... Doh!

I perf speed test remained the same going from my 10gb to 1gb network with the larger MTU so it looks good.

To my knowledge as long as I am on different layer 2 domains (VLAN 10 for MTU9000 and VLAN 1 for MTU1500) I should have no issues.

It is only having MTU mixed on the same subnet that can cause problems.

I am almost there, just need to go configure all my VM's to MTU9000 and change the IP's to be in VLAN10. The one strange thing is the VLAN interface on this switch only has a max MTU of 1570.

Edit: All Done!
Some devices I'll have to go change the gateway, others need to renew the DHCP lease to get it.
Was a pain in the ars when I changed my access port for the ESXi host to the new vlan, it was doing some strange dance where it would not let me set a static IP, and when I had to use the IPMI remote console to get in and fix things it put my management interface on the wrong uplink/vswitch.

End of the night though, its all working well. I get about 9.8gb/s now in my Iperf3 Testing, not a big improvement over 9gb but it's something and less CPU overhead for the VM's now as jumbo packets need less cpu cycles to process on a paravirtulized adapter.

 

[tommybackeast]

Was the static route set correctly?

Yeah the problem with the DHCP is that pfSense only lets you configure DHCP ranges for networks that are directly connected. For now my Windows Server is doing DHCP for me with ip-helper.

reading old posts trying to learn how to configure my 'new' setup of pfsense router + Brocade 7250 +10GB Network +VLANS +l3 and saw your 1yr old comment above.

Can you explain in easy terms your statement of "pfSense only lets you configure DHCP ranges for networks that are directly connected"

Does current version pfSense still have a "bad" DHCP server? or is their DHCP Server now ok for basic home users like myself but who still want VLANS incl 10GB?

 

[StammesOpfer]

So basically if you want pfSense to handle DHCP for an IP range pfSense must have an interface with an IP configured in that subnet. If the only way for pfSense to get to a subnet is via a L3 route handled by another device you can not setup a DHCP range for that subnet. It is just an interface limitation.

What this means is if you have your L3 switch handling the routing then pfSense probably isn't the right way to do DHCP.

 

[kapone]

So basically if you want pfSense to handle DHCP for an IP range pfSense must have an interface with an IP configured in that subnet. If the only way for pfSense to get to a subnet is via a L3 route handled by another device you can not setup a DHCP range for that subnet. It is just an interface limitation.

What this means is if you have your L3 switch handling the routing then pfSense probably isn't the right way to do DHCP.

Not true.

Take an L3 switch, configure all your subnets with gateways on the switch itself, with no presumption about how DHCP/DNS will be handled.

Then lookup "Router-on-a-stick".

 

[StammesOpfer]

Not true.

Take an L3 switch, configure all your subnets with gateways on the switch itself, with no presumption about how DHCP/DNS will be handled.

Then lookup "Router-on-a-stick".

I believe what you are suggesting is using a trunk to pfSense and creating a vlan interface for each subnet in pfSense. Which yes that would work (set the default gateway in DHCP to be the L3 Switch IP) except return traffic from outside your network would be routed via pfsense not the L3 switch. If you read the same sentence before the underlined and bold I say IF the only way to reach it is via a separate L3 device then it won't work.

 

[gregsachs]

I'll take a shot at this, I just was messing with this...
Approach a:
All vlans are trunked to the firewall device, which handles all routing between vlans and external world. In this example, the firewall sits on all vlans and can apply policies between them and the outside world.

Approach B: Vlans terminate at the switch, which handles all routing between vlans.
An exit vlan exists between the switch and the firewall device, which then only routes to external addresses; but it must have static routes configured for the internal vlans that point to the switch IP address.

 

[kapone]

I believe what you are suggesting is using a trunk to pfSense and creating a vlan interface for each subnet in pfSense. Which yes that would work (set the default gateway in DHCP to be the L3 Switch IP) except return traffic from outside your network would be routed via pfsense not the L3 switch. If you read the same sentence before the underlined and bold I say IF the only way to reach it is via a separate L3 device then it won't work.

Got it. My apologies if I misunderstood your comment.

 

[tommybackeast]

I'll take a shot at this, I just was messing with this...
Approach a:
All vlans are trunked to the firewall device, which handles all routing between vlans and external world. In this example, the firewall sits on all vlans and can apply policies between them and the outside world.

Approach B: Vlans terminate at the switch, which handles all routing between vlans.
An exit vlan exists between the switch and the firewall device, which then only routes to external addresses; but it must have static routes configured for the internal vlans that point to the switch IP address.

I had asked the above 'how to' question that started this mini-thread. Basically, I have the hardware now including Brocade 7250, dedicated 1GB pfsense box; but I have 10GB "stuff" like Synology NAS units, and 10GB on Dell Server ESXi VM; and 10GB PC computers.

My end goal : being able to have my 10GB boxes send data to/from each other, a handful of VLANs (something I've never used before), setup OpenVPN so when i'm out on cellphone I can access just one of the Synology NAS boxes. I like using MAC addresses to do DHCP IP Reservations.

many dumb questions shall follow : please restrict your laughter until the end of my post

1. since I have 10GB stuff but my pfsense box only has 1GB NICs; *IF* i made the VLANS on the pfsense box, and did DHCP Server on the pfsense box; can I do intra-VLAN routing on the Brocade 7250 L3 Switch (which has 10GB ports)

2. Must the device setting up the VLANs be the same device controlling intra-VLAN access?

3. Please compare running DHCP Server on pfsense vs DHCP Server on Brocade 7250 (for basic home use).

4. Brocade 7250 WebGui : can the WebGUI be used for DHCP Server? VLAN Creation? intra-VLAN access control?

I guess guys, I have the hardware, I just honestly don't know how to start for until I found this thread, I just assumed pfsense would do the DHCP Server, make me a new 10.10.10.1 Default Gateway, I would make some VLANS and somehow figure out how to allow computerA in VLAN-01 to be able to read files on NAS in VLAN-02....

5. IoT VLAN: my thought was anything in here only has LAN access but no WAN access. So can I put my Laser Printer (which has a RJ45 Network ) in IoT-VLAN; but then how do I config my Workstation PC to have access to it to actually print out ? or should Networked Laser Printer go in same VLAN as my Workstation PC?

Google search on the basic topic of VLANs obviously shows millions of hits - at this point , I have vague understanding of them but am now trying to understand what VLANS i should make up, and where to put my devices. I fully expected to screw shit up while doing this large learning project, lol; but didn't expect to get stuck before I even started

Now after reading more threads here : I don't even know if pfsense or Broade should be the DHCP Server, should I make the VLANS on pfsense or Brocade [ok, now you can laugh for you were holding it in long enough)

 

[tommybackeast]

I'll take a shot at this, I just was messing with this...
Approach a:
All vlans are trunked to the firewall device, which handles all routing between vlans and external world. In this example, the firewall sits on all vlans and can apply policies between them and the outside world.

Approach B: Vlans terminate at the switch, which handles all routing between vlans.
An exit vlan exists between the switch and the firewall device, which then only routes to external addresses; but it must have static routes configured for the internal vlans that point to the switch IP address.

Upon re-reading your post here, since I have 10GB devices on my LAN (NAS, Dell Server ESXi w VMs, two PC Computers) - and have not yet set anything up :

Would best practices for me be VLAN setup and intra-VLAN access to be done on the Brocade 7250? and then the PfSense box is acting only as Router, plus some pfsense packages like pfblockerng, Darkstat, Suricata. In this scenerio, where does DHCP Server run? pfsense or Brocade?