Layer 3 Switch w/ PFSense

Discussion in 'Networking' started by ViciousXUSMC, Jan 17, 2019.

  1. ViciousXUSMC

    ViciousXUSMC Active Member

    Joined:
    Nov 27, 2016
    Messages:
    160
    Likes Received:
    63
    Playing with that Aruba S2500 figured I would try to create some VLANs so that I can create a VLAN with MTU 9000 on my 10gb network and a MTU 1500 on my standard 1gb network and let the fast switch handle the inter-vlan routing and the packet segmentation.

    It too me a bit to get that working but I did.

    One issue I ran into was the PC firewall needed a rule for the other network segment for the ping to work, and I needed the default gateway for each machine to be the Switches layer 3 VLAN IP and let the switches default route be the PFSense IP.

    I have not messed with MTU yet, just one thing at a time.

    So I have
    VLAN1 192.168.1.0/24
    VLAN1 Switch IP 192.168.1.234
    VLAN10 192.168.10.0/24
    VLAN10 Switch IP 192.168.10.234
    PFSense LAN IP 192.168.1.1
    Uplink from the S2500 to PFSense is a Trunk and both VLANS are setup to be tagged from the access ports.

    PC1
    192.168.1.4/24
    GW 192.168.1.234

    PC2
    192.168.10.5/24
    GW 192.168.10.234

    I can ping from one PC to another across the vlans, so I have the inter-vlan routing working.
    From VLAN1 I can reach PFSense and the internet.

    VLAN10 however cant ping the PFSense interface or reach the internet.

    I have been messing with it for a few hours. Surely my traffic is getting there, it just cant get back.
    I think 99% of the articles are focused on using a trunk port and using PFSense to do the routing, so you would create the VLAN interfaces, and then assign them an IP and then assign them to your LAN interface.

    That means what ever my gateway for each PC is now (the switches VLAN IP) would change to be PFSenses VLAN IP.

    That would not be what I want to do since I am sure the Layer 3 switch will be much faster for handling those jumbo frame fragmentations.

    I tried to create a LAN Gateway, I tried a static route that said send 192.168.10.0/24 to LAN
    I tried doing part of the above with creating VLAN interfaces
    I created the outboud NAT rule for 192.168.10.0/24
    I created an allow all Firewall Rule for 192.168.10.0/24

    I feel like I am close, but just missing something, or the right combination of somethings.

    I wonder who here has done this before and knows the magic fix?

    I think IamSpartacus has or was going to do it a long time ago :) - https://forums.servethehome.com/index.php?threads/move-routing-off-pfsense-to-l3-switch.9608/
     
    #1
  2. StammesOpfer

    StammesOpfer Active Member

    Joined:
    Mar 15, 2016
    Messages:
    378
    Likes Received:
    122
    I am doing this with pfSense and a Brocade ICX6610

    You are doing this in a non-standard way however it should work and your static route should be:
    Destination Network = 192.168.10.0/24
    LAN Gateway = 192.168.1.234

    aka any traffic from pfSense destined for 192.168.10.0/24 should be sent to the switch interface on vlan1 for it to route to vlan10

    That gives you the proper return route assuming that is the problem. If that is how it is setup then some more troubleshooting is probably in order.


    In a more normal network layout your external router would not be in a user vlan but in a separate one that only the switch and pfsense is connected to then your 2 users vlans would route to pfsense and again out to the internet.

    More like this:
    Code:
         Internet
            |
          pfSense
            |-----------vlan A
          Switch
         /     \
    vlan b      vlan c
    1gbe         10gbe
    
    DHCP can become a problem with the above situation if you are trying to use pfSense for it since it doesn't seem to support creating DHCP ranges for networks that are not directly connected (using the switch as a relay/helper). If I am wrong about this I would love to know so I can go back to pfSense for me DHCP server.
     
    #2
    Last edited: Jan 17, 2019
    mathiastro likes this.
  3. ViciousXUSMC

    ViciousXUSMC Active Member

    Joined:
    Nov 27, 2016
    Messages:
    160
    Likes Received:
    63
    I was already thinking of doing a 3rd vlan for the more standard layout, so that its a point to point link. Just was doing what I could today to play around without interrupting any traffic on the network.

    I see what I did wrong I for some reason though the LAN gateway had to be the actual IP of my LAN interface (192.168.1.1) I had no idea it was ok to create one with the IP of my VLAN on the switch.

    If you use what is in Cisco a IP Helper I think you should be able to pass the DHCP.

    So far the S2500 is not as easy to configure as the Cisco stuff I am used to. I hate having to create a switching-profile and apply it to a port instead of just configuring a port directly for example.

    So if I do stand up that 3rd VLAN what would the rest of the config look like at that point?
    I assume the switches default gateway (route of last resort) would still be the PFSense LAN IP, but what would PFSense look like for the return traffic (Two static routes and two LAN gateways?)
     
    #3
  4. StammesOpfer

    StammesOpfer Active Member

    Joined:
    Mar 15, 2016
    Messages:
    378
    Likes Received:
    122
    Was the static route set correctly?

    Yeah the problem with the DHCP is that pfSense only lets you configure DHCP ranges for networks that are directly connected. For now my Windows Server is doing DHCP for me with ip-helper.
     
    #4
    Last edited: Jan 17, 2019
  5. ViciousXUSMC

    ViciousXUSMC Active Member

    Joined:
    Nov 27, 2016
    Messages:
    160
    Likes Received:
    63
    Isn't that what the DHCP Relay is for on PFSense? pfSense ip helper [DHCP relay] with multiple subnets : homelab

    As for my problem, not sure my laptop is dead and its time to turn in so will try some more tomorrow, but I am really sure your correct. It didn't make sense to me how it could route back and it would not let me create a true static route where I could simply say Network X to Next Hop Y like I am used to with Cisco devices. It only let me choose a "gateway" and I didn't know I could create a gateway with any IP I wanted I thought it had to match an actual IP used on PFSense.
     
    #5
  6. StammesOpfer

    StammesOpfer Active Member

    Joined:
    Mar 15, 2016
    Messages:
    378
    Likes Received:
    122
    That link gets pfSense to act as the helper. I would like to use my switch as the helper and pfSense as the Server.

    Yeah sounds like you have it now. You just have to add a gateway for x.x.1.234, then set up the static route as indicated.

    Good luck!
     
    #6
  7. ViciousXUSMC

    ViciousXUSMC Active Member

    Joined:
    Nov 27, 2016
    Messages:
    160
    Likes Received:
    63
    I am not willing to give up my static DHCP leases setup in PFSense so I think I'll stick with my "odd" architecture of having my main VLAN on the same subnet as PFSense so it can do DHCP, and just have that server vlan separate. I always put static addresses manually on the servers anyways so loss of DHCP in that subnet is not a big deal, and if somebody somehow managed to get a machine into that subnet I would not want them to get a IP anyways.

    I think it's going to work out really well.

    Now with the Switch doing the routing and no actual vlan interfaces on PFSense I shouldn't even need a trunk port I think? I'll have to test that in more detail after I get the return routing working.
     
    #7
  8. kapone

    kapone Active Member

    Joined:
    May 23, 2015
    Messages:
    616
    Likes Received:
    246
    With a layer 3 switch, the general recommendation is to either let the switch do DHCP duties (it can), or have a dedicated DHCP server that can handle VLANs (pfSense cannot, at this time...).

    With a layer 3 switch, I prefer to let pfSense be a pure firewall appliance. Have a dedicated "transit" VLAN in your layer 3 switch connected to pfSense. As an e.g.

    - You've created a VLAN on your layer 3 switch called TRANSIT, added a single port to it, enabled the virtual interface on it, and the IP is set to 192.168.2.2/xx
    - The physical port from above is connected to a port on your pfSense box on let's say the OPT1 interface.
    - In pfSense this interface (OPT1) is set to a static IP = 192.168.2.1/xx

    At this point, from your console on the switch, you should be able to ping 192.168.2.1 and get a response back. The switch is talking to your pfSense box, but your pfSense box only understands traffic on the 192.168.2.x network.

    Now, in pfSense you create a Gateway (System-->Routing-->Gateways). In the settings for the gateway:
    - Interface should be OPT1
    - Address Family - whatever, v4/v6
    - Name - Something meaningful like ICX6610-1 etc
    - The Gateway IP address should be what the layer 3 switch TRANSIT IP is. In my example, that would be 192.168.2.2

    Now, we have a gateway defined, but pfSense still doesn't know what type of traffic to expect/handle. So, go into System-->Routing-->Static Routes and define one or more static routes. As an e.g.

    - Click Add
    - The Destination Network should be one or more of the VLAN IP ranges on the layer 3 switch. For e.g. 192.168.20.0/24 (The zero is important...)
    - The Gateway should be set to the gateway we just defined, as an e.g. ICX6610-1.
    - Give it some meaningful description.
    - Save.

    At this point, the pfSense box, knows where the 192.168.20.x/xx traffic is coming from and how to talk back to it. But wait...things are probably not working just yet.

    That's because your layer 3 switch is not configured with a "default" route that is pointing to the pfSense box. So, add a default route of 0.0.0.0/xx and point it to 192.168.2.1 which is the pfSense box.

    At this point, all your VLANS on the layer 3 switch should be able to talk to the switch, the pfSense box, the internet (if it's set up in pfSense) and vice versa. What's not working is DHCP and DNS...To solve that, there's many different strategies, and I'm not going to go into the details of them. In my case, I use a dedicated Windows server for both DHCP/DNS and in the layer 3 switch, for EACH virtual interface, an "ip-helper" is setup to point to this Windows box (There's more configuration to be done for DNS to work efficiently/correctly, but that's a different topic).

    What this does, is that all routing is local to the layer 3 switch, and only external networks are routed to the pfSense box. All DHCP is handled at the switch level/dedicated server and not by pfSense. If you've setup DNS correctly, all DNS queries by any clients on any of the VLANS, go to the local DNS server and if the entry is not in the DNS cache, only then does it go outside the network.

    My .02
     
    #8
  9. StammesOpfer

    StammesOpfer Active Member

    Joined:
    Mar 15, 2016
    Messages:
    378
    Likes Received:
    122
    That works and as long as it fits your needs I don't think there is too much wrong with that. You are correct no trunk required.
     
    #9
    ViciousXUSMC likes this.
  10. ViciousXUSMC

    ViciousXUSMC Active Member

    Joined:
    Nov 27, 2016
    Messages:
    160
    Likes Received:
    63
    Props to both responses. I definitely was just one step away frmo having it working, needed that LAN Gateway set to the IP of the switch on the same subnet. In Cisco you can do routing not just by next IP but by interface, so when I was assigning my gateway the IP of 192.168.1.1 I figured it was just saying "your next hop is out this interface" but it obviously does not work that way with PFSense.

    I am surprised it has DHCP and DNS restrictions. I really do like to manage my DNS from PFSense so i can give all my MAC addresses an alias and set them a static lease so I do not think I will want to convert that over to the switch. I will atleast take a look at the switches options however and see.
     
    #10
  11. fohdeesha

    fohdeesha Kaini Industries

    Joined:
    Nov 20, 2016
    Messages:
    1,341
    Likes Received:
    1,079
    dns should work fine, it just has an ancient implementation of DHCP that refuses to hand out leases to non-directly connected subnets (which is how an l3 switch should be set up with a firewall, using a dedicated transit link like the picture below). I and several others have been begging pfsense and related forks to fix this for a while, but no response. Most of us just moved to running isc-dhcp in a VM or similar and have it serve all the seperate vlans you want.

    Feature: DHCP server able to handle non-interface configured subnets · Issue #1105 · opnsense/plugins

    https://i.imgur.com/hl7UPjS.png
     
    #11
    StammesOpfer likes this.
  12. ViciousXUSMC

    ViciousXUSMC Active Member

    Joined:
    Nov 27, 2016
    Messages:
    160
    Likes Received:
    63
    Well the PFSense gateway was the issue, only took a few seconds to fix and setup the static route. Got me up and going.
    I then needed to go into DNS Access Lists, it seems once you have multiple subnets it becomes a requirement as my DNS requests from VLAN10 were denied until I explicitly added them in the ACL.

    Then the next challenge, it was time to take on Jumbo Frames!

    I took my laptop connected to VLAN10, took its port that it was connected to and set the MTU to 9000 and later tried the max (9216 or something) I also set the network card to 9000 MTU on the OS.

    I was using a article how to test jumbo frames and it said to use Ping -f -l 9000 x.x.x.x I could ping normal but this failed... no wonder that -f flag is do not fragment... Doh!

    I perf speed test remained the same going from my 10gb to 1gb network with the larger MTU so it looks good.

    To my knowledge as long as I am on different layer 2 domains (VLAN 10 for MTU9000 and VLAN 1 for MTU1500) I should have no issues.

    It is only having MTU mixed on the same subnet that can cause problems.

    I am almost there, just need to go configure all my VM's to MTU9000 and change the IP's to be in VLAN10. The one strange thing is the VLAN interface on this switch only has a max MTU of 1570.

    Edit: All Done!
    Some devices I'll have to go change the gateway, others need to renew the DHCP lease to get it.
    Was a pain in the ars when I changed my access port for the ESXi host to the new vlan, it was doing some strange dance where it would not let me set a static IP, and when I had to use the IPMI remote console to get in and fix things it put my management interface on the wrong uplink/vswitch.

    End of the night though, its all working well. I get about 9.8gb/s now in my Iperf3 Testing, not a big improvement over 9gb but it's something and less CPU overhead for the VM's now as jumbo packets need less cpu cycles to process on a paravirtulized adapter.
     
    #12
    Last edited: Jan 18, 2019
    StammesOpfer likes this.
Similar Threads: Layer Switch
Forum Title Date
Networking Looking for a (somewhat) Power Efficient 10Gbe Layer 3 Switch for a Homelab - Cisco N3K? Oct 21, 2018
Networking Help Deciding Layer 3 switch Jan 16, 2017
Networking Low power layer 3 switch? Dec 13, 2016
Networking VLAN Routing Between Layer 3 and Layer 2 Switch Aug 16, 2016
Networking Router on a Stick -vs- Layer 3 Routing with Switch Jul 21, 2016

Share This Page