Cable modem + L3 switch + pfsense router-on-a-stick

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

Fallen Kell

Member
Mar 10, 2020
57
23
8
So I have run into a strange issue that I have not being able to figure out. I have been slowly changing over my network from using wifi router to use a pfsense router. I have been slowly configuring all the VLANs I use and have been testing them out to ensure that the L3 switch (a Brocade ICX 6610) is performing all the inter-VLAN routing.

I have 2 VLANs setup simply for the pfsense system, VLAN 99, which is configured as the WAN VLAN and VLAN 98, which is effectively the transit VLAN, on port 1/2/6 set with tagged with VLAN 98, and VLAN 99. I have my pfsense system using IP 192.168.98.2/24 on the VLAN 98, and DHCP for VLAN 99, and a virtual routing interface on the switch of 192.168.98.1/24 (I should probably just make these a /30, since they will be the only 2 IP addresses used).

I have no problem reaching the pfsense from my other VLANs. I have configured static routes on the pfsense system to reach back to the other VLANs through 192.168.98.1 and everything communicates fine. The problem is that once I move the network cable that comes from the cable modem and currently goes into the WAN port on my wifi router over to port 1/1/24 on the brocade which is configured for untagged VLAN 99, I do not ever get an IP address on the pfsense system for the WAN side.

Could the cable modem be effectively locked based on MAC address of my previous router that connected to it? I don't know why they would do something like that since anytime someone replaces their own router they would then have issues (like I am) connecting. Is there something I might have missed?
 
  • Like
Reactions: sevlor

j_h_o

Active Member
Apr 21, 2015
644
179
43
California, US
That's not unusual. Release the DHCP lease on your router first before transferring the cable. Or try power cycle the modem after you transfer the cable. Or spoof the MAC of your router with your pfSense firewall.
 

kapone

Well-Known Member
May 23, 2015
1,095
642
113
There's most likely a few steps missing. I had posted the following a while back, but I'll try to tailor it to your environment.

For your WAN termination on the 6610:
- Create a VLAN - In your case you created VLAN 99 it seems.
- Add the physical port (where the WAN cable will be plugged in) to this VLAN as untagged. - In your case, that would be 1/1/24.
- Add the physical port that will connected to your pfSense machine as... Now, this is where this gets tricky, because there are so many permutations. In your case your port is 1/2/6. But tagged is just fine.
- There should be NO virtual interface on this VLAN, no IP address etc etc. This VLAN is solely for WAN termination, and the only device that can access it is, what's connected to 1/2/6 (with a VLAN of 99, which would be pfSense only). But it doesn't need to be routable, that's what pfSense is doing as the NAT function, NATing between the rest of the network and this "network. Think of it as connecting the WAN cable directly to the pfSense box.

For your TRANSIT link on the 6610:
- Create a VLAN - In your case you created VLAN 98 it seems.
- Add the physical port that will connected to your pfSense machine as... Now, this is where this gets tricky, because there are so many permutations. In your case your port is 1/2/6. But tagged is just fine.
- Create a virtual interface on it, and set the IP to - in your case - 192.168.2.1/xx
- The physical port from above is connected to a port on your pfSense box on some interface.

- Create VLANs in pfSense for WAN (VLAN 99) and TRANSIT (VLAN 98) on the physical interface that is connected to 1/2/6. pfSense will create interfaces for these two VLANS. All configuration must be done on these two interfaces, you're not touching the underlying physical interface.

For the WAN interface:
- Enable Interface = true.
- IPv4 Configuration = DHCP
- IPv6 Configuration = None (I'm not getting into configuring IPv6 for the WAN...completely outside the scope of this)
- Rest defaults.

For the TRANSIT interface:
- Enable Interface = true.
- IPv4 Configuration = static, IPv4 address = 192.168.98.2/24
- IPv6 Configuration = None (I'm not getting into configuring IPv6 for the LAN...completely outside the scope of this)
- IPv4 upstream gateway = None.
- Rest defaults.

Your WAN interface SHOULD get an IP address from your ISP at this point. If it's not, something's wrong. - Sometimes you have to reboot the pfsense box AND the switch (after saving the config) to get this to work.

pfSense config:
- Create a firewall rule in pfSense on the TRANSIT interface to allow ICMP ping from any source to any destination.

At this point, from your console on the switch, you should be able to ping 192.168.2.2 and get a response back. The switch is talking to your pfSense box, but your pfSense box only understands traffic on the 192.168.2.x network.

Now, in pfSense create a Gateway (System-->Routing-->Gateways). In the settings for the gateway:
- Interface should be TRANSIT (VLAN 98)
- Address Family - whatever, v4 (Not getting into IPv6)
- Name - Something meaningful like ICX6610-1 etc
- The Gateway IP address should be what the layer 3 switch TRANSIT IP is. In your example, that would be 192.168.2.1

Now, we have a gateway defined, but pfSense still doesn't know what type of traffic to expect/handle. So, go into System-->Routing-->Static Routes and define one or more static routes. As an e.g.

- Click Add
- The Destination Network should be one or more of the VLAN IP ranges on the layer 3 switch. For e.g. 192.168.20.0/24 (The zero is important...)
- The Gateway should be set to the gateway we just defined, as an e.g. ICX6610-1.
- Give it some meaningful description.
- Save.

At this point, the pfSense box, knows where the 192.168.20.x/xx traffic is coming from and how to talk back to it. But wait...things are probably not working just yet.

That's because your layer 3 switch is not configured with a "default" route that is pointing to the pfSense box. So, add a default route of 0.0.0.0/0 and point it to 192.168.2.2 which is the pfSense box.

Now, the rest of pfSense config (not complete...too many permutations :))
- Create Aliases for each subnet on the other end of that TRANSIT link. Aliases just make life easier. As an e.g. create an alias for 192.168.20.0/24 and name it "testVLAN"
- Create firewall rules on the TRANSIT interface where the source is "single host or alias" and the value is testVLAN (in this example). The rules depend on how you are configuring your network. For starting out and testing, just do any to any to see if things are working.

At this point, all your VLANS on the layer 3 switch should be able to talk to the switch, the pfSense box, the internet (if it's set up in pfSense) and vice versa.

What's not working is DHCP and DNS...To solve that, there's many different strategies, and I'm not going to go into the details of them.

My .02
 
Last edited:

Fallen Kell

Member
Mar 10, 2020
57
23
8
Thanks for some of the suggestions. I basically had everything setup correct. The main issue seems to be a combination of the cable modem I have and pfsense. The cable modem does some strange things in that it will first configure itself as a 192.168.100.1 and give out a DHCP address of 192.168.100.10 to the pfsense system and then after some tests, re-assign itself a different public IP address after confirming something (my guess would be MAC address since once I cloned the MAC used on my other router it re-assigned the IP address). Because of this, I had to do some interesting things, like ditch my internal network route since it was based on 192.168.0.0/16 which overlapped with what this was attempting to use on the external, and I also had to make sure pfsense allowed private nets on the WAN link.

So, in other words a lot of banging of my head over crap that is just bad design...

I am taking a break for a bit, and calling it a weekend on this for now. I got it to the point where I can at least ping out from the pfsense system and do DNS lookups on external servers. I need to now mess with the all my internal routes because of the idiotic 192.168.100.0 net being used by the cable modem temporarily every time it boots.
 
  • Like
Reactions: fphillipeck

Fallen Kell

Member
Mar 10, 2020
57
23
8
So I am getting closer to having this all working. I still have some things not working, and I am not exactly sure why.

Quick overview:
ICX6610 with VLANs 1, 4, 6, 8, 10, 98, and 99, with virtual router interfaces on all but 99, with default route 0.0.0.0/0 pointing to the pfsense system

Netgear R9000 (essentially in AP mode, no routing) with VLANs 1, 8, 10 (running DD-WRT, may add all VLANs depending on if I continue using for DHCP) with DHCP assigned default route pointing to appropriate IP for the virtual router interface depending on VLAN (using dhcp-options)

Pfsense with VLANs 98 and 99 (99 is WAN, 98 is LAN), with a gateway and static routes configured for all the internal networks


Here is the issue I currently have, from VLAN 1, I have a client that can properly ping the router interface for the VLAN on the ICX6610 switch (and it is set as the default route), it can ping the pfsense LAN IP, and can ping external internet DNS servers (by IP address). It can also perform DNS lookups, getting responses from the external DNS servers (obviously routed through the pfsense system). However, when I attempt to go to a website that is external, it just times out. I am guessing this is something pfsense is blocking, but I did not see anything in the pfsense tutorials about needing to open additional ports/protocols for initiating a connection from the LAN side to the WAN side (i.e. I thought outgoing was allowed by default). Did I miss something?
 
Last edited:

kapone

Well-Known Member
May 23, 2015
1,095
642
113
Post a screenshot of your firewall rules section of pfSense. I suspect you've allowed DNS (53) on the TRANSIT interface for "any", but you have to setup additional rules for the subnets on the other side of the TRANSIT connection.

That's kinda what I was hinting at with the Aliases.

As an ex, you'd need any to any rules on the TRANSIT interface (VLAN 98) for 192.168.1.0/24, 192.168.4.0/24, 192.168.6.0/24 etc etc. These are VLANS/subnets on the switch side, not in pfSense, when their traffic comes in over the TRANSIT interface, pfSense doesn't know what to do with it. Your static route(s) back to the switch will also need to include these subnets.
 

Fallen Kell

Member
Mar 10, 2020
57
23
8
That might make sense. I will try and get a screenshot the next time I am working on it. I think the only rule I added was for ICMP on LAN to anywhere (and I think I limited it not by the network LAN, but by the device/interface).
 

Fallen Kell

Member
Mar 10, 2020
57
23
8
I found the problem. It was a netmask issue on pfsense on it's LAN port. I must have missed setting it during the initial setup. Once I changed it, I was able to connect out. Now I just have my other issues with my network (i.e. getting my R9000 properly setup and working... so far the only part that seems to work correctly is DHCP and DHCP settings to wired devices). But this is something I will have to just work through configuring. The biggest thing is that my wired network all seems to be working.
 
  • Like
Reactions: kapone