OPNSense gateway blocking all traffic?

[phekno]

So, based on a couple posts here, I've managed to configure my ICX-6610 to work with OPNSense...partially.

The Brocade is configured as an L3 switch, with OPNSense as it's gateway, similar to this post (OP me), and this post. Switch is 10.100.1.9 (VLAN 1001, management VLAN), 10.199.199.2 (VLAN 999, transit VLAN), OPNSense LAN is 10.199.199.1, laptop is 10.100.1.40 (VLAN 1001, management VLAN). Switch has a default route of 0.0.0.0/0 to 10.199.199.1.

OPNSense has a LAN interface, hooked to the switch (on e 1/2/2) and WAN interface hooked to my ISP. It also has a gateway on the LAN interface pointing to the switch's transit VLAN IP (10.199.199.2). NAT is set to automatic. Static routes for all of my VLANs have been configured to use the gateway.

From the switch on VLAN 1001 (management VLAN), I'm able to hit OPNSense at 10.199.199.1 (either by ping or HTTP/S). I can also get to the switch console, and in there I'm able to ping OPNSense.

Beyond that, my laptop has NO internet access. I get the feeling that it's a firewall issue, but I don't really have any rules in place (other than what comes out-of-the-box with OPNSense). My understanding is that the rules, by default, are enough to at least grant internet access, so I'm not sure what I'm missing. I can even see DNS queries going OUT of the LAN interface, and the firewall passes them, but I can't get responses.

Any suggestions?

 

[phekno]

So, one thing that was also weird is/was that I was unable to check for firmware upgrades on the OPNSense machine. Every time, it would just timeout and fail.

I did a bit more digging, and in the gateways configuration, noticed that the WAN wasn’t set to be the default gateway. Now, at least, I’m able to do a firmware update, and I was even able to ping 1.1.1.1 from the switch, so progress is being made!

 

[phekno]

Just to be on the safe side, I defaulted OPNSense and "started over".

Just defaulted, I got internet access from LAN -> WAN just fine.

I then set up the gateway and routes for it. I set up the gateway/switch on the OPT1 interface, which has a static IP of 10.199.199.1 (the switch is 10.199.199.2) and I also set up the static routes from OPNSense to the switch for the VLANs. I'm able to ping the OPNSense machine's OPT1 interface address from the switch and I'm able to hit the OPNSense console from the switch, but I still have no wider internet access. I did throw some extra rules on my OPT1 interface similar to what I saw on the LAN interface, but still no dice. If I look at the firewall logs, I can see DNS traffic from my laptop trying to hit 1.1.1.1 (hard-coded for the time being) going out the WAN interface, and it's being allowed, but those requests don't seem to make it back in.

 

[LodeRunner]

Can you post screenshots of the gateway, static routes, OPT1 rules, and your Outbound NAT settings from opnSense? It certainly sounds like you've set it up correctly.

Assuming routes in both directions are correct and the switch is passing traffic in both directions (opnSense can ping a client in the VLAN and a client in the VLAN can ping OPT1) then normally a simple Allow any/any outbound rule on the OPT1 interface should make things work.

What switch are you using? Does the switch have a default route set to the OPT1 IP?

 

[phekno]

Can you post screenshots of the gateway, static routes, OPT1 rules, and your Outbound NAT settings from opnSense? It certainly sounds like you've set it up correctly.

Assuming routes in both directions are correct and the switch is passing traffic in both directions (opnSense can ping a client in the VLAN and a client in the VLAN can ping OPT1) then normally a simple Allow any/any outbound rule on the OPT1 interface should make things work.

What switch are you using? Does the switch have a default route set to the OPT1 IP?

Here's all the stuff you asked for. I did add an any/any outbound rule on OPT1, but that didn't seem to help. You should see that in the screenshot.

Also, just to clarify, I am able to ping machines on other VLANs (e.g. something on VLAN1001 can ping something on VLAN1050), I'm able to ping the firewall from something on a VLAN, the switch can ping things on VLANs, OPNSense can ping things on VLANs, etc. It seems like everything LOCALLY is fine...I just can't get anything out to the internet or back. My soon to be former router/firewall is VyOS based, and IMO they have a sort of...backwards...way of thinking of firewall rules, so I think my brain is tangled up with that...

Gateway config:


Static routes:


OPT1 rules:


Outbound NAT:


I'm using a Brocade ICX6610 for my L3 switch. Here's a screenshot of me having run sho ip route, showing that the default route on the switch is 10.199.199.1 (the OPT1 IP address):


Hope this helps!

Thanks!

 

[LodeRunner]

On the face of it, I don't see anything that looks wrong. And just to be clear, these VLANs have subnets that are not reachable by any other route, correct? The firewall can only reach them via the transit VLAN on OPT1? It almost sounds like an asymmetric routing issue after seeing the configs (traffic leaves one gateway, gets returned by another).

 

[BoredSysadmin]

Configuring InterVLAN Routing with a Layer 3 Switch and pfSense

Recently I was tasked with deploying a Layer 3 managed network switch alongside an existing pfSense firewall appliance for a relatively small network. As a quick bit of a background the network con…

greigmitchell.co.uk

 

[phekno]

On the face of it, I don't see anything that looks wrong. And just to be clear, these VLANs have subnets that are not reachable by any other route, correct? The firewall can only reach them via the transit VLAN on OPT1? It almost sounds like an asymmetric routing issue after seeing the configs (traffic leaves one gateway, gets returned by another).

Yeah, that's kinda what I'm thinking too, I guess? For another data point, I'm actually able to get DNS resolution and pings to/from the internet on the switch itself (i.e. using the switch's ping utility). I think that points at some routing, but I don't know what. I'm also able to see traffic from my laptop connected to VLAN 1001 going out to the internet and coming back (mostly DNS queries) in the firewall log, and they're all being passed, so that seems to indicate that it's a routing issue, I guess. My sho run is below if you want to take a look...

Configuring InterVLAN Routing with a Layer 3 Switch and pfSense

Recently I was tasked with deploying a Layer 3 managed network switch alongside an existing pfSense firewall appliance for a relatively small network. As a quick bit of a background the network con…

greigmitchell.co.uk

Kinda helpful. I want to say I've read this, and actually done most of the stuff here, but I seem to be having some routing issue getting back from firewall to the switch.

I'm sure this is a dumb question, but does the transit interface need to be a ve? In my case, that's not how it's configured...

Here's my sho run:

telnet@ICX6610>sho run
Current configuration:
!
ver 08.0.30uT7f3
!
stack unit 1
  module 1 icx6610-48p-poe-port-management-module
  module 2 icx6610-qsfp-10-port-160g-module
  module 3 icx6610-8-port-10g-dual-mode-module
  no legacy-inline-power
  stack-trunk 1/2/1 to 1/2/2
  stack-trunk 1/2/6 to 1/2/7
!
global-stp
!
!
!
vlan 1 name DEFAULT-VLAN by port
 router-interface ve 1
!
vlan 999 name Transit by port
 untagged ethe 1/2/2
!
vlan 1001 name Management by port
 untagged ethe 1/1/11 ethe 1/1/48
 router-interface ve 1001
 spanning-tree 802-1w
 spanning-tree 802-1w priority 0
!
vlan 1010 name Main by port
 tagged ethe 1/1/1 ethe 1/1/3 ethe 1/1/13 ethe 1/1/23
 router-interface ve 1010
 spanning-tree
!
vlan 1020 name Guest by port
!
vlan 1030 name IoT by port
 tagged ethe 1/1/1 ethe 1/1/3 ethe 1/1/13 ethe 1/1/23
 untagged ethe 1/1/9
 spanning-tree
!
vlan 1040 name Video by port
 untagged ethe 1/1/5 ethe 1/1/15 ethe 1/1/19 ethe 1/1/21
 spanning-tree
!
vlan 1050 name Servers by port
 untagged ethe 1/1/2 ethe 1/1/7 ethe 1/2/1 ethe 1/2/3 to 1/2/10
 router-interface ve 1050
 spanning-tree 802-1w
 spanning-tree 802-1w priority 0
!
!
!
!
!
aaa authentication web-server default local
aaa authentication login default local
enable telnet authentication
hostname ICX6610
ip dhcp-client disable
ip dns server-address 1.1.1.1
ip route 0.0.0.0/0 10.199.199.1
ip add-host-route-first
!
username bleh password .....
route-only
!
!
clock summer-time
clock timezone us Central
!
!
ntp
 disable serve
 server 216.239.35.0
 server 216.239.35.4
!
!
no web-management http
!
!
!
!
!
!
!
interface ethernet 1/2/2
 port-name Router Uplink
 ip address 10.199.199.2 255.255.255.240
!
interface ethernet 1/3/1
 speed-duplex 10G-full
!
interface ethernet 1/3/2
 speed-duplex 10G-full
!
interface ethernet 1/3/3
 speed-duplex 10G-full
!
interface ethernet 1/3/4
 speed-duplex 10G-full
!
interface ethernet 1/3/5
 speed-duplex 10G-full
!
interface ethernet 1/3/6
 speed-duplex 10G-full
!
interface ethernet 1/3/7
 speed-duplex 10G-full
!
interface ethernet 1/3/8
 speed-duplex 10G-full
!
interface ve 1
 ip address 192.168.1.9 255.255.255.0
!
interface ve 1001
 ip address 10.100.1.1 255.255.255.0
 ip helper-address 1 10.100.50.5
!
interface ve 1010
!
interface ve 1050
 ip address 10.100.50.1 255.255.255.0
 ip helper-address 1 10.100.50.5
!
!
!
!
!
!
!
!
!
end

 

[LodeRunner]

I think your transit VLAN needs to have a VE instead of sticking the IP on the physical interface. I've never done a setup where the transit link physically had the IP, so I don't know for certain that's the problem.

 

[phekno]

OK. I changed the transit interface to be a VE and assigned the VE an IP (same IP as before). Still same issues. Able to ping everything locally, and can ping the internet from the switch, but can't get any traffic back through the firewall and switch from the internet. I'm pretty sure this is a routing problem, I'm just having trouble tracking it down...

 

[LodeRunner]

What does a traceroute from pfSense to a VLAN client look like? And what does an outbound trace to say, 1.1.1.1 look like? I would expect from pfSense to the client to be a single hop via the transit VLAN VE IP and outbound, I would expect the client VLAN VE IP, then pfSense, then whatever your first ISP hop is.

 

[phekno]

As requested...

Traceroute from VLAN client (laptop on VLAN 1001, 10.100.1.0/24):

Looks like it hits the VE interface (10.100.1.1) and then the OPNSense OPT1 interface (the transit interface 10.199.199.1), and then it's gone. I'd have to look and/or retry the experiment, but I'm GUESSING that I could probably see equivalent entries in the firewall logs, showing ICMP requests going out of OPT1, hitting WAN, and coming back...

Traceroute from OPNSense to a VLAN client (VM on 10.100.50.5):

Nothing too exciting...just about what I'd expect. Hits the switch's transit VLAN VE (10.199.199.2) and then the client.

Traceroute from OPNSense OUT to 1.1.1.1:

Also nothing too exciting, I don't think. First hop is my ISPs first hop (67.4.0.254).

 

[LodeRunner]

Yeah, so inbound route looks fine, but outbound hits OPT1 then disappears. I would normally expect an any/any rule to work, but I guess you could try, if you already have not, the guidance in BoredSysadmin's link where the author wrote a rule for each subnet with the subnet address as the source. I really can't see anything in what you've shared that is wrong per se, so right now this makes no sense.

 

[phekno]

I GOT IT! OMG...I've been working on this for like 2 days, and I got it and now I feel dumb because it was kind of a dumb thing...

IT WAS NAT! I had outbound NAT set to automatic, and the rules that it generated were basically to outbound NAT anything in the "OPT networks". Well...the OPT1 network is 10.199.199.0/30...so there's only 2 IP addresses in there. So, yeah, traffic was getting out, but wasn't being NATed properly, I guess (I don't claim to be an expert on NAT...hell...I'm actually moderately stupid when it comes to NAT, and really only know what "NAT" stands for), so it didn't make it back to the right source. I added manual NAT rules for each subnet, and now everything works.

THANK YOU for helping me. Honestly, I probably couldn't have done it without you. Seriously, this forum has saved my sanity more than once...

EDIT: Now to figure out how to make this switch work with my stupid Ubiquiti setup...

 

[LodeRunner]

I wonder if I never encountered this issue because I do outbound NAT in hybrid mode with static port pinning for various reasons.

Glad you found it!

Edit: Or I’ve just forgotten that I had to do manual outbound NAT. I flattened my networks long ago because routed L3 and mDNS reflection/proxy were not 100% reliable and thus had a negative wife acceptance factor.