Switch inter-vlan routing with only WAN traffic going to a firewall

kapone · Apr 22, 2021

I think there's a few issues with our firewall rule(s). I believe the routing is correct, since the router can send traffic back to a client (on transit), but traffic coming "from" the web, is not being sent to a client.

I need a better picture of how the OpwnWrt is setup. PM me.

Nikotine · Apr 22, 2021

I've done some more testing.
I found it strange that ping didn't work, but traceroute (on the Pi) did.
So I learned that on Linux, ping uses ICMP but traceroute uses UDP packages.
On Windows both use ICMP.
Then I tested if I could do a sudo apt update on the Pi and that worked!

So I believe I have internet access after all, but ICMP is blocked somehow.
That's a separate issue, which I should be able to fix myself.

Thanks for your help!
I will do some more testing e.g. verify that VLAN 100 indeed has no WAN access, and pinging between VLANs.
I'll let you know if there are any issues that pop up.

Nikotine · Apr 22, 2021

Just had an "aha" moment!
I changed this traffic rule from:

Code:

config rule
        option src 'TRANSIT'
        option name 'Allow WAN to 10.25.9.0/24'
        list src_ip '10.25.9.0/24'
        option dest 'wan'
        option target 'ACCEPT'

To:

Code:

config rule
        option src 'TRANSIT'
        option name 'Allow WAN to 10.25.9.0/24'
        list src_ip '10.25.9.0/24'
        option dest 'wan'
        option target 'ACCEPT'
        list proto 'all' <-------- by default it is only UDP and TCP, so not ICMP

Ping is working

kapone · Apr 22, 2021

gah! Excellent!

coxhaus · Apr 22, 2021

So, what does a trace route look like. Last time it was not using route via 172.16.1.1. I guess the same.

Nikotine · Apr 22, 2021

Traceroute indeed still doesn't go over 172.16.1.1, which I don't understand either

But it works as I wanted it to work.

VLAN 9 has WAN access:

Code:

pi@rpi3-sdr:~ $ traceroute -i enxb827eb45a8ec google.com
traceroute to google.com (172.217.17.142), 30 hops max, 60 byte packets
 1  10.25.9.2 (10.25.9.2)  3.293 ms  3.110 ms  3.135 ms
 2  WRT1900AC.lan (10.25.9.1)  0.538 ms  0.417 ms  0.487 ms
 3  192.168.0.1 (192.168.0.1)  1.996 ms  3.177 ms  3.962 ms
 4  <redacted>  20.670 ms  24.491 ms  24.860 ms
 5  * * *
 6  dD5E0FA71.access.telenet.be (213.224.250.113)  42.467 ms  42.375 ms  42.572 ms
 7  74.125.32.88 (74.125.32.88)  43.125 ms  43.007 ms  17.094 ms
 8  * * *
 9  142.251.48.174 (142.251.48.174)  25.357 ms 209.85.240.30 (209.85.240.30)  24.343 ms 142.251.48.180 (142.251.48.180)  26.762 ms
10  108.170.241.205 (108.170.241.205)  25.969 ms 108.170.241.204 (108.170.241.204)  25.028 ms 108.170.236.227 (108.170.236.227)  35.063 ms
11  ams15s30-in-f142.1e100.net (172.217.17.142)  35.488 ms 216.239.42.103 (216.239.42.103)  36.074 ms 209.85.254.49 (209.85.254.49)  36.241 ms

VLAN 100 doesn't have WAN access:

Code:

pi@rpi3-sdr:~ $ traceroute -i enxb827eb45a8ec google.com
traceroute to google.com (142.250.179.206), 30 hops max, 60 byte packets
 1  10.25.100.2 (10.25.100.2)  2.369 ms  2.658 ms  3.221 ms
 2  WRT1900AC.lan (10.25.100.1)  1.601 ms  1.529 ms  1.445 ms
 3  * * *
 4  * WRT1900AC.lan (10.25.100.1)  0.962 ms  0.905 ms <------- stops here

coxhaus · Apr 23, 2021

It looks like VLAN100 is doing layer 2.

It is normal for a layer 3 switch to incur a higher ping rate on the first packet during layer 3 routing but then it stores that info and the rest are run at switching speed. A router does not do this as it is slow all the time.

coxhaus · Apr 23, 2021

Here is what mine looks like on a Cisco L3 SG350-10P switch with couple of trace routes. These are a couple local networks I have running every day 24/7. My wife is Zooming on her iPad on 1 of the networks, 192.168.2.0/24 while I am doing this. Plus, I have security cameras running local and many Apple devices.
If it matters, I am using my wireless laptop. I jump in and out of my 2 wireless networks and run tracert on my Windows 10 machine. My Cisco wireless APs have 2 SSIDs each which each SSID uses a separate VLAN. The Cisco WAP581 APs are setup as single point setup so there is one virtual SSID for each VLAN across multiple APs. 192.168.10.0/30 is a point-to-point connection network with my L3 switch being 192.168.10.2/30 and my router being 192.168.10.1/30. I started with a Class C, 192.168.10.0/24, network as it was easier to migrate over to a layer 3 Cisco SG300-28 switch but that was 10 years ago so I don't remember the details.

C:\Users\leeco>tracert 9.9.9.9

Tracing route to dns9.quad9.net [9.9.9.9]
over a maximum of 30 hops:

1 2 ms 2 ms 2 ms 192.168.2.254
2 1 ms 1 ms 1 ms router44BEE6 [192.168.10.1]
3 9 ms 9 ms 14 ms cpe-72-133-80-1.sw.res.rr.com [72.133.80.1]
4 43 ms 208 ms 32 ms tge0-0-0.elgntx0801h.texas.rr.com [66.68.3.169]
5 16 ms 18 ms 7 ms agg38.ausutxla01r.texas.rr.com [24.175.42.138]
6 20 ms 41 ms 17 ms agg22.dllatxl301r.texas.rr.com [24.175.41.46]
7 19 ms 23 ms 34 ms bu-ether14.dllstx976iw-bcr00.tbone.rr.com [66.109.6.88]
8 23 ms 16 ms 19 ms 209-18-43-77.dfw10.tbone.rr.com [209.18.43.77]
9 30 ms 20 ms 20 ms eqix.da4.packetclearing.com [206.223.118.224]
10 19 ms 21 ms 17 ms dns9.quad9.net [9.9.9.9]

Trace complete.

C:\Users\leeco>

C:\Users\leeco>tracert 8.8.8.8

Tracing route to dns.google [8.8.8.8]
over a maximum of 30 hops:

1 2 ms 1 ms 2 ms 192.168.0.98
2 2 ms 1 ms 1 ms 192.168.10.1
3 15 ms 11 ms 14 ms cpe-72-133-80-1.sw.res.rr.com [72.133.80.1]
4 40 ms 27 ms 29 ms tge0-0-0.elgntx0801h.texas.rr.com [66.68.3.169]
5 15 ms 17 ms 17 ms agg38.ausutxla01r.texas.rr.com [24.175.42.138]
6 16 ms 17 ms 55 ms agg22.dllatxl301r.texas.rr.com [24.175.41.46]
7 18 ms 18 ms 23 ms 66.109.1.216
8 20 ms 19 ms 22 ms 142.250.168.12
9 19 ms 20 ms 24 ms 209.85.243.95
10 37 ms 61 ms 110 ms 209.85.243.255
11 25 ms 18 ms 17 ms dns.google [8.8.8.8]

Trace complete.

kapone · Apr 23, 2021

coxhaus said:
It looks like VLAN100 is doing layer 2.

So, you're saying if he does the "test 2" that I mentioned above, the test will fail?

coxhaus · Apr 23, 2021

I am just looking at his VLAN100 trace route above, it never leaves the 10.25.100.X network. I assume 10.25.100.0/24 but it could be different.

Have him assign an IP address to a client in the VLAN100 so we don't have to worry about DHCP and run a tracert on the client to say 8.8.8.8 or 9.9.9.9. We should see the L3 switch forward the traffic to another network for L3 to be working on the switch.

Nikotine · Apr 23, 2021

kapone said:
So, you're saying if he does the "test 2" that I mentioned above, the test will fail?

Right, I forgot about that.
With the cable to OpenWRT detached, I can still ping between both VLANs.

I should add that in the meantime I am no longer attached with a second cable between 172.16.1.1 on the switch, and 172.16.1.2 on OpenWRT. It's all going via one cable and tagged 1/1/1 now.

coxhaus · Apr 23, 2021

So, what does a tracert look like to 9.9.9.9 from one of your clients on the layer 3 switch. Intervlan routing on the L3 switch may be working between networks in the L3 switch but L3 routing to your OpenWRT may not. Routing to OpenWRT is the important one to me any way. I don't want any traffic to my router unless it is destined internet traffic. Let the L3 switch handle all the local LAN traffic.

Let's also see a VLAN9 to VLAN100 or VLAN100 to VLAN9 trace route. We need to see it from client to client as some of the gateways on the L3 switch may end up being the same.

Nikotine · Apr 23, 2021

So, what does a tracert look like to 9.9.9.9 from one of your clients on the layer 3 switch.

With cable to OpenWRT connected or not? Because without it, it won't go anywhere?

coxhaus · Apr 23, 2021

Well, you need the cable from the L3 switch to the OpenWRTrouter.

Nikotine · Apr 23, 2021

Ok, here you go:

Code:

pi@rpi3-sdr:~ $ traceroute -i enxb827eb45a8ec 9.9.9.9
traceroute to 9.9.9.9 (9.9.9.9), 30 hops max, 60 byte packets
1  10.25.9.2 (10.25.9.2)  2.116 ms  2.580 ms  3.142 ms
2  WRT1900AC.lan (10.25.9.1)  1.303 ms  1.205 ms  1.257 ms
3  192.168.0.1 (192.168.0.1)  6.632 ms  7.433 ms  8.204 ms
4  <redacted>  14.069 ms  16.160 ms  16.921 ms
5  * * *
6  dD5E0FA71.access.telenet.be (213.224.250.113)  26.135 ms  13.320 ms  17.770 ms
7  * * *
8  * * *
9  * * *
10  dns9.quad9.net (9.9.9.9)  23.173 ms !X  20.505 ms !X  18.539 ms !X

And traceroute to a client in another VLAN:

Code:

pi@rpi3-sdr:~ $ traceroute -i enxb827eb45a8ec 10.25.100.122
traceroute to 10.25.100.122 (10.25.100.122), 30 hops max, 60 byte packets
1  10.25.9.2 (10.25.9.2)  5.280 ms  5.695 ms  6.132 ms
2  10.25.100.122 (10.25.100.122)  10.412 ms  10.791 ms  10.932 ms

coxhaus · Apr 23, 2021

What is 192.168.0.1? Is it your local real network? Then it looks like your L3 switch is doing L2 to the OpenWRT router as it never leaves the 10.25.9.0/24 network to me any way. But the only path to the internet is through network 172.172.16.0 so it is strange to me. Is the uplink port defined as a trunk? The uplink port between the router and L3 switch needs to be an access port not a trunk port. An access port will force layer 3. If you are using a trunk port for the uplink then you are using layer 2.

What is the gateway for VLAN100? I think it is working.

What does a trace route look like from a client in the VLAN100 network to 9.9.9.9?

kapone · Apr 23, 2021

There IS something weird about this traceroute. It should be going from 10.25.9.2 (or even 10.25.100.2) --> 172.16.1.2 --> where ever.

@Nikotine - In your brocade config you had 1/1/1 as a mirror port? Is that still there?

However, L3 routing IS working, as he wouldn't be able to reach another VLAN on the switch without it.

coxhaus · Apr 23, 2021

I believe it is his uplink port definition. To use L3 from a switch to a router the port needs to be defined as an access port. Maybe the same on OpenWRT also. I don't define any VLANs on any of my routers so they default to access or single network.

If you uplink as layer 2 by using a trunk port the router is still involved in the L3 switch.

I think what is happening is the trunk port is connecting VLAN9 together from the L3 switch and the router as layer 2 and 172.172.16.0 is not being used at least on the trace route to 9.9.9.9 for VLAN9. I would like to see a trace route for VLAN100 to 9.9.9.9.

Nikotine · Apr 23, 2021

I really appreciate you guys troubleshooting this

coxhaus said:
What is 192.168.0.1? Is it your local real network?

192.168.0.1 is my ISP's router, connected to the WAN port of the OpenWRT router.
OpenWRT is in the DMZ of my ISP router to keep it as dumb as possible.

coxhaus said:
Is the uplink port defined as a trunk? The uplink port between the router and L3 switch needs to be an access port not a trunk port. An access port will force layer 3. If you are using a trunk port for the uplink then you are using layer 2.

The connection between the switch and the OpenWRT router is trunked, yes.
I haven't declared an uplink port, if that's what you mean.
I know I can define a port as uplink in the VLAN settings of the switch, but I'm not sure what that does.
Should I set this?

coxhaus said:
What is the gateway for VLAN100? I think it is working.

Clients in VLAN 100 get 10.25.100.2 as gateway, via option 3 of DHCP.

coxhaus said:
What does a trace route look like from a client in the VLAN100 network to 9.9.9.9?

Code:

pi@rpi3-sdr:~ $ traceroute -i enxb827eb45a8ec 9.9.9.9
traceroute to 9.9.9.9 (9.9.9.9), 30 hops max, 60 byte packets
 1  10.25.100.2 (10.25.100.2)  3.943 ms  3.995 ms  4.566 ms
 2  WRT1900AC.lan (10.25.100.1)  3.190 ms  3.077 ms  2.982 ms
 3  WRT1900AC.lan (10.25.100.1)  2.906 ms  2.812 ms  2.718 ms
pi@rpi3-sdr:~ $

I agree it is strange this doesn't get stopped at 172.16.1.2... But it works.

kapone said:
@Nikotine - In your brocade config you had 1/1/1 as a mirror port? Is that still there?

Not anymore. Initially I had 1/1/1 as mirror port to 1/1/2, so that I could do some Wireshark analysis on everything going in and out of the switch.
But I removed that when I used 1/1/2 as second link to the OpenWRT router.
Now everything goes via one link via trunked 1/1/1.
Mirroring is still disabled.
This is my config:

Code:

Current configuration:
!
ver 08.0.30tT313
!
stack unit 1
  module 1 icx6450-48p-poe-port-management-module
  module 2 icx6450-sfp-plus-4port-40g-module
!
global-stp
!
!
!
vlan 9 name main by port
 tagged ethe 1/1/1
 untagged ethe 1/1/2 to 1/1/36 ethe 1/2/1 to 1/2/4
 router-interface ve 9
!
vlan 100 name IPcams by port
 tagged ethe 1/1/1
 untagged ethe 1/1/37 to 1/1/48
 router-interface ve 100
!
vlan 172 name TRANSIT by port
 tagged ethe 1/1/1
 router-interface ve 172
!
vlan 4095 name DEFAULT-VLAN by port
!
!
!
!
!
aaa authentication web-server default local
aaa authentication enable default local
aaa authentication login default local
default-vlan-id 4095
enable telnet authentication
hostname ICX6450-48P-Router
ip dhcp-client disable
ip dns server-address 172.16.1.2
ip route 0.0.0.0/0 172.16.1.2
!
no telnet server
username root password .....
snmp-server community ..... ro
!
!
clock summer-time
clock timezone gmt GMT+01
!
!
ntp
 disable serve
 server 10.25.9.1
!
!
web-management https
web-management refresh front-panel 30
web-management session-timeout 3600
!
!
!
interface ethernet 1/1/37
 inline power priority 1 power-by-class 3
!
interface ve 9
 ip address 10.25.9.2 255.255.255.0
!
interface ve 100
 ip address 10.25.100.2 255.255.255.0
!
interface ve 172
 ip address 172.16.1.1 255.255.255.0
!
!
!
!
!
!
!
!
!
end

coxhaus · Apr 23, 2021

The trunk port is the problem. In the Cisco world you use an access port not a trunk port. I don't know how to translate to your world. You can try uplink.
You are doing layer 2 from OpenWRT to your L3 switch.

Is uplink for another switch or a router? If it is for a router then it sounds right.

Switch inter-vlan routing with only WAN traffic going to a firewall

Well-Known Member

Member

Member

Well-Known Member

Active Member

Member

Active Member

Active Member

Well-Known Member

Active Member

Member

Active Member

Member

Active Member

Member

Active Member

Well-Known Member

Active Member

Member

Active Member