Switch inter-vlan routing with only WAN traffic going to a firewall

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

kapone

Well-Known Member
May 23, 2015
1,095
642
113
Try something.

1. Disable that global-stp
2. Disable that ntp server 10.25.9.1 setting.

Edit: You may need to reboot the switch.

Try the traceroute again.
 

coxhaus

Active Member
Jul 7, 2020
109
36
28
STP, spanning tree protocol is going to protect you from network storms. But I guess it won't hurt to try.
 

Nikotine

Member
Mar 17, 2021
35
0
6
The trunk port is the problem. In the Cisco world you use an access port not a trunk port. I don't know how to translate to your world. You can try uplink.
You are doing layer 2 from OpenWRT to your L3 switch.
You got me thinking.
If the link between the switch and the OpenWRT router should be untagged, then there should only be one VLAN going over that link, right?

I noticed I can assign more than one subnet to an interface on the OpenWRT router.

Schermafbeelding 2021-04-25 141809.png

What if I do something like this?

homelab transit VLAN untagged.png

I only keep VLAN 172 on the OpenWRT router and remove the rest.
Then I add the two subnets to the same interface, linked to VLAN 172 and activate the DHCP server (currently DHCP is disabled on that interface).
My hope is that this DHCP server will hand out IPs for the three subnets.

I make 1/1/1 untagged and only add that port to VLAN 172 on the switch.

Finally, I add ip helper-address 1 172.16.1.2 to VLAN 9 and VLAN 100 on the switch, so it knows where the DHCP server is.
I don't need this on VLAN 172 as both interfaces have a fixed IP.

Would this work?
 

coxhaus

Active Member
Jul 7, 2020
109
36
28
Seems like to me you need the 172 network also defined on the layer 3 switch otherwise the L3 switch is going to use the router for layer 3 so in affect the switch is doing layer 2. How does the router route to the L3 switch in your example above?

You are trying to substitute untagging for routing.

To me the problem lies with Brocade L3 switch not the OpenWRT router. How does the switch do layer 3?

I have never used either of these products so it could be me.
 
Last edited:

Nikotine

Member
Mar 17, 2021
35
0
6
Seems like to me you need the 172 network also defined on the layer 3 switch
Yes, I wouldn't change anything to the VLANs defined on the switch.

How does the router route to the L3 switch in your example above?
Same as currently, as per the instructions from kapone.
There's a route on the switch for 0.0.0.0/0 to 172.16.1.2 and on the OpenWRT router there's a route as well for 10.25.0.0/16 to 172.16.1.1.

To me the problem lies with Brocade L3 switch not the OpenWRT router. How does the switch do layer 3?
As far as I understand, the Brocade does L3 because each VLAN has a virtual router interface defined.
These are the routes on the switch:
Code:
ICX6450-48P-Router(config-vlan-9)#sh ip route
Total number of IP routes: 4
Type Codes - B:BGP D:Connected O:OSPF R:RIP S:Static; Cost - Dist/Metric
BGP  Codes - i:iBGP e:eBGP
OSPF Codes - i:Inter Area 1:External Type 1 2:External Type 2
        Destination        Gateway         Port          Cost          Type Uptime
1       0.0.0.0/0          172.16.1.2      ve 172        1/1           S    1d22h
2       10.25.9.0/24       DIRECT          ve 9          0/0           D    3d10h
3       10.25.100.0/24     DIRECT          ve 100        0/0           D    3d2h
4       172.16.1.0/24      DIRECT          ve 172        0/0           D    1d22h
 

kapone

Well-Known Member
May 23, 2015
1,095
642
113
This IS a routing problem.

Somewhere somehow the "first hop" for the 10.25.9.x and 10.25.100.x VLANs on the Brocade is being recognized as the .1 IP address respectively in those subnets (still completely on the Brocade). That should not be happening. As an e.g., if I run a traceroute:

Code:
 % traceroute 8.8.8.8
traceroute to 8.8.8.8 (8.8.8.8), 64 hops max, 52 byte packets
1  10.10.10.1 (10.10.10.1)  0.607 ms  0.258 ms  0.263 ms
2  firewall1 (172.16.1.2)  0.433 ms  0.351 ms  0.361 ms
3  * * *
4  ae1316-21.artnvafc-mse01-aa-ie1.verizon-gni.net (100.41.24.204)  3.199 ms
    ae1316-20.washdcdn-mse01-aa-ie1.verizon-gni.net (100.41.24.202)  2.362 ms
    ae1316-21.artnvafc-mse01-aa-ie1.verizon-gni.net (100.41.24.204)  6.999 ms
5  0.ae2.gw16.iad8.alter.net (140.222.3.89)  3.618 ms  4.472 ms
    0.ae1.gw16.iad8.alter.net (140.222.3.87)  4.085 ms
6  &#10.170.148.204.in-addr.arpa (204.148.170.158)  4.095 ms  5.060 ms  5.086 ms
7  10.252.69.254 (10.252.69.254)  4.189 ms
    10.252.242.190 (10.252.242.190)  3.864 ms  3.559 ms
8  dns.google (8.8.8.8)  4.508 ms  4.590 ms  3.735 ms
- I don't have any VLANs defined on firewall (pfsense) because I don't use it for DHCP.
- The 10.10.10.1 IP is the gateway IP given out by my DHCP server (Windows Server 2016) for the 10.10.10.x VLAN. (I'm on a device in this VLAN)
- 172.16.1.2 is the TRANSIT VLAN between my Brocade (6610) and pfSense, where the switch is 172.16.1.1 and pfSense is 172.16.1.2
- I have a default route 0.0.0.0/0 defined on the switch that points to 172.16.1.2
- I do have helper IP addresses on VLANs in the Brocade, which is how my Windows Server does DHCP.
- My global-stp is off on the Brocade.

Notice that the traceroute command immediately jumps to the transit network as the next hop. That's how it should work.
 

kapone

Well-Known Member
May 23, 2015
1,095
642
113
pi@rpi3-sdr:~ $ traceroute -i enxb827eb45a8ec 9.9.9.9
traceroute to 9.9.9.9 (9.9.9.9), 30 hops max, 60 byte packets
1 10.25.100.2 (10.25.100.2) 3.943 ms 3.995 ms 4.566 ms
2 WRT1900AC.lan (10.25.100.1) 3.190 ms 3.077 ms 2.982 ms
3 WRT1900AC.lan (10.25.100.1) 2.906 ms 2.812 ms 2.718 ms pi@rpi3-sdr:~ $
@Nikotine - Notice in your traceroute output, that just your gateway on the Brocade (10.25.100.2) is taking almost 4ms to respond. That's WAY too long. That's the reason I'm focussing on the routing being "out of whack" even though it works for the intended purpose, but not traceroute.

That first hop (assuming you're hardwired) should be really really fast (in my case that's less than 0.3ms)
 

Nikotine

Member
Mar 17, 2021
35
0
6
It is hardwired yes, with cat 6.

Ok, I'm going to save my current config and rip it all up to experiment a bit.

Tomorrow an HP T620 plus arrives and I'll install Opnsense...
Then it will all start over :)
 

kapone

Well-Known Member
May 23, 2015
1,095
642
113
@Nikotine - When you defined that static route on the OpenWrt for the 10.25.0.0/16, did you define it on the transit interface? Can you show me a screenshot of how that looks on the OpenWrt?
 

Nikotine

Member
Mar 17, 2021
35
0
6
Here you go:
Schermafbeelding 2021-04-25 195422.png

Meanwhile I started configuring like this, in an effort to get rid of the different interfaces and VLANs on the OpenWRT switch:
homelab transit VLAN untagged.png

OpenWRT:
Schermafbeelding 2021-04-25 195830.png

Switch config:
Code:
Current configuration:
!
ver 08.0.30tT313
!
stack unit 1
  module 1 icx6450-48p-poe-port-management-module
  module 2 icx6450-sfp-plus-4port-40g-module
!
global-stp
!
!
!
vlan 9 name main by port
 untagged ethe 1/1/2 to 1/1/36 ethe 1/2/1 to 1/2/4
 router-interface ve 9
!
vlan 100 name IPcams by port
 untagged ethe 1/1/37 to 1/1/48
 router-interface ve 100
!
vlan 172 name TRANSIT by port
 untagged ethe 1/1/1
 router-interface ve 172
!
vlan 4095 name DEFAULT-VLAN by port
!
!
!
!
!
aaa authentication web-server default local
aaa authentication enable default local
aaa authentication login default local
default-vlan-id 4095
enable telnet authentication
hostname ICX6450-48P-Router
ip dhcp-client disable
ip dns server-address 172.16.1.2
ip route 0.0.0.0/0 172.16.1.2
!
no telnet server
username root password .....
snmp-server community ..... ro
!
!
clock summer-time
clock timezone gmt GMT+01
!
!
ntp
 disable serve
 server 10.25.9.1
!
!
web-management https
web-management refresh front-panel 30
web-management session-timeout 3600
!
!
!
interface ethernet 1/1/37
 inline power priority 1 power-by-class 3
!
interface ve 9
 ip address 10.25.9.2 255.255.255.0
 ip helper-address 1 172.16.1.2
!
interface ve 100
 ip address 10.25.100.2 255.255.255.0
 ip helper-address 1 172.16.1.2
!
interface ve 172
 ip address 172.16.1.1 255.255.255.0
!
!
!
!
!
!
!
!
!
end
Plenty of errors in OpenWRT log about DHCP...
Schermafbeelding 2021-04-25 195956.png

Not sure if DHCP supposed to work if you define multiple subnets on the same interface in OpenWRT...
 

kapone

Well-Known Member
May 23, 2015
1,095
642
113
This is most likely not going to work.

- On the transit interface on the wrt we have told it that the route for 10.25.0.0/16 is over 172.16.1.1
- If the 10.25.x.x subnets are on that same interface, the static route above will kick in.
- When the DHCP discovery broadcast happens on 10.25.x.x wrt will respond over 172.16.x.x.
- er... :)
 
  • Like
Reactions: Nikotine

Nikotine

Member
Mar 17, 2021
35
0
6
This is most likely not going to work.

- On the transit interface on the wrt we have told it that the route for 10.25.0.0/16 is over 172.16.1.1
- If the 10.25.x.x subnets are on that same interface, the static route above will kick in.
- When the DHCP discovery broadcast happens on 10.25.x.x wrt will respond over 172.16.x.x.
- er... :)
Yeah and I was kind of hoping that the switch would then route this DHCP offer from VLAN 172 to VLAN 9 or 100 :)
 

Nikotine

Member
Mar 17, 2021
35
0
6
@kapone, does your DHCP server offer IP addresses for different subnets?
How did you get that to work?

Current situation is:
- the switch can ping 172.16.1.2, 192.168.0.1 and google.com
- the OpenWRT router can ping 172.16.1.1 but not 10.25.9.2 or 10.25.100.2
- DHCP requests from a client connected to VLAN 9 reaches the DHCP server on OpenWRT, but the DHCP offer never makes it back
 
Last edited:

kapone

Well-Known Member
May 23, 2015
1,095
642
113
does your DHCP server offer IP addresses for different subnets?
Of course. My DHCP/DNS server(s) is on a completely different subnet, isolated from everything else. All it does is DHCP/DNS (and Active Directory in my case). It's on a 10.10.4.x subnet, and scopes have been created in it for the various subnets.
 
  • Like
Reactions: Nikotine

coxhaus

Active Member
Jul 7, 2020
109
36
28
This IS a routing problem.

Somewhere somehow the "first hop" for the 10.25.9.x and 10.25.100.x VLANs on the Brocade is being recognized as the .1 IP address respectively in those subnets (still completely on the Brocade). That should not be happening. As an e.g., if I run a traceroute:

Code:
 % traceroute 8.8.8.8
traceroute to 8.8.8.8 (8.8.8.8), 64 hops max, 52 byte packets
1  10.10.10.1 (10.10.10.1)  0.607 ms  0.258 ms  0.263 ms
2  firewall1 (172.16.1.2)  0.433 ms  0.351 ms  0.361 ms
3  * * *
4  ae1316-21.artnvafc-mse01-aa-ie1.verizon-gni.net (100.41.24.204)  3.199 ms
    ae1316-20.washdcdn-mse01-aa-ie1.verizon-gni.net (100.41.24.202)  2.362 ms
    ae1316-21.artnvafc-mse01-aa-ie1.verizon-gni.net (100.41.24.204)  6.999 ms
5  0.ae2.gw16.iad8.alter.net (140.222.3.89)  3.618 ms  4.472 ms
    0.ae1.gw16.iad8.alter.net (140.222.3.87)  4.085 ms
6  &#10.170.148.204.in-addr.arpa (204.148.170.158)  4.095 ms  5.060 ms  5.086 ms
7  10.252.69.254 (10.252.69.254)  4.189 ms
    10.252.242.190 (10.252.242.190)  3.864 ms  3.559 ms
8  dns.google (8.8.8.8)  4.508 ms  4.590 ms  3.735 ms
- I don't have any VLANs defined on firewall (pfsense) because I don't use it for DHCP.
- The 10.10.10.1 IP is the gateway IP given out by my DHCP server (Windows Server 2016) for the 10.10.10.x VLAN. (I'm on a device in this VLAN)
- 172.16.1.2 is the TRANSIT VLAN between my Brocade (6610) and pfSense, where the switch is 172.16.1.1 and pfSense is 172.16.1.2
- I have a default route 0.0.0.0/0 defined on the switch that points to 172.16.1.2
- I do have helper IP addresses on VLANs in the Brocade, which is how my Windows Server does DHCP.
- My global-stp is off on the Brocade.

Notice that the traceroute command immediately jumps to the transit network as the next hop. That's how it should work.
I explained this problem already. You are using a trunk port between the switch and router so the routing is not happening to 172 network.

Think about it. Network 10.25.9.0/24 and 10.25.100.0/24 are known networks so they will not use the default gateway.
 
Last edited:
  • Like
Reactions: rmflint

coxhaus

Active Member
Jul 7, 2020
109
36
28
Please look at post #50, I'm no longer using a trunk port.
I don't believe that is true. Just because you create a 172 network does not mean you will use it. If you are not using s trunk port then I would say the switch is broke because it still should not do layer 2 unless your trace route has changed.
Has your trace route changed? You should see 172 network between the switch and router on the trace route.

Why don't you try setting up the router with the 172 network only and no DHCP and we can see if the trace route changes. Just hard code the IP addresses manually on the clients that way we can check just the routing. There will not be any 10.25.9.0 network or any 10.25.100 network on the router. This might force layer 3.
 
Last edited:

kapone

Well-Known Member
May 23, 2015
1,095
642
113
I explained this problem already. You are using a trunk port between the switch and router so the routing is not happening to 172 network.

Think about it. Network 10.25.9.0/24 and 10.25.100.0/24 are known networks so they will not use the default gateway.
It's not that simple. Before @Nikotine ripped up his config and started again...

1. He was getting a DHCP IP address from the wrt correctly. His L2 broadcast was working exactly as intended.
2. He had a default route on the switch - 0.0.0.0/0 pointing to 172.16.1.2 (which was the wrt)
3. In the wrt firewall, his allowed access rules were only on the transit interface, not the 10.25.x.x interfaces in wrt. i.e. any packets coming from those VLANs on the wrt will not be accepted by the WAN interface.
4. "10.25.9.0/24 and 10.25.100.0/24 are known networks" is true, but 8.8.8.8 (or any other external IP) is not. When a device on one of the 10.25.x.x VLANs on the brocade tries to access 8.8.8.8 it has to go over the 172.16.x.x network, because the switch doesn't know where else to send it. (Because of the default route on the switch).
5. And it WAS working. He had internet access on the 10.25.9.x VLAN (which had a corresponding allow rule in wrt on the transit interface and nowhere else).

That traceroute issue is perplexing because it shouldn't be happening. Like I said, something somewhere is affecting that. But his external/WAN routing WAS going over the 172.16.x.x interface.
 
  • Like
Reactions: Nikotine

coxhaus

Active Member
Jul 7, 2020
109
36
28
The last trace route I saw was to 9.9.9.9 and it was doing layer 2 to10.25.100.1 on the router.

Why don't you try setting up the router with the 172 network only and no DHCP and we can see if the trace route changes. Just hard code the IP addresses manually on the clients that way we can check just the routing. There will not be any 10.25.9.0 network or any 10.25.100 network on the router. This might force layer 3.
 
  • Like
Reactions: Nikotine