[Solved] One way routing through VLANs and L3 switches - Brocade ICX7150

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

TonyArrr

Active Member
Sep 22, 2021
129
66
28
Straylia
Hi all.
I'm sure I'm missing something obvious, but I'm a hardware guy trying to get into a software world. So this may or may not be a super easy fix.

I currently run a flat network, 192.168.0.0/24, which I am working to replace entirely with a vlan'd up network, coming soon when my other renovations give me wall access to run everything wired.
I have my 3 ICXs that I'll be using around the place, and am trying to build a network to get some practice with VLANs and routing and firewalling in the mean time.

In doing so, I set up the 7150-24 to take the place of the older netgear switch that was linking everything at my desk to my Wireless bridge. Straightforward, all ports untagged VLAN 1, with a VE using 192.168.0.152, set statically on the switch (and outside my current DHCP reservation pool)
I've added a new VLAN, 22, and tagged it on port 8, and connected that to port 1 on my 7250, and gave them both VEs in the 10.0.0.0/24 range, again statically (7150 is 10.0.0.2, 7250 is 10.0.0.1). I then added a static route to my desktop for 10.0.0.0/24 to go via 192.168.0.152

So far so good, I can ssh into the 7250 at 10.0.0.1, interVLAN routing works like a charm, since the 7150 has no ACLs or anything. I put away the 7250's Serial Cable.

So I go ahead and set up a few VLANs and VEs on the 7250, tag and untag a few ports with different VLANs and get comfortable with the syntax. Time for something new.

I connect up my little SuperMicro 1U server's IPMI port to the 7250, on a port that is untagged VLAN 22. Use it's BIOS to set a static IP, 10.0.0.4/24. Reboot, and bam, I can remote in from my desktop. Now in the IPMI web interface, I change it so it's ethernet port is tagging everything VLAN 22, and then go back to the 7250 and remove the untagged 22 and tag it 22 instead. Success, can still log into the IPMI from my desktop.

So I go ahead and install Proxmox using the IPMI's virtual media function. I connect a DAC between the SFP+ port on the 7250 and the SFP+ port on the 1U, and using the IPMI's KVM, set a static IP, 10.0.0.5/24.

Now from the Desktop, I can both ssh into Proxmox, as well as access it's web interface, both at 10.0.0.5, no problems. As I go about configuring it though, I find that Proxmox cannot connect to the internet. Oh, it's DNS server is itself. Cool, just change that to my DNS server on 192.168.0.200, easy.
Nope, still can't resolve domains. Can't ping the DNS server either, huh.

Right, static routes. Configure a static route to 192.168.0.0/24 via 10.0.0.2, but no change, still can't reach my DNS server.
But I can ping my desktop, which is 192.168.0.2, which I guess makes sense, since the SSH connection to it would tell each device along the path to it where it came from. And the DNS server is on the other side of the Wireless bridge, so that's gonna be a bit tricky to route to, perhaps.

So I try to ping my NUC, which is directly connected to the wireless bridge (192.168.0.203) via a VLAN 1 port, at 192.168.0.10, and just get time outs. So the Wireless bridge is the problem here? Nope. I SSH from the NUC to the IPMI of the 1U, and now proxmox can ping the NUC but still NOT the wireless bridge.

I've tried setting a static route on the 7150, as "0.0.0.0/0 via 192.168.0.203", "192.168.0.0/24 via 192.168.0.203", and "0.0.0.0/0 via 192.168.0.201", but no change. The 7150 can ping the gateway (192.168.0.201) and the DNS server fine, but Proxmox behind it gets told there is no route.

So.... what little thing did I miss?

Network topology

Gateway (192.168.0.201) and DNS Server (192.168.0.200)
Wired to
Wireless Router (and DHCP Server) (192.168.0.1)
Wireless to
Wireless Bridge (192.168.0.203)
Wired to
NUC (192.168.0.10) and 7150 (VLAN 1: 192.168.0.152, Static IP; VLAN 22: 10.0.0.2, Static IP)

7150, VLAN 1 untagged
Wired to
Desktop (192.168.0.2)

7150, VLAN 22 tagged
Wired to
7250 (10.0.0.1)
Wired to
IPMI (10.0.0.4, tagged VLAN 22) and Proxmox (10.0.0.5, untagged VLAN 22)

Code:
Current configuration:
!
ver 08.0.95hT213
!
stack unit 1
  module 1 icx7150-24-port-management-module
  module 2 icx7150-2-copper-port-2g-module
  module 3 icx7150-4-sfp-plus-port-40g-module
!
!
!
!
!
vlan 1 name DEFAULT-VLAN by port
router-interface ve 1
!
vlan 22 by port
tagged ethe 1/1/8
router-interface ve 22
!
!
!
!
!                                                                
!
!
!
!
!
!
!
!
!
aaa authentication web-server default local
aaa authentication login default local
enable aaa console
ip dhcp-client disable
!
no telnet server
username super password .....
!
!
!
!
clock timezone gmt GMT+10
!
manager disable                                                  
!
!
manager port-list 987
!
!
!
!
!
!
!
!
!
interface ethernet 1/3/1
speed-duplex 10G-full
!
interface ethernet 1/3/2
speed-duplex 10G-full
!
interface ethernet 1/3/3
speed-duplex 10G-full
!
interface ethernet 1/3/4
speed-duplex 10G-full                                          
!
interface ve 1
ip address 192.168.0.152 255.255.255.0
!
interface ve 22
ip address 10.0.0.2 255.255.255.0
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
end

Code:
Total number of IP routes: 2
Type Codes - B:BGP D:Connected O:OSPF R:RIP S:Static; Cost - Dist/Metric
BGP  Codes - i:iBGP e:eBGP
OSPF Codes - i:Inter Area 1:External Type 1 2:External Type 2
STATIC Codes - v:Inter-VRF
        Destination        Gateway         Port          Cost          Type Uptime
1       10.0.0.0/24      DIRECT          ve 22         0/0           D    2h34m
2       192.168.0.0/24    DIRECT          ve 1          0/0           D    95d0h

Code:
Current configuration:
!
ver 08.0.95hT213
!
stack unit 1
  module 1 icx7250-24p-poe-port-management-module
  module 2 icx7250-sfp-plus-8port-80g-module
!
!
!
!
!
vlan 1 name DEFAULT-VLAN by port
router-interface ve 1
!
vlan 22 by port
tagged ethe 1/1/1 ethe 1/1/24 ethe 1/2/1 ethe 1/2/3
untagged ethe 1/2/2
router-interface ve 22
!
vlan 50 by port
tagged ethe 1/2/1 to 1/2/3
router-interface ve 50
!                                                                
vlan 60 by port
tagged ethe 1/2/1 to 1/2/3
router-interface ve 60
!
vlan 70 by port
tagged ethe 1/2/1 to 1/2/3
router-interface ve 70
!
vlan 80 by port
tagged ethe 1/2/1 to 1/2/3
router-interface ve 80
!
vlan 107 by port
tagged ethe 1/2/1 to 1/2/3
router-interface ve 107
!
!
!
!
!
!
!
!                                                                
!
!
!
!
!
!
optical-monitor
optical-monitor non-ruckus-optic-enable
aaa authentication web-server default local
aaa authentication login default local
enable aaa console
ip dhcp-client disable
!
no telnet server
username super password .....
!
!
!
!
!
manager disable
!
!                                                                
manager port-list 987
!
!
!
!
!
!
!
!
!
interface ethernet 1/1/1
no inline power
!
interface ethernet 1/1/2
no inline power
!
interface ethernet 1/1/3
no inline power
!
interface ethernet 1/1/4
no inline power
!
interface ethernet 1/1/5                                        
no inline power
!
interface ethernet 1/1/6
no inline power
!
interface ethernet 1/1/7
no inline power
!
interface ethernet 1/1/8
no inline power
!
interface ethernet 1/1/9
no inline power
!
interface ethernet 1/1/10
no inline power
!
interface ethernet 1/1/11
no inline power
!
interface ethernet 1/1/12
no inline power
!                                                                
interface ethernet 1/1/13
no inline power
!
interface ethernet 1/1/14
no inline power
!
interface ethernet 1/1/15
no inline power
!
interface ethernet 1/1/16
no inline power
!
interface ethernet 1/1/17
no inline power
!
interface ethernet 1/1/18
no inline power
!
interface ethernet 1/1/19
no inline power
!
interface ethernet 1/1/20
no inline power                                                
!
interface ethernet 1/1/21
no inline power
!
interface ethernet 1/1/22
no inline power
!
interface ethernet 1/1/23
no inline power
!
interface ethernet 1/1/24
no inline power
!
interface ethernet 1/2/1
speed-duplex 10G-full
!
interface ethernet 1/2/2
no optical-monitor
speed-duplex 10G-full
!
interface ethernet 1/2/3
speed-duplex 10G-full
!                                                                
interface ethernet 1/2/4
speed-duplex 10G-full
!
interface ethernet 1/2/5
speed-duplex 10G-full
!
interface ethernet 1/2/6
speed-duplex 10G-full
!
interface ethernet 1/2/7
speed-duplex 10G-full
!
interface ethernet 1/2/8
speed-duplex 10G-full
!
interface ve 1
ip address 192.168.0.151 255.255.255.0
!
interface ve 22
ip address 10.0.0.1 255.255.255.0
!
interface ve 50
!                                                                
interface ve 60
!
interface ve 70
!
interface ve 80
!
interface ve 107
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
end

Code:
Total number of IP routes: 2
Type Codes - B:BGP D:Connected O:OSPF R:RIP S:Static; Cost - Dist/Metric
BGP  Codes - i:iBGP e:eBGP
OSPF Codes - i:Inter Area 1:External Type 1 2:External Type 2
STATIC Codes - v:Inter-VRF
        Destination        Gateway         Port          Cost          Type Uptime
1       10.0.0.0/24      DIRECT          ve 22         0/0           D    2h33m
2       192.168.0.0/24    DIRECT          ve 1          0/0           D    2h33m

Code:
auto lo
iface lo inet loopback

iface eno7 inet manual

iface eno1 inet manual

iface eno2 inet manual

iface eno3 inet manual

iface eno4 inet manual

iface eno5 inet manual

iface eno6 inet manual

iface eno8 inet manual

auto vmbr0
iface vmbr0 inet static
        address 10.0.0.5/24
        gateway 10.0.0.2
        bridge-ports eno7
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 2-4094

Code:
default via 10.0.0.2 dev vmbr0 proto kernel onlink
10.0.0.0/24 dev vmbr0 proto kernel scope link src 10.0.0.5
 
  • Like
Reactions: Lews

DavidWJohnston

Active Member
Sep 30, 2020
242
188
43
There's a lot going on here but I think there are a couple of major issues:

- If your gateway has its LAN connection only on the 192.168.0.0/24 subnet, is it configured/allowed to route stuff coming in from 10.0.0.x? Does the gateway have a static route pointing back to 10.0.0.x for the return traffic?

- More generally: How does your internet gateway know about the 10.0.0.x network?

- What is the wireless bridge doing exactly? If it's literally just a bridge (wireless to wired) why are you routing packets to it since it's not a router?

- Is your "Desktop" the same as the NUC?

I think creating a diagram and mapping out hops is the way to go. For example, if you ping from Proxmox to Internet, what are all the hops (both routers and L2 devices) in the egress (request), and back through the ingress (response) direction?

If you answer the 4 questions and create a diagram I think we can get it sorted eventually.
 

TonyArrr

Active Member
Sep 22, 2021
129
66
28
Straylia
There's a lot going on here but I think there are a couple of major issues:

- If your gateway has its LAN connection only on the 192.168.0.0/24 subnet, is it configured/allowed to route stuff coming in from 10.0.0.x? Does the gateway have a static route pointing back to 10.0.0.x for the return traffic?

- More generally: How does your internet gateway know about the 10.0.0.x network?

- What is the wireless bridge doing exactly? If it's literally just a bridge (wireless to wired) why are you routing packets to it since it's not a router?

- Is your "Desktop" the same as the NUC?

I think creating a diagram and mapping out hops is the way to go. For example, if you ping from Proxmox to Internet, what are all the hops (both routers and L2 devices) in the egress (request), and back through the ingress (response) direction?

If you answer the 4 questions and create a diagram I think we can get it sorted eventually.
I’ll answer what I can now, and flesh out the rest when I get home and can check the details.

1. This is a good point, I will need to add a Static Route at the gateway pointing back as well. It’s a ISP standard modem/router with its wifi and DHCP server disabled. That said, traffic isn’t getting anywhere near that far through the network yet…

2. It does not. I’ll add a return Static Route when I get home in prep for when I can get traffic that far.

3. Correct, it is a wireless to wired bridge. I was sending packets there to see where in my network I can reach, so just trying to ping it. I can ping it from my other computers.
When I try to ping it, or the gateway, or my DNS server, the output was along the lines of “address unknown” from memory. It was saying the response was from 10.0.0.2 (the ICX7150)

4. No, the desktop (192.168.0.2) is a separate computer to the NUC (192.168.0.10). The NUC is directly connected to the wireless bridge via a cat6, and the desktop is directly connected to the ICX7150, and the wireless bridge is cabled to the ICX7150 as well

I’ll draw up a diagram when I get home as well
 

TonyArrr

Active Member
Sep 22, 2021
129
66
28
Straylia
Sorry, didn't get home till very late so didn't look again until now.

Turns out your questions answered the problem, @DavidWJohnston. Added the static route to my gateway (192.168.0.201) back to the ICX7150, and now Proxmox can reach everything on my network and receive replies. Added a 0.0.0.0/0 static route to the ICX7150, and now I can ping the whole internet! Huzzah!

My next problem is that Proxmox cannot resolve domain names. I have the IP of my DNS server in Proxmox's settings, and can ping the same IP, but it will not resolve names regardless.
This might not be a huge problem since I intend to run a VM or LXC that will be just for BIND9 to host the local DNS zone, and act as a recursive server for the network, so I'll get that going and make sure it can query the net before pointing Proxmox itself at the VM/LXC...

Thanks for your help!
 

DavidWJohnston

Active Member
Sep 30, 2020
242
188
43
Ok cool so routing is working now that's good.

That's strange you can ping your DNS server but not resolve from it. Maybe try dig from the command like this:

dig 1.1.1.1 myname.local

Where 1.1.1.1 is the IP of the DNS server to use for resolution and myname.local is the name you're trying to resolve. Try yours and 1.1.1.1, 8.8.8.8.

It might be a policy to only allow resolutions from a certain subnet, not sure.

On my homelab network I use 1.1.1.1 as my DNS resolver because it has a faster response than my ISP's own DNS. For local DNS I run an MS AD domain and external I use Afraid.org FreeDNS premium. I also have an external subdomain NS pointing to my AD for certain things.

Depending on your needs, you could try something similar.

Good Luck
 
  • Like
Reactions: Aluminat

TonyArrr

Active Member
Sep 22, 2021
129
66
28
Straylia
Thanks, it was indeed a ACL on recursive queries I had set back when I set up BIND in the first place, probably 6-7 years ago.
Dig gave me the answer to that one, it helpfully reported exactly that when I upped the output verbosity.

One thing I am wondering, is there a way to deliver the network info your normally get along with DHCP leases (like routes, dns servers) to populate on hosts where you set the IP and subnet mask staticly? Or is it all or nothing?
 

DavidWJohnston

Active Member
Sep 30, 2020
242
188
43
There are mechanisms like UPnP, mDNS and RAs for IPv6, dynamic routing protocols, GPOs, etc but I don't think any of that is what you really want. DHCP and static is the way to go.

Maybe consider using DHCP reservations. For most permanent important servers, I manually set them. On transitory or pop-up infra, I rely on DHCP, sometimes with a reservation. You can also limit your subnet, so DHCP supplies IPs starting at 192.168.0.100 only. Then all your statics go below .100. When you first provision a new server, you'll get something > 100, then if it's a permanent server, re-config it to something under 100 manually.

I also push a DNS suffix via DHCP Option 015 but this only applies to Linux servers. With Windows you usually join an AD domain and receive the DNS suffix that way, or set it manually. You can also push PXE (network boot) settings with DHCP.

Also one wouldn't normally push routes via DHCP, but doing so is possible but not on all OSs. Generally you push a default gateway, then all static routes are configured on the gateway. If you need to have special routes for individual machines, you could use policy-based routing on the gateway which takes into account the source IP when making routing decisions.

Also note the unique identifier used by Windows for DHCP is the adapter MAC address, but Linux uses the value in /etc/machine-id. If you manually create a reservation you need to take that into account.
 

TonyArrr

Active Member
Sep 22, 2021
129
66
28
Straylia
I was planning on using DHCP reservations on everything “client” that I wanted having fixed addresses. Good to know about /etc/machine-id. I’d always just used whatever was printed/stickered on the NIC and haven’t had problems yet, but I’ll keep an eye on that.

By static routes, I was meaning for directing traffic for inter-Vlan routing, so that traffic that wasn’t going to leave my home wouldn’t need to go to my gateway/firewall, since several of my machines have 10GBit NICs and in my testing are able to get well over the halfway point to saturation.

I suppose I could set my Core Switch as the default gateway for each subnet, both by DHCP and in the static setups, and then set the route for each network with appropriate ACLs there, and set it’s default gateway as the gateway/firewall That should have the desired effect, no?
 

DavidWJohnston

Active Member
Sep 30, 2020
242
188
43
Yes, that is the normal way to do it - Use your L3 switch interfaces as the default gateway for all your devices, which does inter-VLAN routing, and create a default route on your L3 switch to forward everything else upstream to your next router towards the internet.

Routing decisions always use the most specific destination scope that's in the table which matches. So it will only hit the 0.0.0.0/0 (default) rule if no other table entry matches as 0.0.0.0/0 is the least specific destination scope possible. Most specific would be like 1.2.3.4/32, which forwards traffic if and only if the dest IP is exactly 1.2.3.4.

When you create an interface on a L3 switch with an IP and subnet mask, a routing table entry is automatically configured for that subnet - They are called "Directly Connected" routes.

You only need ACLs if you want to restrict some VLANs from communicating with each other.

Here is my L3 switch routing table:

Code:
$ show ip route vrf all
Codes: K - kernel route, C - connected, S - static, R - RIP,
       O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
       T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP,
       F - PBR, f - OpenFabric,
       > - selected route, * - FIB route, q - queued, r - rejected, b - backup

VRF Vrf_Prod:
S>* 0.0.0.0/0 [1/0] via 10.41.255.2, Vlan50, weight 1, 06w0d17h
C>* 10.41.0.0/24 is directly connected, Vlan1000, 06w0d17h
C>* 10.41.1.0/24 is directly connected, Vlan1001, 06w0d17h
C>* 10.41.2.0/24 is directly connected, Vlan1002, 06w0d17h
C>* 10.41.3.0/24 is directly connected, Vlan1003, 06w0d17h
C>* 10.41.8.0/24 is directly connected, Vlan1008, 06w0d17h
C>* 10.41.9.0/24 is directly connected, Vlan1009, 06w0d17h
C>* 10.41.255.0/30 is directly connected, Vlan50, 06w0d17h
C>* 192.168.7.0/24 is directly connected, Vlan107, 06w0d17h

VRF default:
C>* 10.1.0.1/32 is directly connected, Loopback0, 06w0d17h
S>* 10.41.2.0/24 [1/0] is directly connected, Vlan1002 (vrf Vrf_Prod), weight 1, 06w0d17h
S>* 10.41.3.0/24 [1/0] is directly connected, Vlan1003 (vrf Vrf_Prod), weight 1, 06w0d17h
VRFs are different "realms" which have separate routing tables.

And here is the routing table (statics only) on my upstream internet gateway:
1685130209130.png

You can see I use a transit VLAN, which is a small 2-IP subnet for point-to-point router connections. This is the best way. I could have also done 10.41.0.0/16 to cover my whole internal subnet range but I wanted them separate.

For more info, see this post where I answered a similar question: https://forums.servethehome.com/index.php?threads/noob-question-on-l3-switch-routing.39875/
 

TonyArrr

Active Member
Sep 22, 2021
129
66
28
Straylia
Thank you so much for the detail, this is really helpful.
And I was going to ask about using a Transit VLAN, cause I’ve seen it mentioned and how it is “best practice”, “standard fare” and so on, but never articulated why it was so. Your post on that previous thread makes that super clear, so I will make use of that now. One of the other commenters hit the mail on the head when they said your reply was one of the most useful and actionable responses they had had online, I’m completely agree with that characterisation :D

I’m a little curious, if you have the time to keep talking it through with me, but would I be correct in saying that inter-VLAN routing does not actually lead to a “double-NAT” effect, even with a Transit VLAN in the mix?

And in doing DHCP, how does the DHCP server on the far side of the Transit VLAN know what broadcast domain DHCP requests are coming from? Is that all part of the DHCP Helper service in the level 3 switch? (I’m currently planning to use Vyos as my gateway/firewall/router, which I believe uses ISC-DHCPd)

Finally, for DNS, I planned for that to be BIND in a small VM alongside Vyos, with a interface in each VLAN, partially so that Avahi can also run there to perform mDNS reflector duties as well. Does that still make sense in a Transit VLAN setup or should I move it elsewhere on my network? (The host is just for Vyos and BIND, all my other VMs and containers are on a different server)
 

DavidWJohnston

Active Member
Sep 30, 2020
242
188
43
I'm glad it's helpful for people, lots of us have this type of network, all with their own unique attributes. Sure I can answer your other questions.

Routing on its own does not NAT; the source IP of the packet stays original all the way through its routing hops.

For DHCP, you generally use a DHCP relay and a subnet-aware DHCP server. A relay hears the broadcast, and re-transmits the DHCP request as unicast with the source IP as itself on to the DHCP server, and that packets routes its way through the network. The DHCP server then responds back to the relay, and the relay switches the source IP back and passes the answer to the original requestor. Most L3 switches have a DHCP relay feature built-in, you can see in my screenshots that's what I'm using.

You need a subnet-aware DHCP server, and create a scope for each subnet like in my screenshots. A quick Google search reveals Vyos can support multiple subnet DHCP scopes. It's also worth noting that DHCP is generally broadcast on the request-side, but unicast on the answer-side. With DHCP relays, the request-side has a unicast segment between the relay and the server in most configurations, but multiple configurations are possible.

Your DNS server(s) can be anywhere on your network, and your routing tables will pass around DNS traffic just like any other packet. For mDNS, as you noted, you will need a listener in every subnet and a reflector to somewhere. The DNS servers delivered by DHCP could be something else. In my case, I use MS Active Directory which also provides DNS on its domain controllers (DCs) so those are my 2 redundant DNS servers delivered by DHCP. If the request needs to hit a different DNS server, the DCs just forward it along to where it needs to go, for instance public DNS 1.1.1.1.

If you have a box that straddles multiple subnets, it can introduce asymmetric routing; where the request and response traffic take a different path through the network. But using this straddling to only to handle mDNS is reasonable. Using it as a router could be problematic, so it should never be the default gateway for devices unless you understand and accept the asymmetric routing.

Hope this helps, good luck!