Brocade Inter-VLAN routing on some but not all? Help

[BoGs]

I have recently been thinking about splitting up my main POE switch into multiple VLANs and I am trying to figure out how to do what I have in my head. I am trying to do intervlan routing on some but not all the vlans. The main point here is that I want VLAN 30, 40, 50 to go upstream to the router to handle in ip firewall rules. I do not allow any of vlan 40 to anything else and vlan 30 and 50 have very limited access patters to each other.

I only want the default vlan(1), 10 and 20 to be routed at the switch. Even if a computer on IoT does scan on vlan 10 would need to go to the router.

Here is what I have so far but I am unsure if a device from non intervlan vlan requests a device if the switch will route it or send it to the router. This is what I have in my show ip route currently before setting up vlans.

Total number of IP routes: 2
Type Codes - B:BGP D:Connected O:OSPF R:RIP S:Static; Cost - Dist/Metric
BGP  Codes - i:iBGP e:eBGP
OSPF Codes - i:Inter Area 1:External Type 1 2:External Type 2
        Destination        Gateway         Port          Cost          Type Uptime
1       0.0.0.0/0          192.168.1.1     ve 1          1/1           S    9m14s
2       192.168.1.0/24     DIRECT          ve 1          0/0           D    9m15s

vlan 10 name Home
untag ethernet 1/1/1
router-interface ve 10
interface ve 10
ip address 192.168.10.5/24

vlan 20 name Servers
untag ethernet 1/1/2
router-interface ve 20
interface ve 20
ip address 192.168.20.5/24

vlan 30 name IoT
untag ethernet 1/1/3

vlan 40 name Work
untag ethernet 1/1/4

vlan 50 name Cameras
untag ethernet 1/1/5

Thanks in advance.

 

[kpfleming]

Well, it would be useful to use the usual terminology... there is no 'requesting a device' involved For IPv4, ARP is used to find hardware (MAC) addresses which correspond to IP addresses, and ARP is a layer 2 protocol so it does not cross VLANs.

The difference between routing VLANs and non-routing VLANs is the presence of a VE on that VLAN. A VLAN without a VE will not participate in routing (layer 3), because it cannot do so without having an address on the ICX (which is what a VE provides). If a VLAN does not have a VE, the ICX will not respond to ARP queries on that VLAN.

Given that, devices connected to VLANs 30, 40, and 50 will either not have access to other subnets (routing), or will, depending on whether they have been given a default gateway address which they can reach (which would have to be an address on your router). The router will need to have VLAN subinterfaces on VLANs 30, 40, and 50 in order for it to be able to handle traffic on those VLANs.

 

[BoGs]

Awesome thanks so much appreciate it a bunch. Already have the VLAN subinterfaces on my opnsense as it does all the routing and maxes out around 2.5G trying to bypass that for the bandwidth heavy machines.

 

[sic0048]

Already have the VLAN subinterfaces on my opnsense as it does all the routing and maxes out around 2.5G trying to bypass that for the bandwidth heavy machines.

So you have set up your OPNsense device to handle the VLAN management which means your network switch is functioning as a managed layer 2 device. It's "VLAN aware", but not functioning as a layer 3 device. Honestly this is how I would recommend a typical home network to be set up unless you are an IT professional that regularly deals with setting up network switches as layer 3 devices.

As noted, traffic traveling on the same VLAN is handled by the switch (ARP level) regardless of where the VLAN management is handled (on router or on switch). Therefore if you plan your VLANs properly, it should be easy to keep bandwidth heavy traffic on the same VLAN and therefore help prevent the network bottlenecks that can occur when the firewall/router is handling VLAN management/routing.

In other words, even through your switch is functioning as a managed layer 2 device, if you need to do some very bandwidth intensive data transfers (perhaps you are a video editor and need to move large files between your editing machine and a series of backup storage devices), all you need to do is ensure those devices are on the same VLAN. Doing this simple step will ensure that the traffic will be routed at the switch level and not have to travel out to the router and back like it would if the data was passing between two different VLANs.

 

[BoGs]

Why do you say that? I was looking to setup some basic VLAN which is really L2 aside from inter vlan routing. I could probably merge the VLANs if I wanted but overall just like the setup makes it easy to see what computer is where and what permissions the LAN has.

Is it because of security holes? or too difficult to set it up properly? I am not really looking into setting up ACLs and such. I do not really worry about it until dealing with large files from desktop to truenas server.

Personally love the domain (internal) that I can run for all the vlans and connect to computers using the hostname it is a joy instead of trying to remember all IP addresses

 

[Blue)(Fusion]

Oh dear, I think you're missing some necessary steps.

For starter, I'm going to define my terms here to avoid confusion.

ER= Edge Router, whatever router+firewall device/software you are using to reach the WAN/Internet and enforcing firewall policies.
ICX = the Brocade ICX switch. Note: once you are using the L3 functionality of the switch, it is, by definition, also a router.

So.....
You desire the ICX to route traffic between VLAN 10 and 20 and all other traffic go to the ER, got it. You need to tell any devices connecting to VLANs 10 and 20 to use the appropriate ICX VLAN IP address (192.168.10.5 and 192.168.20.5 respectively) as their default gateway. You likely have a DHCP server configured on your ER and it is supplying the ER's address (on the respective VLAN) as the default gateway. This is desired behavor on VLANs 30, 40, and 50 but NOT 10 or 20. If you can not modify your DHCP server to give this alternate default gateway address, you will either need to use a different DHCP server or statically set the default gateway on each device on those VLANs (impractacle).

Oh this can also be another issue...the ER should NOT be aware of VLAN 10 or 20. DO not configure VLAN 10 or 20 on the ER. You can still add firewall rules using the appropriate subnets if you desire to for reaching the WAN through the ER on the interface that can directly access the ICX.

Okay, so you have that part done. You also need to make sure routing is correct in both directions between the ICX and the ER. You can automate this with OSPF or RIP if both ER and ICX support it or you can statically route. The ICX, you already have configured corectly - default gateway is your ER. Make sure you have a route that sends traffic for the appropriate subnets from the ER to the ICX or else the ER will not know where to send packets destined for those networks.

On the ER you must have these routes:

192.168.10.0/24 via ${ICX IP ADDR}
192.168.20.0/24 via ${ICX IP ADDR}

${ICX IP ADDR} should be an IP address that is directly accessible on a VLAN interface of of the ER. Probably whatever the IP address you use in the default VLAN and should be on the same subnet as the ER.

EDIT TO ADD:
Don't forget about doing the same routing setup for your IPv6 address space as well (Get a /56 from your ISP and each VLAN should have it's own /64).

In my case, my ER has no VLANs configured on it and all LAN subnets are routed on the ICX, so it's easy for me to not use any VLANs between the ICX and the ER. Specifically, I just set the LAN port on my OPNSense box to a private /30 (2 usable IPs) 10.254.254.1/30 and the port that it is plugged into as 10.254.254.2/30. The routers above would look like 192.168.1.0/24 via 10.254.254.2 in my example case.

For your case, whatever IP you initially set up and access the ICX through from your ER, you should probably be using that as the gateway for those routes.

 

[BoGs]

Thanks so much for taking the time to write this all @Blue)(Fusion appreciate the explanation. Time to tinker. I am not a network guy as you can tell, more on the software side of things with k8s

 

[sic0048]

Why do you say that?

Not sure exactly what "that" is referring to, but I am going to guess it was my statement that I would recommend that most people set their home networks up as "router on stick" configurations (like you have set up) and not attempt to set their switches up for true layer 3 functionality unless they are already experienced with these setups/techniques.

I say that because for most residential situations, a non-IT person trying to learn how to properly set up layer3 switches using a live network is going to be a bigger headache than it is worth. The big disadvantage to running a "router on a stick" configuration is the potential data bottlenecks when data has to traverse to the router. But that can be greatly mitigated simply by carefully planning your VLAN to minimize the situations where data has to traverse to the router.

I mean it's great if a non-IT person wants to set up a test environment to learn about layer 3 switching. I think that is a great way to learn new techniques and I definitely recommend doing that. That person may eventually feel confident enough to change over their home network to utilize layer 3 switching. But I wouldn't recommend a person try to jump into learning about layer 3 switching with their main/production home network. That is a recipe for disaster IMHO and is likely to cause a lot of frustration and downtime before they finally get it working correctly or decide to throw in the towel go back to the "router on a stick" design.

And just to note, the fact that you use OPNsense and a Brocade switch actually makes setting up layer 3 switching harder than some other situations. I say that because the pfSense/OPNsense software doesn't allow for a DHCP server except on gateways defined in their software. If you are moving the gateways to the switch for layer 3 functionality, this means you cannot use DHCP services on the router for those network segments. Of course the switch has DHCP functionality, but the DHCP server in many of the Brocade switches doesn't act as an "authoritative" DHCP server. This means there are some devices that won't be able to get a DHCP ip address assigned to them by the switch's DHCP server and you'll have to find an alternative solution in those situations. These are not insurmountable issues, but it isn't the "easiest" system to try to learn layer 3 switching on either.