Brocade, how to test if L3 is working?

Caennanu

Member
May 18, 2021
78
3
8
Good day all,

As i can't seem to find an answer with some google and forum searching, i create this post.

Not all to long ago i have purchased an brocade ICX 6450-48P. I have finally managed to switch everything over to the brocade so that it can take over the L3 switching from my EdgeRouter. However, i doesn't seem to do that.
Inter-Vlan routing speeds still seem dependant on the router. (i test this with iperf between a VM, server and standalone pc, as well as with ookla local and external speedtests, and getting varying results, with V1 firmware on router Vlan hardware offloading is enabled, with V2 it doens't work, and as such i lose 300mbits between the 2 firmwares. But the v1 firmware with the proper speeds has issues with iTV, so right now i'm chosing stable iTV over upload speed).

Now the question really is. How can i test if L3 switching is enabled on a brocade?
 

kapone

Well-Known Member
May 23, 2015
1,049
621
113
Turn off router, if need be, give them static IPs/DNS, use iperf (or whatever) between two machines.
 

Caennanu

Member
May 18, 2021
78
3
8
ah right, so disable DHCP, set static ip and do the testing. See if i can connect to another vlan.
No other, less invasive, way?
 

Caennanu

Member
May 18, 2021
78
3
8
setting static ips in itsself is not. disconnecting the router with its dhcp is . . .
couple vm's that will die, CCTV system that will die, internet that will die, and that is obviously the worst. not that internet dies, but that that the miss can't use it ;)
 

Caennanu

Member
May 18, 2021
78
3
8
So i found a video explaining how to enable router mode. Which is done by loading the secondary flash.

As i probably am booted to primary ... Will booting to secondary 'reset' the port configs already done?
 

Blue)(Fusion

Active Member
Mar 1, 2017
124
40
28
Chicago
It may or may not have the L3 image in secondary flash. It may be in both or it may be in none. If you follow the guide by @fohdeesha in the OP, you will have copied the latest L3 firmware into primary flash. You can copy it to secondary flash as well if you wish.

Look at the ouput of show version and show flash.

The important part:
Code:
SSH@ks-sw-01#show ver
  Copyright (c) 1996-2016 Brocade Communications Systems, Inc. All rights reserved.
UNIT 1: compiled on Apr 23 2020 at 13:17:12 labeled as FCXR08030u
(10545591 bytes) from Primary FCXR08030u.bin
Note the FCXR in the version. That is the Router firmware.

Here is an example of my ICX6450 running the L2 firmware:
Code:
SSH@ks-icx-02#show ver
  Copyright (c) 1996-2016 Brocade Communications Systems, Inc. All rights reserved.
UNIT 1: compiled on Apr 23 2020 at 10:57:26 labeled as ICX64S08030u
(8563580 bytes) from Primary ICX64S08030u.bin
Note the S in the firmware image name/version.

The firmware version and type (L2 vs L3) can be determined with the show flash command.

In this example, my ICX6450 has the same L2 firmware installed in both primary and secondary flash (again note the S in the .bin file name):
Code:
SSH@ks-icx-02#show flash
Stack unit 1:
Compressed Pri Code size = 8563580, Version:08.0.30uT311 (ICX64S08030u.bin)
Compressed Sec Code size = 8563580, Version:08.0.30uT311 (ICX64S08030u.bin)
Compressed Boot-Monitor Image size = 786944, Version:10.1.05T310
Code Flash Free Space = 32489472
 

Caennanu

Member
May 18, 2021
78
3
8
@Blue)(Fusion now this is the information and guidance i was hoping to get! Thanks a ton! Will get on it ASAP.

And yes, i followed that guide. But unfortunately the follow-up guides are 'comming soon'
 

dswartz

Active Member
Jul 14, 2011
529
55
28
setting static ips in itsself is not. disconnecting the router with its dhcp is . . .
couple vm's that will die, CCTV system that will die, internet that will die, and that is obviously the worst. not that internet dies, but that that the miss can't use it ;)
WAF => Wife Acceptance Factor
 

Caennanu

Member
May 18, 2021
78
3
8
@Blue)(Fusion
So finally got around to accessing the pooter.

Code:
Stack unit 1:
  Compressed Pri Code size = 9871112, Version:08.0.30uT313 (ICX64R08030u.bin)
  Compressed Sec Code size = 8526668, Version:08.0.30kT311 (ICX64S08030k.bin)
  Compressed Boot-Monitor Image size = 786944, Version:10.1.05T310
  Code Flash Free Space = 32481280
And if i'm getting you right. Primary code for me is router, secondary is switch.

Than comes the question. Why is my WAN router still doing the routing? What do i need to add in the brocade to actually start routing.
 

klui

Active Member
Feb 3, 2019
431
181
43
How are you determining your router is performing routing duties instead of your switch?
 

Caennanu

Member
May 18, 2021
78
3
8
@klui in my EdgeRouter 12, i see traffic for Vlan increasing almost equal to the ipv4 traffic.
I've learned to look at this, since hardware offloading for Vlan's on my ER12 doesn't work with V2 firmware, but my iTV doesn't work with V1 firmware. So thats why i'm trying to offload everything to the brocade, so i can have good iTV and full internet speeds :p
1636962456985.png

And i'm thinking, i have probably missed this part in the configuration of the brocade:

And this video also talks about the configuration of the VE.

Now i'm wondering, if i assign an VE to the Vlan's, and give them an ip address. but the DHCP for it is on my WAN router. will it actually work? Because when i do a similar thing on my EdgeMax router, where i add vlan's to interfaces 7 - 11, they basically kill all Vlan traffic as soon as i assign an ip address to them.
 
Last edited:

Blue)(Fusion

Active Member
Mar 1, 2017
124
40
28
Chicago
I think you have a misunderstanding of how L3 routing occurs. Dropping in an L3 switch does not make L3 routing occur on the switch. Each VLAN must have a virtual interface (VE) and assigned an IP (IPv4/Pv6 as required), and that virtual interface IP becomes the Default Gateway on each VLAN device.

This requires manual intervention in DHCP (or static IP/routes) to ensure the proper gateway is assigned.
 
  • Like
Reactions: fohdeesha

Caennanu

Member
May 18, 2021
78
3
8
Allright, no i did not know that.
So let me get this straight to learn.

Right now my er12 hosts the vlan, the dhcp for each vlan and the vlans have an static ip assigned on the switch interface. I have Them tagged or untagged depending on the ports.

By assigning a ve to each vlan on the l3 router and assigning an static ip, they will become the gateway for everything behind that l3 router.

When a new device enters the vlan, the l3 router will forward the dhcp request to the er12?

When an device with ip from vlan2 wants access to vlan3 for which the l3 router has an ve and static ip, it wont access the er12 and route by itsself, this includes the vlan internet is on, say vlan1. It will forward the dns request to the er12 on vlan1 which will forward to the isp.

Or will the dns request stay on the original vlan untill it reaches the er12 and then get forwarded?

Or, because i assigned the vlans on the er12 an static ip, they are the default gateway and i will need to remove those before the l3 router can become the gateway?
 
Last edited:

Blue)(Fusion

Active Member
Mar 1, 2017
124
40
28
Chicago
A few important points:
  • No switch or router "hosts" a VLAN. A switch or router may be aware of a VLAN and allow certain VLAN traffic to pass. A router (L3) must have an IP on any VLANs you wish that router to route between VLANs. For the purposes of this post, I am using L3 switch and router interchangeably.
  • Default gateways on IPv4 are certainly not automatic. You must configure your DHCP server to assign the IP used on each respective VE interface as the default gateway.
  • Ideally, you will use a DHCP server that is capable of being subnet aware (i.e. ISC-DHCP). I am not sure if the EdgeRouter DHCP server is capable of that as I have never used it.
  • Ideally, you will not transport most (if any) of the VLAN traffic to the L3 switch. You want presumably only internet traffic going to the EdgeRouter device. In this scenario, your EdgeRouter would have no VLANs configured on it.

Here's an example:

Code:
10.0.1.0/24     10.0.2.0/24      10.0.3.0/24
VLAN1              VLAN2              VLAN3
   \                 |                 /
    \                |                /
10.0.1.1         10.0.2.1       10.0.3.1
    --------------------------------
    |          ICX SWITCH          |
    --------------------------------
                 10.1.0.2
                    |
                    |
                 10.1.0.1
             ----------------
             |  EdgeRouter  |
             ----------------
                    |
                    |
                Modem/ISP
The IPs on each VE in this example are 10.0.x.1 and each VLAN is assumed to be a subnet of /24 (255.255.255.0). A device in VLAN 3 needs to have a default gateway of 10.0.3.1 in the example above. Only then will traffic destined to any network that is NOT 10.0.3.0/24 will the L3 routing of the switch do it's job and figure out where to send it based on it's own routing table. And that routing table will include all of the other VLAN networks. What about internet traffic? That goes to the "default" route on the switch which should be 0.0.0.0/0 via 10.1.0.1 (the LAN IP of the EdgeRouter) in this example. This sends any traffic that is not directly routed on the ICX switch to the EdgeRouter, which then likely sends it to the internet - unless you have additional networks configured on the EdgeRouter for, for example, a homelab setup that you don't want affecting your main home network incase you eff it all up.
 
Last edited:

Caennanu

Member
May 18, 2021
78
3
8
@Blue)(Fusion Learning more every day, and i appreciate that you take your time to explain this to me.

Right so, i get this, mostly. However, there is a little caveat.
The EdgeRouter has an SFP, that is directly connected to my ISP.
The connection type is PPPOE, using Vlan 6 (external) for Internet and Vlan 4 (external) for IpTV.
Currently, the edgerouter routes Vlan 6 with internet to Vlan 1 on the EdgeRouter, and Vlan 4 external to Vlan 4 internal, both with their local DHCP.

In your diagram i would then simply untag range 10.1.0.1. and tag the range for vlan 4 to the switch no?

Then the next question. if there is no Vlan 1 - 3 connection towards the EdgeRouter. I cannot possibly serve their DHCP there can i?
And if i cannot serve their DHCP there, i also cannot use the NAT, Firewall rules etc. there either?
This is something i'd like to keep on the EdgeRouter, else it will be a pretty useless piece of equipment :p

This is roughly the network i had in mind. Where i would offload as much routing to the Brocade as possible.
1637048601157.png
The reason for this is simple. I was able to get an L2/L3 switch with 10GB functionality and Power over Ethernet, for less than 100 bucks. It just so happens to have 48+4 ports, while i could do with 12.
 
Last edited:

Caennanu

Member
May 18, 2021
78
3
8
p.s. while i await response, i'm watching youtube and brushing up my information ;)

and i think the dhcp on my ER-12 is subnet aware, due to the /24?
1637067321558.png

So . . . i gotta configure a DHCP relay on the brocade?
Would this be the UDP helper? (as i'm assuming right now, that DHCP broadcast requests are udp)
And if so. how would i configure this? Select the port where the dhcp hides behind, and state the ip of the dhcp?

(screenshot of webinterface, as it makes me understand better than clean text)
1637074668484.png
 
Last edited:

LodeRunner

Active Member
Apr 27, 2019
225
98
28
Where DHCP addresses are served from has no effect on whether or not NAT works, so long as NAT has valid addresses to use.

On the DHCP side, you'll need a DHCP server capable of having multiple pools if you want to serve DHCP into more than one VLAN. I don't know how the ER-12 handles multiple DHCP pools, if they can be attached to the same interface or if you must define a physical or VLAN interface on the ER-12 for each one. You might be better served running DHCP from a container on unRAID.

I don't bother with the web UI, so at the command line, you'd do something like the following:
Code:
SSH@core#conf t
SSH@core(config)#int ve4
SSH@core(config-vif-4)#ip helper-address 1 192.168.0.18
SSH@core(config-vif-4)#ip helper-address 2 192.168.0.19
VE4 is the VLAN interface for VLAN4 on my network. .18 and .19 are redundant DHCP servers, hence indexed as 1 and 2; if you have only one, you still need to pass the index position. You would repeat this for each VLAN interface.

How the DHCP pools are configured to deal with this is platform specific, I used to do it with Windows. I flattened my network because VLANing created WAF problems (uPNP required by many games we play together and my switch at the time did not support multicast routing; mDNS isn't routable so I had to maintain an avahi-proxy which was flaky).
 

Caennanu

Member
May 18, 2021
78
3
8
@LodeRunner thanks for the reply.
At this moment i have the er12 setup to already serve multiple dhcp pools, so that should be good.

I apply the vlans with a tag / untag per port on the switch interface.

Ok, so when i configure the helper. I point to the router vlan1 address or to the address that is currently set as gateway on the er12 for each specific vlan?
 

LodeRunner

Active Member
Apr 27, 2019
225
98
28
Assuming your ER-12 has a single internal facing IP, then that is the IP you would use as the helper address.