Brocade, how to test if L3 is working?

Blue)(Fusion

Active Member
Mar 1, 2017
124
40
28
Chicago
Again, I am not familiar with the EdgeRouter software including the DHCP server's capabilities. Based solely on the image you showed, it will not fit your purposes. It only will offer IPs in a single subnet range (one /24). You need to investigate if you can add additional subnets on the DHCP server. One subnet per VLAN is what you will need - the size of each subnet is up to you (whether /24, /16, /28, etc.).

Regarding the VLANs you described, only the WAN side of the EdgeRouter should have VLANs configured based on you ISPs requirements. Make things easy and avoid VLANs on the LAN port(s) of your EdgeRouter.

Another point, you need to use the command line for the ICX switch to learn and to effectively manage it. The web interface is hard to navigate, sometimes buggy, and not very helpful in understanding what you are doing. 9 times out of 10 you will see other members here giving you advice in the form of text commands usually because we never use the web interface.
 

Caennanu

Member
May 18, 2021
78
3
8
@Blue)(Fusion thanks again for the reply.
As the following image will show, the ER-12 is capable of offering more than 1 DHCP.
1637091127621.png
As you can see in the picture, it is already running 5 DHCP services, each with their own range / subnet (192.168.Vlan#.0/24)

If i do not configure the vlan's on the EdgeRouter, i wouldn't know currently how to add Hairpin Nat (i believe port forwarding rules are an DHCP function no?) or where to add firewall rules. The Brocade doesn't seem to offer such functionality, unless it is commandline only. And to be frank, i do not find myself experienced enough for command line.

And yes, i am aware people will tell me to do this and that on the command line. The thing is, i can't really find easy to digest documentation to learn me all its features, how they are called or when to use them. When people say, do this on the command line, it will create this and that, i will figure it out what it means and can fairly easily get the hang of it (i'm an autodidact, i learn by doing). As such, the guides made by fohdeesha were great. Just lacking the parts that i need currently.

When it comes to the Vlan on ER-12 part, i don't think i can really escape it, as i will need atleast 2. 1 for internet and the other for iTV. (yes i'm keen on keeping them seperate).
 

Caennanu

Member
May 18, 2021
78
3
8
@LodeRunner right. not sure what you mean with that . . .

But let me try and figure this out.

So, on the ER, i have 5 Vlan's. ranging 1 to 5.
Vlan 1 is my actual internet connection. 2 - 5 have to route to 1 to actually get internet.
Let's say i use port 0 on the ER-12. I untag Vlan 1 (where the router has 192.168.1.1 as ip) on that and tag 2 - 5.
And now i do the same on port 1 of the brocade. i untag Vlan 1, and tag 2 - 5. (after first 'creating' them ofcourse . . .)
I will now have an 'open' internet connection to the Brocade switch. Acting as a 'dumb' switch. (this is current situation, EdgeRouter does the routing)

I then create a VE on Vlan 1 on the switch and assign it 192.168.1.2. This will enable routing on that switch.
I then assign vlan 1 to port 2 as untagged. The device on this port will contact the Brocade switch to try and get an DHCP, but it won't.
So i configure the dhcp helper. With a line similar to ip helper-address 1 192.168.1.1. as this is the ip of the EdgeRouter in that Vlan.
Now when the client on port 2 of the brocade requests an ip from the DHCP, the brocade switch will relay that to the EdgeRouter on Vlan 1. Vlan 1 has DHCP coupled, it will reply and the client will get an IP. Right?

Now, if i got this right. Next i will create vlan 2 on the brocade switch.
I will create a ve on Vlan 2, with ip 192.168.2.2 (as 2.1 is taken by the EdgeRouter).
Then i create the DHCP helper, this time using address 192.168.2.1 (EdgeRouter ip on that Vlan).
And the Brocade switch will relay any DHCP request to the EdgeRouter for that specific Vlan. Correct?
 

klui

Active Member
Feb 3, 2019
431
181
43
There's some good documentation on L3 configuration from Ruckus. Start with Technical Documents | Ruckus Wireless Support. Choose your product (Ruckus ICX 6430 and 6450 Campus Switches). You'll be shown some "web 2.0" search results but click on the partially hidden left-most box behind the upper left hand corner of the "Search" field to get a better presented list of all documents. Download the last "FastIron 08.0.30r Software Documentation (.zip)" documentation bundle. Inside the zip file is the layer 3 routing guide.

1637092915189.png
 

Caennanu

Member
May 18, 2021
78
3
8
@klui thanks for the reply.

ok, got that file.
Lots of information on a lot of pages.
Which page should describe what i need? cause i'm lost . . .
 

LodeRunner

Active Member
Apr 27, 2019
225
98
28
In generic terms your setup to have the switch do all the L3 routing would be as follows:
  • Each VLAN has a defined interface, e.g. "router-interface ve <VLAN #>"
  • Each VE has an IP assigned
  • There is a transit VLAN between the switch and firewall; using a /30 (2 usable addresses) keeps things simple
  • The switch has a default route to the firewall, e.g. "ip route 0.0.0.0/0 <FIREWALL IP>"
  • The firewall has one or more routes for your VLANS set, using the transit VLAN IP of the switch; if all your VLANS are using 192.168.x.y/24 for addressing, you could make it a single route to 192.168.0.0/16 so long as your transit VLAN is not using 192.168 addresses.
  • The firewall does not need any awareness of the VLANS; the switch is handling everything at L3 and routing to the firewall. VLANS are a L2 thing.
  • Unless trunking multiple VLANS on a single interface, your setup has no need for any VLAN tagging, so all ports should be untagged to the desired VLAN. This simplifies things as it means not having to touch each device and configure VLAN tagging (assuming said device even supports it)
Sloppy diagram:
1637097449698.png

As far as commands on the switch, you can always press "?" and it will list the commands available in your current context. In the configuration terminal ("conf t") end exits configuration mode entirely, exit leaves the current context and moves up one level. Commands can be shortened to the minimum characters required to be unique, thus "int" works when you need to use the "interface" context.

For example, to modify a VLAN and define a VE as I described:
Code:
conf t
VLAN 1
router-interface ve 1
int ve 1
ip address 192.168.1.1 255.255.255.0
end
To set the default route:
Code:
conf t
ip route 0.0.0.0/0 172.31.255.1
end
All your DHCP pools are on the ER-12, so going by my quick layout, your helper statements would be as follows:
Code:
conf t
int ve 1
ip helper-address 1 172.31.255.1
int ve 2
ip helper-address 1 172.31.255.1
< ... and so on ... >
end
When done making changes, execute "wr me" to write the changes to the startup file, or your work will be lost upon switch restart (handy if you fat finger something; just pull power and you'll be good to go).

Quite frankly, unless you have a pressing security or configuration need for this, or are doing it for lab experience to apply to a work environment, I would not do routed VLANs in a home environment. I did it before and it broke a number of things, particularly uPNP/AT-PMP and mDNS, because enterprise switches normally are not aware of and do not care about these protocols.
 
Last edited:

Caennanu

Member
May 18, 2021
78
3
8
@LodeRunner thanks for the reply.
This really helps and was what i was looking for, i think.

So basically, the only thing i need to do on the ER12 is have 1 port with ip 172.31.255.1/30 assigned. It won't needs its DHCP pool as we are just manually assigning it. And give 1 port on the brocade the 172.31.255.2/30 address. connect them together and continu with the rest of the configuration.

As for trunking, i need trunking for my acces points, LGS318P Switch, and the unraid server. i don't want my game servers or CCTV setup to interfere with my Home LAN. And for SURE i don't want new IOT devices on any of them, at all.

Ill give it a shot, as soon as the WAF is acceptable :p
 
Last edited:

LodeRunner

Active Member
Apr 27, 2019
225
98
28
@LodeRunner thanks for the reply.
This really helps and was what i was looking for, i think.

So basically, the only thing i need to do on the ER12 is have 1 port with ip 172.31.255.1/30 assigned. It won't needs its DHCP pool as we are just manually assigning it. And give 1 port on the brocade the 172.31.255.2/30 address. connect them together and continu with the rest of the configuration.

As for trunking, i need trunking for my acces points, LGS318P Switch, and the unraid server. i don't want my game servers or CCTV setup to interfere with my Home LAN. And for SURE i don't want new IOT devices on any of them, at all.

Ill give it a shot, as soon as the WAF is acceptable :p
Right so you'd do VLAN tagging on the ports for the APs, downstream switch, and unRAID; since it's a 6450, you might need to set 'dual-mode' on the ports, but I have no direct experience with that model or the OS version that it runs; I have 7150 and 7450 switches running v 8.0.95; dual-mode was removed in v8.0.60 and forward (or maybe earlier, but the 6xxx series can only run up to 8.0.30).

VLANing off the IoT devices doesn't really change your security posture unless you're setting strict firewall rules. Which you could just as easily do on your main interfaces using DHCP reservations or static IPs + aliases for the rules. I'm at a loss as to how game servers would interfere with the LAN as well, unless you allow 3rd party remote access to them for administrative reasons. The VM I run game servers in has it's own interface to the firewall because I do allow a small, trusted user pool to connect by VPN and log into the management interfaces, so it's off in its own little world with some pretty strict firewalling between that interface and my LAN.

CCTV I would consider VLANing, depending on how chatty the devices are; otherwise, again, strict firewall rules do the job.

And uPNP on my firewall is restricted to specific IPs of gaming computers, so other devices can't just add their own NAT entries.

I do have an advantage that my firewall is a VM, so I can just add arbitrary interfaces to it with VLAN tagging and the inter-VLAN traffic is so low that having the firewall handle routing is not a problem.

You'll need to set ACLs on the switch as well, otherwise any VLAN can reach into any other VLAN. I have not configured ACLs on a ICX unit before, so the manual or other forum goers will have to help you out there.
 

Caennanu

Member
May 18, 2021
78
3
8
Yeah, the Dual mode i already figured out. i have that set up for my current setup. was a bit of a puzzle but managed.

New IoT devices and new game servers, i prefer to have on seperate networks untill i know what they are doing. after that i will 'migrate' them to one of the permanent vlan's, depending on its usecase, and add security to the ip's specifically. Currently have a temp Vlan setup, that i only enable when i'm hooking up new devices. That Vlan cannot talk anywhere but inside its own Vlan, i also use a repurposed old laptop that i can spool a new install / image on. (yes yes, maybe a little too paranoid for an home user, but i like to work this way).

Alright, good to know about the ACL's, ill have to look into that too then being taking the plunge.
First things first. time to write out a game plan :D
 

Caennanu

Member
May 18, 2021
78
3
8
@LodeRunner
So i started doing some prep work, figured i could atleast prime the ports i want to start using for this setup.
However . . . i seem to be unable to assign the port an ip address. any idea?

Code:
SSH@10g48p#config t
SSH@10g48p(config)#int e 1/2/1
SSH@10g48p(config-if-e10000-1/2/1)#ip address 172.31.255.2
Invalid input -> address 172.31.255.2
Type ? for a list
SSH@10g48p(config-if-e10000-1/2/1)#ip address 172.31.255.2/30
Invalid input -> address 172.31.255.2/30
Type ? for a list
SSH@10g48p(config-if-e10000-1/2/1)#
When trying this on the webgui, it gives me the message 'invalid port number'. Maybe that helps?

--edit--
Or do i need to add this ip to the Vlan 1 interface, and only untag this port to it?
 
Last edited:

LodeRunner

Active Member
Apr 27, 2019
225
98
28
Use Tab or ? at the end of a command to see if there are more parameters you need to supply (which is what the error message is telling you to do). It's possible that version of the OS only allows you to enter an IP using the format "172.31.255.2 255.255.255.52" rather than the easier CIDR notation. While 8.0.95 support CIDR notation, you'll see in my code example, I wrote out the subnet.

As you are using VLANs, for consistency's, set IPs on the VLAN interface, not physical ports. This also means that should you need to move a cable, you simply update the VLAN membership and call it a day.

What is output of "sh ver"? If you're not running the routing firmware, that will also cause you grief (SPR firmware vs SPS).
 

Caennanu

Member
May 18, 2021
78
3
8
I have used the ?, Didn't tell me anything. I van use the command just fine to set IT for the vlan interface :s

I am using the router firmware, i checked that.

Yeah ill Just add it to vlan then
 

LodeRunner

Active Member
Apr 27, 2019
225
98
28
I just double checked against a port on 8.0.95 and "ip" followed by ? shows a command list and "address" is not on that list. So yes, you can only attach an IP to the VLAN ve interface.
Code:
SSH@core(config-if-e1000-1/1/32)#ip 
  access-group                     Apply IPv4 access control list
  add-host-route-first             Add host route before sending buffered
                                   packets
  arp                              Set ARP option
  arp-age                          Set ARP aging period
  as-path                          Set BGP AS Path filter
  bootp-use-intf-ip                Use incoming interface IP as source IP
 

rnavarro

Active Member
Feb 14, 2013
181
28
28
A few important points:
  • No switch or router "hosts" a VLAN. A switch or router may be aware of a VLAN and allow certain VLAN traffic to pass. A router (L3) must have an IP on any VLANs you wish that router to route between VLANs. For the purposes of this post, I am using L3 switch and router interchangeably.
  • Default gateways on IPv4 are certainly not automatic. You must configure your DHCP server to assign the IP used on each respective VE interface as the default gateway.
  • Ideally, you will use a DHCP server that is capable of being subnet aware (i.e. ISC-DHCP). I am not sure if the EdgeRouter DHCP server is capable of that as I have never used it.
  • Ideally, you will not transport most (if any) of the VLAN traffic to the L3 switch. You want presumably only internet traffic going to the EdgeRouter device. In this scenario, your EdgeRouter would have no VLANs configured on it.

Here's an example:

Code:
10.0.1.0/24     10.0.2.0/24      10.0.3.0/24
VLAN1              VLAN2              VLAN3
   \                 |                 /
    \                |                /
10.0.1.1         10.0.2.1       10.0.3.1
    --------------------------------
    |          ICX SWITCH          |
    --------------------------------
                 10.1.0.2
                    |
                    |
                 10.1.0.1
             ----------------
             |  EdgeRouter  |
             ----------------
                    |
                    |
                Modem/ISP
The IPs on each VE in this example are 10.0.x.1 and each VLAN is assumed to be a subnet of /24 (255.255.255.0). A device in VLAN 3 needs to have a default gateway of 10.0.3.1 in the example above. Only then will traffic destined to any network that is NOT 10.0.3.0/24 will the L3 routing of the switch do it's job and figure out where to send it based on it's own routing table. And that routing table will include all of the other VLAN networks. What about internet traffic? That goes to the "default" route on the switch which should be 0.0.0.0/0 via 10.1.0.1 (the LAN IP of the EdgeRouter) in this example. This sends any traffic that is not directly routed on the ICX switch to the EdgeRouter, which then likely sends it to the internet - unless you have additional networks configured on the EdgeRouter for, for example, a homelab setup that you don't want affecting your main home network incase you eff it all up.
This is such a great explanation and drawing.

Kudos!


I wish these ICX switches had better support for dynamic IPv6 prefixes handed out from the ISP.

I was able to get IPv6 routing all working on mine but I had to hard code the prefixes, who knows when that'll break :(
 

Caennanu

Member
May 18, 2021
78
3
8
@LodeRunner so . . .
The misses is to work, and i have some time to tinker with the network.
So . . . i assigned every ve an ip and helper ip.
1637568736045.png
Ve 2 in this case is going to be my uplink to the ER.
To double check everything is configured, i ran the command on every VE to make sure they all have the helper address set.
1637568789605.png
I also set the default ip route. when requesting the info on this however, i'm getting the following:
1637568997134.png

time to . . . disable the old uplink from the ER-12, and see if the new uplink takes over.
 

Caennanu

Member
May 18, 2021
78
3
8
well . . . .
it took over, but speed is . . . near to non existent :p
was getting 5/5mb/ps with a ping of . . 300+, both interally and externally.
needles to say, i pulled the power plug and reset the config for now.

Now i'm geussing, this all was because i need to add a static route on my ER-12. but not sure.
This is the option i have in the ER-gui.
1637574212758.png
 
Last edited:

LodeRunner

Active Member
Apr 27, 2019
225
98
28
Gateway is the correct option, since your transit network is in a different RFC1918 space, you should just be able to do
Code:
Destination: 192.168.0.0/16
Next hop address: 172.31.255.2
 

Caennanu

Member
May 18, 2021
78
3
8
Gateway is the correct option, since your transit network is in a different RFC1918 space, you should just be able to do
Code:
Destination: 192.168.0.0/16
Next hop address: 172.31.255.2
Thanks. Hopefully i can do that tommorrow. Feels weird tho. Having a destination but no source :p terminology i geuss!
 

Caennanu

Member
May 18, 2021
78
3
8
@LodeRunner
So i finally got around to setting the static route and 'redoing' the configuration of the brocade.
But, when i do the 'switch over', there's little to no data passing the brocade. everything behind it becomes irresponsive.

The only thing i have noticed, is that there's another route being automatically configured in the ER for the 172 address. but instead of 172.31.255.2 its configured for 172.31.255.0.

So . . . what am i missing here?

All VE's have an IP
Default route 0.0.0.0/0 set to 172.31.255.1 (as i asume 172.16.31.255.1 was a typo?)
all helper addresses set to 172.31.255.1
 

LodeRunner

Active Member
Apr 27, 2019
225
98
28
Yeah, that address having 5 octets is certainly a typo. I should write for TV: IP 461.994.370.1, am I right?
Does the ER12 automatically add routes for each defined subnet?

Post output of "sh run" in a spoiler here. It all sounds like it's setup like I would expect. I have a 7150 that I keep as a spare, I'll try to lab this out later. Syntax should be mostly portable to the 6450 (I think the only exception would be dual-mode for trunk interfaces)

Edit: also from a device in a VLAN with a static IP, what's the output of say, "traceroute -d 1.1.1.1"?