Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

[noduck]

It looks like ERSPAN does not seem to find new outgoing port after STP change.

I have an ERSPAN session configured from a 7150-c12. Normally, this traffic flows through an upstream 7250 switch. However, if that switch is not available there is another path that becomes available through STP.

ping/sFlow all work in both situations. ERSPAN can work in either situation, but will not flip itself (need to reboot switch, or recreate the ERSPAN configuration).

Is this an ICX bug? If so, anyway to report it?

#show erspan 
 
 Profile 3 
 Type             ERSPAN 
 Mirror destination Reachable. 
 Destination IP   172.31.x.y
 Destination MAC  52xx.xxxx.xxxx
 Source IP        172.31.x.z
 Source MAC       78xx.xxxx.xxxx
 Outgoing port    1/3/1 
 Outgoing VLAN    999 
 Outgoing VE      999 

#show arp | include 172.31.x.y
2     172.31.x.y       52xx.xxxx.xxxx Dynamic  2    1/1/9             Valid 
#show version 
  Copyright (c) Ruckus Networks, Inc. All rights reserved. 
    UNIT 1: compiled on Dec 28 2020 at 20:20:54 labeled as SPR08092e 
      (33554432 bytes) from Primary SPR08092e.bin (UFI) 
        SW: Version 08.0.92eT213

 

[JoJoMan]

Update: POE+ hats arrived today, poe magically works flawlessly now

 

[ZFSZealot]

Another ICX6610

Not sure if you are in the US but there are a ton of listings on the bay - you might also post in the WTB thread and someone may have one for you too? think of the one you have as a parts donor for fan(s) and PSU(S)

This. Those fan trays and PSU's bring an absurd amount of money on eBay if you can find them.

 

[nw60312]

Setting up a new ICX 7150-24p. Out of the box 1/2/1 is up can SSH into the box, upgrade firmware etc. Cannot get the USB console to work, or OOB network port. Would like to create a LAG on both uplink ports to the router. Can create the lag and add 1/2/2, however when I try to add 1/2/1 it fails as that has an IP address assigned (How I am able to SSH to the box). How do I get out of the race condition?

Anyone had success getting the USB console to work with a 2018 MacBook Pro? Tried installing the Ruckus drivers but those are blocked during install, tried version 6.0.1 direct for SI Labs but still no joy.

 

[nickf1227]

I have only ever used a USB to DB9 serial adapter with a Cisco DB9 to RJ45 or Brocade DB9 to "mini-usb" for certain models on my MacBook. I used the program Serial. I would recommend investing in a console cable.
I've never tried any other method.

OOB interface maybe set to a static IP and not DHCP so that may be why you can't get in that way.

As for your other issue, the ICX line won't let you add two interfaces to a lag that have differing configurations. The easiest way to solve the problem you are having is to make a transit vlan.

Make up anew VLAN that will be used exclusively for routing between your router and your switch at L3. Tag that VLAN on the LAG between your switch and router, and on the LAG on your routers side. Make a VE on your Brocade in that VLAN and give it an address. Pick an address in that same subnet on the router side and assign it to the VLAN. You want to use a /30 for this type of link.

Setup your routing on both sides, and viola. You have separated your access network from your wan edge at L3. You can make this type of link as wide as you want. However, by the nature of doing things this way You are doing all of your inter-vlan routing in your Brocade switch and not sending that traffic north to your router for now reason.

 

[mfolnovic]

Hello everyone! I'm setting up ICX 6450-48P and I'm trying to follow @phil9878's instructions.

I'm beginner at this, but read as much as I could (first ~20 and last ~40 pages here and watched some of Terry Henry's videos), and I'm aware there are easier ways to use this switch (e.g. router on a stick, running isc-dhcp separately). I already have router on a stick running so I'm trying to learn and improve from there. I've also tried running isc-dhcp on separate VM (ip helper-address) and got basic setup working. I would have stayed at this setup if I didn't stumble upon @phil9878's instructions.

Current issue is my PCs (Windows and Linux) can't get IP address from pfsense's DHCP when I connect them to port 1/1/39. Pfsense is connected to port 1/1/1.

Cleaned up sh run:

vlan 1 name DEFAULT-VLAN by port
 router-interface ve 1
!
vlan 90 name Guest by port
 untagged ethe 1/1/39
 router-interface ve 90
!
vlan 300 by port
 tagged ethe 1/1/1
 router-interface ve 300
!
ip dhcp-client disable
ip route 0.0.0.0/0 172.26.1.1
!
interface ve 1
 ip address 192.168.2.1 255.255.255.0
!
interface ve 90
 ip address 192.168.90.2 255.255.255.0
!
interface ve 300
 ip address 172.26.1.2 255.255.255.0

And here are screenshots from pfsense:



And DHCP server for VLAN 90 (didn't touch other options):


VLAN 300 doesn't have DHCP server.

I've tried adding IP helper like:

interface ve 90
 ip address 192.168.90.2 255.255.255.0
 ip helper-address 1 192.168.90.1

but it didn't help.

Not sure how to debug this any further.
Thank you all very much for all the information on this forum and for future help!

 

[itronin]

Hello everyone! I'm setting up ICX 6450-48P and I'm trying to follow @phil9878's instructions.

I'm beginner at this, but read as much as I could (first ~20 and last ~40 pages here and watched some of Terry Henry's videos), and I'm aware there are easier ways to use this switch (e.g. router on a stick, running isc-dhcp separately). I already have router on a stick running so I'm trying to learn and improve from there. I've also tried running isc-dhcp on separate VM (ip helper-address) and got basic setup working. I would have stayed at this setup if I didn't stumble upon @phil9878's instructions.

Current issue is my PCs (Windows and Linux) can't get IP address from pfsense's DHCP when I connect them to port 1/1/39. Pfsense is connected to port 1/1/1.

This is key. pfSense does NOT serve up DHCP except on directly connected interfaces.
I can't tell from your screenshots if you have defined your pfSense interface as VLAN tagged 90.
For sure you are not passing VLAN 90 traffic to PFsense at the switch since your configuration shows the only member is untagged ethe 1/1/39

!
vlan 90 name Guest by port
untagged ethe 1/1/39
router-interface ve 90
[/CODE]

you'll need to add this to your VLAN 90 definition and you'll need to make sure you have a VLAN 90 tagged (sub) interface on pfSense. this note is on the drawing in the instructions you linked above.

tagged ethe 1/1/1

I've tried adding IP helper like:

interface ve 90
ip address 192.168.90.2 255.255.255.0
ip helper-address 1 192.168.90.1

but it didn't help.

Not sure how to debug this any further.
Thank you all very much for all the information on this forum and for future help!

IP helper isn't going to do much for you because pfSense serves up DHCP on directly connected interfaces only.

My advice - break up your troubleshooting into functional components.

First test basic IP connectivity (this is more switch and pfSense interface config)
statically configure your PC with a an IP - say 192.168.90.9 (which is within your reserved DHCP block)
Based on your config you should be able to ping 192.168.90.2 since that is the switch's ve and ethe 1/1/39 is untagged in VLAN 90.
Once you can ping 192.168.90.2 move on to pinging 192.168.90.1 - as I said I can't see in your pfSense screenshots whether your VLAN 90 interface is actually defined correctly.

Second you can then move on to testing DHCP.
Then test routing directly out of pfSense first, make sure you have Internet access, NAT working correctly etc. Ie. define the def gw as the pfense box in your DHCP configuration.
Once that works move on to the heart of what you trying to do: which is use the ICX as the core router for LAN and and outbound Internet traffic via the transit VLAN ie. change the DHCP def gw for VLAN 90 pool to your ICX ve90 IP 192.168.90.2[/QUOTE]

 

[mfolnovic]

Thank you very much for your help, setting missing tagged ethe 1/1/1 helped with pretty much everything!

DHCP now works:


So, everything (pinging public IP, pinging 192.168.90.1, reaching pfsense console) but DNS works, which is expected because I haven't even played with that yet.

With static IP, I couldn't ping 192.168.90.1 nor public IP. When I picked static IP from DHCP range, it worked.

change the DHCP def gw for VLAN 90 pool to your ICX ve90 IP 192.168.90.2

I think that's on last screenshot, right? Or did I set it up wrong?

Again, thank you very much!

 

[nw60312]

Got OOB network running. What concepts am I missing to read up on. Device is running Fastiron 0900 layer 3 image. Uplink ports are configured into a LAG, only 1 cable connected at the moment. Both sides of the LAG show active. I can't get traffic to the switch uplink lag. I think this is the relevant config:



ver 09.0.00T213
!
stack unit 1
module 1 icx7150-24p-poe-port-management-module
module 2 icx7150-2-copper-port-2g-module
module 3 icx7150-4-sfp-plus-port-40g-module
!
global-stp
!
lag core dynamic id 1
ports ethe 1/2/1 to 1/2/2
disable ethe 1/2/2
!
vlan 1 name DEFAULT-VLAN by port
spanning-tree
!
chassis fanless 1
hostname core-switch
!

interface management 1
ip address 192.168.1.168 255.255.255.0
!

interface lag 1
ip address 172.16.10.5 255.255.255.0
!
!

no telnet server

!
end




 

[nickf1227]

Can you type
show Ip Route?

Also wht is the config of the other side? What kind of device is it?

 

[nw60312]

Other end is an opnsense box. I think you hit the issue there is no route:

SSH@core-switch>show ip route


Total number of IP routes: 1


Type Codes - B:BGP D:Connected O:OSPF R:RIP S:Static; Cost - Dist/Metric


BGP  Codes - i:iBGP e:eBGP


OSPF Codes - i:Inter Area 1:External Type 1 2:External Type 2


STATIC Codes - v:Inter-VRF


        Destination        Gateway         Port          Cost          Type Uptime


1       192.168.1.0/24     DIRECT          e mgmt1       0/0           D    0m23s 


SSH@core-switch>

 

[clcorbin]

This might be a silly question, but I haven't found an answer so far: Is there a way to permanently disable the paging, like the "skip-page-display" command does? For some reason that irritates me every single time I log in. Small problem, but if anyone has a solution...

 

[nickf1227]

Other end is an opnsense box. I think you hit the issue there is no route:

SSH@core-switch>show ip route


Total number of IP routes: 1


Type Codes - B:BGP D:Connected O:OSPF R:RIP S:Static; Cost - Dist/Metric


BGP  Codes - i:iBGP e:eBGP


OSPF Codes - i:Inter Area 1:External Type 1 2:External Type 2


STATIC Codes - v:Inter-VRF


        Destination        Gateway         Port          Cost          Type Uptime


1       192.168.1.0/24     DIRECT          e mgmt1       0/0           D    0m23s


SSH@core-switch>

So you are not doing any routing on your switch?
Are you trying to connect at L2 or L3?

What does OPN Sense configuration look like?

What are you trying to do?

 

[Jason Antes]

VDX 6740 is an option

I even have one still that I want to sell, but it ain't cheap. Of course some of that is I have multiple spare parts to go with it and it's fully licensed.

 

[fohdeesha]

This might be a silly question, but I haven't found an answer so far: Is there a way to permanently disable the paging, like the "skip-page-display" command does? For some reason that irritates me every single time I log in. Small problem, but if anyone has a solution...

nothing permanent you can save to the config sadly, just the skip command for that session

 

[aaroneaton]

I am a bit confused on what a "stacking cable" is. Id like to stack two brocades together (2x icx 6610, using the port 1 40Gb QSFP+.
Can I use any old 40G QSFP DAC, or is there a specific one that needs to be used? The guides/docs I find seem to be confusing.

thanks

 

[itronin]

I am a bit confused on what a "staking cable" is. Id like to stack two brocades together (2x icx 6610, using the port 1 40Gb QSFP+.
Can I use any old 40G QSFP DAC, or is there a specific one that needs to be used? The guides/docs I find seem to be confusing.

thanks

A "staking cable" is a common vampire hunter accessory used to immobilize a vampire.

I personally use a couple of "shorty" Netapp QSFP to QSFP cable that I think cost $10USD from the bay to stack 2 ICX6610's.

 

[aaroneaton]

A "staking cable" is a common vampire hunter accessory used to immobilize a vampire.

I personally use a couple of "shorty" Netapp QSFP to QSFP cable that I think cost $10USD from the bay to stack 2 ICX6610's.

Ha ha thanks. Typed that too fast it seems.
I was looking around so at some options and kept reading that i needed a "stacking cable" and could not use a reg 40G QSFP+ DAC. cool beans. thanks!

 

[NablaSquaredG]

Regular QSFP+ DAC is perfectly fine.

Just take care which ports you link! If you mess up, you may get weird errors and behaviour.

 

[aaroneaton]

right. Thanks

Regular QSFP+ DAC is perfectly fine.

Just take care which ports you link! If you mess up, you may get weird errors and behaviour.