Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

[tls]

I see the mstp name command, but from the documentation synopsis it makes no sense, claiming that it is a name "for the device", and that "the name applies to the switch that can have many different VLANs that can belong to many different MSTP regions". This makes no sense - names in MSTP are per-region, not per-switch. Is the documentation just wrong? It gives this example, too, which sure looks like whoever wrote it thought this was a "switch name" not a "region name" - but there is no other way I can find to set the region name!

device(config)# mstp name Device1

I did some guesswork and testing. The documentation (of course?) is wrong. mstp name sets the region name, just like on everyone else's switches; it does not set the "device name" which is nonsense from a MSTP point of view. The example is bogus and misleading, too, of course; it should probably be changed to use a region name of "mymst" since that's what's used in all of Cisco's configuration examples, or something more clear like "Region1".

Watch out for the MSTP "revision" also - FastIron defaults this to 0 like most other switches, but there are a few brands and product lines out there that default it to 1. If both name and revision don't match, you will end up with multiple MST regions you did not design, all communicating by RSTP or even old-school STP across the "boundaries", and are likely to lose redundancy you designed into your network, since links beyond boundaries are not visible to each accidentally configured region's root bridge.

 

[ProxmoxProphet]

I bought an ICX7250-48P a few months ago, and, other than some strange IPv6 issues that resolved once I upgraded to 09.0.10 it's been great, but recently I noticed another frustrating issue that I don't know how to fix and I hope someone who encountered this issue can reply with how they fixed it.

I noticed a problem recently with the switch not being able to handle wifi devices roaming between different APs after moving to a new house where I had to buy multiple APs to get connectivity in the whole house (my old house was smaller so I had only 1 AP and therefore never encountered this problem). So here is an explanation of how the problem presents itself: Let's say I am downstairs, connected to my home wifi via the downstairs AP on my phone, I can access the internet just fine, I can access locally-hosted services like nextcloud, jellyfin, etc just fine BUT when I go upstairs to my bedroom, and my phone roams from the downstairs AP to the AP in the upstairs hallway, now the phone can no longer reach anything that is more than one hop away. It can reach other devices in the same VLAN just fine, it can reach other wifi devices, the switch on that VLAN, etc, but it can no longer reach the internet or any locally-hosted services (because the locally-hosted servers are on a different VLAN, and the uplink from the switch to my firewall is also on its own VLAN, so its 2 hops away to those). And if I statically assign the phone a different IP address, then all of a sudden now it can reach the internet and locally-hosted services again (until it roams to a different AP once again and the problem happens again). And this issue is not just with my phone but with any Wi-Fi connected device

Also, another problem I noticed a while ago as well that frustrates me: Sometimes, when a device on the network is assigned an IP that was previously leased to some other device by the DHCP server, then that device cannot reach the internet or anything else thats more than one hop away unless I give it a different IP. I'm not sure but maybe these two issues are related, because the problematic behavior (not being able to reach anything more than 1 hop away) is present in both


Here is a brief description of my network topology in case it would help anyone understand something better:

ISP fiber cable comes into my home and connects to the ONT, then the ONT connects to my opnsense firewall with ethernet. From there, my opnsense firewall connects to a 2.5G/10G managed switch via SFP+ DAC. The 2.5G ports on the switch are used to connect downstairs AP and other things on the user VLAN, and the other SFP+ port on that switch is connected to a singlemode fiber cable which runs to the upstairs hallway closet where my ICX7250 is. This cable carries userVLAN traffic from downstairs to the ICX7250 and the traffic for the transit VLAN between the ICX7250 and Opnsense is also sent down this cable. Then, there are two more APs upstairs connected to the ICX7250 (on the same VLAN as the downstairs APs) as well as some other servers and PCs, a fiber cable that connects to a Mikrotik 10G switch on the third floor, and other things connected to the ICX7250 as well.

 

[jode]

I noticed a problem recently with the switch not being able to handle wifi devices roaming between different APs after moving to a new house where I had to buy multiple APs to get connectivity in the whole house (my old house was smaller so I had only 1 AP and therefore never encountered this problem).

I have quite a similar setup which works fine on a ICX-6450. So, I don't think it's necessarily a switch limitation.

In my setup, I can see devices moving from one port (AP1) to another port (AP2) in the log.
Try sh log and look for that activity.

I would expect some configuration issue, possibly on the 2.5G/10G switch (wild guess, based on my setup experience). Make sure that all VLANs you use with your APs are connected via both switches.

 

[bgubs]

Wired up the other fan inline with fan #2 using a new connector. It works great. I'm now idling around 46-47C. I wish I had a benchmark to go off of for when I hook my network back up, then I could've really compared the difference in temps.

I'm still curious though about fan port #3. I'm wondering if I found the 3 components that were missing and soldered them on to the board whether or not the fan would actually work (with PWM too) and recognized in `show chassis`. Absolutely a waste of money and time regardless, but sounds like a fun and neat experience.

Well experiment was a partial bust. I was able to find a similar mosfet and had some capacitors on hand. Soldered them in today and the fan will turn on, but seems there's no logic sent to the 3rd pin. I'm going to trace a little further up the line, but I think unless someone with more know-how is able to enable the 3rd pin in the firmware the other half of the experiment is over. Oh well, was a fun test and worth a try.

 

[ProxmoxProphet]

I have quite a similar setup which works fine on a ICX-6450. So, I don't think it's necessarily a switch limitation.

In my setup, I can see devices moving from one port (AP1) to another port (AP2) in the log.
Try sh log and look for that activity.

I would expect some configuration issue, possibly on the 2.5G/10G switch (wild guess, based on my setup experience). Make sure that all VLANs you use with your APs are connected via both switches.

Hi, thanks for sharing your experience, it's good to hear that you don't have this problem, but also strange, I'm not sure what is causing this problem for me.

I walked with my phone to downstairs and back (confirming in the TP link app that the phone did actually roam to the downstairs AP and back to the office AP) and then I ran that command, but I didn't see any output about my phone, or any other devices, roaming between APs. The only output was stuff about failed and successful SSH logins, some stuff about STP, and some stuff about the switch disabling PoE on certain ports because a non-PoE devices was connected, but nothing about AP roaming.

I also thought that the 2.5G/10G switch may be responsible for the issues, but I checked the configuration on that and the appropriate VLANs were passed, the ethernet ports on that switch are all untagged for my User VLAN (VLAN 110) and the SFP+ ports are both tagged for the User VLAN and the Transit VLAN.

Also, I also notice this issue not just when roaming between downstairs and second floor APs, but also when roaming between the second floor APs and the AP on the third floor in my office, and the 2.5G/10G switch is on the first floor, so I don't think that 2.5/10G switch would be involved when roaming between second and third floor APs. Also I checked the configuration on the Mikrotik switch on the 3rd floor (which connects to the 3rd floor AP) and the VLANs were configured appropriately on there as well

 

[fohdeesha]

I bought an ICX7250-48P a few months ago, and, other than some strange IPv6 issues that resolved once I upgraded to 09.0.10 it's been great, but recently I noticed another frustrating issue that I don't know how to fix and I hope someone who encountered this issue can reply with how they fixed it.

I noticed a problem recently with the switch not being able to handle wifi devices roaming between different APs after moving to a new house where I had to buy multiple APs to get connectivity in the whole house (my old house was smaller so I had only 1 AP and therefore never encountered this problem). So here is an explanation of how the problem presents itself: Let's say I am downstairs, connected to my home wifi via the downstairs AP on my phone, I can access the internet just fine, I can access locally-hosted services like nextcloud, jellyfin, etc just fine BUT when I go upstairs to my bedroom, and my phone roams from the downstairs AP to the AP in the upstairs hallway, now the phone can no longer reach anything that is more than one hop away. It can reach other devices in the same VLAN just fine, it can reach other wifi devices, the switch on that VLAN, etc, but it can no longer reach the internet or any locally-hosted services (because the locally-hosted servers are on a different VLAN, and the uplink from the switch to my firewall is also on its own VLAN, so its 2 hops away to those). And if I statically assign the phone a different IP address, then all of a sudden now it can reach the internet and locally-hosted services again (until it roams to a different AP once again and the problem happens again). And this issue is not just with my phone but with any Wi-Fi connected device

Also, another problem I noticed a while ago as well that frustrates me: Sometimes, when a device on the network is assigned an IP that was previously leased to some other device by the DHCP server, then that device cannot reach the internet or anything else thats more than one hop away unless I give it a different IP. I'm not sure but maybe these two issues are related, because the problematic behavior (not being able to reach anything more than 1 hop away) is present in both


Here is a brief description of my network topology in case it would help anyone understand something better:

ISP fiber cable comes into my home and connects to the ONT, then the ONT connects to my opnsense firewall with ethernet. From there, my opnsense firewall connects to a 2.5G/10G managed switch via SFP+ DAC. The 2.5G ports on the switch are used to connect downstairs AP and other things on the user VLAN, and the other SFP+ port on that switch is connected to a singlemode fiber cable which runs to the upstairs hallway closet where my ICX7250 is. This cable carries userVLAN traffic from downstairs to the ICX7250 and the traffic for the transit VLAN between the ICX7250 and Opnsense is also sent down this cable. Then, there are two more APs upstairs connected to the ICX7250 (on the same VLAN as the downstairs APs) as well as some other servers and PCs, a fiber cable that connects to a Mikrotik 10G switch on the third floor, and other things connected to the ICX7250 as well.

this is an arp table issue most likely, have you changed anything weird in your config? previous fastiron versions had some bugs when it came to ipv6 ND, it wouldn't update the ND table when devices moved between physical switch ports, but that was fixed in like 8095 as far as I know

 

[ProxmoxProphet]

this is an arp table issue most likely, have you changed anything weird in your config? previous fastiron versions had some bugs when it came to ipv6 ND, it wouldn't update the ND table when devices moved between physical switch ports, but that was fixed in like 8095 as far as I know

Hi, I haven't changed my config recently, and I'm also on the version 09.0.10j_cd2T213 , so definitely newer than 8095
I made a comment in this thread a few months ago about an IPv6 issue that I resolved and my config in there is the same as my current config (with only minor changes like creating VLANs, adding VLANs to ports and such)

Is there any fix for the arp table issue you mentioned?

 

[logixworx]

anyone know if Brocade 10G-SFPP-TWX-0101 cables are cisco compatible? can I use them with brocade icx 6450 as well as intel x520-da cards?

 

[dbvader]

And if I statically assign the phone a different IP address, then all of a sudden now it can reach the internet and locally-hosted services again

It seems assigning a different rather than just a static address is what's fixing the issue.

When a client (e.g. 10.10.10.1) roams from an AP to a different AP, and those APs are attached to different ports on the Brocade switch (e.g. ports 1/1/4 and 1/1/5), you should see log messages akin to;

<timestamp>:next hop router 10.10.10.1 moved from port 1/1/4 to port 1/1/5

which is something to look for.

I'd run a traceroute from the client to the internet, and from the expected hop after where traffic is being blocked back to the client, or at least from further upstream if you can't run a traceroute on the hop immediately after where traffic is being blocked.

 

[dbvader]

@ProxmoxProphet Ah, I see @jode already suggested checking logs. Also agree there's a good chance it's due to the other switch or even the opnsense node.

Also, where is the DHCP server running and how intelligent are the APs? Are they just doing L2 or also L3?

 

[Koop]

Hey, what the dog doin?

 

[nlj]

Hi everybody. Someone over on Reddit asked me to document my fan mod for my ICX 7150-24; I hope this is helpful to someone. See my previous post in this thread for my research on various different makes/models of fans. This is a soft mod, meaning it's easily reversible and you can put the stock fan back without any issues.

Items needed:

• Delta FFB0412VHN-F00 fan [Digikey]
• A 3-pin Molex fan header [Digikey]
  • You can cut off the existing fan's connector, but then this becomes a hard mod that you can't easily revert
• [optional] if soldering: solder and flux
• electrical tape or [optional] heat shrink tubing (especially if soldering)

Equipment needed:

• Wire cutter and/or wire stripper
• [optional] if soldering: soldering iron and related supplies
• [optional] if using heat shrink tubing: heat source like hairdryer or heat gun

Process:

The motherboard fan pinout on the ICX-7150 does not match the standard ATX pinout, so pay careful attention to the pinout to make sure you connect the right wires. The motherboard pins are: 1) fan speed, 2) 12VDC+, 3) 12VDC–.

• Cut the fan header so you have a single, appropriately-sized "pigtail" that has a Molex connector on one end.
• Strip the pigtail wires, and if not already stripped, the fan wires. You may consider trimming the wire lengths so there's less excess wiring in the case.
• If using heat shrink tubing, thread onto the wires BEFORE soldering them.
• Attach the fan to the header pigtail using the following pinout chart, either by electrical tape or soldering.
  • If soldering, cover with electrical tape or heat shrink tubing.
• Remove the old fan and replace in the same way with the new fan.

Pinout (for ATX-style Molex connectors):

Pin	Molex ATX fan wire color	Fan wire color	Purpose
1	Black	Blue	Speed/Tach
2	Red	Red	12VDC+
3	Yellow	Black	12VDC–

Finally, I've attached a photo of a completed project for reference. I hope this helps all you fan modders out there! Cheers!

 

[anomaly]

Back to this behemoth of a thread: I am helping a friend with a 6450 and doing the intra-vlan routing for him. I'm beyond rusty since I touched ACLs and forgot how to assign them/limit them to specific VEs. I have cleared all the ACLs, added VEs for the VLANs that are meant to be routed by the switch, and yet, trying to reach any host between any two VLANs (with the correct VEs) isn't working.

I'm testing with a static route using the IP of the VE in the VLAN of the test host (ex. 10.7.6.3 in VLAN 3), for the address of the target host (ex. 10.7.7.3 in VLAN 2). THe switch has the VEs at .2 for both VLANs (ex. 10.7.6.2). And yes, it is the routing-enabled firmware variant

What am I missing?

Nevermind, had to adjust the gateway for the clients in VLAN 3.

This is still something I have not sorted out:

SSH@switch-1(config-vif-10)#ip access-group "test_traffic" out                                 

ERROR: ACL test_traffic has filters with "log" action. Output Access Lists cannot be configured.

I was able to set an ACL with log directives before. Running ICX64R08030t.

 

[anomaly]

Pinout (for ATX-style Molex connectors):

Pin	Molex ATX fan wire color	Fan wire color	Purpose
1	Black	Blue	Speed/Tach
2	Red	Red	12VDC+
3	Yellow	Black	12VDC–

Finally, I've attached a photo of a completed project for reference. I hope this helps all you fan modders out there! Cheers!

View attachment 38358

You can get a cheap crimp tool and connectors from Aliexpress. Once you get the gist of it it's quicker than bothering yourself with solder joints and cutting wires. Search for "crimping JST kit", I believe the pitch in these is 2.0mm. @fohdeesha can probably confirm.

 

[DangerMc]

I'm having some issues connecting 3 ICX7450s together using the QSFP port on the back. I have one with two qsfp ports and then from there a MTP cable out to each of the two other switches. (I'm using brocade transceivers from FS along with MTP cables from them too) However, they don't seem to be showing as active/up. Im trying to configure these in a uplink mode and not stack, so I have gone in and disabled the stacking feature but that didn't change anything.
Is there anything else I should have done or should this just work as an uplink port "out of the box?", i.e. is there some likely hood that the cable is broken?
Thanks

 

[Fluxor]

Having an issue with upgrade on a 7150-C12P, Seems stuck in a boot loop. I get this message during boot and then it reboots

CPLD is upgraded Successfully !!!Applying factory defaults..
sil_ramfs_creat: Failed to create file /var/log/pdc/pdc_errorlog.txt

Could not create ramfs file!
pdc_errorlog
Could not open ramfs file!

Got my answer from a thread titled "Brocade ICX 7150-C12P stuck in boot loop after upgrade"

If this happens to you, just hard power cycle the unit and it will continue

 

[anomaly]

Setting up a 7250-48p for a friend, the fans are stuck on speed 2:

Fan 1 ok, speed (auto): 1<->[[2]]

Fan 2 ok, speed (auto): 1<->[[2]]

Fan 3 ok, speed (auto): 1<->[[2]]

I tried 'chassis fanless' but it is not supported on the 7250. Is there a reason the fans are not ramping down? (case is open)

 

[anomaly]

Putting the cover on:

The stack unit 1 chassis info:

Power supply 1 (AC - PoE) present, status ok
Power supply 2 not present
Power supply 3 not present

Fan 1 ok, speed (auto): [[1]]<->2
Fan 2 ok, speed (auto): [[1]]<->2
Fan 3 ok, speed (auto): [[1]]<->2

Fan controlled temperature:
        Rule 1/2 (MGMT THERMAL PLANE): 60.7 deg-C
        Rule 2/2 (AIR OUTLET NEAR PSU): 28.5 deg-C

Fan speed switching temperature thresholds:
        Rule 1/2 (MGMT THERMAL PLANE):
                Speed 1: NM<-----> 95       deg-C
                Speed 2:        85<----->105 deg-C (shutdown)
        Rule 2/2 (AIR OUTLET NEAR PSU):
                Speed 1: NM<-----> 41       deg-C
                Speed 2:        34<----->105 deg-C (shutdown)

Fan 1 Air Flow Direction:  Front to Back 
Fan 2 Air Flow Direction:  Front to Back                          
Fan 3 Air Flow Direction:  Front to Back 
Slot 1 Current Temperature: 61.2 deg-C (Sensor 1), 28.5 deg-C (Sensor 2)

So, without the cover:

Slot 1 Current Temperature: 63.5 deg-C (Sensor 1), 56.0 deg-C (Sensor 2)

How can we configure the rules?? Are these somewhere we can reach with commands or inside the baremetal Linux OS? Obviously the threshold of 63.5 is too low.

 

[RuckusVol]

I'm sorry for not following up on this sooner. We have a newborn and there's been some time-consuming challenges and I had to set this aside for a few weeks.

There are multiple ways to do this, but since your eventual goal is to have the SFP provide routes to the Internet (which it will do over one or more VLANs), you don't want to use the layer 3 (router-only) method.

Which version of the ICX software are you running? Is it the 'S' (layer 2) or 'R' (layer 3) flavor? If it's 'R', are you currently using the ICX as a router, or are all of your ports and VLANs configured for layer 2 connectivity only?

You say the 'recommended' address for the SFP is 192.168.1.200, but what's critical to know is how the device will get its address. It will either be hardcoded in the device, or it will use BOOTP or DHCP to obtain an address (in which case there is no real need to use the 'recommended' address).

With the answers to all of those questions, it is possible to figure out a working configuration.

I'm not sure whether I'm on the S or R version of software. show run ver 08.0.95mT213.

I've been able to access the SFP module on 192.168.1.1. I've configured it and get the O5 status indicating it's getting what it wants from the ISP. I have not been able to see any traffic on the back side of the SFP that's trying to communicate with my router so I can get a DHCP lease.

I'm not intentionally using the switch as a router. I'm not very good at networking and I'm trying really hard to learn. I believe there are a few IP routes in there, from the guide here and maybe one I added to get to the SFP. I've spent 12+ hours on this this weekend and I'm still stuck haha. But currently, I'm trying to use port 1/2/8 for the GPON SFP and 1/1/22 for the ethernet to router WAN. I created a VLAN 1202 (which is the VLAN I get from the ISP) and I've tried creating it with both tagged and untagged ports for the members, with and without any other vlans assigned to the ports. When I do a tcp dump I don't see any traffic from the GPON.

I have the GPON set to auto use the ISP VLAN. I also tried the forward anything option from the guides I've seen.

which SFP ONT? you'll most likely not want to do any IP addressing or routing on the icx itself. Just plug the SFP in, then on a computer on the same network (plugged into an ICX port that's on the same VLAN as the sfp+ port), and assign that computer the .200 address. the ICX shouldn't have an IP in that SFP+ vlan. Which leads me to - if you haven't, the sfp should be in its own isolated vlan, as it will be providing your internet feed. then you can connect that vlan to your routers WAN port, by either putting another icx port in that vlan, or trunking the vlan over something to your router. This is what I'm doing with my WAS-110 ONT in my icx6610

I'm using a DFP-34X-2C2 and following the PON Madness guide. I currently have VLAN 1202 setup with all ports tagged and the only traffic on that vlan I see is from the router requesting a DHCP lease. When I do a tcpdump on the whole interface I still don't see other traffic from the GPON SFP. I also tried untagged ports.

What's the proper setup? Just those two ports on VLAN 1202 and no other VLAN? Tagged or Untagged? Should a third port used to access the SFP GPON be tagged or untagged to be able to get to it? Any other ideas why I'm not getting traffic through to the ethernet side that goes to the router WAN? I really appreciate the input so far. You guys take care.

 

[R3Z3N]

snip....Fan speed 2 on complaint....snip

There is no software/firmware logic to change on these ICX models beside the model that supports fanless mode. Either add more cooling via some hackey method, run ac to where this is located, or buy a more energy efficient router/switch.