Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

[safado]

Need some help as I'm stuck and not sure where to go to next.

I have a 6610 that is licensed, has the stack removed per Fodeesha etc. I added a 6650 about a year ago and used a 40GB port to connect to the 6610 and it worked perfectly. No issues at all. I had a reboot a couple of days ago and since then I've been unable to get access to the 6650 over the network. I've re-ran setup and assigned the IP, single VLAN, checked licensing and everything else. Wiped back to the default config and started over. I've found that if I connect a 10gb port on the front to a 10gb port of the 6610 it works. The minute I try to use a 40gb qsfp port in the rear (and remove the 10gb port) it will fail. It won't fail by plugging in a qsfp. On the 6650 it shows the port active and state forwarding but on the 6610 it will show it connected and negotiated at 40gb but under state it will show "blocked". Any idea how to get the 6650 and the 6610 chained together (separate switches not a stack) leveraging the 40gb qsfp connection (1/21 or 1/2/6)? Thanks for any help!

 

[DangerMc]

I have a ICX7750 that I've yet to configure. I've updated it to the latest firmware per the fohdeesha docs, but not connected it to any devices yet. If I let it sit turned on, it will periodically reboot (the front display shuts off and then the fans ramp up before they quiet down again), every 10-15 mins or so. I've tried searching to see what the reason for this is but cannot seem to find anything that points to a direct cause. Anyone else experience this with any of their switches?

 

[itronin]

I have a ICX7750 that I've yet to configure. I've updated it to the latest firmware per the fohdeesha docs, but not connected it to any devices yet. If I let it sit turned on, it will periodically reboot (the front display shuts off and then the fans ramp up before they quiet down again), every 10-15 mins or so. I've tried searching to see what the reason for this is but cannot seem to find anything that points to a direct cause. Anyone else experience this with any of their switches?

Is this plugged into a UPS?

 

[Dave Corder]

Is this plugged into a UPS?

Yeah, I had an almost identical issue with my ICX6610-48P when using a consumer-grade CyberPower UPS. I switched it to my rack-mount APC SmartUPS and it's been fine ever since.

 

[andrea87]

The power supplies in these devices are very heavy lifters, they're rated at 1kW output. Also they have to be active PFC due to regulations and such.

as of that, any input waveform that isn't sinusoidal will make the PFC circuit go haywire and possibly trigger very large current spikes on the UPS. It's basically made with a boost converter that charges the "tank" capacitors with an input current that is adjusted to be perfectly sinusoidal. Apply a non-sine input to it and all kinds of mayhem will happen.

I bet your issues are either with a modified sine wave or a square wave ups.

 

[itronin]

I have a ICX7750 that I've yet to configure. I've updated it to the latest firmware per the fohdeesha docs, but not connected it to any devices yet. If I let it sit turned on, it will periodically reboot (the front display shuts off and then the fans ramp up before they quiet down again), every 10-15 mins or so. I've tried searching to see what the reason for this is but cannot seem to find anything that points to a direct cause. Anyone else experience this with any of their switches?

here's a practical recommendation - just plug it into street power and see if it continues the reboot cycle.

 

[DangerMc]

here's a practical recommendation - just plug it into street power and see if it continues the reboot cycle.

Sorry for the late reply, both street power and Eaton 2000VA UPS forces continuous reboot cycles. I have a couple of different UPS' that I will try but never occurred to me that that might be the culprit. (However, I have two different 7450 plugged into the same UPS that does not exhibit the issue)

EDIT : I will try using a different APC SmartUPS and see if that fixes the issue.

Thanks for all the help!

 

[safado]

Need some help as I'm stuck and not sure where to go to next.

I have a 6610 that is licensed, has the stack removed per Fodeesha etc. I added a 6650 about a year ago and used a 40GB port to connect to the 6610 and it worked perfectly. No issues at all. I had a reboot a couple of days ago and since then I've been unable to get access to the 6650 over the network. I've re-ran setup and assigned the IP, single VLAN, checked licensing and everything else. Wiped back to the default config and started over. I've found that if I connect a 10gb port on the front to a 10gb port of the 6610 it works. The minute I try to use a 40gb qsfp port in the rear (and remove the 10gb port) it will fail. It won't fail by plugging in a qsfp. On the 6650 it shows the port active and state forwarding but on the 6610 it will show it connected and negotiated at 40gb but under state it will show "blocked". Any idea how to get the 6650 and the 6610 chained together (separate switches not a stack) leveraging the 40gb qsfp connection (1/21 or 1/2/6)? Thanks for any help!

Well I ended up resolving it. Somewhere between disabling STP and removing a LAG I know I didn’t create the port state changed and it’s now working. Of course after I ordered another 6610 to get around the issue. Ha!

 

[codyjd1]

Awesome thanks for the reply! I will check it out and hope I can get it all configured when they arrive! Thanks again!

I have received the switches and have them setup. I can confirm that you can stack them with ports 1 and 3 on each switch and still have ports 2 and 4 available as regular data ports.

 

[madmoot]

I should have the switch in my hands in about a week - will check fan voltage then. Just trying to get some replacement cooling lined up to have on hand asap. This thing will be living in my home office about 8 ft away, otherwise I wouldn't really care about the noise level

So just a quick update... Ended up swapping out the single stock 7250 Nidec fan for a Delta EFB0412VHD-F00 and put a Sunon MF60101V2-1000U-A99 on the ASIC (V3s were out of stock on Digikey), which has a slightly higher CFM. Sensor 1 temps settled out at ~65 degs and barely audible about a foot away from me. Very pleased to say the least. FWIW, temps were about 3-4 degs cooler with the ASIC fan blowing down, so i've left it like that for now. Thanks @RoachedCoach and @anomaly for all the guidance.

 

[bpye]

What's going on with FI 10 and the 7150s? Just noticed on the doc for "chassis fanless" - Commscope Technical Content Portal -

10.0.10d	This command was re-introduced for RUCKUS ICX 7150-ES devices.
10.0.10e	This command was re-introduced for ICX 7150 devices.

The release notes still show the ICX 7150 as unsupported - so one of them is wrong.

 

[anomaly]

What's going on with FI 10 and the 7150s? Just noticed on the doc for "chassis fanless" - Commscope Technical Content Portal -

10.0.10d	This command was re-introduced for RUCKUS ICX 7150-ES devices.
10.0.10e	This command was re-introduced for ICX 7150 devices.

The release notes still show the ICX 7150 as unsupported - so one of them is wrong.

Does this mean they re-introduced fanless support for some models? It might be interesting to run a BinDiff/binary diffing tool between those firmware versions, and figure out what they changed. That will give up where the fan logic is located.

 

[bpye]

Well it's less the fact that fanless is re-introduced, and more that it suggests 10.0.10e should run on ICX 7150 switches, and not just the ICX 7150-ES switches.

Of course it could just be a typo.

 

[anomaly]

If you have both firmware packages, I would love to be able to download them

 

[fohdeesha]

If you have both firmware packages, I would love to be able to download them

they're all freely available on ruckus' site, you just need to create a free account

 

[GOSHKU]

Hi,

i bought two used Brocade ICX 6450-48

I used erase command i dont have any license. (or its basic, i dont know, i am not so familliar)

What i need to do to be able to use the 2 or all 4 10G ports?

When i am booting - i have this message in the serial console -
"No license present for port 1/2/2
No license present for port 1/2/4"

Does that mean that i can use 2 or no one 10G SFP+ ports?

Thanks a lot,
Greetings

 

[mshook]

I used erase command i dont have any license. (or its basic, i dont know, i am not so familliar)

What i need to do to be able to use the 2 or all 4 10G ports?

Hi,

everything you need is in the first post of @fohdeesha , namely this link Setup / Config / Licensing Guide v2

 

[ProxmoxProphet]

I bought an ICX7250-48P a few months ago, and, other than some strange IPv6 issues that resolved once I upgraded to 09.0.10 it's been great, but recently I noticed another frustrating issue that I don't know how to fix and I hope someone who encountered this issue can reply with how they fixed it.

I noticed a problem recently with the switch not being able to handle wifi devices roaming between different APs after moving to a new house where I had to buy multiple APs to get connectivity in the whole house (my old house was smaller so I had only 1 AP and therefore never encountered this problem). So here is an explanation of how the problem presents itself: Let's say I am downstairs, connected to my home wifi via the downstairs AP on my phone, I can access the internet just fine, I can access locally-hosted services like nextcloud, jellyfin, etc just fine BUT when I go upstairs to my bedroom, and my phone roams from the downstairs AP to the AP in the upstairs hallway, now the phone can no longer reach anything that is more than one hop away. It can reach other devices in the same VLAN just fine, it can reach other wifi devices, the switch on that VLAN, etc, but it can no longer reach the internet or any locally-hosted services (because the locally-hosted servers are on a different VLAN, and the uplink from the switch to my firewall is also on its own VLAN, so its 2 hops away to those). And if I statically assign the phone a different IP address, then all of a sudden now it can reach the internet and locally-hosted services again (until it roams to a different AP once again and the problem happens again). And this issue is not just with my phone but with any Wi-Fi connected device

Also, another problem I noticed a while ago as well that frustrates me: Sometimes, when a device on the network is assigned an IP that was previously leased to some other device by the DHCP server, then that device cannot reach the internet or anything else thats more than one hop away unless I give it a different IP. I'm not sure but maybe these two issues are related, because the problematic behavior (not being able to reach anything more than 1 hop away) is present in both


Here is a brief description of my network topology in case it would help anyone understand something better:

ISP fiber cable comes into my home and connects to the ONT, then the ONT connects to my opnsense firewall with ethernet. From there, my opnsense firewall connects to a 2.5G/10G managed switch via SFP+ DAC. The 2.5G ports on the switch are used to connect downstairs AP and other things on the user VLAN, and the other SFP+ port on that switch is connected to a singlemode fiber cable which runs to the upstairs hallway closet where my ICX7250 is. This cable carries userVLAN traffic from downstairs to the ICX7250 and the traffic for the transit VLAN between the ICX7250 and Opnsense is also sent down this cable. Then, there are two more APs upstairs connected to the ICX7250 (on the same VLAN as the downstairs APs) as well as some other servers and PCs, a fiber cable that connects to a Mikrotik 10G switch on the third floor, and other things connected to the ICX7250 as well.

Hi, thanks for sharing your experience, it's good to hear that you don't have this problem, but also strange, I'm not sure what is causing this problem for me.

I walked with my phone to downstairs and back (confirming in the TP link app that the phone did actually roam to the downstairs AP and back to the office AP) and then I ran that command, but I didn't see any output about my phone, or any other devices, roaming between APs. The only output was stuff about failed and successful SSH logins, some stuff about STP, and some stuff about the switch disabling PoE on certain ports because a non-PoE devices was connected, but nothing about AP roaming.

I also thought that the 2.5G/10G switch may be responsible for the issues, but I checked the configuration on that and the appropriate VLANs were passed, the ethernet ports on that switch are all untagged for my User VLAN (VLAN 110) and the SFP+ ports are both tagged for the User VLAN and the Transit VLAN.

Also, I also notice this issue not just when roaming between downstairs and second floor APs, but also when roaming between the second floor APs and the AP on the third floor in my office, and the 2.5G/10G switch is on the first floor, so I don't think that 2.5/10G switch would be involved when roaming between second and third floor APs. Also I checked the configuration on the Mikrotik switch on the 3rd floor (which connects to the 3rd floor AP) and the VLANs were configured appropriately on there as well

It seems assigning a different rather than just a static address is what's fixing the issue.

When a client (e.g. 10.10.10.1) roams from an AP to a different AP, and those APs are attached to different ports on the Brocade switch (e.g. ports 1/1/4 and 1/1/5), you should see log messages akin to;

<timestamp>:next hop router 10.10.10.1 moved from port 1/1/4 to port 1/1/5

which is something to look for.

I'd run a traceroute from the client to the internet, and from the expected hop after where traffic is being blocked back to the client, or at least from further upstream if you can't run a traceroute on the hop immediately after where traffic is being blocked.

I have not been able to update on this as I have been away, but I have tried both running a traceroute from affected clients who cant reach the internet, to an IP on the internet, and also a traceroute from the opnsense firewall to the affected devices, and here are the results:

When I run a traceroute from an affected client (a phone) on the User VLAN to something on the internet (1.1.1.1) , the traceroute reaches the switch's IP on the same VLAN as the device, but then the traceroute dies there, it times out at the second hop (which should be the opnsense router's IP on the transit VLAN). When I run a traceroute from the opnsense firewall to the client device's IP, it reaches the switch's IP on the transit VLAN, but then the traffic dies there and does not go to the client device. So it seems the traffic is dying at the switch.

Also, when a device roams between APs, I can see that it's IP is now on a different port using the "show arp ve 150" command but there are no log messages generated when I run the log command.

Any advice on how to resolve this would be greatly appreciated!

 

[fohdeesha]

I have not been able to update on this as I have been away, but I have tried both running a traceroute from affected clients who cant reach the internet, to an IP on the internet, and also a traceroute from the opnsense firewall to the affected devices, and here are the results:

When I run a traceroute from an affected client (a phone) on the User VLAN to something on the internet (1.1.1.1) , the traceroute reaches the switch's IP on the same VLAN as the device, but then the traceroute dies there, it times out at the second hop (which should be the opnsense router's IP on the transit VLAN). When I run a traceroute from the opnsense firewall to the client device's IP, it reaches the switch's IP on the transit VLAN, but then the traffic dies there and does not go to the client device. So it seems the traffic is dying at the switch.

Also, when a device roams between APs, I can see that it's IP is now on a different port using the "show arp ve 150" command but there are no log messages generated when I run the log command.

Any advice on how to resolve this would be greatly appreciated!

Idk how to read so I can't read the history of this issue but have you added a static route on opnsense pointed at your lan with a nexthop of the switches transit vlan ip

 

[ProxmoxProphet]

Idk how to read so I can't read the history of this issue but have you added a static route on opnsense pointed at your lan with a nexthop of the switches transit vlan ip

Yes, I have static routes on my opnsense firewall for every one of my VLANs' subnets, with a nexthop of the switch's transit vlan IP.
And when a device is not experiencing this problem the opnsense can reach stuff in the VLANs/vice versa just fine and the connection works normally, its just intermittently this issue happens where certain devices can't reach the opnsense or anything on the internet (until a different local IP is assigned to the device, which temporarily relieves the problem until the next time it happens)