Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

[Zombielinux]

I doubt they would have a separate PCB for POE vs non POE but it's pretty likely that some components for POE are depop-ed on non POE boards

Its possible. They didn't populate the connectors, but those are typically very expensive components, and two of them are through hole (not something that a PnP can do very easily. The only thing I can think of is maybe they have some 0 ohm resistors somewhere, but a quick visual didn't show anything. The board is MASSIVE so I could have missed something.

 

[Wolfcastle]

My first check would probably be to make sure that the Microsemi ICs on the POE board are all getting power, I think they're all 3.3V VDD ICs and on the top side so that should be easy enough to probe. I had dug up the datasheets to figure out which pin is which when I was trying to troubleshoot my switch a few days ago but I didn't save them.

Checking for movement on the I2C pins on the POE board after that would be ideal but I think that's going to be hard to do without a scope which is not something people typically have at home!

edit: have you considered moving the non-functional POE board to the working POE switch for a quick test to see if it works there?

 

[Zombielinux]

My first check would probably be to make sure that the Microsemi ICs on the POE board are all getting power, I think they're all 3.3V VDD ICs and on the top side so that should be easy enough to probe. I had dug up the datasheets to figure out which pin is which when I was trying to troubleshoot my switch a few days ago but I didn't save them.

Checking for movement on the I2C pins on the POE board after that would be ideal but I think that's going to be hard to do without a scope which is not something people typically have at home!

edit: have you considered moving the non-functional POE board to the working POE switch for a quick test to see if it works there?

I'll probe around with my meter a bit more.

As it so happens, I DO have a 100mhz scope and an old logic analyzer sitting next to my healing bench.

I have considered doing the POE board swap the next chance I can to take the core network down. Might be a while for that though.

Since they're responding to commands (I've seen the error when it doesn't due to a missing resistor I had to replace) I'd think its maybe port 1 or 2 (the only ones I've tried so far) that have the issue.

I'll get the others turned on and see if they're fine or broken.

The unit the PoE board came from died unexpectedly, and even with @fohdeesha 's magic JTAG box, we weren't able to bring it back.

 

[Wolfcastle]

I'll probe around with my meter a bit more.

As it so happens, I DO have a 100mhz scope and an old logic analyzer sitting next to my healing bench.

I have considered doing the POE board swap the next chance I can to take the core network down. Might be a while for that though.

Since they're responding to commands (I've seen the error when it doesn't due to a missing resistor I had to replace) I'd think its maybe port 1 or 2 (the only ones I've tried so far) that have the issue.

I'll get the others turned on and see if they're fine or broken.

The unit the PoE board came from died unexpectedly, and even with @fohdeesha 's magic JTAG box, we weren't able to bring it back.

If you see my post above, ports 1-4 are dead on my 24P switch but others seem to be OK (haven't tested POE since the POE board in my switch was fried beyond the amount of effort I was willing to put into fix it). I assume you do have ethernet comms (non-POE) on ports 1 and 2 on this switch? Might be worth trying the other ports since it's pretty quick and low effort compared to other troubleshooting

 

[Wolfcastle]

@fohdeesha I tried following your configuration on my ICX 7450 but the primary boot doesn't seem to work, any ideas? This is what I get. It seems like it is still using the 1.1.05 bootloader when it tries to load the primary, this seems wrong?

Brocade Bootloader: 10.1.05T215 (Mar 19 2015 - 16:39:20)

Validate Shmoo parameters stored in flash ..... OK
Restoring Shmoo parameters from flash .....
Running simple memory test ..... OK
ICX7450-24 Copper (POE), PVT1
SYS CPLD VER: 0x10, Released Ver: 0

Enter 'b' to stop at boot monitor:  0
bootdelay: ===
Booting image from Primary
.......................................................................................................................................................................................................................................................................................................................................Wrong Image Format for bootm command
ERROR: can't get kernel image!
could not boot from primary, no valid image; trying to boot from secondary
BOOTING image from Secondary

System initialization completed...console going online.
  Copyright (c) 1996-2015 Brocade Communications Systems, Inc. All rights reserved.
    UNIT 1: compiled on Jan 26 2016 at 22:35:15 labeled as SPR08030f
      (31662276 bytes) from Secondary SPR08030f.bin
        SW: Version 08.0.30fT213
      Compressed Boot-Monitor Image size = 786944, Version:10.1.05T215 (spz10118)

Boots fine into the secondary image though...

edit: it did work previous to the update however (edit2 removed the log, not necessary).

edit 2: OK fixed the problem. I had to run update_uboot immediately before running update_primary otherwise I kept getting the old bootloader after reset. This fixed my issue where the primary boot kept failing because it was using bootloader 10.1.05 instead of 10.1.18 with the newer firmware. The instructions probably could use this minor update

 

[Rain]

Has anyone tried stacking ICX switches with SFP+ 10GBASE-T transceivers? If not, can someone with an existing stack and some spare 10GBASE-T transceivers give it a try?

I was planning on running the two switches I purchased in a stack but the cabling between the switches is unfortunately only CAT6; running fiber is possible, but not easy. Thankfully the existing cabling between the two locations is SSTP CAT6 and the distance shouldn't be an issue.

I purchased a 4-pack of QSFPTEK SFP+ 10GBASE-T transceivers and they all appear to work perfectly between NICs and switches of various brands. Connecting the two switches together with CAT6 seems to work perfectly as well if stacking isn't enabled or the transceivers are in non-stacking ports; no dropped packets with intense iperf testing, rated 10Gb speeds.

As soon as stacking is enabled, the ports refuse to come up, regardless if a stack is created or not. If the 10GBASE-T transceivers are plugged into the stacking ports on either switch and communicating perfectly, simply running "stack enable" causes them to flap and then drop out.

My current assumption is that the transceivers first connect at 1Gb (or some NBASE-T speed since they're multi-gigabit compatible) before switching to 10Gb and since 10Gb is required for stacking, the switch resets the port a few times before giving up. I'm hoping this is simply an incompatibility with these specific transceivers and another make/model transceiver will work.

If anyone has a different brand set and can take a few minutes to test for me, I would really appreciate it! Likewise, is anyone aware of any configuration changes I may be able to make to resolve this?

edit 2: OK fixed the problem. I had to run update_uboot immediately before running update_primary otherwise I kept getting the old bootloader after reset. This fixed my issue where the primary boot kept failing because it was using bootloader 10.1.05 instead of 10.1.18 with the newer firmware.

Seems similar to what I ran into above with a 7250 that was running the 10.1.06 bootloader (similar 2015 build date). The uboot update didn't stick until after booting into the newly primary image and the only way to boot into the new primary image was to re-flash the new bootloader after flashing the primary image. Weird chicken & egg problem.

 

[infoMatt]

As a general advice, avoid stacking unless more ports are needed in the same IDF. Definitely I wouldn't span stacked switches across multiple floors or buildings. Let L3 do it's job, the smaller your L2 domains are, the better.

 

[juju]

As a general advice, avoid stacking unless more ports are needed in the same IDF. Definitely I wouldn't span stacked switches across multiple floors or buildings. Let L3 do it's job, the smaller your L2 domains are, the better.

Interesting. I got a 7250 a few months back and have been planning to get another one for redundancy - in a stack. Are you saying it's a bad idea? I don't really need the ports.

btw - can I stack a 7250-24 and a 7250-24P ?

 

[infoMatt]

Interesting. I got a 7250 a few months back and have been planning to get another one for redundancy - in a stack. Are you saying it's a bad idea? I don't really need the ports.

Yes in this case you can do it

btw - can I stack a 7250-24 and a 7250-24P ?

I think that you can, but the most expert is by far @fohdeesha .

 

[Didomir]

I was wondering how the price/performance ratio looks like in 2022 ? I was looking to buy 7250/7450/6450/6610 24P models, however prices seems to be high. I'll need a pair of 24P in Q3/22 - Q1/22 shall I wait for good price of 7250/7450 (price and noise are in consideration) or I can go with 6610/6450 (8080vs8030) ?

 

[Zombielinux]

@fohdeesha @Wolfcastle

The final conclusion.

You CANNOT convert a non-PoE ICX6610 to PoE.

Reason #1 The headers aren't on the non-PoE boards.
Reason #2 (the real show stopper). The RJ45 jacks are not connected to PoE because the part number is different from the PoE models (Bel-Fuse 1840860 non-PoE vs Bel-Fuse 2250308 PoE).

Unless you desolder all the RJ-45 ports, and reinstall them on a different board, there is no way to make this work.

I may try that if I ever get bored. It will probably be a while.

 

[Rain]

As a general advice, avoid stacking unless more ports are needed in the same IDF. Definitely I wouldn't span stacked switches across multiple floors or buildings. Let L3 do it's job, the smaller your L2 domains are, the better.

Well, I'm either going to LAG two ports between the switches or stack. I figured a stack would be better & simpler (one logical switch vs two). Regardless of which is better in this instance, I'm still trying to figure out if stacking is possible with SFP+ Base-T transceivers that seem to work perfectly in non-stacking ports.

 

[Wolfcastle]

@fohdeesha @Wolfcastle

The final conclusion.

You CANNOT convert a non-PoE ICX6610 to PoE.

Reason #1 The headers aren't on the non-PoE boards.
Reason #2 (the real show stopper). The RJ45 jacks are not connected to PoE because the part number is different from the PoE models (Bel-Fuse 1840860 non-PoE vs Bel-Fuse 2250308 PoE).

Unless you desolder all the RJ-45 ports, and reinstall them on a different board, there is no way to make this work.

I may try that if I ever get bored. It will probably be a while.

Interesting, and it makes sense, I’m sure the POE connectors are more expensive than the non-PoE ones!

 

[kpfleming]

You can stack any units from the same family (71xx, 72xx, etc.)

I went the other direction: rather than getting a single large switch, I got multiple 7150-C12Ps, and stacked them. This provides power supply redundancy (among others), and they are also fanless. I have a few devices using two-port LAGs and those are split across switch units so they'll continue to operate even if one of those units fails.

I've also got a unit in the garage, which is stacked with the ones in the house, since I didn't feel like managing it as a separate unit.

 

[Wolfcastle]

I was wondering how the price/performance ratio looks like in 2022 ? I was looking to buy 7250/7450/6450/6610 24P models, however prices seems to be high. I'll need a pair of 24P in Q3/22 - Q1/22 shall I wait for good price of 7250/7450 (price and noise are in consideration) or I can go with 6610/6450 (8080vs8030) ?

I bought both a 6610-24P ($180) and a 7450-24P ($260). The 7450 came in a bundle that included a 4x10G module and 2 40G modules making it about equivalent to the 6610 performance wise. My 7450 is almost as loud as my 6610 but I just got it on Friday and haven’t spent too much time on it yet. The 7450 has some higher power PoE ports and is still supported hardware which is pretty nice IMO

 

[RobstarUSA]

I just picked up a 6610 48 port with POE, dual power supply for about $150 shipped...I though that was pretty good. Both my 6610 not poe with one power supply were about $90 or so.

Anyone know where to pick up spare fan parts? not the fans themself but the entire assembly. When I look on the 'bay they are $40-$50 each which is crazy when the whole switch can be picked up for the price of 2 fans.

 

[Rain]

Has anyone tried stacking ICX switches with SFP+ 10GBASE-T transceivers? If not, can someone with an existing stack and some spare 10GBASE-T transceivers give it a try?

... snip ...

As soon as stacking is enabled, the ports refuse to come up, regardless if a stack is created or not. If the 10GBASE-T transceivers are plugged into the stacking ports on either switch and communicating perfectly, simply running "stack enable" causes them to flap and then drop out.

I've still been playing around and trying to get stacking working with 10GBASE-T transceivers between two 7250-48p switches. Nothing I've tried seems to work.

Just to test, I created a stack with DACs and after everything was running normally I swapped them with the 10GBASE-T transceivers. The link lights turn on for both ports, flicker a few times, and then turn off; after a bit of time, this repeats. If optics or DACs are swapped back in everything links back up.

Again, if the switches are not setup in a stack, a 10GBASE-T connection between the switches works perfectly (without dropped packets or anything); it's only stacking that causes issues.

I've tried both "sets" of stacking ports (1/2/1, 1/2/3 vs 1/2/5, 1/2/7) and enabled various items in the debugging menu. Nothing seems to indicate an error. That said, there are a ton of things in the debugging menu (and sub-menus) so I may not have found the correct option...

Anyone have any ideas?

 

[kpfleming]

Which version of the ICX software are you using? Also, have you considered asking about this on the Ruckus Wireless support forum? The tech team here is pretty responsive.

 

[Rain]

Which version of the ICX software are you using?

Currently, 08.0.95f.

Also, have you considered asking about this on the Ruckus Wireless support forum? The tech team here is pretty responsive.

I have not. I figured since this is likely far from a supported mode of operation (stacking with BASE-T adapters) they wouldn't provide much help. There's also this thread (link) where they indicate stacking on their own 10G RJ45 transceivers isn't even supported. I may give it a shot though, I suppose.

 

[brb78]

how do sth members get the licensing to upgrade from sfp to sfp+ as mentioned in post#6?