Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

[fohdeesha]

I recently picked up a 24 port ICX7150 after finding a good eBay deal on a brand new one (I liked the fanless nature of it and the even lower power draw than the 6450). I've been doing some configuration on it and had a question that I wondered whether anyone could help with.

It seems to come with two of the 10 gig ports configured for stacking by default. I don't want to do stacking and would rather use them to connect devices. I was able to remove one, but now if I do "no stack-port 1/3/1" to try to remove the remaining one, I get "Error! cannot remove the only stack-port 1/3/1."

I've already disabled stacking. Is there a restriction where one of the ports has to be used for stacking, or is there something else I'm missing (I'm new to these enterprise type switches)? What happens if you just connect a regular device to a stacking port?

Also thanks to Fohdeesha for the excellent guides and resources!

it might be because they'll only work for stacking unless the 10gb license is installed? Haven't used a full size 7150. Make sure you're running the latest bootloader and firmware, (use the 7150 config guide all the way through, jit's the same firmware: ICX7150 - Fohdeesha Docs ). However the licenses available will differ from the guide examples. at the enable level type "license install perpetual 1" and hit tab a bunch, it should show you the available license name arguments for that command, run the full command with each of them to install all the lics, then you need to set the ports to 10gb which I believe on that switch would be:

conf t
int e 1/3/1 to 1/3/4
no speed-duplex 1000-full
speed-duplex 10g-full
exit
write mem
exit
reload


when it comes back up you should be able to run "no stack-port xxx" on everything. If you still can't, paste the output of "show stack". Looking at their command example for the full size 7150, it might just always have 1/3/1 as a stacking port, but you can still use it for devices I guess: Converting stacking ports to data ports on ICX 7150 devices

 

[Vorwrath]

it might be because they'll only work for stacking unless the 10gb license is installed? Haven't used a full size 7150. Make sure you're running the latest bootloader and firmware, (use the 7150 config guide all the way through, jit's the same firmware: ICX7150 - Fohdeesha Docs ). However the licenses available will differ from the guide examples. at the enable level type "license install perpetual 1" and hit tab a bunch, it should show you the available license name arguments for that command, run the full command with each of them to install all the lics, then you need to set the ports to 10gb which I believe on that switch would be:

conf t
int e 1/3/1 to 1/3/4
no speed-duplex 1000-full
speed-duplex 10g-full
exit
write mem
exit
reload


when it comes back up you should be able to run "no stack-port xxx" on everything. If you still can't, paste the output of "show stack". Looking at their command example for the full size 7150, it might just always have 1/3/1 as a stacking port, but you can still use it for devices I guess: Converting stacking ports to data ports on ICX 7150 devices

I'd already updated the firmware (used 08.0.90d which seemed to be their current recommendation) as well as figured the licensing out and set the ports to 10g. It's "license install perpetual 1 4x10gr" if you want it for the guide.

SSH@typhon#conf t
SSH@typhon(config)#stack unit 1
SSH@typhon(config-unit-1)#show stack

***** Warning! stack is not enabled. *****

T=3m16.8: alone: standalone, D: dynamic cfg, S: static
ID   Type          Role    Mac Address    Pri State   Comment
1  S ICX7150-24    alone   609c.9ff5.0cb4   0 local   None:0


     +---+
  3/1| 1 |
     +---+
Current stack management MAC is 609c.9ff5.0cb4
SSH@typhon(config-unit-1)#no stack-port 1/3/1
Error! cannot remove the only stack-port 1/3/1.

It just seems like there always has to be one stack port set. I'm still waiting on some cables so haven't tried plugging anything into it yet, so I don't know whether that matters. I've only really got a use for three 10g ports at the moment anyway, was just curious about it.

 

[fohdeesha]

the manual states in several places all 4 ports can be used as data ports, so the question is just how. I have a feeling the lone remaining stack port will just default to functioning as a data port until you actually run the stack setup command. Also FYI 8090 and beyond is not recommended by ruckus for production (which is why it's not on the guide site) but for home use it's probably fine. Ran into some fun stuff (random reboots) on 8090 on our 7750s. Ruckus has a pdf somewhere of recommended codetrain info, if I recall correctly even 8080 isn't recommended for production yet but it has the most licensing goodies

 

[fohdeesha]

oh neat, it looks like on december 10th they finally moved the target path from 8070 to 8090 for most 7 series: https://fohdeesha.com/data/other/brocade/ruckus-icx-target-path.pdf

granted the version they're recommending is the exact version that has known random reboot issues (at least on the 7750), so I'm not sure how much I would trust that

 

[Vorwrath]

oh neat, it looks like on december 10th they finally moved the target path from 8070 to 8090 for most 7 series: https://fohdeesha.com/data/other/brocade/ruckus-icx-target-path.pdf

granted the version they're recommending is the exact version that had known random reboot issues, so I'm not sure how much I would trust that

Yeah, I was going on Ruckus ICX 7150 Campus Switches | Products | Ruckus Wireless Support where they seem to suggest that version, so I guess it's changed recently.

I've only got it for home use, so I might well not hit the same issues (or if I do it's easy enough to swap to another version). I'll give that port a try once I've got some cables and will find out whether it can run as a data port or not.

 

[M2Shawning]

Hello! I recently picked up an ICX 6450 and am having problems with the fan noise. The fan does not seem to throttle down after booting up on my stock fan nor my third party 10 CFM fan. I followed the provided guides with no results. Any and all help would be much appreciated!

6450#show cha
The stack unit 1 chassis info:

Power supply 1 (NA - AC - Regular) present, status ok
Power supply 2 not present

Fan ok, speed (auto): [[1]]<->2

Fan speed switching temperature thresholds:
        1 -> 2 @ 69 deg-C
        1 <- 2 @ 64 deg-C

Sensor B Temperature Readings:
        Current temperature : 34.0 deg-C
Sensor A Temperature Readings:
        Current temperature : 40.5 deg-C
        Warning level.......: 66.0 deg-C
        Shutdown level......: 76.0 deg-C

6450#show ver
  Copyright (c) 1996-2016 Brocade Communications Systems, Inc. All rights reserved.
    UNIT 1: compiled on Feb 13 2019 at 17:44:29 labeled as ICX64R08030t
                (9868556 bytes) from Primary ICX64R08030t.bin
        SW: Version 08.0.30tT313
  Boot-Monitor Image size = 786944, Version:10.1.05T310 (kxz10105)
  HW: Stackable ICX6450-48
==========================================================================
UNIT 1: SL 1: ICX6450-48 48-port Management Module
         Serial  #: BZU2522H04P
         License: ICX6450_BASE_ROUTER_SOFT_PACKAGE   (LID: XXXXXXXXXXX)
         P-ENGINE  0: type DEF0, rev 01
         P-ENGINE  1: type DEF0, rev 01
==========================================================================
UNIT 1: SL 2: ICX6450-SFP-Plus 4port 40G Module
==========================================================================
  800 MHz ARM processor ARMv5TE, 400 MHz bus
65536 KB flash memory
  512 MB DRAM
STACKID 1  system uptime is 18 minute(s) 10 second(s)
The system started at 00:00:39 GMT+00 Thu Jan 01 1970
 The system : started=warm start         reloaded=by "reload"

6450#dm fan-speed
All fans in Low speed (0xff)

 

[u238]

Hello! I recently picked up an ICX 6450 and am having problems with the fan noise. The fan does not seem to throttle down after booting up on my stock fan nor my third party 10 CFM fan. I followed the provided guides with no results. Any and all help would be much appreciated!

6450#show cha
The stack unit 1 chassis info:

Power supply 1 (NA - AC - Regular) present, status ok
Power supply 2 not present

Fan ok, speed (auto): [[1]]<->2

Fan speed switching temperature thresholds:
        1 -> 2 @ 69 deg-C
        1 <- 2 @ 64 deg-C

Sensor B Temperature Readings:
        Current temperature : 34.0 deg-C
Sensor A Temperature Readings:
        Current temperature : 40.5 deg-C
        Warning level.......: 66.0 deg-C
        Shutdown level......: 76.0 deg-C

6450#show ver
  Copyright (c) 1996-2016 Brocade Communications Systems, Inc. All rights reserved.
    UNIT 1: compiled on Feb 13 2019 at 17:44:29 labeled as ICX64R08030t
                (9868556 bytes) from Primary ICX64R08030t.bin
        SW: Version 08.0.30tT313
  Boot-Monitor Image size = 786944, Version:10.1.05T310 (kxz10105)
  HW: Stackable ICX6450-48
==========================================================================
UNIT 1: SL 1: ICX6450-48 48-port Management Module
         Serial  #: BZU2522H04P
         License: ICX6450_BASE_ROUTER_SOFT_PACKAGE   (LID: XXXXXXXXXXX)
         P-ENGINE  0: type DEF0, rev 01
         P-ENGINE  1: type DEF0, rev 01
==========================================================================
UNIT 1: SL 2: ICX6450-SFP-Plus 4port 40G Module
==========================================================================
  800 MHz ARM processor ARMv5TE, 400 MHz bus
65536 KB flash memory
  512 MB DRAM
STACKID 1  system uptime is 18 minute(s) 10 second(s)
The system started at 00:00:39 GMT+00 Thu Jan 01 1970
 The system : started=warm start         reloaded=by "reload"

6450#dm fan-speed
All fans in Low speed (0xff)

I have the exact issue with my 6450. The fan is always at full speed while show chassis reports low fan speed. I just swapped in a quieter fan and haven't had an issue. No idea what the problem is.

 

[M2Shawning]

I have the exact issue with my 6450. The fan is always at full speed while show chassis reports low fan speed. I just swapped in a quieter fan and haven't had an issue. No idea what the problem is.

Damn that sucks to hear. I could opt for a quieter fan but that's obviously less than optimal. If I find anything I'll be sure to report back.

 

[fohdeesha]

Hello! I recently picked up an ICX 6450 and am having problems with the fan noise. The fan does not seem to throttle down after booting up on my stock fan nor my third party 10 CFM fan. I followed the provided guides with no results. Any and all help would be much appreciated!

6450#show cha
The stack unit 1 chassis info:

Power supply 1 (NA - AC - Regular) present, status ok
Power supply 2 not present

Fan ok, speed (auto): [[1]]<->2

Fan speed switching temperature thresholds:
        1 -> 2 @ 69 deg-C
        1 <- 2 @ 64 deg-C

Sensor B Temperature Readings:
        Current temperature : 34.0 deg-C
Sensor A Temperature Readings:
        Current temperature : 40.5 deg-C
        Warning level.......: 66.0 deg-C
        Shutdown level......: 76.0 deg-C

6450#show ver
  Copyright (c) 1996-2016 Brocade Communications Systems, Inc. All rights reserved.
    UNIT 1: compiled on Feb 13 2019 at 17:44:29 labeled as ICX64R08030t
                (9868556 bytes) from Primary ICX64R08030t.bin
        SW: Version 08.0.30tT313
  Boot-Monitor Image size = 786944, Version:10.1.05T310 (kxz10105)
  HW: Stackable ICX6450-48
==========================================================================
UNIT 1: SL 1: ICX6450-48 48-port Management Module
         Serial  #: BZU2522H04P
         License: ICX6450_BASE_ROUTER_SOFT_PACKAGE   (LID: XXXXXXXXXXX)
         P-ENGINE  0: type DEF0, rev 01
         P-ENGINE  1: type DEF0, rev 01
==========================================================================
UNIT 1: SL 2: ICX6450-SFP-Plus 4port 40G Module
==========================================================================
  800 MHz ARM processor ARMv5TE, 400 MHz bus
65536 KB flash memory
  512 MB DRAM
STACKID 1  system uptime is 18 minute(s) 10 second(s)
The system started at 00:00:39 GMT+00 Thu Jan 01 1970
 The system : started=warm start         reloaded=by "reload"

6450#dm fan-speed
All fans in Low speed (0xff)

the PWM fan control circuitry is fried and sending wide open 12v to the fans even though the switch is trying to send fan speed 1 (~4.5v). Seen it on a couple other 6450's @u238 included but it seems quite rare. You can either swap in a slow enough fan that full speed is quiet, or contact the ebay seller for a return

 

[juey]

the PWM fan control circuitry is fried and sending wide open 12v to the fans even though the switch is trying to send fan speed 1 (~4.5v). Seen it on a couple other 6450's @u238 included but it seems quite rare. You can either swap in a slow enough fan that full speed is quiet, or contact the ebay seller for a return

That said, having some basic smd soldering skills, couldnt the PWM ic be replaced ? I think i ran into this issue in one of my 6450, too. RMA no longer possible.

 

[fohdeesha]

That said, having some basic smd soldering skills, couldnt the PWM ic be replaced ? I think i ran into this issue in one of my 6450, too. RMA no longer possible.

sure assuming that's the actual failure, I suppose the PWM driver could also be part of some larger IC as well like all the I2C switchers and crap in that section of the board, and that would make it a PITA to replace. on the other hand it could be as simple as the big 3 pin transistor right above the fan header, you'd have to test

 

[Vorwrath]

the manual states in several places all 4 ports can be used as data ports, so the question is just how. I have a feeling the lone remaining stack port will just default to functioning as a data port until you actually run the stack setup command. Also FYI 8090 and beyond is not recommended by ruckus for production (which is why it's not on the guide site) but for home use it's probably fine. Ran into some fun stuff (random reboots) on 8090 on our 7750s. Ruckus has a pdf somewhere of recommended codetrain info, if I recall correctly even 8080 isn't recommended for production yet but it has the most licensing goodies

I've given it a test now and you were right - the port works fine as a data port without doing anything special. So all 4 are usable for devices. Seems like a great switch for my needs, I'm happy with it!

 

[M2Shawning]

the PWM fan control circuitry is fried and sending wide open 12v to the fans even though the switch is trying to send fan speed 1 (~4.5v). Seen it on a couple other 6450's @u238 included but it seems quite rare. You can either swap in a slow enough fan that full speed is quiet, or contact the ebay seller for a return

Damn, well I've just purchased some lower speed fans and "silent fan adapters" so that should work around the issue. Thank you for the insight!

 

[blinkenlights]

After upgrading and tweaking, I've been mostly satisfied with the ICX7450-48 I rescued from a datacenter. One exception is occasional port errors that seem to be associated with 100M devices on the embedded 48-port (not module) side. I have a large mix of 100M/1G/10G copper devices, all running at FDX. The 100M devices include two SiliconDust HDHomeRun tuners and two Sony Bravia XBR series televisions; cabling is fine. The switch is running the latest switch code train (SPS08092a.bin) but these errors were also seen in earlier versions.

Throughout the day, errors like this (debug was enabled for this one) appear in the log:
ICX7450-48 Switch>
stack: 01864cac 01865bf4 01869698 b6e2abd0 b6f29195
1:_bcm_esw_linkscan_update_port: Port ge18: bcm_port_info_get failed: Operation failed
1:_bcm_esw_linkscan_update_port: Port ge18: Link Up with Speed 0M!

I assume 'ge18' corresponds to the Sony XBR wired to port 1/1/19 and not the empty 1/1/18. Hardcoding speed/duplex makes no difference.

Questions: Is this anything other than a nuisance that's associated with 100M devices? And if more than just a nuisance, should I consider throwing them onto a separate switch?

 

[iotapi322]

So after a few weeks I can tell you that the 6610 was just too loud. I will blank the licenses out of it and try to sell it on ebay ( or here if anyone wants it). However I did pickup a 6450-48 and was able to flash it and unlimit license the chassis. ( Thanks for those great guides @fohdeesha )

Oh and to put a cap on the above messages it was totally the cisco card that was causing the issue. The mellanox cards ( after upgrading all of them worked great).

 

[iotapi322]

one more thing... anyone got a set of rack mounts for the 6450 the want to sell or send me the dimensions too? That would be AMAZING!
Please and thank you


Can anyone comment on whether any of the "Cisco Compatible" rack kits will work with the 6450-48?

something like this?

 

[HotRodNerd]

I just want to add the long list and say thanks for making this forum thread, HUGE help!

 

[fohdeesha]

After upgrading and tweaking, I've been mostly satisfied with the ICX7450-48 I rescued from a datacenter. One exception is occasional port errors that seem to be associated with 100M devices on the embedded 48-port (not module) side. I have a large mix of 100M/1G/10G copper devices, all running at FDX. The 100M devices include two SiliconDust HDHomeRun tuners and two Sony Bravia XBR series televisions; cabling is fine. The switch is running the latest switch code train (SPS08092a.bin) but these errors were also seen in earlier versions.

Throughout the day, errors like this (debug was enabled for this one) appear in the log:
ICX7450-48 Switch>
stack: 01864cac 01865bf4 01869698 b6e2abd0 b6f29195
1:_bcm_esw_linkscan_update_port: Port ge18: bcm_port_info_get failed: Operation failed
1:_bcm_esw_linkscan_update_port: Port ge18: Link Up with Speed 0M!

I assume 'ge18' corresponds to the Sony XBR wired to port 1/1/19 and not the empty 1/1/18. Hardcoding speed/duplex makes no difference.

Questions: Is this anything other than a nuisance that's associated with 100M devices? And if more than just a nuisance, should I consider throwing them onto a separate switch?

Do you still get errors in the log when not in debug mode? When they happen, does the link flap or otherwise disturb the actual device on the other end? Do you see any CRC errors, inerrors or outerrors when running show stat e 1/1/1? If not I would just ignore it, not ideal but yet another reason I don't have the 7450's listed in the OP lmao

 

[UKEE]

you can open it up and physically reverse the fans, but I would expect less airflow this way than just leaving it stock upside down (in my experience intake fans need to run much faster to achieve the same airflow as exhaust fans in these switch layouts), I would test each way

Just for kicks, I tried this with the ICX7250-24. First, I placed the slower Sunon fan into the unit but it caused the unit to go into a boot loop (any slower fan that I tried did the same). The unit would do the high speed fan startup but when finally going to the slower running speed, it would sit for a few seconds and just reboot (over and over and over again) With that in mind, I recplaced the Foxconn fan with the Nidec that I pulled from the ICX6450. It was noticeably quieter and moved less air than the original 7250 Foxconn fan but was much faster, louder and moved far more air than the Sunon. Just monitoring from the WebGUI, the temperature was over 20C LOWER with the fan blowing in than exhausting. 62.5 vs 85.9C. The fan blowing actually blows across the CPU heatsink of the switch (at least in the case of the ICS7250-24). I cut the fan guard from the back of the unit as well. That was responsible from another 2-3C drop (I suppose more airflow into the case).

I also tried a fan controller on the Foxconn fan. I could turn the speed down (and noise too) on the initial high speed startup but the fan would quit when going into regular slowdown mode no matter the dial setting. At that point, the switch would simply reboot starting the whole process over and over and over again. I guess the voltage was too low to run the Foxconn fan.

 

[blinkenlights]

Do you still get errors in the log when not in debug mode? When they happen, does the link flap or otherwise disturb the actual device on the other end? Do you see any CRC errors, inerrors or outerrors when running show stat e 1/1/1? If not I would just ignore it, not ideal but yet another reason I don't have the 7450's listed in the OP lmao

Thanks. Yes, the errors occur when not in debug mode. Not seeing any link flaps, errors, and unclear whether the device is disturbed (Sony's diagnostic stack is not terribly helpful, LOL). My gut tells me to just ignore these shenanigans and blame the entertainment device manufacturer for putting substandard silicon in their devices.

Oh, just for the record, same errors on the ICX7250 ;-p