Drag to reposition cover

Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

fohdeesha

Kaini Industries
Nov 20, 2016
2,277
2,358
113
30
fohdeesha.com
So when I said page 300, I meant page 301. The new guide is finally out, 112 git commits later. There's too many changes for me to list them all but I'll list the big ones. The big "surprise" is licensing information for the EoL icx6xxx series is now public, and integrated into the guide. Changes I can think of offhand:

  • Changed color theme to orange for halloween (most important & most arduous change)

  • The big one: Integrated licensing/unlock information for all models into the guide, so it's no longer necessary to contact me. I've generated a "master set" of licenses for all models, tied to a custom serial and license ID of my choosing, so it will be incredibly easy to see if these pop up on ebay etc. There's a bit of an easter egg in the serial, see if you can find it after updating your chassis and then viewing your licenses

  • The other big one: Came up with a set of instructions to get ICX7xxx series models onto the latest 8095 UFI images. This one took some poking around on some lab switches, but I believe I found the cleanest method that doesn't require a massive list of "if" statements. The 8090 train can be flashed from *any* version bootloader, older or newer. The 8090 train is also capable of flashing the new UFI image types. So, the guide has you drop into the bootloader of your new switch, and flash 8090m. Then do a quick boot into 8090m, and flash the 8095d UFI image

  • If your switch happens to already be on an 8090+ version, this could be seen as an extra step, but it's the only method that will work *regardless* of the state your switch comes in. If it comes with 8040, 8050, 8060, 8070, 8080, 8090, 8092, 8095, no OS, all these combinations can follow the same guide page without traversing down a list of "if this version, then go here and do this" options. The other big advantage here is once you flash the 8095 UFI image, it automatically handles updating the bootloader and PoE firmware to the latest on first boot, so it's not necessary to have the user touch these at all, so the overall update guide for ICX7xxx is now shorter. If you already have a switch updated and configured using the old guide (so probably version 8080), you can just follow the new guide starting from the "Load The Latest UFI Image" section, as the 8080 image you're on supports UFI firmware upgrades.

  • So, as mentioned above, the ICX7xxx guides have been updated to the latest stable codetrain Ruckus now recommends, which is 8095d

  • ICX6xxx guides have been updated to the latest release for these, 8030u

  • The other other big one: Split all the switch guides into two parts: the model-specific firmware update stuff, and then a separate page for further configuration, setup etc. That stuff is all common to the entire switch line, so this allowed me to aggregate all that into a single "further configuration" page. So, each switch guide page has the basic wipe/update/give it an IP instructions. Then you switch to a common "ICX6xxx Config" page for ssh etc setup. There's a common ICX7xxx config page for that series as well. Two different common setup pages were required as there's quite a lot of differences between the 8030 and 8095 firmware. For example, the 8095 firmware has a mandatory default login, has smartzone crap enabled by default that you want to disable usually, etc. Having common pages for config stuff makes it much easier for me to:

  • Add a lot of info and config options. HTTPS webserver setup, PoE configuration and monitoring, LACP setup, helpful commands, optics info, etc. I will be adding more to these sections in the following week (VLANs, interVLAN routing, stacking, etc)

  • Migrated all switch upgrade methods to occur in the bootloader. Most were already like this (and of course now the ICX7xxx series is, as described above). However a couple ICX6xxx models still had you boot whatever mystery OS your switch came with and update from there. This led to a lot of "if" statements to handle weird situations like your switch coming with layer 2 firmware, in a stack configuration, etc. None of this is necessary and has been cleaned up now that the guide has you just drop into the bootloader, wipe everything, and flash a known latest L3 image

  • Removed the split "if access protection is required / if access protection is not required" option sections for the icx6xxx guides. In the newer ICX7xxx images it's no longer optional, as it comes by default secured with the super user. This means the ICX7xxx config guide does not have the choice of "if no access protection is required", and I wanted the icx6xxx config guide to match, as there's really no reason to not have a basic user/password set

  • Integrated all firmware packages, license files, etc into a single zip download - same single download regardless of what series switch you have. This means when new firmware is pushed I just have one zip to update. this also allowed me to do the following:

  • Add a "Brocade Overview" landing page, with the main firmware/licensing zip download. This is the new "starting point" regardless of what model you have. On this page, I've added decently detailed info/instructions on setting up a TFTP server on both windows and *nix. Having this in a single starting page allowed me to remove all the extraneous tftp instructions/notes from every single switch guide page

  • Added a ready to go portable copy of Tftpd64 to the main firmware zip, it's even set to the correct directory. Windows users can just launch this and have a TFTP server properly configured and running (assuming your windows firewall doesn't interfere)

  • Split the old combo icx6430/icx6450 config page into two distinct pages for each switch - the icx6430 cannot run the layer3 image, so the combo guide with layer3 instructions for the icx6450 was not very easy to follow (impossible, really)

  • Reorganized the left-hand menu to make more sense, and group common stuff

  • Add note to ICX6xxx config page about them not supporting 4096 bit SSH key pairs, which caused some issues with users in here before

  • Moved the ICX6610 stack ports page into its own section in the fcx/6610 update/config page

  • Add all the up to now private reverse engineering info I had on my old private site most of you had. It's all under the "brocade fun" section

  • Went through and updated all URLs, file links, etc, to relative paths to a folder within the docs folder. This allowed me to:

  • Add an archive download of the entire guide site with all files, zips, etc, so if something happens, you can use the site and all downloads entirely offline. you'll find it under the "Home" menu


There's ~100 other small changes that I don't recall offhand or aren't worth listing out, but I think the guide is pretty clean now, covers more use cases, leaves less room for error, and answers more common questions that we repeatedly answer in here. Some stats from the past 3 years of this thread for fun:

  • 780,000 thread views (most viewed thread on STH)
  • 6,000 thread replies (most replied thread on STH)
  • 1,392 private messages to my STH inbox (license requests, questions that should have been asked in the thread)
  • 1,102 emails to my email inbox (license requests, questions that should have been asked in the thread)
  • 282 commits to the documentation github repo
  • 5 contributors to the github repo other than me (thank you guys!)
  • 900 billion million switches bought by all of you nerds

Please let me know if you find any typos etc in the new guide, it was thousands of new lines so I'm sure I missed just one somewhere. I'm now hopping on the midnight train out of beeftown and I think I can consider my duty done. this whole time I was the brocade CEO and you've all pumped up my stock numbers enough, thanks!

Big thanks to @Patrick for allowing this monster of a thread to operate "hands off" for three years, and providing a site with a mature enough user base to make it to 300 pages without a meltdown

Lastly a huge thanks to the handful of STH members in here who have stuck around in this thread and helped users over and over for hundreds of pages now. I wanted to list you all by name but then realized I'd forget someone then feel like an asshole, so: you know who you are, thank you!
 
Last edited:

RoachedCoach

Member
Feb 4, 2020
26
30
13
So when I said page 300, I meant page 301. The new guide is finally out, 112 git commits later. There's too many changes for me to list them all but I'll list the big ones. The big "surprise" is licensing information for the EoL icx6xxx series is now public, and integrated into the guide. Changes I can think of offhand:

  • Changed color theme to orange for halloween (most important & most arduous change)

  • The big one: Integrated licensing/unlock information for all models into the guide, so it's no longer necessary to contact me. I've generated a "master set" of licenses for all models, tied to a custom serial and license ID of my choosing, so it will be incredibly easy to see if these pop up on ebay etc. There's a bit of an easter egg in the serial, see if you can find it after updating your chassis and then viewing your licenses

  • The other big one: Came up with a set of instructions to get ICX7xxx series models onto the latest 8095 UFI images. This one took some poking around on some lab switches, but I believe I found the cleanest method that doesn't require a massive list of "if" statements. The 8090 train can be flashed from *any* version bootloader, older or newer. The 8090 train is also capable of flashing the new UFI image types. So, the guide has you drop into the bootloader of your new switch, and flash 8090m. Then do a quick boot into 8090m, and flash the 8095d UFI image

  • If your switch happens to already be on an 8090+ version, this could be seen as an extra step, but it's the only method that will work *regardless* of the state your switch comes in. If it comes with 8040, 8050, 8060, 8070, 8080, 8090, 8092, 8095, no OS, all these combinations can follow the same guide page without traversing down a list of "if this version, then go here and do this" options. The other big advantage here is once you flash the 8095 UFI image, it automatically handles updating the bootloader and PoE firmware to the latest on first boot, so it's not necessary to have the user touch these at all, so the overall update guide for ICX7xxx is now shorter. If you already have a switch updated and configured using the old guide (so probably version 8080), you can just follow the new guide starting from the "Load The Latest UFI Image" section, as the 8080 image you're on supports UFI firmware upgrades.

  • So, as mentioned above, the ICX7xxx guides have been updated to the latest stable codetrain Ruckus now recommends, which is 8095d

  • ICX6xxx guides have been updated to the latest release for these, 8030u

  • The other other big one: Split all the switch guides into two parts: the model-specific firmware update stuff, and then a separate page for further configuration, setup etc. That stuff is all common to the entire switch line, so this allowed me to aggregate all that into a single "further configuration" page. So, each switch guide page has the basic wipe/update/give it an IP instructions. Then you switch to a common "ICX6xxx Config" page for ssh etc setup. There's a common ICX7xxx config page for that series as well. Two different common setup pages were required as there's quite a lot of differences between the 8030 and 8095 firmware. For example, the 8095 firmware has a mandatory default login, has smartzone crap enabled by default that you want to disable usually, etc. Having common pages for config stuff makes it much easier for me to:

  • Add a lot of info and config options. HTTPS webserver setup, PoE configuration and monitoring, LACP setup, helpful commands, optics info, etc. I will be adding more to these sections in the following week (VLANs, interVLAN routing, stacking, etc)

  • Migrated all switch upgrade methods to occur in the bootloader. Most were already like this (and of course now the ICX7xxx series is, as described above). However a couple ICX6xxx models still had you boot whatever mystery OS your switch came with and update from there. This led to a lot of "if" statements to handle weird situations like your switch coming with layer 2 firmware, in a stack configuration, etc. None of this is necessary and has been cleaned up now that the guide has you just drop into the bootloader, wipe everything, and flash a known latest L3 image

  • Removed the split "if access protection is required / if access protection is not required" option sections for the icx6xxx guides. In the newer ICX7xxx images it's no longer optional, as it comes by default secured with the super user. This means the ICX7xxx config guide does not have the choice of "if no access protection is required", and I wanted the icx6xxx config guide to match, as there's really no reason to not have a basic user/password set

  • Integrated all firmware packages, license files, etc into a single zip download - same single download regardless of what series switch you have. This means when new firmware is pushed I just have one zip to update. this also allowed me to do the following:

  • Add a "Brocade Overview" landing page, with the main firmware/licensing zip download. This is the new "starting point" regardless of what model you have. On this page, I've added decently detailed info/instructions on setting up a TFTP server on both windows and *nix. Having this in a single starting page allowed me to remove all the extraneous tftp instructions/notes from every single switch guide page

  • Added a ready to go portable copy of Tftpd64 to the main firmware zip, it's even set to the correct directory. Windows users can just launch this and have a TFTP server properly configured and running (assuming your windows firewall doesn't interfere)

  • Split the old combo icx6430/icx6450 config page into two distinct pages for each switch - the icx6430 cannot run the layer3 image, so the combo guide with layer3 instructions for the icx6450 was not very easy to follow (impossible, really)

  • Reorganized the left-hand menu to make more sense, and group common stuff

  • Add note to ICX6xxx config page about them not supporting 4096 bit SSH key pairs, which caused some issues with users in here before

  • Moved the ICX6610 stack ports page into its own section in the fcx/6610 update/config page

  • Add all the up to now private reverse engineering info I had on my old private site most of you had. It's all under the "brocade fun" section

  • Went through and updated all URLs, file links, etc, to relative paths to a folder within the docs folder. This allowed me to:

  • Add an archive download of the entire guide site with all files, zips, etc, so if something happens, you can use the site and all downloads entirely offline. you'll find it under the "Home" menu


There's ~100 other small changes that I don't recall offhand or aren't worth listing out, but I think the guide is pretty clean now, covers more use cases, leaves less room for error, and answers more common questions that we repeatedly answer in here. Some stats from the past 3 years of this thread for fun:

  • 780,000 thread views (most viewed thread on STH)
  • 6,000 thread replies (most replied thread on STH)
  • 1,392 private messages to my STH inbox (license requests, questions that should have been asked in the thread)
  • 1,102 emails to my email inbox (license requests, questions that should have been asked in the thread)
  • 282 commits to the documentation github repo
  • 5 contributors to the github repo other than me (thank you guys!)
  • 900 billion million switches bought by all of you nerds

Please let me know if you find any typos etc in the new guide, it was thousands of new lines so I'm sure I missed just one somewhere. I'm now hopping on the midnight train out of beeftown and I think I can consider my duty done. this whole time I was the brocade CEO and you've all pumped up my stock numbers enough, thanks!

Big thanks to @Patrick for allowing this monster of a thread to operate "hands off" for three years, and providing a site with a mature enough user base to make it to 300 pages without a meltdown

Lastly a huge thanks to the handful of STH members in here who have stuck around in this thread and helped users over and over for hundreds of pages now. I wanted to list you all by name but then realized I'd forget someone then feel like an asshole, so: you know who you are, thank you!
You absolute monster. Bravo!
 

Drewy

Active Member
Apr 23, 2016
183
37
28
51
Thank You, you’re a star.

what’s the recommended method of upgrading a stack? Un-stack and do them one at a time?
ive got a pair of stacked 7250’s.
 
  • Like
Reactions: fohdeesha

LodeRunner

Active Member
Apr 27, 2019
177
74
28
Thank You, you’re a star.

what’s the recommended method of upgrading a stack? Un-stack and do them one at a time?
ive got a pair of stacked 7250’s.
Since it's the same firmware as my 7450 stack, just update the active master and it will handle updating stack members and performing a rolling restart. Anything that's cross-chassis LAG'd will have no downtime.
 
  • Like
Reactions: Drewy

fohdeesha

Kaini Industries
Nov 20, 2016
2,277
2,358
113
30
fohdeesha.com
Since it's the same firmware as my 7450 stack, just update the active master and it will handle updating stack members and performing a rolling restart. Anything that's cross-chassis LAG'd will have no downtime.
indeed, you can follow the guide from just the UFI subsection to flash the new 8095d ufi image, it'll handle flashing it to all stack members ICX7250 / ICX7450 - Fohdeesha Docs
 

ZFSZealot

New Member
Aug 16, 2021
19
3
3
Does anyone have a scrap mainboard? I need two replacement push pins for the main routing IC heatsinks. I guess they failed with heat and age.

View attachment 9590
This is really digging up bones, sorry, this is one of the few mentions I've found for this happening, this is specific to the ICX6610.

With my first 6610, it was just the chip on the right side as viewed from the front, and it was obvious that it got hotter than the other one. I wonder if this is because that group of 24 ports got used and the other one didn't. Differences in heatsink color and PCB under the chip in question are pretty clear in the photos I'm posting. Chip on the right in the first pic is obviously mounted to the area on the left in the second pic, board was just flipped left to right:

IMG_0314Small.JPG
IMG_0315Small.JPG

This seems to have happened also to a lesser degree on the QSFP+ board, with the heatsinks on the full 40G port side being a little more brown than the black ones to their left.

Noted this as I was disassembling and cleaning dust out of the switch, primarily because I heard parts rattling around inside on receipt which turned out to be pieces of the heatsink pins and springs. Complete teardown was in order to make sure the pieces weren't going to short something.

I also found the same thing as mentioned further down from the quoted message, even though the pins are gone, the heatsink cannot be removed with any reasonable amount of twisting force.

Does anyone know exactly where/how to order the pins and springs? Unless these are glued well enough that the pins aren't really a concern? And just as a curiosity thing, why one chip lived life hotter than the other? Switch seems to work just fine as far as I've tested it.
 

Attachments

texteditor

New Member
Oct 8, 2019
15
2
3
Before I start bricking stuff, I'm looking to upgrade my 7150-C12 to 8090k from the fohdeesha default 8080. Guides/videos talk about usb upgrade, but before I brick it, they mention the security of NOT having a cfg installed when going to upgrade.

So... how do you do that? I have the ICX7150 and manifest files on my usb, but how else do I prepare the switch for the upgrade? Is there a doc that I didn't see about "simple" upgrade paths?

I for whatever reason could not get my 7150-c12 to take a TFTP'd image, but the USB method worked for me very well. I even thought I botched it at one point but was able to fix whatever I thought I messed up
 

nickf1227

Active Member
Sep 23, 2015
154
79
28
30
Since it's the same firmware as my 7450 stack, just update the active master and it will handle updating stack members and performing a rolling restart. Anything that's cross-chassis LAG'd will have no downtime.
Just make sure if you are pre8080 you go to 8080 first then to current.
 
  • Like
Reactions: Drewy

mmx

New Member
Dec 18, 2015
9
9
3
I love the revisions done to the guide!

Also, hack the planet! :D

EDIT: The serial number has 2 interpretations. Can I share my findings? :)
 
Last edited:
  • Haha
  • Like
Reactions: Yunia and ZFSZealot

mmx

New Member
Dec 18, 2015
9
9
3
Go for it although it looks like you noticed the LID hash already :p
First one I came up with was: "a beef box jk" lol

My second interpretation/guess was a word similar to your username, but wasn't able to make it work. :)
 

rootwyrm

Member
Mar 25, 2017
52
60
18
www.rootwyrm.com
I also found the same thing as mentioned further down from the quoted message, even though the pins are gone, the heatsink cannot be removed with any reasonable amount of twisting force.

Does anyone know exactly where/how to order the pins and springs? Unless these are glued well enough that the pins aren't really a concern? And just as a curiosity thing, why one chip lived life hotter than the other? Switch seems to work just fine as far as I've tested it.
The heatsink being unable to be removed is not a good sign. It means the TIM has likely failed. These do not use paste or thin pads, they use special impregnated fabric which has limited adhesive properties. When it fails it can turn into the consistency of glue or epoxy. The only way to remove it is to heat things up (carefully) until it lets go again.
Do NOT use thermal paste, silicone, or closed cell pads as a replacement. This way baked devices lie, because the already marginal interface will be terrible. I recommend Fujipoly SARCON Form-In-Place SPG-25B or SPG-30 as an alternative, the ODM equivalent (superior) is Fujipoly SARCON NR-Tc, or Fujipoly Extremely Compressible PG25A or above.

The pushpin setup is almost always specific to the device, but they are a very common part. You can use a nut-and-bolt setup (USE LOCTITE! they will back out) in a pinch as long as you're extremely careful not to over-tighten. If you have to go metallic, you must insulate because the mounting holes have ground plane pads. It's almost impossible to find pushpins in anything less than qty 100, but you can try ATS at Digikey.
 
  • Like
Reactions: klui and fohdeesha

ZFSZealot

New Member
Aug 16, 2021
19
3
3
The heatsink being unable to be removed is not a good sign. It means the TIM has likely failed. These do not use paste or thin pads, they use special impregnated fabric which has limited adhesive properties. When it fails it can turn into the consistency of glue or epoxy. The only way to remove it is to heat things up (carefully) until it lets go again.
Do NOT use thermal paste, silicone, or closed cell pads as a replacement. This way baked devices lie, because the already marginal interface will be terrible. I recommend Fujipoly SARCON Form-In-Place SPG-25B or SPG-30 as an alternative, the ODM equivalent (superior) is Fujipoly SARCON NR-Tc, or Fujipoly Extremely Compressible PG25A or above.

The pushpin setup is almost always specific to the device, but they are a very common part. You can use a nut-and-bolt setup (USE LOCTITE! they will back out) in a pinch as long as you're extremely careful not to over-tighten. If you have to go metallic, you must insulate because the mounting holes have ground plane pads. It's almost impossible to find pushpins in anything less than qty 100, but you can try ATS at Digikey.
Oh wow. I'm assuming TIM is "thermal interface material"? If the device appears to still work properly, this is still something that has to be addressed, correct?

Is this a common issue with the 6610's or did I just get lucky?

Also, I had thought about using a bolt and nut setup, but with a nyloc nut and the springs that I was able to recover, is that a reasonable option?

Here are temperatures, it's idle but nothing seems too far out of line?

Code:
ICX6610-48_01#sh chassis
The stack unit 1 chassis info:

Power supply 1 (AC - Regular) present, status ok
        Model Number:   23-0000144-01
        Serial Number:  0VH    
        Firmware Ver:    B
Power supply 1 Fan Air Flow Direction:  Front to Back
Power supply 2 (AC - Regular) present, status ok
        Model Number:   23-0000144-01
        Serial Number:  04S    
        Firmware Ver:    B
Power supply 2 Fan Air Flow Direction:  Front to Back

Fan 1 ok, speed (auto): [[1]]<->2
Fan 2 ok, speed (auto): [[1]]<->2

Fan controlled temperature: 62.0 deg-C

Fan speed switching temperature thresholds:
                Speed 1: NM<----->78       deg-C
                Speed 2:       73<-----> 84 deg-C (shutdown)

Fan 1 Air Flow Direction:  Front to Back
Fan 2 Air Flow Direction:  Front to Back                        
MAC 1 Temperature Readings:
        Current temperature : 43.5 deg-C
MAC 2 Temperature Readings:
        Current temperature : 56.0 deg-C
CPU Temperature Readings:
        Current temperature : 62.5 deg-C
sensor A Temperature Readings:
        Current temperature : 46.5 deg-C
sensor B Temperature Readings:
        Current temperature : 49.0 deg-C
sensor C Temperature Readings:
        Current temperature : 40.5 deg-C
stacking card Temperature Readings:
        Current temperature : 48.0 deg-C
        Warning level.......: 81.0 deg-C
        Shutdown level......: 84.0 deg-C
 

Nnyan

Active Member
Mar 5, 2012
137
40
28
I’m thinking of getting a 6450 since I really don’t like junos on the 3300ex I just picked up. I’m not doing anything too fancy on it (some L3 routing and VLANs) but I prefer a workable UI. What’s the brocades like?