Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

fohdeesha

Kaini Industries
Nov 20, 2016
1,906
1,713
113
29
fohdeesha.com
they are referencing mixing optics slots like sfp/qsfp with built in copper ports. The QSFP and SFP ports are both "fiber ports". As long as the ports are optics slots (qsfp, sfp, etc) you can mix module types no problem. Ran a lag between an SR optic module and an RJ45 10GBASE-T module without issue:


Code:
SSH@GG-CORE-N1>sh lag test2
Total number of LAGs:          2
Total number of deployed LAGs: 2
Total number of trunks created:2 (118 available)
LACP System Priority / ID:     1 / 748e.f8f7.8498
LACP Long timeout:             120, default: 120
LACP Short timeout:            3, default: 3
=== LAG "test2" ID 1 (dynamic Deployed) ===
Deployment: HW Trunk ID 2
Port       Link    State   Dupl Speed Trunk Tag Pvid Pri MAC             Name
1/3/2      Up      Forward Full 10G   1     No  1    0   748e.f8f7.8498
2/3/2      Up      Forward Full 10G   1     No  1    0   748e.f8f7.8498
Port       [Sys P] [Port P] [ Key ] [Act][Tio][Agg][Syn][Col][Dis][Def][Exp][Ope]
1/3/2           1        1   20001   Yes   L   Agg  Syn  Col  Dis  No   No   Ope
2/3/2           1        1   20001   Yes   L   Agg  Syn  Col  Dis  No   No   Ope

1/3/2 is an SR optics module, 2/3/2 is a mikrotik copper rj45 module
 
  • Like
Reactions: juey

gigogig

New Member
Oct 29, 2018
19
7
3
i just got 2 6450-24p's in and trying to update their firmware both are having problems,
unit one is bootlooping with:

| | | | | __ ) ___ ___ | |_ | | | |___| _ \ / _ \ / _ \| __| | |_| | - Pastebin.com

"after rebooting to secondary boot partition

| | | | | __ ) ___ ___ | |_ | | | |___| _ \ / _ \ / _ \| __| | |_| |__ - Pastebin.com



after booting from secondary partition unit 1 is up, but as its stuck in router not switch mode i don't know how to properly update it to switch mode with new firmware?







_____________________________________________________
and unit two was throwing :

For port 1/2/1 4.0x9004 = 0x100

For port 1/2/2 4.0x9003 = 0x0
For port 1/2/2 4.0x9004 = 0x100

For port 1/2/3 4.0x9003 = 0x0
For port 1/2/3 4.0x9004 = 0x100

For port 1/2/4 4.0x9003 = 0x0
For port 1/2/4 4.0x9004 = 0x100

scratch unit 2, firmware fixed it from doing that constantly, so unit 2 look all good now
 
Last edited:

fohdeesha

Kaini Industries
Nov 20, 2016
1,906
1,713
113
29
fohdeesha.com
that is dev/testing firmware (especially the bootloader), I've seen this come on ebay units (supposedly from brocade hq) a couple times, it's fixable, you definitely don't want to run that software and OS

On both units, can you follow the guide to flash new bootloader and OS? it should NOT have the big ascii u-boot logo during boot. If that doesn't work, there's a more "hardcore" way of overwriting the dev software from u-boot, I would probably have to do that remotely over google hangouts or teamviewer as it's too much to write out

Either way, once you're fully flashed to the normal bootloader, poe firmware, and OS, can you please run the following commands in the bootloader (hit b during boot to get into it), and paste the results on Pastebin.com - #1 paste tool since 2002! ?

printenv
printenv bootcmd
printenv defbootcmd
printenv extra_bootargs

Most of the ICX's I've seen come like this also had dev/chassis polling/test mode flags set in u-boot, this will tell me if there are, then I can tell you how to get rid of them. please do not skip this step. If it's quicker for you, just PM me teamviewer info and I can do all of it including the flashing, this dev software can be tricky to replace
 

gigogig

New Member
Oct 29, 2018
19
7
3
that is dev/testing firmware (especially the bootloader), I've seen this come on ebay units (supposedly from brocade hq) a couple times, it's fixable, you definitely don't want to run that software and OS

On both units, can you follow the guide to flash new bootloader and OS? it should NOT have the big ascii u-boot logo during boot. If that doesn't work, there's a more "hardcore" way of overwriting the dev software from u-boot, I would probably have to do that remotely over google hangouts or teamviewer as it's too much to write out

Either way, once you're fully flashed to the normal bootloader, poe firmware, and OS, can you please run the following commands in the bootloader (hit b during boot to get into it), and paste the results on Pastebin.com - #1 paste tool since 2002! ?

printenv
printenv bootcmd
printenv defbootcmd
printenv extra_bootargs

Most of the ICX's I've seen come like this also had dev/chassis polling/test mode flags set in u-boot, this will tell me if there are, then I can tell you how to get rid of them. please do not skip this step. If it's quicker for you, just PM me teamviewer info and I can do all of it including the flashing, this dev software can be tricky to replace
unit 2 took the firmwares without any fuss, the were constantly outputting odd data before the update but after the firmware upgrade, they seem good, here is there boot up, Bootloader Version: 10.1.05T310 (Mar 19 2015 - 16:39:59) Model ID: 1.0.0. - Pastebin.com and env Bootloader Version: 10.1.05T310 (Mar 19 2015 - 16:39:59) Model ID: 1.0.0. - Pastebin.com

unit one on the other hand , seemed to be in working order until i tried to update the firmware, that's when i got stuck in the reboot loop here | | | | | __ ) ___ ___ | |_ | | | |___| _ \ / _ \ / _ \| __| | |_| | - Pastebin.com

and here is unit one env Bootloader Version: 10.1.05T310 (Mar 19 2015 - 16:39:59) Model ID: 1.0. - Pastebin.com

what should i do to unit one? retry the flash?

edit i retried the flash, this is what i got
UNIT 1: compiled on Nov 18 2011 at 14:51:58 labeled as ICXR07400b1 - Pastebin.com
 
Last edited:
  • Like
Reactions: kache

fohdeesha

Kaini Industries
Nov 20, 2016
1,906
1,713
113
29
fohdeesha.com
unit 2 looks good - make sure to update it's PoE firmware as well if you haven't

unit 1 definitely still has a dev bootloader which is why it can't boot. I can fix this for you, but I need remote access (teamviewer is usually easiest). it involves low level NAND flash writing commands that if screwed up will brick it, so it's too risky for me to write them out and then you try to follow them
 

realtomatoes

Active Member
Oct 3, 2016
244
31
28
40
got me one of those HP 649281-B21 and a qsfp cable recently to connect my 2 nodes directly for VM storage traffic.
am now looking at plugging the other port to the 6450 which am still shopping for.
newb question: i am gonna need something like the MAM1Q00A-QSA to connect the HP nic to the 6450, right?
 

fohdeesha

Kaini Industries
Nov 20, 2016
1,906
1,713
113
29
fohdeesha.com
one of these GENUINE 655902-001 655874-B21 HP MELLANOX QSFP to SFP ADAPTER 886111656585 | eBay

with an sfp+ optic like (brand does not matter) Brocade 10GB 57-0000075-01 10G-SFPP-SR 10Gbase-sr/sw 850nm SFP+ Transceiver >500 689719088136 | eBay

(unless you plan on using a DAC, never tried an SFP+ DAC with a qsfp to sfp adapter, but it should probably work)

follow this post to get the proper firmware on the NIC https://forums.servethehome.com/ind...net-dual-port-qsfp-adapter.20525/#post-198015
 

realtomatoes

Active Member
Oct 3, 2016
244
31
28
40
one of these GENUINE 655902-001 655874-B21 HP MELLANOX QSFP to SFP ADAPTER 886111656585 | eBay

with an sfp+ optic like (brand does not matter) Brocade 10GB 57-0000075-01 10G-SFPP-SR 10Gbase-sr/sw 850nm SFP+ Transceiver >500 689719088136 | eBay

(unless you plan on using a DAC, never tried an SFP+ DAC with a qsfp to sfp adapter, but it should probably work)

follow this post to get the proper firmware on the NIC https://forums.servethehome.com/ind...net-dual-port-qsfp-adapter.20525/#post-198015
thanks, for this and all the knowledge you've shared. it is greatly appreciated.
 
  • Like
Reactions: fohdeesha

fohdeesha

Kaini Industries
Nov 20, 2016
1,906
1,713
113
29
fohdeesha.com
thanks to @gigogig 's patience I was able to copy out the internal-use only dev bootloader before overwriting it with stock - been wanting to do that for a while but haven't found another ICX with such a bootloader all year. Seems to have a ton of custom routines and commands, programming voltages and fans etc, will load it on one of my bench 6450's sometime soon and see if there's anything useful
 

mattaw

Member
Jul 30, 2018
51
15
8
@fohdeesha , do I need the brocade serial cable? Googling around I have found the pinout but I wasn't sure if it was necessary with the dedicated management port. Maybe there is a cheap alternative?
 

fohdeesha

Kaini Industries
Nov 20, 2016
1,906
1,713
113
29
fohdeesha.com
@fohdeesha , do I need the brocade serial cable? Googling around I have found the pinout but I wasn't sure if it was necessary with the dedicated management port. Maybe there is a cheap alternative?
For what switch? the ICX6xxx series use the industry standard RJ45 serial - https://www.amazon.com/Console-Essential-Accesory-Ubiquity-Switches/dp/B01AFNBC3K

You'll need serial access to reset the switch to factory defaults and clear any old passwords it comes with, you can't do that from the management ethernet port (especially if the past config never set it up with an IP anyway)
 
  • Like
Reactions: mattaw

anlin

New Member
Dec 8, 2016
29
6
3
27
Just pulled the trigger on a 7250-24P, can't wait for it to arrive! Do you know if the 7250 is also brand agnostic when it comes to SFP+ modules? Cisco 10-2415-03 from China are the cheapest ones available to me at about $7. The brocade modules you linked to above cost $40 each after shiping and customs charges :(
 

mattaw

Member
Jul 30, 2018
51
15
8
Of the 3x icx6610's coming there are 4 psus but only 2x fans out of a possible 6. The cost of new fan units is >$175 each?!?
 

fohdeesha

Kaini Industries
Nov 20, 2016
1,906
1,713
113
29
fohdeesha.com
well you're only getting 1 working unit, so you actually have a spare fan tray :p (looking on the bright side)

fan trays pop up around 20 or 30 dollars on ebay, you just have to be patient (really patient, sometimes)
 

Skud

Active Member
Jan 3, 2012
109
47
28
So...who bought the 3x 7150's and 7250?
I grabbed one 7150 and one 7250-24P.

Gonna see how the 7250 compares to the 6610-24P I just received. I may use the 7250 instead if the noise and power is a lot less. I don’t really have a use for the QSFPs anyways.

Not that my 6610 is loud. Is there much of a difference in noise between the rev b and c PSUs of the 6610?

Riley
 
  • Like
Reactions: fohdeesha

fohdeesha

Kaini Industries
Nov 20, 2016
1,906
1,713
113
29
fohdeesha.com
nope, 99% of them were sold with 1, the fan tray is already redundant with the PSU fans (it will run with zero fan trays, it will just complain and tell you to replace the failed tray ASAP)

zero difference between rev b and rev c sound wise
 
  • Like
Reactions: mattaw