Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

[fohdeesha]

they are referencing mixing optics slots like sfp/qsfp with built in copper ports. The QSFP and SFP ports are both "fiber ports". As long as the ports are optics slots (qsfp, sfp, etc) you can mix module types no problem. Ran a lag between an SR optic module and an RJ45 10GBASE-T module without issue:

SSH@GG-CORE-N1>sh lag test2
Total number of LAGs:          2
Total number of deployed LAGs: 2
Total number of trunks created:2 (118 available)
LACP System Priority / ID:     1 / 748e.f8f7.8498
LACP Long timeout:             120, default: 120
LACP Short timeout:            3, default: 3
=== LAG "test2" ID 1 (dynamic Deployed) ===
Deployment: HW Trunk ID 2
Port       Link    State   Dupl Speed Trunk Tag Pvid Pri MAC             Name
1/3/2      Up      Forward Full 10G   1     No  1    0   748e.f8f7.8498
2/3/2      Up      Forward Full 10G   1     No  1    0   748e.f8f7.8498
Port       [Sys P] [Port P] [ Key ] [Act][Tio][Agg][Syn][Col][Dis][Def][Exp][Ope]
1/3/2           1        1   20001   Yes   L   Agg  Syn  Col  Dis  No   No   Ope
2/3/2           1        1   20001   Yes   L   Agg  Syn  Col  Dis  No   No   Ope

1/3/2 is an SR optics module, 2/3/2 is a mikrotik copper rj45 module

 

[gigogig]

i just got 2 6450-24p's in and trying to update their firmware both are having problems,
unit one is bootlooping with:

| | | | | __ ) ___ ___ | |_ | | | |___| _ \ / _ \ / _ \| __| | |_| | - Pastebin.com

"after rebooting to secondary boot partition

| | | | | __ ) ___ ___ | |_ | | | |___| _ \ / _ \ / _ \| __| | |_| |__ - Pastebin.com



after booting from secondary partition unit 1 is up, but as its stuck in router not switch mode i don't know how to properly update it to switch mode with new firmware?







_____________________________________________________
and unit two was throwing :
For port 1/2/1 4.0x9004 = 0x100

For port 1/2/2 4.0x9003 = 0x0
For port 1/2/2 4.0x9004 = 0x100

For port 1/2/3 4.0x9003 = 0x0
For port 1/2/3 4.0x9004 = 0x100

For port 1/2/4 4.0x9003 = 0x0
For port 1/2/4 4.0x9004 = 0x100

scratch unit 2, firmware fixed it from doing that constantly, so unit 2 look all good now

 

[fohdeesha]

that is dev/testing firmware (especially the bootloader), I've seen this come on ebay units (supposedly from brocade hq) a couple times, it's fixable, you definitely don't want to run that software and OS

On both units, can you follow the guide to flash new bootloader and OS? it should NOT have the big ascii u-boot logo during boot. If that doesn't work, there's a more "hardcore" way of overwriting the dev software from u-boot, I would probably have to do that remotely over google hangouts or teamviewer as it's too much to write out

Either way, once you're fully flashed to the normal bootloader, poe firmware, and OS, can you please run the following commands in the bootloader (hit b during boot to get into it), and paste the results on Pastebin.com - #1 paste tool since 2002! ?

printenv
printenv bootcmd
printenv defbootcmd
printenv extra_bootargs

Most of the ICX's I've seen come like this also had dev/chassis polling/test mode flags set in u-boot, this will tell me if there are, then I can tell you how to get rid of them. please do not skip this step. If it's quicker for you, just PM me teamviewer info and I can do all of it including the flashing, this dev software can be tricky to replace

 

[gigogig]

that is dev/testing firmware (especially the bootloader), I've seen this come on ebay units (supposedly from brocade hq) a couple times, it's fixable, you definitely don't want to run that software and OS

On both units, can you follow the guide to flash new bootloader and OS? it should NOT have the big ascii u-boot logo during boot. If that doesn't work, there's a more "hardcore" way of overwriting the dev software from u-boot, I would probably have to do that remotely over google hangouts or teamviewer as it's too much to write out

Either way, once you're fully flashed to the normal bootloader, poe firmware, and OS, can you please run the following commands in the bootloader (hit b during boot to get into it), and paste the results on Pastebin.com - #1 paste tool since 2002! ?

printenv
printenv bootcmd
printenv defbootcmd
printenv extra_bootargs

Most of the ICX's I've seen come like this also had dev/chassis polling/test mode flags set in u-boot, this will tell me if there are, then I can tell you how to get rid of them. please do not skip this step. If it's quicker for you, just PM me teamviewer info and I can do all of it including the flashing, this dev software can be tricky to replace

unit 2 took the firmwares without any fuss, the were constantly outputting odd data before the update but after the firmware upgrade, they seem good, here is there boot up, Bootloader Version: 10.1.05T310 (Mar 19 2015 - 16:39:59) Model ID: 1.0.0. - Pastebin.com and env Bootloader Version: 10.1.05T310 (Mar 19 2015 - 16:39:59) Model ID: 1.0.0. - Pastebin.com

unit one on the other hand , seemed to be in working order until i tried to update the firmware, that's when i got stuck in the reboot loop here | | | | | __ ) ___ ___ | |_ | | | |___| _ \ / _ \ / _ \| __| | |_| | - Pastebin.com

and here is unit one env Bootloader Version: 10.1.05T310 (Mar 19 2015 - 16:39:59) Model ID: 1.0. - Pastebin.com

what should i do to unit one? retry the flash?

edit i retried the flash, this is what i got
UNIT 1: compiled on Nov 18 2011 at 14:51:58 labeled as ICXR07400b1 - Pastebin.com

 

[fohdeesha]

unit 2 looks good - make sure to update it's PoE firmware as well if you haven't

unit 1 definitely still has a dev bootloader which is why it can't boot. I can fix this for you, but I need remote access (teamviewer is usually easiest). it involves low level NAND flash writing commands that if screwed up will brick it, so it's too risky for me to write them out and then you try to follow them

 

[realtomatoes]

got me one of those HP 649281-B21 and a qsfp cable recently to connect my 2 nodes directly for VM storage traffic.
am now looking at plugging the other port to the 6450 which am still shopping for.
newb question: i am gonna need something like the MAM1Q00A-QSA to connect the HP nic to the 6450, right?

 

[fohdeesha]

one of these GENUINE 655902-001 655874-B21 HP MELLANOX QSFP to SFP ADAPTER 886111656585 | eBay

with an sfp+ optic like (brand does not matter) Brocade 10GB 57-0000075-01 10G-SFPP-SR 10Gbase-sr/sw 850nm SFP+ Transceiver >500 689719088136 | eBay

(unless you plan on using a DAC, never tried an SFP+ DAC with a qsfp to sfp adapter, but it should probably work)

follow this post to get the proper firmware on the NIC https://forums.servethehome.com/ind...net-dual-port-qsfp-adapter.20525/#post-198015

 

[realtomatoes]

one of these GENUINE 655902-001 655874-B21 HP MELLANOX QSFP to SFP ADAPTER 886111656585 | eBay

with an sfp+ optic like (brand does not matter) Brocade 10GB 57-0000075-01 10G-SFPP-SR 10Gbase-sr/sw 850nm SFP+ Transceiver >500 689719088136 | eBay

(unless you plan on using a DAC, never tried an SFP+ DAC with a qsfp to sfp adapter, but it should probably work)

follow this post to get the proper firmware on the NIC https://forums.servethehome.com/ind...net-dual-port-qsfp-adapter.20525/#post-198015

thanks, for this and all the knowledge you've shared. it is greatly appreciated.

 

[fohdeesha]

thanks to @gigogig 's patience I was able to copy out the internal-use only dev bootloader before overwriting it with stock - been wanting to do that for a while but haven't found another ICX with such a bootloader all year. Seems to have a ton of custom routines and commands, programming voltages and fans etc, will load it on one of my bench 6450's sometime soon and see if there's anything useful

 

[mattaw]

@fohdeesha , do I need the brocade serial cable? Googling around I have found the pinout but I wasn't sure if it was necessary with the dedicated management port. Maybe there is a cheap alternative?

 

[fohdeesha]

@fohdeesha , do I need the brocade serial cable? Googling around I have found the pinout but I wasn't sure if it was necessary with the dedicated management port. Maybe there is a cheap alternative?

For what switch? the ICX6xxx series use the industry standard RJ45 serial - https://www.amazon.com/Console-Essential-Accesory-Ubiquity-Switches/dp/B01AFNBC3K

You'll need serial access to reset the switch to factory defaults and clear any old passwords it comes with, you can't do that from the management ethernet port (especially if the past config never set it up with an IP anyway)

 

[fohdeesha]

So...who bought the 3x 7150's and 7250?

 

[anlin]

Just pulled the trigger on a 7250-24P, can't wait for it to arrive! Do you know if the 7250 is also brand agnostic when it comes to SFP+ modules? Cisco 10-2415-03 from China are the cheapest ones available to me at about $7. The brocade modules you linked to above cost $40 each after shiping and customs charges

 

[fohdeesha]

yup the 7 series is identical to the 6 series in those regards, no locks of any kind

 

[mattaw]

Of the 3x icx6610's coming there are 4 psus but only 2x fans out of a possible 6. The cost of new fan units is >$175 each?!?

 

[fohdeesha]

well you're only getting 1 working unit, so you actually have a spare fan tray (looking on the bright side)

fan trays pop up around 20 or 30 dollars on ebay, you just have to be patient (really patient, sometimes)

 

[fohdeesha]

I'll see if I can get an icx7xxx update guide and firmware dl up before all of your guys switches arrive

 

[Skud]

So...who bought the 3x 7150's and 7250?

I grabbed one 7150 and one 7250-24P.

Gonna see how the 7250 compares to the 6610-24P I just received. I may use the 7250 instead if the noise and power is a lot less. I don’t really have a use for the QSFPs anyways.

Not that my 6610 is loud. Is there much of a difference in noise between the rev b and c PSUs of the 6610?

Riley

 

[mattaw]

well you're only getting 1 working unit, so you actually have a spare fan tray (looking on the bright side)

I'll keep my eyes open. So you don't need two fan trays per switch then? That's a relief.

 

[fohdeesha]

nope, 99% of them were sold with 1, the fan tray is already redundant with the PSU fans (it will run with zero fan trays, it will just complain and tell you to replace the failed tray ASAP)

zero difference between rev b and rev c sound wise