Drag to reposition cover

Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

itronin

Well-Known Member
Nov 24, 2018
1,234
794
113
Denver, Colorado
Guys. Finally I got my hands on my ICX -6610-48p and it could not be different being a Cisco guy I'm having some trouble to configure the 2 rear stacking ports.
I want to use as a regular port to connect to my windows 10 box but no joy.
I'm using the fllowing hardware and cables:
NIC: MCX354A-FCBT 649281-B21 656089-001 VPI FDR 40GbE Mellanox from HP 544QSFP MCX354A-FCBT 649281-B21 656089-001 VPI FDR 40GbE Mellanox OEMFirmware | eBay
Cable: Netapp 112-00178 5M 16ft QSFP-QSFP External SAS Cable X6559-R6 from NetApp X 6559 R 6 SAS Cable 5 M QSFP for sale online | eBay
You are missing a couple of things. I'm assuming you are running a single switch and will be doing no stacking at all. I'm assuming your vlan is untagged, it is vlan 1 and your are using 1/2/1 for your data port.

1) you need to disable the stack and reboot the switch
conf t
disable stack
exit
write mem
reload


2) you need to add the 40Gbe port to your vlan
conf t
vlan 1
untagged ethe 1/2/1
# don't forget to add any other ports (tagged or untagged) to vlan 1.
exit
exit
 

Bell System

New Member
May 26, 2019
4
1
3
Southwest USA
...Unfortunately, that was about 20 or 30 pages before people started posting reports of busted switches from eBay seller "g-electronic," which of course is who I purchased mine from. They haven't arrived yet, but when they do it looks like I'll be putting them through a VERY intense round of testing and burn-in, especially the PoE stuff. Ugh. I'll be sure to report back on their condition once they arrive.
As promised, I wanted to update everyone on my 6450-24P switches from eBay seller "G-electronic."

I ordered two switches and unfortunately, both PoE modules are bad. They just keep printing the following over and over again:

Code:
PoE Error: Device 0 failed to start on PoE module.
PoE Error: Device 1 failed to start on PoE module.
Resetting module in slot 1 again to recover from dev fault
PoE Info: Hard Resetting in slot 1....
PoE Info: Resetting module in slot 1....completed.
Oddly, some ports do provide PoE power, but doing a
Code:
sh inline power
indicates that the switch doesn't even detect a connected PoE device.

One switch even has at least one port that is totally dead.

Seems like this guy is either reselling the same bad PoE switches hoping no one will notice, or he has a huge lot full of damaged switches. In any case, mine are going back, and I recommend staying away from this seller if you're interested in getting a switch with working PoE.
 
  • Like
Reactions: fohdeesha

Joao Paulo Mello

New Member
Jan 16, 2019
14
1
3
Thanks for you reply.

1) you need to disable the stack and reboot the switch
conf t
disable stack
exit
write mem
reload
I did but I had to "stack disable" instead of "disable stack".
2) you need to add the 40Gbe port to your vlan
conf t
vlan 1
untagged ethe 1/2/1
# don't forget to add any other ports (tagged or untagged) to vlan 1.
exit
exit
The port is on vlan 1 since it's default vlan and is already untagged
#show interfaces ethernet 1/2/1
40GigabitEthernet1/2/1 is down, line protocol is down
Port down for 6 minutes 58 seconds
Hardware is 40GigabitEthernet, address is cc4e.24f8.ef2c (bia cc4e.24f8.ef5d)
Interface type is 40Gig Fiber
Configured speed 40Gbit, actual unknown, configured duplex fdx, actual unknown
Configured mdi mode AUTO, actual unknown
Member of L2 VLAN ID 1, port is untagged, port state is BLOCKING
BPDU guard is Disabled, ROOT protect is Disabled, Designated protect is Disabled
Link Error Dampening is Disabled
STP configured to ON, priority is level0, mac-learning is enabled
Flow Control is enabled
Mirror disabled, Monitor disabled
Mac-notification is disabled
Not member of any active trunks
Not member of any configured trunks
No port name
MTU 1500 bytes, encapsulation ethernet
300 second input rate: 0 bits/sec, 0 packets/sec, 0.00% utilization
300 second output rate: 0 bits/sec, 0 packets/sec, 0.00% utilization
0 packets input, 0 bytes, 0 no buffer
Received 0 broadcasts, 0 multicasts, 0 unicasts
0 input errors, 0 CRC, 0 frame, 0 ignored
0 runts, 0 giants
0 packets output, 0 bytes, 0 underruns
Transmitted 0 broadcasts, 0 multicasts, 0 unicasts
0 output errors, 0 collisions
Relay Agent Information option: Disabled

Egress queues:
Queue counters Queued packets Dropped Packets
0 0 0
1 0 0
2 0 0
3 0 0
4 0 0
5 0 0
6 0 0
7 0 0


The problem remains, no link.
I read something mentioning that I need to have the latest firmware running in order to use the stack ports as data port.
That's my show version:

#show ver
Copyright (c) 1996-2014 Brocade Communications Systems, Inc. All rights reserved.
UNIT 1: compiled on Jul 08 2015 at 16:29:14 labeled as FCXR08010m
(9716466 bytes) from Primary FCXR08010m.bin
SW: Version 08.0.10mT7f3
Boot-Monitor Image size = 370695, Version:10.1.00T7f5 (grz10100)
HW: Stackable ICX6610-48-HPOE
==========================================================================
UNIT 1: SL 1: ICX6610-48P POE 48-port Management Module
Serial #: xxxxxxxxxxxxx
License: ICX6610_ADV_ROUTER_SOFT_PACKAGE (LID: xxxxxxxxxxxxxxxxxxxx)
P-ENGINE 0: type E02B, rev 01
P-ENGINE 1: type E02B, rev 01
==========================================================================
UNIT 1: SL 2: ICX6610-QSFP 10-port 160G Module
==========================================================================
UNIT 1: SL 3: ICX6610-8-port Dual Mode(SFP/SFP+) Module
==========================================================================
800 MHz Power PC processor 8544E (version 0021/0023) 400 MHz bus
65536 KB flash memory
512 MB DRAM
STACKID 1 system uptime is 13 minutes 7 seconds
The system : started=warm start reloaded=by "reload"
 
Last edited:

Joao Paulo Mello

New Member
Jan 16, 2019
14
1
3
Solved!!!
Going back to the past one more time I noticed the @arglebargle post: https://forums.servethehome.com/ind...gbe-40gbe-switching.21107/page-15#post-198883

That solved for me :).
Thanks @arglebargle, thanks everyone!

Edit:
I'm back with one more question: My unit came with 2 PSU's and just one fan tray. Does make sense to assume that installing another fan tray the fans will calm down a bit? The PSU's fan spin down after boot and I can live with that but if there's any way to slow down those boys a little that would be great. Have anyone had such experience?
 
Last edited:

Joao Paulo Mello

New Member
Jan 16, 2019
14
1
3
with those adapters you REALLY need to update the firmware using this guide as mentioned in the original post, it also points out to set the ports to ethernet: https://forums.servethehome.com/ind...net-dual-port-qsfp-adapter.20525/#post-198015

the firmware version your switch is using is also ancient, update it using the guide linked in the original post
Will do.

if your PSUs are already spinning down, adding a second fan tray will not make them spin down further
Damn! But I can live with the current noise level when the PSU's spins down...I have other stuff on the rack, some of them modded to be quiet and some couldn't be modded. So it's just another addition to the choir.

I was just testing the 40g adapter on Windows, in fact it will be used at my NAS (Ubuntu), the default protocol mode on linux is Infiniband as well? Do I'll need to install the driver and set to Ethernet somewhere?
 

bkvamme

New Member
Sep 23, 2018
11
4
3
Hi,
Might be a long shot, but does anyone know the ECCN and CCATS number for the ICX-6610-48P? My freight forwarder has been unable to get the numbers.

Based on what I see online, the ECCN should be: 5A002, but I am unable to find the CCATS number.

Edit: OK, this is getting a bit ridiculous. Contacted Ruckus support, and the rep stated that I since I purchased the switch through an unauthorised vendor (eBay) I would need to purchase a support contract for them to provide me with this number. Wth. Ubiquiti and Cisco are listing these online.
 
Last edited:

fohdeesha

Kaini Industries
Nov 20, 2016
2,728
3,078
113
33
fohdeesha.com
was digging around in @britinpdx 's ICX7250, it came with some dev software, was unbricking it for him and came across a hidden environment variable: setenv diag_mode_enable=1

Enabling this from the bootloader by running:

setenv diag_mode_enable 1
saveenv
reset

makes it reset into the bootloader again, except now it will be full featured u-boot with everything available, all the i2c raw read and write commands etc, all the stuff that's hidden in the "normal" bootloader prompt.

to get out of it and go back to booting normally just run:

setenv diag_mode_enable
saveenv
reset

Should work on any brocade switch running a u-boot based bootloader (NOT the 6610)
 

cuco

Member
Feb 13, 2018
57
4
8
38
Hey,
does anyone know a good place to buy cheap rackmount ears for the 6610?
Cant find anything on ebay
Thanks
 

HomelabN0rth

New Member
Jan 30, 2019
3
2
3
Hey all, thanks for all the info here. It made me jump on the ICX7250 train via ebay but I F*ed up and ordered from G-Electronics and got a switch with corrupted images on both the primary and secondary. I can get into ICX7250-Boot> but can't figure out how to do anything else. This is what happens:

ICX7250-Boot> tftpboot
ERROR: Ethernet external port not connected
at bcmiproc_eth.c:124/bcmiproc_eth_open()
could not boot from tftp, trying to boot from primary
Error: The header crc ffffffff and the calculated crc 4007851a mismatch
Fatal: Firmware integrity checksum failed
Booting image from Primary
no current device selected
Wrong Image Format for bootm command
ERROR: can't get kernel image!
could not boot from primary, no valid image; trying to boot from secondary
Error: The header crc ffffffff and the calculated crc 4007851a mismatch
Fatal: Firmware integrity checksum failed
Booting image from Secondary
no current device selected
Wrong Image Format for bootm command
ERROR: can't get kernel image!
Wrong Image Format for tftpboot command
ERROR: can't get kernel image!
ICX7250-Boot>

When the switch boots, it boots to a DIAG> prompt. From there, I can type linux_shell and get the switch to reboot, and if I press B during the beginning of that process, I can get to this ICX7250-Boot> prompt. Any suggestions how I can get from this boot menu to upload the new firmware to the device? Or am I just totally borked?
 

fohdeesha

Kaini Industries
Nov 20, 2016
2,728
3,078
113
33
fohdeesha.com
yours came with the exact same diag bootloader and broadcom test image that @britinpdx did (also from g-electronic). install teamviewer and send me credentials, I can fix it. you'll need a putty window up connected to the switch over serial and the management port plugged into a network with a tftp server

we can definitively rule out g-electronic doing ANY testing, what a liar
 

HomelabN0rth

New Member
Jan 30, 2019
3
2
3
Actually, I got it figured out.
I had to setenv the ip settings in the Boot>, used TFTP then to mount a new boot image. Restarted the switch. After that, did the same using update_primary and update_ secondary to update the main image and afterwards was back in business. What a pain in the everything tho.
 
  • Like
Reactions: fohdeesha

HomelabN0rth

New Member
Jan 30, 2019
3
2
3
damn, I was hoping to extract those development images before wiping them :p
Oh sorry, I was in TERMINALDAD> mode - I wasn't going to let it "win", ya know? I have to admit, the documentation from Rukus is what saved the day. It wasn't all where I was looking for it but some Google-fu with the error messages and commands was enough to bring up pages where they posted how to do each individual part, even if each part was on a separate document LOL
 
  • Like
Reactions: fohdeesha

fohdeesha

Kaini Industries
Nov 20, 2016
2,728
3,078
113
33
fohdeesha.com
:p I will say you got lucky, half of the "development" switches I have seen with these bootloaders have an "update_boot" routine that lays the bootloader file down unmodified at the beginning of flash (which worked for whatever development bootloaders they were using). However the "production" bootloader files that we all have have around 30 lines or so of metadata at the beginning that needs to be stripped off first, if you flash those using the update_boot of a dev bootloader and it lays it down raw, it'll brick it. It sounds like yours already had a production bootloader though, that just happened to be in diag mode with a bunch of strange boot environment variables set for the broadcom image (similar to @britinpdx ). I almost want to ask g-electronic where he is getting these because they keep coming with the strangest stuff I've seen
 
Last edited:
  • Like
Reactions: HomelabN0rth

fohdeesha

Kaini Industries
Nov 20, 2016
2,728
3,078
113
33
fohdeesha.com
I checked their source and it appears in the patch file for the 6650 and 7250 but not the 6450 or 7650.
Nice find. after skimming through the patch files some more it turns out the secret mode is still there, they just renamed the routine and environment variable.

for the ICX6450:
(it also requires setting of pp_init for this mode to occur)

setenv pp_init_enable 1
setenv diag_cmd_on 1
saveenv
reset

##go back to normal
setenv pp_init_enable
setenv diag_cmd_on
saveenv
reset


For the ICX7650:

setenv diag_mode_on 1
saveenv
reset

##go back to normal
setenv diag_mode_on
saveenv
reset
 
  • Like
Reactions: sean

Skippy96

New Member
May 14, 2019
19
7
3
Is there any way to leverage all the access we have with these dev builds to modify the firmware to allow for fan modifications?
My 6610 is getting too loud for my office :)