Drag to reposition cover

Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

spali

Member
Nov 4, 2018
32
3
8
So what I am going to do with mine is stack across the breakout ports, and use the 4 total ports across my 2x 6610's for 40g.
Does this also work if I use the 40g's as uplink to a 40g switch?
My plan is to uplink two 6610 to my arista 40 g "core" switch.
But if I understand it right, this forces me to not stack or stack with the breakout ports. I wonder how I could force the main traffic to the core switch over the 40g uplinks in a "breakout" stack.
Any ideas or suggestions?
 

Terry Wallace

PsyOps SysOp
Aug 13, 2018
138
79
28
52
Allen, TX
I’d take 2 of the 40 gig ports on each 6610 and make them a stack
Then I’d take the other 2 ports off the 6610s and make an mlag to your arista. That way you get failover stack connections.
 

spali

Member
Nov 4, 2018
32
3
8
I’d take 2 of the 40 gig ports on each 6610 and make them a stack
Then I’d take the other 2 ports off the 6610s and make an mlag to your arista. That way you get failover stack connections.
I'm new to mlag, but aren't the links then limited to 8x 10g per switch?
 

Joao Paulo Mello

New Member
Jan 16, 2019
14
1
3
Guys. Finally I got my hands on my ICX -6610-48p and it could not be different being a Cisco guy I'm having some trouble to configure the 2 rear stacking ports.
I want to use as a regular port to connect to my windows 10 box but no joy.
I'm using the fllowing hardware and cables:
NIC: MCX354A-FCBT 649281-B21 656089-001 VPI FDR 40GbE Mellanox from HP 544QSFP MCX354A-FCBT 649281-B21 656089-001 VPI FDR 40GbE Mellanox OEMFirmware | eBay
Cable: Netapp 112-00178 5M 16ft QSFP-QSFP External SAS Cable X6559-R6 from NetApp X 6559 R 6 SAS Cable 5 M QSFP for sale online | eBay

Windows 10 recognized the NIC right away but no link and the ICX6610 says the same:

#show interfaces ethernet 1/2/1
40GigabitEthernet1/2/1 is down, line protocol is down
Port down for 1 hours 13 minutes 24 seconds
Hardware is 40GigabitEthernet, address is cc4e.24f8.ef2c (bia cc4e.24f8.ef5d)
Interface type is 40Gig Fiber
Configured speed 40Gbit, actual unknown, configured duplex fdx, actual unknown
Configured mdi mode AUTO, actual unknown
Member of L2 VLAN ID 1, port is untagged, port state is BLOCKING
BPDU guard is Disabled, ROOT protect is Disabled, Designated protect is Disabled
Link Error Dampening is Disabled
STP configured to ON, priority is level0, mac-learning is enabled
Flow Control is enabled
Mirror disabled, Monitor disabled
Mac-notification is disabled
Not member of any active trunks
Not member of any configured trunks
No port name


The same happens with the 1/2/6.
The device came factory reset so it was pretty clean, that's my running config, pretty short:

#show running-config
Current configuration:
!
ver 08.0.10mT7f3
!
stack unit 1
module 1 icx6610-48p-poe-port-management-module
module 2 icx6610-qsfp-10-port-160g-module
module 3 icx6610-8-port-10g-dual-mode-module
stack-trunk 1/2/1 to 1/2/2
stack-trunk 1/2/6 to 1/2/7
!
!
!
!
vlan 1 name DEFAULT-VLAN by port
router-interface ve 1
!
!
!
!
aaa authentication web-server default local
aaa authentication enable default local
aaa authentication login default local
enable telnet authentication
!
username root password .....
!
!
!
!
!
!
!
!
!
interface management 1
ip address 192.168.1.1 255.255.255.0
!
interface ve 1
ip address 10.10.240.250 255.255.255.0
!
!
!
!
!
!
!
!
!
end




Any help would be greatly appreciated.

 

itronin

Well-Known Member
Nov 24, 2018
441
276
63
Denver, Colorado
Guys. Finally I got my hands on my ICX -6610-48p and it could not be different being a Cisco guy I'm having some trouble to configure the 2 rear stacking ports.
I want to use as a regular port to connect to my windows 10 box but no joy.
I'm using the fllowing hardware and cables:
NIC: MCX354A-FCBT 649281-B21 656089-001 VPI FDR 40GbE Mellanox from HP 544QSFP MCX354A-FCBT 649281-B21 656089-001 VPI FDR 40GbE Mellanox OEMFirmware | eBay
Cable: Netapp 112-00178 5M 16ft QSFP-QSFP External SAS Cable X6559-R6 from NetApp X 6559 R 6 SAS Cable 5 M QSFP for sale online | eBay
You are missing a couple of things. I'm assuming you are running a single switch and will be doing no stacking at all. I'm assuming your vlan is untagged, it is vlan 1 and your are using 1/2/1 for your data port.

1) you need to disable the stack and reboot the switch
conf t
disable stack
exit
write mem
reload


2) you need to add the 40Gbe port to your vlan
conf t
vlan 1
untagged ethe 1/2/1
# don't forget to add any other ports (tagged or untagged) to vlan 1.
exit
exit
 

Bell System

New Member
May 26, 2019
4
1
3
Southwest USA
...Unfortunately, that was about 20 or 30 pages before people started posting reports of busted switches from eBay seller "g-electronic," which of course is who I purchased mine from. They haven't arrived yet, but when they do it looks like I'll be putting them through a VERY intense round of testing and burn-in, especially the PoE stuff. Ugh. I'll be sure to report back on their condition once they arrive.
As promised, I wanted to update everyone on my 6450-24P switches from eBay seller "G-electronic."

I ordered two switches and unfortunately, both PoE modules are bad. They just keep printing the following over and over again:

Code:
PoE Error: Device 0 failed to start on PoE module.
PoE Error: Device 1 failed to start on PoE module.
Resetting module in slot 1 again to recover from dev fault
PoE Info: Hard Resetting in slot 1....
PoE Info: Resetting module in slot 1....completed.
Oddly, some ports do provide PoE power, but doing a
Code:
sh inline power
indicates that the switch doesn't even detect a connected PoE device.

One switch even has at least one port that is totally dead.

Seems like this guy is either reselling the same bad PoE switches hoping no one will notice, or he has a huge lot full of damaged switches. In any case, mine are going back, and I recommend staying away from this seller if you're interested in getting a switch with working PoE.
 
  • Like
Reactions: fohdeesha

Joao Paulo Mello

New Member
Jan 16, 2019
14
1
3
Thanks for you reply.

1) you need to disable the stack and reboot the switch
conf t
disable stack
exit
write mem
reload
I did but I had to "stack disable" instead of "disable stack".
2) you need to add the 40Gbe port to your vlan
conf t
vlan 1
untagged ethe 1/2/1
# don't forget to add any other ports (tagged or untagged) to vlan 1.
exit
exit
The port is on vlan 1 since it's default vlan and is already untagged
#show interfaces ethernet 1/2/1
40GigabitEthernet1/2/1 is down, line protocol is down
Port down for 6 minutes 58 seconds
Hardware is 40GigabitEthernet, address is cc4e.24f8.ef2c (bia cc4e.24f8.ef5d)
Interface type is 40Gig Fiber
Configured speed 40Gbit, actual unknown, configured duplex fdx, actual unknown
Configured mdi mode AUTO, actual unknown
Member of L2 VLAN ID 1, port is untagged, port state is BLOCKING
BPDU guard is Disabled, ROOT protect is Disabled, Designated protect is Disabled
Link Error Dampening is Disabled
STP configured to ON, priority is level0, mac-learning is enabled
Flow Control is enabled
Mirror disabled, Monitor disabled
Mac-notification is disabled
Not member of any active trunks
Not member of any configured trunks
No port name
MTU 1500 bytes, encapsulation ethernet
300 second input rate: 0 bits/sec, 0 packets/sec, 0.00% utilization
300 second output rate: 0 bits/sec, 0 packets/sec, 0.00% utilization
0 packets input, 0 bytes, 0 no buffer
Received 0 broadcasts, 0 multicasts, 0 unicasts
0 input errors, 0 CRC, 0 frame, 0 ignored
0 runts, 0 giants
0 packets output, 0 bytes, 0 underruns
Transmitted 0 broadcasts, 0 multicasts, 0 unicasts
0 output errors, 0 collisions
Relay Agent Information option: Disabled

Egress queues:
Queue counters Queued packets Dropped Packets
0 0 0
1 0 0
2 0 0
3 0 0
4 0 0
5 0 0
6 0 0
7 0 0


The problem remains, no link.
I read something mentioning that I need to have the latest firmware running in order to use the stack ports as data port.
That's my show version:

#show ver
Copyright (c) 1996-2014 Brocade Communications Systems, Inc. All rights reserved.
UNIT 1: compiled on Jul 08 2015 at 16:29:14 labeled as FCXR08010m
(9716466 bytes) from Primary FCXR08010m.bin
SW: Version 08.0.10mT7f3
Boot-Monitor Image size = 370695, Version:10.1.00T7f5 (grz10100)
HW: Stackable ICX6610-48-HPOE
==========================================================================
UNIT 1: SL 1: ICX6610-48P POE 48-port Management Module
Serial #: xxxxxxxxxxxxx
License: ICX6610_ADV_ROUTER_SOFT_PACKAGE (LID: xxxxxxxxxxxxxxxxxxxx)
P-ENGINE 0: type E02B, rev 01
P-ENGINE 1: type E02B, rev 01
==========================================================================
UNIT 1: SL 2: ICX6610-QSFP 10-port 160G Module
==========================================================================
UNIT 1: SL 3: ICX6610-8-port Dual Mode(SFP/SFP+) Module
==========================================================================
800 MHz Power PC processor 8544E (version 0021/0023) 400 MHz bus
65536 KB flash memory
512 MB DRAM
STACKID 1 system uptime is 13 minutes 7 seconds
The system : started=warm start reloaded=by "reload"
 
Last edited:

Joao Paulo Mello

New Member
Jan 16, 2019
14
1
3
Solved!!!
Going back to the past one more time I noticed the @arglebargle post: https://forums.servethehome.com/ind...gbe-40gbe-switching.21107/page-15#post-198883

That solved for me :).
Thanks @arglebargle, thanks everyone!

Edit:
I'm back with one more question: My unit came with 2 PSU's and just one fan tray. Does make sense to assume that installing another fan tray the fans will calm down a bit? The PSU's fan spin down after boot and I can live with that but if there's any way to slow down those boys a little that would be great. Have anyone had such experience?
 
Last edited:

Joao Paulo Mello

New Member
Jan 16, 2019
14
1
3
with those adapters you REALLY need to update the firmware using this guide as mentioned in the original post, it also points out to set the ports to ethernet: https://forums.servethehome.com/ind...net-dual-port-qsfp-adapter.20525/#post-198015

the firmware version your switch is using is also ancient, update it using the guide linked in the original post
Will do.

if your PSUs are already spinning down, adding a second fan tray will not make them spin down further
Damn! But I can live with the current noise level when the PSU's spins down...I have other stuff on the rack, some of them modded to be quiet and some couldn't be modded. So it's just another addition to the choir.

I was just testing the 40g adapter on Windows, in fact it will be used at my NAS (Ubuntu), the default protocol mode on linux is Infiniband as well? Do I'll need to install the driver and set to Ethernet somewhere?
 

bkvamme

New Member
Sep 23, 2018
7
1
3
Hi,
Might be a long shot, but does anyone know the ECCN and CCATS number for the ICX-6610-48P? My freight forwarder has been unable to get the numbers.

Based on what I see online, the ECCN should be: 5A002, but I am unable to find the CCATS number.

Edit: OK, this is getting a bit ridiculous. Contacted Ruckus support, and the rep stated that I since I purchased the switch through an unauthorised vendor (eBay) I would need to purchase a support contract for them to provide me with this number. Wth. Ubiquiti and Cisco are listing these online.
 
Last edited:

fohdeesha

Kaini Industries
Nov 20, 2016
1,994
1,813
113
29
fohdeesha.com
was digging around in @britinpdx 's ICX7250, it came with some dev software, was unbricking it for him and came across a hidden environment variable: setenv diag_mode_enable=1

Enabling this from the bootloader by running:

setenv diag_mode_enable 1
saveenv
reset

makes it reset into the bootloader again, except now it will be full featured u-boot with everything available, all the i2c raw read and write commands etc, all the stuff that's hidden in the "normal" bootloader prompt.

to get out of it and go back to booting normally just run:

setenv diag_mode_enable
saveenv
reset

Should work on any brocade switch running a u-boot based bootloader (NOT the 6610)
 

cuco

Member
Feb 13, 2018
52
2
8
35
Hey,
does anyone know a good place to buy cheap rackmount ears for the 6610?
Cant find anything on ebay
Thanks
 

HomelabN0rth

New Member
Jan 30, 2019
3
2
3
Hey all, thanks for all the info here. It made me jump on the ICX7250 train via ebay but I F*ed up and ordered from G-Electronics and got a switch with corrupted images on both the primary and secondary. I can get into ICX7250-Boot> but can't figure out how to do anything else. This is what happens:

ICX7250-Boot> tftpboot
ERROR: Ethernet external port not connected
at bcmiproc_eth.c:124/bcmiproc_eth_open()
could not boot from tftp, trying to boot from primary
Error: The header crc ffffffff and the calculated crc 4007851a mismatch
Fatal: Firmware integrity checksum failed
Booting image from Primary
no current device selected
Wrong Image Format for bootm command
ERROR: can't get kernel image!
could not boot from primary, no valid image; trying to boot from secondary
Error: The header crc ffffffff and the calculated crc 4007851a mismatch
Fatal: Firmware integrity checksum failed
Booting image from Secondary
no current device selected
Wrong Image Format for bootm command
ERROR: can't get kernel image!
Wrong Image Format for tftpboot command
ERROR: can't get kernel image!
ICX7250-Boot>

When the switch boots, it boots to a DIAG> prompt. From there, I can type linux_shell and get the switch to reboot, and if I press B during the beginning of that process, I can get to this ICX7250-Boot> prompt. Any suggestions how I can get from this boot menu to upload the new firmware to the device? Or am I just totally borked?
 

fohdeesha

Kaini Industries
Nov 20, 2016
1,994
1,813
113
29
fohdeesha.com
yours came with the exact same diag bootloader and broadcom test image that @britinpdx did (also from g-electronic). install teamviewer and send me credentials, I can fix it. you'll need a putty window up connected to the switch over serial and the management port plugged into a network with a tftp server

we can definitively rule out g-electronic doing ANY testing, what a liar
 

HomelabN0rth

New Member
Jan 30, 2019
3
2
3
Actually, I got it figured out.
I had to setenv the ip settings in the Boot>, used TFTP then to mount a new boot image. Restarted the switch. After that, did the same using update_primary and update_ secondary to update the main image and afterwards was back in business. What a pain in the everything tho.
 
  • Like
Reactions: fohdeesha