Drag to reposition cover

Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

Propaganda

Active Member
Dec 6, 2017
103
25
28
42
I am green to switch stacking and have some probably stupid questions here. I have been reading the stacking documentation for 08.0.90(k) firmware running on 2x 7250. I just want to make sure I am understanding correctly because I haven't seen definitive answers. I am using a single 10G port on each switch 1/2/1 and 2/2/1 for stacking, do I need to make a trunk or can I just use a single port on each like this? Can I reclaim the use of 1/2/3 and 2/2/3 as a data ports? Do I need to provide additional uplink ports between the switches or is all data transferred between using the stack ports?

Code:
ICX7250-24P Router>show stack
T=19h39m5.5: alone: standalone, D: dynamic cfg, S: static
ID   Type          Role    Mac Address    Pri State   Comment
1  S ICX7250-24P   active  mmmm.aaaa.cccc   0 local   Ready
2  S ICX7250-24    standby mmmm.aaaa.cccc   0 remote  Ready

    active       standby
     +---+        +---+
  2/3| 1 |2/1--2/1| 2 |2/3
     +---+        +---+
Standby u2 - sent role but not acked
Role history: N: standalone, A: active, S: standby, M: member
U1: N->A, U2: M->S->N->M->S
 
Last edited:

Spearfoot

Active Member
Apr 22, 2015
113
53
28
I am having issues with the 40G breakout ports, where only the 4th port in each group (1/2/5 and 1/2/10) will link when using a QSFP-to-4xSFP+ breakout DAC (specific cable is a FS Brocade 40G-QSFP-4SFP-C-0201).

My Setup:
Switch - ICX 6610-48P with licenses, breakout QSFP plugged into XL2-5
Server 1 - Intel X520 PCIe NIC - ports 1 and 2 connected to breakout SFP+ transceivers 1 and 2 (1/2/2 and 1/2/3)
Server 2 - Intel 82599ES daughter card - ports 1 and 2 connected to breakout SFP+ transceivers 3 and 4 (1/2/4 and 1/2/5)

The successful link follows the switch port (changing the order of transceivers in a NIC, and trying different NICs results in 1/2/5 always linking). Plugging only one SFP+ breakout at a time into a NIC does not change the behavior, only the 1/2/5 or 1/2/10 ports will link up. Using a new cable (of the same model) does not resolve this, using XL7-10 (the second QSFP breakout port) also does not resolve (only 1/2/10 links up).

Any help figuring this out would be appreciated!

show interface brief ethernet 1/2/1 to 1/2/5
Code:
Port       Link    State   Dupl Speed Trunk Tag Pvid Pri MAC         Name
1/2/1      Up      Forward Full 40G   13    Yes N/A  0   redacted    redacted
1/2/2      Down    None    None None  None  No  1    0   redacted    redacted
1/2/3      Down    None    None None  None  No  1    0   redacted    redacted
1/2/4      Down    None    None None  None  No  1    0   redacted    redacted
1/2/5      Up      Forward Full 10G   None  No  1    0   redacted
show stack
Code:
***** Warning! stack is not enabled. *****

T=22d20h53m31.8: alone: standalone, D: dynamic cfg, S: static
ID   Type          Role    Mac Address    Pri State   Comment
1  S ICX6610-48P   alone   redacted         0 local   None:0


     +---+
  2/1| 1 |2/6
     +---+
Current stack management MAC is redacted
show run
Code:
stack unit 1
  module 1 icx6610-48p-poe-port-management-module
  module 2 icx6610-qsfp-10-port-160g-module
  module 3 icx6610-8-port-10g-dual-mode-module
ETA: I did a fair bit of googling and keyword searching this thread and found the combined stacking + breakout config instructions, but my setup is not stacking, just 100% data. The XL1 and XL6 ports are connected directly to a 40G NIC in another box.
I don't know if it makes a difference... but do you have 'stack disable' in your setup? Like this:
Code:
stack unit 1
  module 1 icx6610-48p-poe-port-management-module
  module 2 icx6610-qsfp-10-port-160g-module
  module 3 icx6610-8-port-10g-dual-mode-module
stack disable
 
  • Like
Reactions: neodied

fohdeesha

Kaini Industries
Nov 20, 2016
2,577
2,775
113
31
fohdeesha.com
When I got home today my entire network was down and my ICX 6610 had the fans at 100%, so I unplugged and replugged it into power; the fans still stuck at full bore, and the activity lights on the SFP ports are solid green. I cannot see it on the network anymore and I tried using a console cable but it's no longer working either.

Anyone have any troubleshooting suggestions? I feel like it may just be a noisy brick now.
That's odd, sounds like it is indeed bricked, which in my experience is usually a bad/incomplete bootloader, but it typically doesn't happen for no reason - could be a bad bootloader flash/upgrade from a while back and it just hadn't been rebooted yet or similar. Depending on how invested you are and how close to indiana you are, I might be able to mail you my jtag unit if you want to flash a fresh bootloader via jtag and see if that's the issue
 

neodied

New Member
Aug 1, 2021
2
3
3
I don't know if it makes a difference... but do you have 'stack disable' in your setup? Like this:
Code:
stack unit 1
  module 1 icx6610-48p-poe-port-management-module
  module 2 icx6610-qsfp-10-port-160g-module
  module 3 icx6610-8-port-10g-dual-mode-module
stack disable
You are a lifesaver! Adding that, then a reboot solved the issue, now I'm getting link on all 4 QSFP breakout ports.
 

MassiveMeatMissile

New Member
Aug 22, 2020
3
1
3
That's odd, sounds like it is indeed bricked, which in my experience is usually a bad/incomplete bootloader, but it typically doesn't happen for no reason - could be a bad bootloader flash/upgrade from a while back and it just hadn't been rebooted yet or similar. Depending on how invested you are and how close to indiana you are, I might be able to mail you my jtag unit if you want to flash a fresh bootloader via jtag and see if that's the issue
I appreciate the offer but I've decided to buy a new one already.
 
  • Like
Reactions: fohdeesha

OKGolombRuler

New Member
Mar 13, 2020
16
6
3
7150-c12p, MOCA, and Campus Fabric-- oh, my!

Hi folks--

First off, thanks to Fohdeesha and all the forum folks as a result of whose work and words I've learned so much (at least, failed so much less!). :)

Second, I've trawled this thread a fair bit but haven't seen much discussion of campus fabric, which is today's puzzle.

Requests:
  • Corrections for any of the below
  • Suggestions on smarter/alternative ways to do it? Stupid/crazy but practicable ideas welcomed.
  • Is this as simple as finding either a 7150 (to stack?) or a ... 7450? 7650? to serve as a CB?

Environment:
My house is rather hostile to 5ghz wifi and lacks ethernet and cell service but has cable TV jacks. Most of my network traffic is latency-sensitive, whether it's endless Zoom calls, VOIP/SIP traffic, SDR, etc. I do isolate IOT, guests, etc into VLANs.

I have a 6610 in the basement and a couple 7150-c12ps which I put in distant rooms, attached client devices and a POE AP, and home-run by way of individual pairs of MOCA bridges. And it's great, and mostly even works! but I've now got enough VLAN segmentation, firewall shenanigans, etc to want to manage the L2 switching fabric as one coherent unit. I'm tired of trunking vlans around on switch after switch each time I want to rewicker an experiment.

Obstacles:
  • The network topology is hub-and-spoke; there will be no redundant links here. (Which rules out Stacking but should work with Campus Fabric mode, I think?)
  • The 7150s will supposedly act as Port Extenders (which I read as akin to Cisco FEXes) if paired with a suitable (7150? 7x50?) Control Bridge, but my 6610 will not cut the mustard.
  • The 7150s do not support the 2.5gbps ethernet the MOCA bridges offer; so I'm leaving bandwidth on the table. (This is a modest annoyance, but I mention it in case my ignorance not silicon is the limiting factor).
  • It 'feels weird' to push every frame down a 1gbps link to the Control Bridge only to have it come right back up that same link. Maybe I misunderstand, and only frames destined for a different physical switch get pushed to the BC for processing? In which case they'd be traversing it anyway, no harm done

I'm still noodling on the 8.0.95 campus fabric guide, please feel free to point me to docs (or your own running examples!) if I missed anything.
 

LodeRunner

Active Member
Apr 27, 2019
455
197
43
I could be wrong, but I believe you're going to need a 7650, 7750, or 7850 to be a CB. My 7450, running 8.0.92e, certainly does not have the commands present to configure it as a CB.
 
  • Sad
Reactions: OKGolombRuler

OKGolombRuler

New Member
Mar 13, 2020
16
6
3
note that campus fabric is dead tech, they abandoned it and retired it completely in the latest firmware train
Crud, I haven't looked beyond 8.0.95. Hopefully they replaced it with something that will accept single, full-duplex links. I'll have to go do some digging. Thanks!

And, while you're eyes-on, Fohdeesha, thanks again for all your effort and research into these devices!
 
  • Like
Reactions: fohdeesha

fohdeesha

Kaini Industries
Nov 20, 2016
2,577
2,775
113
31
fohdeesha.com
Crud, I haven't looked beyond 8.0.95. Hopefully they replaced it with something that will accept single, full-duplex links. I'll have to go do some digging. Thanks!

And, while you're eyes-on, Fohdeesha, thanks again for all your effort and research into these devices!
Hmm stacking should definitely work with single links! (otherwise when a redundant two link stack connection lost a link, it would become...quite useless)
 

LodeRunner

Active Member
Apr 27, 2019
455
197
43
Rats! Thanks for letting me know. :) May I ask about what your 7450 ran you?
My ICX7450-48P came with both PSUs and 2 40G modules, but no 10G module and missing a fan, for $300. You might do better right now, eBay stock is always changing obviously. The fan and 4x10 module cost me another $90 ($60 for fan, $29 for 10G); the 4x10's are cheaper now, fan supply is ... erratic; right now, ICX-FAN10-E modules are $100+
 
  • Like
Reactions: OKGolombRuler

liv3010m

New Member
Jul 22, 2018
21
2
3
Buenos Aires, Argentina
Hi guys, I just jumped into the ICX bandwagon, may I be the last wagon? Hopefully not, because it's great hardware for the price! :p
Joke aside, anyone running firmware version 8030u on ICX6450? I saw some posts from like a year ago, on this same thread, where it was mentioned it had some bugs or it was "being tested", so my question is: anybody running it without any abnormalities?
 

Vesalius

Active Member
Nov 25, 2019
224
160
43
Hi guys, I just jumped into the ICX bandwagon, may I be the last wagon? Hopefully not, because it's great hardware for the price! :p
Joke aside, anyone running firmware version 8030u on ICX6450? I saw some posts from like a year ago, on this same thread, where it was mentioned it had some bugs or it was "being tested", so my question is: anybody running it without any abnormalities?
Thats the latest/last firmware for the ICX6450, so pretty much everyone in this thread using a ICX-6*** is using it, I am on my 6450. I am sure there are a few bugs, but none that affect me and all is well for the last year or 2.