Drag to reposition cover

Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

Nnyan

Active Member
Mar 5, 2012
137
40
28
I'm sure you've figured out I've spent (and am still spending) a lot of time with the IOS CLI. ;)
I don't mind going to the CLI when I need to but junos just feels like unneeded pain. I'm by far not an expert but I've always managed to get a switch up and running pretty quickly, the EX3300? not so much.
 

fohdeesha

Kaini Industries
Nov 20, 2016
2,278
2,359
113
30
fohdeesha.com
I don't mind going to the CLI when I need to but junos just feels like unneeded pain. I'm by far not an expert but I've always managed to get a switch up and running pretty quickly, the EX3300? not so much.
JunOS is...verbose. Which is amazing in a production environment where I need to create nested lists that feed into BGP filters etc, but at home, yeah you're probably gunna feel like you're wasting time if you don't already have a bunch of junOS experience. Wanna trade for an ICX6450? lmao
 

klui

Active Member
Feb 3, 2019
395
169
43
I very much like commit confirmed exists in JunOS even though I typically don't use it. I normally use commit check. I've been bit before where issuing a change in conf t immediately borked my config forcing me to use the console or power cycle.

Aside from that, if one doesn't use it regularly it's difficult to remember the higher level node names and once you do see the config scrolling up is necessary to see them--a pain in the ass. For that show config|display set displays the full command in a non-structured format.
 
  • Like
Reactions: fohdeesha

jahsoul

Active Member
Dec 13, 2013
260
33
28
War Eagle Country
JunOS is...verbose. Which is amazing in a production environment where I need to create nested lists that feed into BGP filters etc, but at home, yeah you're probably gunna feel like you're wasting time if you don't already have a bunch of junOS experience. Wanna trade for an ICX6450? lmao
I swear you just described my experience with my EX2300. lol. The initial configuration was fine but dang, creating Firewall Filters are like writing a novel. I can create Cisco ACLs in my sleep but these firewall filters have me stuck.
 
  • Like
Reactions: fohdeesha

aaroneaton

New Member
Jan 15, 2021
12
0
1
www.rfehosting.com
Ok ive got a weird issue going on.
So i had issues getting breakout working, so ended up getting a new 6610. I was able to get the breakouts working on the new one.
I then setup stacking between the new and old. And am now not able to get breakout working on the new one. And it only works on the old one.

I used the directions above. So its like the breakout port fixed itself on the old and broke on the new. So it has to do with stacking somehow.

I have 2 diff breakout cables. one is FS, and one is Brocade from FS.
I have the exact same config/setup as you used here.

Any other thoughts on how to make breakout work on unit 1? It was working before stacking.
ID Type Role Mac Address Pri State Comment
1 S ICX6610-48P active 748e.f8fb.b8b6 128 local Ready
2 S ICX6610-48P standby cc4e.2413.2d44 0 remote Ready

active standby
+---+ +---+
-2/6| 1 |2/1--2/1| 2 |2/6-
| +---+ +---+ |
| |
|------------------------|
Standby u2 - protocols ready, can failover

stack unit 1
module 1 icx6610-48p-poe-port-management-module
module 2 icx6610-qsfp-10-port-160g-module
module 3 icx6610-8-port-10g-dual-mode-module
priority 128
stack-port 1/2/1 1/2/6
stack unit 2
module 1 icx6610-48p-poe-port-management-module
module 2 icx6610-qsfp-10-port-160g-module
module 3 icx6610-8-port-10g-dual-mode-module
stack-port 2/2/1 2/2/6
stack enable


Unit 1 is the new one, unit 2 is the old. Breakout never worked on the old one pre stacking. It worked fine on the new one prestacking. Now it is reversed, and does not work on the new one, but works on the old.

Thoughts?




OK - considering the rear ports are not intended to be used for data ports, you won't find any information on this in the manuals etc. Using them as just data ports is easy enough, but splitting them up and using some for stacking and some for data ports requires some finangling to make the stacking code not try and take over all 4 ports.

However it's possible and stable across reboots etc. basically follow this, it might take you a couple times. This will enable stacking between two units just using the 40gbE qsfp ports, leaving 4x breakout QSFPs total available for data use


Code:
#unplug all stacking cables except for 1, going between port 1/2/1 on unit 1 and 1/2/1 on unit two
#don't plug anything else into unit 2 or you'll get a loop
#first you have to unstack all the units and remove any existing stack configuration:
enable
stack unconfigure all
write mem
reload
#both switches should reload, wait till they come back up

#when they're back up, on unit 1:
enable
conf t
stack unit 1
no stack-trunk 1/2/1 to 1/2/2
stack-port 1/2/1 1/2/6
#ignore the warning that pops up about not showing up in "show run"

#still running the below on unit 1!
stack unit 2
no stack-trunk 2/2/1 to 2/2/2
stack-port 2/2/1 2/2/6
#ignore the warning that pops up about not showing up in "show run"
exit
stack enable
write mem
exit
stack secure-setup

#When the setup is done, do "show run"
#stack unit 2 section probably has "stack-trunk 2/2/1 to 2/2/2" added back to it

#the second switch is currently rebooting to rejoin the stack, wait for it to do so
#once it comes back online, do the following to remove the stack unit 2 trunk:
#still on unit 1:
conf t
stack unit 2
no stack-trunk 2/2/1 to 2/2/2
#will probably reboot unit 2 again
#wait for it to come back online, then connect the second pair of 40gbE ports
Once you connect the second pair of 40gbe ports (1/2/6 on unit 1 to 1/2/6 on unit 2) you should be done. Check by running "show stack", you should see both units "ready" with the connection diagram showing both 40gbE ports linked up like below:

Code:
ICX1#sh stack
T=6m28.6: alone: standalone, D: dynamic cfg, S: static
ID   Type          Role    Mac Address    Pri State   Comment
1  S ICX6610-48P   active  cc4e.24b8.d9d0 128 local   Ready
2  S ICX6610-48P   standby cc4e.243e.aa74   0 remote  Ready

    active       standby
     +---+        +---+
-2/6| 1 |2/1--2/1| 2 |2/6-
|   +---+        +---+   |
|                        |
|------------------------|
Standby u2 - protocols ready, can failover
Current stack management MAC is cc4e.24b8.d9d0
If you have the same output, do a "write mem" then a "reload" to reload the whole stack. It should come back up fresh in the correct state and running "show stack" again should have the same output as before.

Your final stacking config should look like this:

Code:
stack unit 1
  module 1 icx6610-48p-poe-port-management-module
  module 2 icx6610-qsfp-10-port-160g-module
  module 3 icx6610-8-port-10g-dual-mode-module
  priority 128
  stack-port 1/2/1 1/2/6
stack unit 2
  module 1 icx6610-48p-poe-port-management-module
  module 2 icx6610-qsfp-10-port-160g-module
  module 3 icx6610-8-port-10g-dual-mode-module
  stack-port 2/2/1 2/2/6
stack enable
 

ZFSZealot

New Member
Aug 16, 2021
20
3
3
Having trouble using the breakout stack ports on a 6610 to a server. Stacking is not enabled. Using a decent quality breakout DAC cable.

I'm sure I'm missing something stupid.

I configured a static LAG to go to an ESXi server using 1/2/9 and 1/2/10, primary port 1/2/9. Deployed. Links were up, lights on the back of the server were lit. Then I added 1/2/9 as tagged to one of the vlans and the link went down on both ports. I have not been able to get them to come back up. I know the cable/nic/switchports are physically fine because I saw link before. The link dropped the instant that I did the "tagged ethe 1/2/9" in the vlan config. I saw it out of the corner of my eye because I was tailing the vmkernel.log on the server. I have since completely removed the ports from all vlans, the lag itself, added the two ports to my main vlan (happens to be 101) and rebooted the ESXi. There is no chance of a loop as the NICs are "unused adapters" on the esxi side. STP appears to be enabled on the port but not on any of my vlans.

I cannot for the life of me get these links to come back up. I have not reloaded the switch because it is passing other traffic right now but I can try that if necessary. It doesn't seem like this is the sort of thing that should require reboot of the switch. What stupid thing am I missing?

Code:
  10GigabitEthernet 1/2/9 is down, line protocol is down      
  Port down for 11 minute(s) 48 second(s)                       
  Hardware is   10GigabitEthernet , address is 748e.f8e9.a6d0 (bia 748e.f8e9.a709)
  Configured speed 10Gbit, actual unknown, configured duplex fdx, actual unknown
  Configured mdi mode AUTO, actual unknown                     
  Member of L2 VLAN ID 101, port is untagged, port state is BLOCKING
    BPDU guard is Disabled, ROOT protect is Disabled, Designated protect is Disabled
  Link Error Dampening is Disabled                             
  STP configured to ON, priority is level0, mac-learning is enabled
  Openflow is Disabled, Openflow Hybrid mode is Disabled,  Flow Control is config enabled, oper enabled, negotiation disabled
  Mirror disabled, Monitor disabled                             
  Mac-notification is disabled                                 
  Not member of any active trunks                               
  Not member of any configured trunks                           
  Port name is Trunk vSphere Umbra 10G                         
  MTU 1500 bytes, encapsulation ethernet                       
  300 second input rate: 0 bits/sec, 0 packets/sec, 0.00% utilization
  300 second output rate: 0 bits/sec, 0 packets/sec, 0.00% utilization
  0 packets input, 0 bytes, 0 no buffer                         
  Received 0 broadcasts, 0 multicasts, 0 unicasts               
  0 input errors, 0 CRC, 0 frame, 0 ignored                     
  0 runts, 0 giants                                             
  0 packets output, 0 bytes, 0 underruns                       
  Transmitted 0 broadcasts, 0 multicasts, 0 unicasts           
  0 output errors, 0 collisions                                 
  Relay Agent Information option: Disabled
 
Last edited:

nickf1227

Active Member
Sep 23, 2015
158
83
28
30
Protip:
If you upgrade/migrate from the "switch mode" firmware to the "router mode" firmware in @fohdeesha 's guides, you will break your management interface and have to console into the switch ;)
 

fohdeesha

Kaini Industries
Nov 20, 2016
2,278
2,359
113
30
fohdeesha.com
Protip:
If you upgrade/migrate from the "switch mode" firmware to the "router mode" firmware in @fohdeesha 's guides, you will break your management interface and have to console into the switch ;)
reason #12324 why it's easier to just start on the layer3 FW in the first place, even if you don't need l3 features at the moment :p zero downside

For the two people above having breakout link issues, once the other end is plugged in and the server is booted up etc, can you reboot all the switches? On some chassis I've seen some weird stuff where the breakout stack ports only want to link up if the stuff is all plugged in and ready on switch boot
 
  • Like
Reactions: nickf1227

aaroneaton

New Member
Jan 15, 2021
12
0
1
www.rfehosting.com
reason #12324 why it's easier to just start on the layer3 FW in the first place, even if you don't need l3 features at the moment :p zero downside

For the two people above having breakout link issues, once the other end is plugged in and the server is booted up etc, can you reboot all the switches? On some chassis I've seen some weird stuff where the breakout stack ports only want to link up if the stuff is all plugged in and ready on switch boot
Ill give it a try. Thanks
 

ZFSZealot

New Member
Aug 16, 2021
20
3
3
For the two people above having breakout link issues, once the other end is plugged in and the server is booted up etc, can you reboot all the switches? On some chassis I've seen some weird stuff where the breakout stack ports only want to link up if the stuff is all plugged in and ready on switch boot
Regarding mine, the following comes out of the serial console on startup. A clue? I'm using 1/2/2, 1/2/3 in a static LAG, and same for 1/2/7, 1/2/8. No physical connections on 1/2/1, 1/2/4, 1/2/5, 1/2/6, 1/2/9, 1/2/10 - yet. Do all four lanes in the QSFP+ on these breakout stacking ports have to be physically connected to something?

Code:
Parsing Config Data ...
------------------------------------------------------------------
M:9 L:0 - chow_qsfp_read, qsfp 2, error in seting up mux
------------------------------------------------------------------
M:9 L:0 - link_40G_4x10G_get_media: qsfp 2, port 1/2/2 error in reading qsfp
chow_40G_4x10G_get_media: error in reading qsfp 1/2/2
------------------------------------------------------------------
M:9 L:0 - chow_qsfp_read, qsfp 3, error in seting up mux
------------------------------------------------------------------
M:9 L:0 - link_40G_4x10G_get_media: qsfp 3, port 1/2/7 error in reading qsfp
chow_40G_4x10G_get_media: error in reading qsfp 1/2/7
EDIT: And all four of 1/2/2, 1/2/3, 1/2/7 and 1/2/8 have links up after the reboot despite the errors I pasted above. The errors must be something to do with stacking code unifying those broken out ports for stacking or something?

If this switch doesn't like servers at the end of the breakout QSFP+ ports going up and down without itself being reloaded too this may not meet my use case - I like to keep one ESXi up most of the time and only spin up the others if I need them - power use and all... Going to have to experiment and will report.
 

ZFSZealot

New Member
Aug 16, 2021
20
3
3
I'm assuming

M:9 L:0 - chow_qsfp_read, qsfp 2, error in seting up mux
------------------------------------------------------------------
M:9 L:0 - link_40G_4x10G_get_media: qsfp 2, port 1/2/2 error in reading qsfp
chow_40G_4x10G_get_media: error in reading qsfp 1/2/2
------------------------------------------------------------------
M:9 L:0 - chow_qsfp_read, qsfp 3, error in seting up mux
------------------------------------------------------------------
M:9 L:0 - link_40G_4x10G_get_media: qsfp 3, port 1/2/7 error in reading qsfp
chow_40G_4x10G_get_media: error in reading qsfp 1/2/7


is a safe bet that the QSFP to SFP+ DAC breakout I got is no bueno? Any possible way of troubleshooting this further?

EDIT: Upon booting looks like I'm seeing activity on the links and they do show up as up in the switch. Initially when I plugged them in the switch was on and I got nothing, so I reloaded. I guess might be a case of it not liking the modules being hotplugged?
The conversation starting here sounds almost exactly like what I'm experiencing - including adding tagged vlans to them causing the links to go down and never come back up until the switch is reloaded. I think the conclusion was that the switch was bad, but could the fact that I'm seeing the same thing show this is a bug instead? I do have another 6610 to try if it's necessary but they came from the same place so who knows if trying it will show anything conclusive. I'm running the latest 08.0.30u firmware. I do have some different breakouts coming in the mail supposedly today too - another thing to try.
 
Last edited:

chickenparm555

New Member
Sep 22, 2021
3
0
1
Hi all - two questions about the ICX7xxx series:

1. Can the SFP+ "uplink" ports be used just like any other port on the switch? Or are there limitations on what they can/cannot be used for. (I know only certain ports can be uplinks, but can uplinks be "normal" ports)?

2. Are you able to stack any model ICX7xxx with any other model ICX7xxx? For example a ICX7150-C12P at my desk and a "bigger beef" ICX7450 in a sever rack?

I found this, but it ONLY mentions the 7150:

And this, but it doesn't mention the 7150:

So I'm concerned I can't mix the entry-level switches with the higher-end ones in a single stack.



Thank you!!
 

LodeRunner

Active Member
Apr 27, 2019
181
78
28
Hi all - two questions about the ICX7xxx series:

1. Can the SFP+ "uplink" ports be used just like any other port on the switch? Or are there limitations on what they can/cannot be used for. (I know only certain ports can be uplinks, but can uplinks be "normal" ports)?

2. Are you able to stack any model ICX7xxx with any other model ICX7xxx? For example a ICX7150-C12P at my desk and a "bigger beef" ICX7450 in a sever rack?

I found this, but it ONLY mentions the 7150:

And this, but it doesn't mention the 7150:

So I'm concerned I can't mix the entry-level switches with the higher-end ones in a single stack.



Thank you!!
Uplink ports can be use as normal switchports.

No cross-model stacking. Must be in same family. So 71xx, 72xx, 74xx, etc.

The only exception is if you are using them as 802.1br SPX extenders with a 76, 77, or 78 series as the CB, but that's apparently gone away in v9 of the firmware, possibly indicating EoL/discontinuation of 802.1br support.
 
  • Like
Reactions: fohdeesha