Drag to reposition cover

Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

fohdeesha

Kaini Industries
Nov 20, 2016
2,117
1,988
113
30
fohdeesha.com
Haha. I would never doubt you! ;)

I want to understand how the def'n works and


The documentation states that stack-port "selects only one of the two stacking ports as a stacking port, which allows you to use the other port as a data port." So I thought stack-trunk 1/2/1 to 1/2/2 means the stack consists of the 40G and breakout 40G port and stack-port 1/2/1 means use the 40G as the stacking port. Then on stack unit 2, if I want to use the breakout 40G, I would reference 2/2/2 using stack-port. Basically connect between stack unit 1, port XL1 (1/2/1) and stack unit 2, port XL2-5 (1/2/2).

I feel it would really help if you could clarify on how cables are connected between stack units 1 and 2 based on the configuration you gave.

Thanks!
You must be reading stacking information regarding a different model, as the ICX6610 has 4 stacking ports, not 2, and they definitely don't talk about using them for data - in fact it explicitly states they *can't* be used as data ports ever (thankfully this *feature* was snuck in in 8010 if I recall).

The top two ports on each switch are connected to each other for a redundant stacking connection in my setup. One 40gbE port, one breakout port. this leaves one free 40gbe port and one free breakout port per switch for me:

 
  • Like
Reactions: klui

fohdeesha

Kaini Industries
Nov 20, 2016
2,117
1,988
113
30
fohdeesha.com
So - Just to be sure:
An ICX6610 has four QSFP+ ports on the rear. Two of them are regular 40GBe ports, but the other two are special.

They can either be used as stacking ports or as QSFP+->4x10G SFP+ breakout ports, but not as regular 40GBe Ports, right?
check the third reply to this forum thread, it has a diagram, and explains it (at least what I thought) was pretty clearly. two are 40gbE only, two are 4x 10gbE breakout only. any of the ports can be used for regular ethernet ports, or stacking ports
 

NablaSquaredG

Active Member
Aug 17, 2020
285
103
43
check the third reply to this forum thread, it has a diagram, and explains it (at least what I thought) was pretty clearly. two are 40gbE only, two are 4x 10gbE breakout only. any of the ports can be used for regular ethernet ports, or stacking ports
So yeah, basically what I said with the exception that the two regular 40GBe ports can also be used for stacking.

I'd like to use all four ports as uplink ports to other, non ICX6610 switches - I guess there is no way to build a reverse breakout cable?
ICX 6610 -> 40G QSFP+ to 4x10G Breakout -> 4x10G SFP+ to 40G QSFP+ Reverse Breakout -> QSFP+ port on another switch (not ICX6610)

I suppose replacing the Breakout - Reverse Breakout combination with a single QSFP+ cable wouldn't work either?
 

fohdeesha

Kaini Industries
Nov 20, 2016
2,117
1,988
113
30
fohdeesha.com
So yeah, basically what I said with the exception that the two regular 40GBe ports can also be used for stacking.

I'd like to use all four ports as uplink ports to other, non ICX6610 switches - I guess there is no way to build a reverse breakout cable?
ICX 6610 -> 40G QSFP+ to 4x10G Breakout -> 4x10G SFP+ to 40G QSFP+ Reverse Breakout -> QSFP+ port on another switch (not ICX6610)

I suppose replacing the Breakout - Reverse Breakout combination with a single QSFP+ cable wouldn't work either?
You can absolutely use a single regular qsfp cable to connect the breakout port to another switch (assuming the other switches port is configured for breakout / 4x 10gbe). You can even connect a breakout port with a qsfp cable to a non breakout port on something, but only one lane will link up so it will run at 10gbps

Both 40gbe and 4x 10gbe breakout qsfp+ are electrically identical, it's 4x 10gbps lanes. Difference is in how the ASIC/PHY decides to split them (or mux them)
 

NablaSquaredG

Active Member
Aug 17, 2020
285
103
43
You can absolutely use a single regular qsfp cable to connect the breakout port to another switch (assuming the other switches port is configured for breakout / 4x 10gbe).
Whaaaaaaat o_O
Okay, that is cool.

Assuming I use an ICX6610 on one side and a SX6036 on the other side

ICX6610 4x 10G breakout port in breakout mode -> 40G QSFP+ cable -> SX6036 regular QSFP+ port in 4x10G breakout mode would link up with 4x 10G links?
It should then be easy to pack the 4x 10G links and another "regular" QSFP+ link into an Link Aggretation group and get "virtual" 80G bandwidth
 

darthray

New Member
Apr 11, 2021
5
1
3
Thank you, @fohdeesha for the info and @klui and @NablaSquaredG for asking some of the questions I had in my head :). Certainly not doubting any of your information, @fohdeesha, but there's enough I don't know I feel I need to ask more questions. If you couldn't tell, I'm a fiber newbie. Oh, and I've read the post about stacking a few times already. Probably need to read it a few more times.

I had a feeling using an MPO/MPO cable on the breakout ports would work, so it's nice to get confirmation of that. Now, in your example you created a truck with ports 1/2/1 and 1/2/2. What if I just want to use the breakout port alone? Can I just set 1/2/2 as my only connection to the other 6610? My MPO fiber + transceivers are only going to be here late next week, otherwise I'd be experimenting with this stuff.

Edit: just saw your other reply saying that:
two [ports] are 40gbE only, two are 4x 10gbE breakout only. any of the ports can be used for regular ethernet ports, or stacking ports
so, if I can stack the 6610s just using one of the breakout ports (e.g.: 1/2/2 on each one) I should be golden.
 
Last edited:

klui

Active Member
Feb 3, 2019
295
124
43
You must be reading stacking information regarding a different model, as the ICX6610 has 4 stacking ports, not 2, and they definitely don't talk about using them for data - in fact it explicitly states they *can't* be used as data ports ever (thankfully this *feature* was snuck in in 8010 if I recall).

The top two ports on each switch are connected to each other for a redundant stacking connection in my setup. One 40gbE port, one breakout port. this leaves one free 40gbe port and one free breakout port per switch for me:

Ah, OK. You connect the ports like-for-like? 40G-to-40G and 40G breakout-to-40G breakout?

I just pasted the description from the 08.0.30p command line reference. The 08.0.30b stacking guide (w/in the 08030r manuals, which also includes the 08030p command reference) does have an ICX6610 section but does not elaborate on the stack-port command's uniqueness to the 6610 other than "The ICX 6610 does not allow users to change the stack-port command configuration."
 

eduncan911

Active Member
Jul 27, 2015
192
94
28
eduncan911.com
Exactly how would someone load a "boot image" into an ICX6610, where there is no boot image?

Got a pair from eBay that stated there is no boot image in the desc, and sure enough nothing is displayed on the screen and the fans/PSU screams at what sounds like max RPMs forever.

Read the website again, and been searching the thread but not finding much.

Do I really need to use JTAG, and the BDI2000 to recover these?

Is there a Raspberry Pi SPI method I could instead with like flashrom?
 

darthray

New Member
Apr 11, 2021
5
1
3
Exactly how would someone load a "boot image" into an ICX6610, where there is no boot image?

Got a pair from eBay that stated there is no boot image in the desc, and sure enough nothing is displayed on the screen and the fans/PSU screams at what sounds like max RPMs forever.

Read the website again, and been searching the thread but not finding much.
We bought it from the same place . Just follow the guide (link in the OP). You'll need a console cable. No need for JTAG or anything else.
 
  • Like
Reactions: eduncan911

mintchipmadness

New Member
Nov 27, 2020
7
2
3
Hello Everyone,
I have the ICX 7250-48p switch and I am looking for a 4 post 1U rail kit that will work. The ruckus part number is XBR-R000295. Has anyone found a reasonably priced alternatives to that kit other than a shelf? The cost of the ruckus part is more than I paid for the switch. Thank you for your help.
 
Last edited:

eduncan911

Active Member
Jul 27, 2015
192
94
28
eduncan911.com
Hello Everyone,
I have the ICX 7250-48p switch and I am looking for a 4 post 1U rail kit that will work. The ruckus part number is XBR-R000295. Has anyone found a reasonably priced alternatives to that kit other than a shelf? The cost of the ruckus part is more than I paid for the switch. Thank you for your help.
What about one of those 4-post shelves? Like this one:

 
Last edited:
  • Like
Reactions: mintchipmadness

scot1297

New Member
Apr 17, 2021
3
0
1
So I need a little bit of help pointing me in the right direction. I have went through most of the forums for my answer and I am 100% sure I read it, but didn't do it right. So the trouble I am having is other vlan's on the brocade 6610 switch can pull a ip from the dhcp server in pfsense, but I can not ping out or get internet.

My setup is internet to pfsense box to brocade 6610 switch. Its a fresh install of pfsense with vlan 20 vlan 100 vlan 101 setup. They were all provided dhcp servers with the ranges I needed. I also have a unifi AP that is plugged into the switch and get internet on it with the default vlan of 1. When ever i switch it to a different vlan I get internet issues. I am assuming I missed a setting so that I can reach the internet from these other vlans. I pasted below my brocade config. Any help would be great or if I am totally off base then tell me to...thanks

Code:
telnet@coreswitch(config)#show run
Current configuration:
!
ver 08.0.30tT7f3
!
stack unit 1
  module 1 icx6610-48p-poe-port-management-module
  module 2 icx6610-qsfp-10-port-160g-module
  module 3 icx6610-8-port-10g-dual-mode-module
!
global-stp
!
!
lag pvec1 dynamic id 1
 ports ethernet 1/3/1 to 1/3/2
 primary-port 1/3/1
 deploy
!
lag pvec2 dynamic id 2
 ports ethernet 1/3/3 to 1/3/4
 primary-port 1/3/3
 deploy
!
lag pvec3 dynamic id 3
 ports ethernet 1/3/5 to 1/3/6                                   
 primary-port 1/3/5
 deploy
!
lag pvec4 dynamic id 4
 ports ethernet 1/3/7 to 1/3/8
 primary-port 1/3/7
 deploy
!
lag synologynas dynamic id 5
 ports ethernet 1/1/3 to 1/1/6
 primary-port 1/1/3
 deploy
!
!
vlan 1 name DEFAULT-VLAN by port
 router-interface ve 1
 spanning-tree
 spanning-tree rstp
!
vlan 5 name corosync by port
 untagged ethe 1/1/25 to 1/1/28
!
vlan 10 name cephpublic by port                                   
 tagged ethe 1/3/1 to 1/3/8
!
vlan 15 name cephcluster by port
 tagged ethe 1/3/1 to 1/3/8
!
vlan 17 name pvemigration by port
 tagged ethe 1/3/1 to 1/3/8
!
vlan 20 name wifi by port
 tagged ethe 1/1/1 ethe 1/1/13 to 1/1/16 ethe 1/1/43 ethe 1/1/47 to 1/1/48
 spanning-tree
 spanning-tree rstp
!
vlan 100 name proxmox100 by port
 tagged ethe 1/1/1 ethe 1/1/13 to 1/1/16 ethe 1/1/43 ethe 1/1/47 to 1/1/48
 spanning-tree
 spanning-tree rstp
!
vlan 101 name proxmox101 by port
 tagged ethe 1/1/1 ethe 1/1/13 to 1/1/16 ethe 1/1/43 ethe 1/1/47 to 1/1/48
 router-interface ve 101
 spanning-tree
 spanning-tree rstp                                               
!
!
!
!
!
aaa authentication web-server default local
aaa authentication enable default local
aaa authentication login default local
jumbo
hostname coreswitch
ip dhcp-client disable
ip dns server-address 1.1.1.1
ip route 0.0.0.0/0 192.168.1.1
!
username ... password .....
snmp-server community ..... ro
!
!
clock summer-time
clock timezone us Central
!
!
ntp                                                               
 disable serve
 server 216.239.35.0
 server 216.239.35.4
!
!
!
!
!
!
!
!
!
interface ethernet 1/1/1
 dual-mode
!
interface ethernet 1/1/13
 dual-mode
!
interface ethernet 1/1/14
 dual-mode
!
interface ethernet 1/1/15
 dual-mode                                                       
!
interface ethernet 1/1/16
 dual-mode
!
interface ethernet 1/1/43
 dual-mode
 inline power
!
interface ethernet 1/1/44
 inline power
!
interface ethernet 1/1/45
 inline power
!
interface ethernet 1/1/46
 inline power
!
interface ethernet 1/1/47
 dual-mode
 inline power
!
interface ethernet 1/1/48
 dual-mode                                                       
 inline power
!
interface ethernet 1/3/1
 speed-duplex 10G-full
!
interface ethernet 1/3/3
 speed-duplex 10G-full
!
interface ethernet 1/3/5
 speed-duplex 10G-full
!
interface ethernet 1/3/7
 speed-duplex 10G-full
!
interface ve 1
 ip address 192.168.1.2 255.255.255.0
!
interface ve 101
 ip address 192.168.101.254 255.255.255.0
!
!
!
!                                                                 
!
!
!
!
!
end
 

LodeRunner

Active Member
Apr 27, 2019
127
57
28
I don’t see IPs or virtual router definitions for anything other than VLAN 1 and 100. Also, do you have a return route from pfSense? Do you have to proper rules in both directions on the LANS on the pfSense side?
 

scot1297

New Member
Apr 17, 2021
3
0
1
I don’t have a return route in pfsense. That I am struggling with, can you point me in the direction of setting up a return route? Thanks!
 

LodeRunner

Active Member
Apr 27, 2019
127
57
28
Last time I tried this, I had a dedicated routing VLAN. So each VLAN had an IP, switch had a default route over a separate VLAN (/30). The switch had the pfSense IP as the destination, in the pfSense routing page, I had to manually create a route from pfSense with the switch IP as the gateway.

I haven't actually done VLANs on pfSense directly and was using ACLs between the VLANs instead of firewall rules. But since the switch is the gateway for all networks and pfSense is the upstream gateway, then you need a reverse route in pfSense pointing back to the switch IP, or maybe the VLAN IPs? That's what I'm not clear on, but I haven't sat down to try and make VLANs on pfSense work.
 

mintchipmadness

New Member
Nov 27, 2020
7
2
3
What about one of those 4-post shelves? Like this one:

Thank you for your help. I was thinking of that exact one but I believe it won't work for my switch since the main airflow is side to back. The OEM kit comes in two pieces, one for the front of the switch and the other at the back which leaves the side intake open for airflow.
 
Last edited:
  • Like
Reactions: eduncan911

DavidRa

Infrastructure Architect
Aug 3, 2015
283
128
43
Central Coast of NSW
www.pdconsec.net
I can't find a way to search the thread - are there models other than the ones in the first few posts worth looking at? I may have to replace my Dell 8024F pair (each 20 x SFP+ and 4x combo SFP+/10T, and stacked).

I'd need a combination of SFP+ and 10GBaseT ports - ideally ~16-20 SFP+ and 4-8 10G-T each would be sufficient. No need for 40Gbps (my two are currently stacked with 4x 10Gb in LACP). Is there a ... IDK a 7300 or 7500 or something?
 

fohdeesha

Kaini Industries
Nov 20, 2016
2,117
1,988
113
30
fohdeesha.com
Thank you, @fohdeesha for the info and @klui and @NablaSquaredG for asking some of the questions I had in my head :). Certainly not doubting any of your information, @fohdeesha, but there's enough I don't know I feel I need to ask more questions. If you couldn't tell, I'm a fiber newbie. Oh, and I've read the post about stacking a few times already. Probably need to read it a few more times.

I had a feeling using an MPO/MPO cable on the breakout ports would work, so it's nice to get confirmation of that. Now, in your example you created a truck with ports 1/2/1 and 1/2/2. What if I just want to use the breakout port alone? Can I just set 1/2/2 as my only connection to the other 6610? My MPO fiber + transceivers are only going to be here late next week, otherwise I'd be experimenting with this stuff.

Edit: just saw your other reply saying that:


so, if I can stack the 6610s just using one of the breakout ports (e.g.: 1/2/2 on each one) I should be golden.
FYI I vaguely remember the stack config needing to use at least one 40gbE port, (eg it would complain about the user only specifying a "secondary stack port" when only entering a breakout port). I would search the thread, I can't remember conclusively
 
  • Like
Reactions: darthray