Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

[fohdeesha]

Haha. I would never doubt you!

I want to understand how the def'n works and


The documentation states that stack-port "selects only one of the two stacking ports as a stacking port, which allows you to use the other port as a data port." So I thought stack-trunk 1/2/1 to 1/2/2 means the stack consists of the 40G and breakout 40G port and stack-port 1/2/1 means use the 40G as the stacking port. Then on stack unit 2, if I want to use the breakout 40G, I would reference 2/2/2 using stack-port. Basically connect between stack unit 1, port XL1 (1/2/1) and stack unit 2, port XL2-5 (1/2/2).

I feel it would really help if you could clarify on how cables are connected between stack units 1 and 2 based on the configuration you gave.

Thanks!

You must be reading stacking information regarding a different model, as the ICX6610 has 4 stacking ports, not 2, and they definitely don't talk about using them for data - in fact it explicitly states they *can't* be used as data ports ever (thankfully this *feature* was snuck in in 8010 if I recall).

The top two ports on each switch are connected to each other for a redundant stacking connection in my setup. One 40gbE port, one breakout port. this leaves one free 40gbe port and one free breakout port per switch for me:

 

[fohdeesha]

So - Just to be sure:
An ICX6610 has four QSFP+ ports on the rear. Two of them are regular 40GBe ports, but the other two are special.

They can either be used as stacking ports or as QSFP+->4x10G SFP+ breakout ports, but not as regular 40GBe Ports, right?

check the third reply to this forum thread, it has a diagram, and explains it (at least what I thought) was pretty clearly. two are 40gbE only, two are 4x 10gbE breakout only. any of the ports can be used for regular ethernet ports, or stacking ports

 

[NablaSquaredG]

check the third reply to this forum thread, it has a diagram, and explains it (at least what I thought) was pretty clearly. two are 40gbE only, two are 4x 10gbE breakout only. any of the ports can be used for regular ethernet ports, or stacking ports

So yeah, basically what I said with the exception that the two regular 40GBe ports can also be used for stacking.

I'd like to use all four ports as uplink ports to other, non ICX6610 switches - I guess there is no way to build a reverse breakout cable?
ICX 6610 -> 40G QSFP+ to 4x10G Breakout -> 4x10G SFP+ to 40G QSFP+ Reverse Breakout -> QSFP+ port on another switch (not ICX6610)

I suppose replacing the Breakout - Reverse Breakout combination with a single QSFP+ cable wouldn't work either?

 

[fohdeesha]

So yeah, basically what I said with the exception that the two regular 40GBe ports can also be used for stacking.

I'd like to use all four ports as uplink ports to other, non ICX6610 switches - I guess there is no way to build a reverse breakout cable?
ICX 6610 -> 40G QSFP+ to 4x10G Breakout -> 4x10G SFP+ to 40G QSFP+ Reverse Breakout -> QSFP+ port on another switch (not ICX6610)

I suppose replacing the Breakout - Reverse Breakout combination with a single QSFP+ cable wouldn't work either?

You can absolutely use a single regular qsfp cable to connect the breakout port to another switch (assuming the other switches port is configured for breakout / 4x 10gbe). You can even connect a breakout port with a qsfp cable to a non breakout port on something, but only one lane will link up so it will run at 10gbps

Both 40gbe and 4x 10gbe breakout qsfp+ are electrically identical, it's 4x 10gbps lanes. Difference is in how the ASIC/PHY decides to split them (or mux them)

 

[NablaSquaredG]

You can absolutely use a single regular qsfp cable to connect the breakout port to another switch (assuming the other switches port is configured for breakout / 4x 10gbe).

Whaaaaaaat
Okay, that is cool.

Assuming I use an ICX6610 on one side and a SX6036 on the other side

ICX6610 4x 10G breakout port in breakout mode -> 40G QSFP+ cable -> SX6036 regular QSFP+ port in 4x10G breakout mode would link up with 4x 10G links?
It should then be easy to pack the 4x 10G links and another "regular" QSFP+ link into an Link Aggretation group and get "virtual" 80G bandwidth

 

[eduncan911]

As we found out recently, they were probably too busy paying sketchball felons...

??

 

[jjacobs]

Buffer overruns, license violations, and bad code: FreeBSD 13’s close call

40,000 lines of flawed code almost made it into FreeBSD's kernel—we examine how.

arstechnica.com

 

[darthray]

Thank you, @fohdeesha for the info and @klui and @NablaSquaredG for asking some of the questions I had in my head . Certainly not doubting any of your information, @fohdeesha, but there's enough I don't know I feel I need to ask more questions. If you couldn't tell, I'm a fiber newbie. Oh, and I've read the post about stacking a few times already. Probably need to read it a few more times.

I had a feeling using an MPO/MPO cable on the breakout ports would work, so it's nice to get confirmation of that. Now, in your example you created a truck with ports 1/2/1 and 1/2/2. What if I just want to use the breakout port alone? Can I just set 1/2/2 as my only connection to the other 6610? My MPO fiber + transceivers are only going to be here late next week, otherwise I'd be experimenting with this stuff.

Edit: just saw your other reply saying that:

two [ports] are 40gbE only, two are 4x 10gbE breakout only. any of the ports can be used for regular ethernet ports, or stacking ports

so, if I can stack the 6610s just using one of the breakout ports (e.g.: 1/2/2 on each one) I should be golden.

 

[klui]

You must be reading stacking information regarding a different model, as the ICX6610 has 4 stacking ports, not 2, and they definitely don't talk about using them for data - in fact it explicitly states they *can't* be used as data ports ever (thankfully this *feature* was snuck in in 8010 if I recall).

The top two ports on each switch are connected to each other for a redundant stacking connection in my setup. One 40gbE port, one breakout port. this leaves one free 40gbe port and one free breakout port per switch for me:

Ah, OK. You connect the ports like-for-like? 40G-to-40G and 40G breakout-to-40G breakout?

I just pasted the description from the 08.0.30p command line reference. The 08.0.30b stacking guide (w/in the 08030r manuals, which also includes the 08030p command reference) does have an ICX6610 section but does not elaborate on the stack-port command's uniqueness to the 6610 other than "The ICX 6610 does not allow users to change the stack-port command configuration."

 

[eduncan911]

Exactly how would someone load a "boot image" into an ICX6610, where there is no boot image?

Got a pair from eBay that stated there is no boot image in the desc, and sure enough nothing is displayed on the screen and the fans/PSU screams at what sounds like max RPMs forever.

Read the website again, and been searching the thread but not finding much.

Do I really need to use JTAG, and the BDI2000 to recover these?

Is there a Raspberry Pi SPI method I could instead with like flashrom?

 

[darthray]

Exactly how would someone load a "boot image" into an ICX6610, where there is no boot image?

Got a pair from eBay that stated there is no boot image in the desc, and sure enough nothing is displayed on the screen and the fans/PSU screams at what sounds like max RPMs forever.

Read the website again, and been searching the thread but not finding much.

We bought it from the same place . Just follow the guide (link in the OP). You'll need a console cable. No need for JTAG or anything else.

 

[mintchipmadness]

Hello Everyone,
I have the ICX 7250-48p switch and I am looking for a 4 post 1U rail kit that will work. The ruckus part number is XBR-R000295. Has anyone found a reasonably priced alternatives to that kit other than a shelf? The cost of the ruckus part is more than I paid for the switch. Thank you for your help.

 

[eduncan911]

Hello Everyone,
I have the ICX 7250-48p switch and I am looking for a 4 post 1U rail kit that will work. The ruckus part number is XBR-R000295. Has anyone found a reasonably priced alternatives to that kit other than a shelf? The cost of the ruckus part is more than I paid for the switch. Thank you for your help.

What about one of those 4-post shelves? Like this one:

https://www.amazon.com/gp/product/B0060RUVBA

 

[scot1297]

So I need a little bit of help pointing me in the right direction. I have went through most of the forums for my answer and I am 100% sure I read it, but didn't do it right. So the trouble I am having is other vlan's on the brocade 6610 switch can pull a ip from the dhcp server in pfsense, but I can not ping out or get internet.

My setup is internet to pfsense box to brocade 6610 switch. Its a fresh install of pfsense with vlan 20 vlan 100 vlan 101 setup. They were all provided dhcp servers with the ranges I needed. I also have a unifi AP that is plugged into the switch and get internet on it with the default vlan of 1. When ever i switch it to a different vlan I get internet issues. I am assuming I missed a setting so that I can reach the internet from these other vlans. I pasted below my brocade config. Any help would be great or if I am totally off base then tell me to...thanks

telnet@coreswitch(config)#show run
Current configuration:
!
ver 08.0.30tT7f3
!
stack unit 1
  module 1 icx6610-48p-poe-port-management-module
  module 2 icx6610-qsfp-10-port-160g-module
  module 3 icx6610-8-port-10g-dual-mode-module
!
global-stp
!
!
lag pvec1 dynamic id 1
 ports ethernet 1/3/1 to 1/3/2
 primary-port 1/3/1
 deploy
!
lag pvec2 dynamic id 2
 ports ethernet 1/3/3 to 1/3/4
 primary-port 1/3/3
 deploy
!
lag pvec3 dynamic id 3
 ports ethernet 1/3/5 to 1/3/6                                   
 primary-port 1/3/5
 deploy
!
lag pvec4 dynamic id 4
 ports ethernet 1/3/7 to 1/3/8
 primary-port 1/3/7
 deploy
!
lag synologynas dynamic id 5
 ports ethernet 1/1/3 to 1/1/6
 primary-port 1/1/3
 deploy
!
!
vlan 1 name DEFAULT-VLAN by port
 router-interface ve 1
 spanning-tree
 spanning-tree rstp
!
vlan 5 name corosync by port
 untagged ethe 1/1/25 to 1/1/28
!
vlan 10 name cephpublic by port                                   
 tagged ethe 1/3/1 to 1/3/8
!
vlan 15 name cephcluster by port
 tagged ethe 1/3/1 to 1/3/8
!
vlan 17 name pvemigration by port
 tagged ethe 1/3/1 to 1/3/8
!
vlan 20 name wifi by port
 tagged ethe 1/1/1 ethe 1/1/13 to 1/1/16 ethe 1/1/43 ethe 1/1/47 to 1/1/48
 spanning-tree
 spanning-tree rstp
!
vlan 100 name proxmox100 by port
 tagged ethe 1/1/1 ethe 1/1/13 to 1/1/16 ethe 1/1/43 ethe 1/1/47 to 1/1/48
 spanning-tree
 spanning-tree rstp
!
vlan 101 name proxmox101 by port
 tagged ethe 1/1/1 ethe 1/1/13 to 1/1/16 ethe 1/1/43 ethe 1/1/47 to 1/1/48
 router-interface ve 101
 spanning-tree
 spanning-tree rstp                                               
!
!
!
!
!
aaa authentication web-server default local
aaa authentication enable default local
aaa authentication login default local
jumbo
hostname coreswitch
ip dhcp-client disable
ip dns server-address 1.1.1.1
ip route 0.0.0.0/0 192.168.1.1
!
username ... password .....
snmp-server community ..... ro
!
!
clock summer-time
clock timezone us Central
!
!
ntp                                                               
 disable serve
 server 216.239.35.0
 server 216.239.35.4
!
!
!
!
!
!
!
!
!
interface ethernet 1/1/1
 dual-mode
!
interface ethernet 1/1/13
 dual-mode
!
interface ethernet 1/1/14
 dual-mode
!
interface ethernet 1/1/15
 dual-mode                                                       
!
interface ethernet 1/1/16
 dual-mode
!
interface ethernet 1/1/43
 dual-mode
 inline power
!
interface ethernet 1/1/44
 inline power
!
interface ethernet 1/1/45
 inline power
!
interface ethernet 1/1/46
 inline power
!
interface ethernet 1/1/47
 dual-mode
 inline power
!
interface ethernet 1/1/48
 dual-mode                                                       
 inline power
!
interface ethernet 1/3/1
 speed-duplex 10G-full
!
interface ethernet 1/3/3
 speed-duplex 10G-full
!
interface ethernet 1/3/5
 speed-duplex 10G-full
!
interface ethernet 1/3/7
 speed-duplex 10G-full
!
interface ve 1
 ip address 192.168.1.2 255.255.255.0
!
interface ve 101
 ip address 192.168.101.254 255.255.255.0
!
!
!
!                                                                 
!
!
!
!
!
end

 

[LodeRunner]

I don’t see IPs or virtual router definitions for anything other than VLAN 1 and 100. Also, do you have a return route from pfSense? Do you have to proper rules in both directions on the LANS on the pfSense side?

 

[scot1297]

I don’t have a return route in pfsense. That I am struggling with, can you point me in the direction of setting up a return route? Thanks!

 

[LodeRunner]

Last time I tried this, I had a dedicated routing VLAN. So each VLAN had an IP, switch had a default route over a separate VLAN (/30). The switch had the pfSense IP as the destination, in the pfSense routing page, I had to manually create a route from pfSense with the switch IP as the gateway.

I haven't actually done VLANs on pfSense directly and was using ACLs between the VLANs instead of firewall rules. But since the switch is the gateway for all networks and pfSense is the upstream gateway, then you need a reverse route in pfSense pointing back to the switch IP, or maybe the VLAN IPs? That's what I'm not clear on, but I haven't sat down to try and make VLANs on pfSense work.

 

[mintchipmadness]

What about one of those 4-post shelves? Like this one:

https://www.amazon.com/gp/product/B0060RUVBA

Thank you for your help. I was thinking of that exact one but I believe it won't work for my switch since the main airflow is side to back. The OEM kit comes in two pieces, one for the front of the switch and the other at the back which leaves the side intake open for airflow.

 

[DavidRa]

I can't find a way to search the thread - are there models other than the ones in the first few posts worth looking at? I may have to replace my Dell 8024F pair (each 20 x SFP+ and 4x combo SFP+/10T, and stacked).

I'd need a combination of SFP+ and 10GBaseT ports - ideally ~16-20 SFP+ and 4-8 10G-T each would be sufficient. No need for 40Gbps (my two are currently stacked with 4x 10Gb in LACP). Is there a ... IDK a 7300 or 7500 or something?

 

[fohdeesha]

Thank you, @fohdeesha for the info and @klui and @NablaSquaredG for asking some of the questions I had in my head . Certainly not doubting any of your information, @fohdeesha, but there's enough I don't know I feel I need to ask more questions. If you couldn't tell, I'm a fiber newbie. Oh, and I've read the post about stacking a few times already. Probably need to read it a few more times.

I had a feeling using an MPO/MPO cable on the breakout ports would work, so it's nice to get confirmation of that. Now, in your example you created a truck with ports 1/2/1 and 1/2/2. What if I just want to use the breakout port alone? Can I just set 1/2/2 as my only connection to the other 6610? My MPO fiber + transceivers are only going to be here late next week, otherwise I'd be experimenting with this stuff.

Edit: just saw your other reply saying that:


so, if I can stack the 6610s just using one of the breakout ports (e.g.: 1/2/2 on each one) I should be golden.

FYI I vaguely remember the stack config needing to use at least one 40gbE port, (eg it would complain about the user only specifying a "secondary stack port" when only entering a breakout port). I would search the thread, I can't remember conclusively