Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

[NablaSquaredG]

You can absolutely use a single regular qsfp cable to connect the breakout port to another switch (assuming the other switches port is configured for breakout / 4x 10gbe).

Whaaaaaaat
Okay, that is cool.

Assuming I use an ICX6610 on one side and a SX6036 on the other side

ICX6610 4x 10G breakout port in breakout mode -> 40G QSFP+ cable -> SX6036 regular QSFP+ port in 4x10G breakout mode would link up with 4x 10G links?
It should then be easy to pack the 4x 10G links and another "regular" QSFP+ link into an Link Aggretation group and get "virtual" 80G bandwidth

 

[eduncan911]

As we found out recently, they were probably too busy paying sketchball felons...

??

 

[jjacobs]

Buffer overruns, license violations, and bad code: FreeBSD 13’s close call

40,000 lines of flawed code almost made it into FreeBSD's kernel—we examine how.

arstechnica.com

 

[darthray]

Thank you, @fohdeesha for the info and @klui and @NablaSquaredG for asking some of the questions I had in my head . Certainly not doubting any of your information, @fohdeesha, but there's enough I don't know I feel I need to ask more questions. If you couldn't tell, I'm a fiber newbie. Oh, and I've read the post about stacking a few times already. Probably need to read it a few more times.

I had a feeling using an MPO/MPO cable on the breakout ports would work, so it's nice to get confirmation of that. Now, in your example you created a truck with ports 1/2/1 and 1/2/2. What if I just want to use the breakout port alone? Can I just set 1/2/2 as my only connection to the other 6610? My MPO fiber + transceivers are only going to be here late next week, otherwise I'd be experimenting with this stuff.

Edit: just saw your other reply saying that:

two [ports] are 40gbE only, two are 4x 10gbE breakout only. any of the ports can be used for regular ethernet ports, or stacking ports

so, if I can stack the 6610s just using one of the breakout ports (e.g.: 1/2/2 on each one) I should be golden.

 

[klui]

You must be reading stacking information regarding a different model, as the ICX6610 has 4 stacking ports, not 2, and they definitely don't talk about using them for data - in fact it explicitly states they *can't* be used as data ports ever (thankfully this *feature* was snuck in in 8010 if I recall).

The top two ports on each switch are connected to each other for a redundant stacking connection in my setup. One 40gbE port, one breakout port. this leaves one free 40gbe port and one free breakout port per switch for me:

Ah, OK. You connect the ports like-for-like? 40G-to-40G and 40G breakout-to-40G breakout?

I just pasted the description from the 08.0.30p command line reference. The 08.0.30b stacking guide (w/in the 08030r manuals, which also includes the 08030p command reference) does have an ICX6610 section but does not elaborate on the stack-port command's uniqueness to the 6610 other than "The ICX 6610 does not allow users to change the stack-port command configuration."

 

[eduncan911]

Exactly how would someone load a "boot image" into an ICX6610, where there is no boot image?

Got a pair from eBay that stated there is no boot image in the desc, and sure enough nothing is displayed on the screen and the fans/PSU screams at what sounds like max RPMs forever.

Read the website again, and been searching the thread but not finding much.

Do I really need to use JTAG, and the BDI2000 to recover these?

Is there a Raspberry Pi SPI method I could instead with like flashrom?

 

[darthray]

Exactly how would someone load a "boot image" into an ICX6610, where there is no boot image?

Got a pair from eBay that stated there is no boot image in the desc, and sure enough nothing is displayed on the screen and the fans/PSU screams at what sounds like max RPMs forever.

Read the website again, and been searching the thread but not finding much.

We bought it from the same place . Just follow the guide (link in the OP). You'll need a console cable. No need for JTAG or anything else.

 

[mintchipmadness]

Hello Everyone,
I have the ICX 7250-48p switch and I am looking for a 4 post 1U rail kit that will work. The ruckus part number is XBR-R000295. Has anyone found a reasonably priced alternatives to that kit other than a shelf? The cost of the ruckus part is more than I paid for the switch. Thank you for your help.

 

[eduncan911]

Hello Everyone,
I have the ICX 7250-48p switch and I am looking for a 4 post 1U rail kit that will work. The ruckus part number is XBR-R000295. Has anyone found a reasonably priced alternatives to that kit other than a shelf? The cost of the ruckus part is more than I paid for the switch. Thank you for your help.

What about one of those 4-post shelves? Like this one:

https://www.amazon.com/gp/product/B0060RUVBA

 

[scot1297]

So I need a little bit of help pointing me in the right direction. I have went through most of the forums for my answer and I am 100% sure I read it, but didn't do it right. So the trouble I am having is other vlan's on the brocade 6610 switch can pull a ip from the dhcp server in pfsense, but I can not ping out or get internet.

My setup is internet to pfsense box to brocade 6610 switch. Its a fresh install of pfsense with vlan 20 vlan 100 vlan 101 setup. They were all provided dhcp servers with the ranges I needed. I also have a unifi AP that is plugged into the switch and get internet on it with the default vlan of 1. When ever i switch it to a different vlan I get internet issues. I am assuming I missed a setting so that I can reach the internet from these other vlans. I pasted below my brocade config. Any help would be great or if I am totally off base then tell me to...thanks

telnet@coreswitch(config)#show run
Current configuration:
!
ver 08.0.30tT7f3
!
stack unit 1
  module 1 icx6610-48p-poe-port-management-module
  module 2 icx6610-qsfp-10-port-160g-module
  module 3 icx6610-8-port-10g-dual-mode-module
!
global-stp
!
!
lag pvec1 dynamic id 1
 ports ethernet 1/3/1 to 1/3/2
 primary-port 1/3/1
 deploy
!
lag pvec2 dynamic id 2
 ports ethernet 1/3/3 to 1/3/4
 primary-port 1/3/3
 deploy
!
lag pvec3 dynamic id 3
 ports ethernet 1/3/5 to 1/3/6                                   
 primary-port 1/3/5
 deploy
!
lag pvec4 dynamic id 4
 ports ethernet 1/3/7 to 1/3/8
 primary-port 1/3/7
 deploy
!
lag synologynas dynamic id 5
 ports ethernet 1/1/3 to 1/1/6
 primary-port 1/1/3
 deploy
!
!
vlan 1 name DEFAULT-VLAN by port
 router-interface ve 1
 spanning-tree
 spanning-tree rstp
!
vlan 5 name corosync by port
 untagged ethe 1/1/25 to 1/1/28
!
vlan 10 name cephpublic by port                                   
 tagged ethe 1/3/1 to 1/3/8
!
vlan 15 name cephcluster by port
 tagged ethe 1/3/1 to 1/3/8
!
vlan 17 name pvemigration by port
 tagged ethe 1/3/1 to 1/3/8
!
vlan 20 name wifi by port
 tagged ethe 1/1/1 ethe 1/1/13 to 1/1/16 ethe 1/1/43 ethe 1/1/47 to 1/1/48
 spanning-tree
 spanning-tree rstp
!
vlan 100 name proxmox100 by port
 tagged ethe 1/1/1 ethe 1/1/13 to 1/1/16 ethe 1/1/43 ethe 1/1/47 to 1/1/48
 spanning-tree
 spanning-tree rstp
!
vlan 101 name proxmox101 by port
 tagged ethe 1/1/1 ethe 1/1/13 to 1/1/16 ethe 1/1/43 ethe 1/1/47 to 1/1/48
 router-interface ve 101
 spanning-tree
 spanning-tree rstp                                               
!
!
!
!
!
aaa authentication web-server default local
aaa authentication enable default local
aaa authentication login default local
jumbo
hostname coreswitch
ip dhcp-client disable
ip dns server-address 1.1.1.1
ip route 0.0.0.0/0 192.168.1.1
!
username ... password .....
snmp-server community ..... ro
!
!
clock summer-time
clock timezone us Central
!
!
ntp                                                               
 disable serve
 server 216.239.35.0
 server 216.239.35.4
!
!
!
!
!
!
!
!
!
interface ethernet 1/1/1
 dual-mode
!
interface ethernet 1/1/13
 dual-mode
!
interface ethernet 1/1/14
 dual-mode
!
interface ethernet 1/1/15
 dual-mode                                                       
!
interface ethernet 1/1/16
 dual-mode
!
interface ethernet 1/1/43
 dual-mode
 inline power
!
interface ethernet 1/1/44
 inline power
!
interface ethernet 1/1/45
 inline power
!
interface ethernet 1/1/46
 inline power
!
interface ethernet 1/1/47
 dual-mode
 inline power
!
interface ethernet 1/1/48
 dual-mode                                                       
 inline power
!
interface ethernet 1/3/1
 speed-duplex 10G-full
!
interface ethernet 1/3/3
 speed-duplex 10G-full
!
interface ethernet 1/3/5
 speed-duplex 10G-full
!
interface ethernet 1/3/7
 speed-duplex 10G-full
!
interface ve 1
 ip address 192.168.1.2 255.255.255.0
!
interface ve 101
 ip address 192.168.101.254 255.255.255.0
!
!
!
!                                                                 
!
!
!
!
!
end

 

[LodeRunner]

I don’t see IPs or virtual router definitions for anything other than VLAN 1 and 100. Also, do you have a return route from pfSense? Do you have to proper rules in both directions on the LANS on the pfSense side?

 

[scot1297]

I don’t have a return route in pfsense. That I am struggling with, can you point me in the direction of setting up a return route? Thanks!

 

[LodeRunner]

Last time I tried this, I had a dedicated routing VLAN. So each VLAN had an IP, switch had a default route over a separate VLAN (/30). The switch had the pfSense IP as the destination, in the pfSense routing page, I had to manually create a route from pfSense with the switch IP as the gateway.

I haven't actually done VLANs on pfSense directly and was using ACLs between the VLANs instead of firewall rules. But since the switch is the gateway for all networks and pfSense is the upstream gateway, then you need a reverse route in pfSense pointing back to the switch IP, or maybe the VLAN IPs? That's what I'm not clear on, but I haven't sat down to try and make VLANs on pfSense work.

 

[mintchipmadness]

What about one of those 4-post shelves? Like this one:

https://www.amazon.com/gp/product/B0060RUVBA

Thank you for your help. I was thinking of that exact one but I believe it won't work for my switch since the main airflow is side to back. The OEM kit comes in two pieces, one for the front of the switch and the other at the back which leaves the side intake open for airflow.

 

[DavidRa]

I can't find a way to search the thread - are there models other than the ones in the first few posts worth looking at? I may have to replace my Dell 8024F pair (each 20 x SFP+ and 4x combo SFP+/10T, and stacked).

I'd need a combination of SFP+ and 10GBaseT ports - ideally ~16-20 SFP+ and 4-8 10G-T each would be sufficient. No need for 40Gbps (my two are currently stacked with 4x 10Gb in LACP). Is there a ... IDK a 7300 or 7500 or something?

 

[fohdeesha]

Thank you, @fohdeesha for the info and @klui and @NablaSquaredG for asking some of the questions I had in my head . Certainly not doubting any of your information, @fohdeesha, but there's enough I don't know I feel I need to ask more questions. If you couldn't tell, I'm a fiber newbie. Oh, and I've read the post about stacking a few times already. Probably need to read it a few more times.

I had a feeling using an MPO/MPO cable on the breakout ports would work, so it's nice to get confirmation of that. Now, in your example you created a truck with ports 1/2/1 and 1/2/2. What if I just want to use the breakout port alone? Can I just set 1/2/2 as my only connection to the other 6610? My MPO fiber + transceivers are only going to be here late next week, otherwise I'd be experimenting with this stuff.

Edit: just saw your other reply saying that:


so, if I can stack the 6610s just using one of the breakout ports (e.g.: 1/2/2 on each one) I should be golden.

FYI I vaguely remember the stack config needing to use at least one 40gbE port, (eg it would complain about the user only specifying a "secondary stack port" when only entering a breakout port). I would search the thread, I can't remember conclusively

 

[andreathing]

Has anyone done any progress on this error?

brocade02(config)#M:9 L:0 - chow_qsfp_read, qsfp 0, error in reading qsfp
M:9 L:0 - link_40G_4x10G_get_media: qsfp 0, port 1/2/1 error in reading qsfp

I thought i had faulty 40g ports, but it turns out the link work perfectly using dac cables. I've tried 4 different xbr-00232 modules so far on both ports of two different switches and i get the same error every time. It seems highly unlikely to me that all 4 modules are broken and since the link work perfectly with the dac cable (even connecting to a mellanox sx6036) i doubt the port are fried either. Btw config is vanilla as per @fohdeesha guide, just installed latest firmware and removed stacking.

 

[klui]

Has anyone done any progress on this error?

brocade02(config)#M:9 L:0 - chow_qsfp_read, qsfp 0, error in reading qsfp
M:9 L:0 - link_40G_4x10G_get_media: qsfp 0, port 1/2/1 error in reading qsfp

Although I get something similar during insertion, show media ethernet .... shows the vendor and serial and the port links up. Those are the important behaviors. I suppose it's due to the type of transceiver.

ICX6610-48P Router>sh media eth 1/2/1
Port 1/2/1:Type : 40G QSFP Module
Vendor Name: BROCADE Serial Num: HTA11402000010N Revision: A

 

[andreathing]

Although I get something similar during insertion, show media ethernet .... shows the vendor and serial and the port links up. Those are the important behaviors. I suppose it's due to the type of transceiver.

ICX6610-48P Router>sh media eth 1/2/1
Port 1/2/1:Type : 40G QSFP Module
Vendor Name: BROCADE Serial Num: HTA11402000010N Revision: A

Ok, i get the same result. I must confess i panicked at the error and stopped there, afraid to have just bought 4 not-so-expensive paperweights... I have the same result as you:

SSH@brocade02#show media eth 1/2/6
Port   1/2/6:Type  : 40G QSFP Module
Vendor Name: BROCADE          Serial Num: HTA214320000041 Revision: A

Now i have to understand why the link doesn't come up... I thought the cable was good (b polaryty MTP).

Edit: just a bad cable, the error has been a red herring. As soon as i changed it i got the link up.

Thank you again @klui

 

[klui]

Now i have to understand why the link doesn't come up... I thought the cable was good (b polaryty MTP).

Edit: just a bad cable, the error has been a red herring. As soon as i changed it i got the link up.

Bad cable or bad part #? I purchased the exact model @datanet recommended at https://forums.servethehome.com/index.php?threads/mto-questions.31800/post-298142.