Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

[i386]

I used the routing firmware because it has more features. But I'm not using them

 

[LodeRunner]

What is recommended for firmware L2(sps) or L3(spr)? I will be running some vlans which pfsense is handling the routing on at the moment so should I just stick to L2?

Just install SPR. That way if you ever want to experiment, it doesn't require a firmware update and reload.

 

[Propaganda]

Is the sps easier to setup or perform better?

 

[Oodaloop]

Looking for a little help creating a console cable for the brocade switches. I've searched through the thread and saw other examples but for whatever reason my cable does not appear to work.

I hacked up a micro usb cable and attached its wiring to a RJ45 keystone. Green to pin 6, White to pin 3. Red & Black to nothing. And connected that to a cisco RJ45 to USB console cable.

Am I doing something wrong with the wiring?

 

[klui]

Swap your green and white cables. Next time use test clips to confirm before you punch down.

 

[LodeRunner]

Is the sps easier to setup or perform better?

Not in any meaningful way that I'm aware of. Fohdeesha's guides assume SPR since it has you assign an IP to a VE. I haven't had to reference it in a while, so I don't recall if he has the L2 only steps in there as well. Probably does, it's pretty thorough.

 

[Propaganda]

Looking for a little help creating a console cable for the brocade switches. I've searched through the thread and saw other examples but for whatever reason my cable does not appear to work.

I hacked up a micro usb cable and attached its wiring to a RJ45 keystone. Green to pin 6, White to pin 3. Red & Black to nothing. And connected that to a cisco RJ45 to USB console cable.

Am I doing something wrong with the wiring?

Not sure if the other switches are different but the 7250 has the following pinout for the miniusb port.
So it looks like you are missing a ground wire?

 

[Oodaloop]

Swap your green and white cables. Next time use test clips to confirm before you punch down.

That worked! Thank you!

 

[fohdeesha]

Is the sps easier to setup or perform better?

no, just follow the guide

 

[fohdeesha]

90000 git commits this week, new docs build with cool new secrets should be pushed out soon(tm)

 

[Propaganda]

That worked! Thank you!

I still think you are missing a ground which could cause you issues. It probably works because you have an electrical ground through your home that commons up the 2 devices but I wouldn't count on it and could even cause damage if you get too much of a differential between the devices.

 

[ben255]

Picked up a 7150-24p and got it updated and licensed using your guide! Thanks!

I also managed to get Brocade Network Advisor installed on linux (centos 7). Turns out that all the config files/scripts were written on a Windows system with DOS line endings, which linux does NOT like. Luckily there is a handy command to convert them.

Run install.bin, but when you reach the end, choose not to run the Configuration Wizard. Once that is done, install dos2unix from the standard repo (yum install dos2unix). Next, cd to /opt/Network_Advisor_IP14_2_12 and run "find . -type f -print0 | xargs -0 dos2unix" which will search through the entire directory and convert all text files to the correct Unix format.

Once that completes, run "bin/configwizard" and follow the instructions to configure BNA!

If you want to run it as a server, you'll need to modify the firewall to allow access.

 

[MassiveMeatMissile]

When I got home today my entire network was down and my ICX 6610 had the fans at 100%, so I unplugged and replugged it into power; the fans still stuck at full bore, and the activity lights on the SFP ports are solid green. I cannot see it on the network anymore and I tried using a console cable but it's no longer working either.

Anyone have any troubleshooting suggestions? I feel like it may just be a noisy brick now.

 

[LodeRunner]

When I got home today my entire network was down and my ICX 6610 had the fans at 100%, so I unplugged and replugged it into power; the fans still stuck at full bore, and the activity lights on the SFP ports are solid green. I cannot see it on the network anymore and I tried using a console cable but it's no longer working either.

Anyone have any troubleshooting suggestions? I feel like it may just be a noisy brick now.

If you power cycle it with console cable active, do you even see the boot loader?

 

[MassiveMeatMissile]

If you power cycle it with console cable active, do you even see the boot loader?

No, it doesn't show anything at all.

 

[deeceesth]

I tried searching this thread for ideas on expanding sfp+ ports on these switches. Would a cheap L2 only SFP+ switch uplinked to a icx7250 do the trick or is there an affordable brocade option for getting more sfp+ ports?

 

[neodied]

I am having issues with the 40G breakout ports, where only the 4th port in each group (1/2/5 and 1/2/10) will link when using a QSFP-to-4xSFP+ breakout DAC (specific cable is a FS Brocade 40G-QSFP-4SFP-C-0201).

My Setup:
Switch - ICX 6610-48P with licenses, breakout QSFP plugged into XL2-5
Server 1 - Intel X520 PCIe NIC - ports 1 and 2 connected to breakout SFP+ transceivers 1 and 2 (1/2/2 and 1/2/3)
Server 2 - Intel 82599ES daughter card - ports 1 and 2 connected to breakout SFP+ transceivers 3 and 4 (1/2/4 and 1/2/5)

The successful link follows the switch port (changing the order of transceivers in a NIC, and trying different NICs results in 1/2/5 always linking). Plugging only one SFP+ breakout at a time into a NIC does not change the behavior, only the 1/2/5 or 1/2/10 ports will link up. Using a new cable (of the same model) does not resolve this, using XL7-10 (the second QSFP breakout port) also does not resolve (only 1/2/10 links up).

Any help figuring this out would be appreciated!

show interface brief ethernet 1/2/1 to 1/2/5

Port       Link    State   Dupl Speed Trunk Tag Pvid Pri MAC         Name
1/2/1      Up      Forward Full 40G   13    Yes N/A  0   redacted    redacted
1/2/2      Down    None    None None  None  No  1    0   redacted    redacted
1/2/3      Down    None    None None  None  No  1    0   redacted    redacted
1/2/4      Down    None    None None  None  No  1    0   redacted    redacted
1/2/5      Up      Forward Full 10G   None  No  1    0   redacted

show stack

***** Warning! stack is not enabled. *****

T=22d20h53m31.8: alone: standalone, D: dynamic cfg, S: static
ID   Type          Role    Mac Address    Pri State   Comment
1  S ICX6610-48P   alone   redacted         0 local   None:0


     +---+
  2/1| 1 |2/6
     +---+
Current stack management MAC is redacted

show run

stack unit 1
  module 1 icx6610-48p-poe-port-management-module
  module 2 icx6610-qsfp-10-port-160g-module
  module 3 icx6610-8-port-10g-dual-mode-module

ETA: I did a fair bit of googling and keyword searching this thread and found the combined stacking + breakout config instructions, but my setup is not stacking, just 100% data. The XL1 and XL6 ports are connected directly to a 40G NIC in another box.

 

[Jason Antes]

Picked up a 7150-24p and got it updated and licensed using your guide! Thanks!

I also managed to get Brocade Network Advisor installed on linux (centos 7). Turns out that all the config files/scripts were written on a Windows system with DOS line endings, which linux does NOT like. Luckily there is a handy command to convert them.

Run install.bin, but when you reach the end, choose not to run the Configuration Wizard. Once that is done, install dos2unix from the standard repo (yum install dos2unix). Next, cd to /opt/Network_Advisor_IP14_2_12 and run "find . -type f -print0 | xargs -0 dos2unix" which will search through the entire directory and convert all text files to the correct Unix format.

Once that completes, run "bin/configwizard" and follow the instructions to configure BNA!

If you want to run it as a server, you'll need to modify the firewall to allow access.

Should make that a VM template and get it uploaded.

 

[Jason Antes]

I tried searching this thread for ideas on expanding sfp+ ports on these switches. Would a cheap L2 only SFP+ switch uplinked to a icx7250 do the trick or is there an affordable brocade option for getting more sfp+ ports?

You could by any of several of the ICX line that are fairly cheap to do that.

 

[anemoiac]

For my 7250-24P, I started with replacing the rear fans with 2x MF40201V3-1000U-G99. I found that the fans were near silent when running at speed 1, but it would intermittently kick up into speed 2 when the ASIC hit a high enough temp. This sits in a rack right next to me, so I added 1x MF60101V3-1000U-A99 that sits directly on top of the ASIC heatsink and is tapped into the constant 12v to the mainboard, and it never kicks into speed 2 now, even sitting in my rack that runs ~80F inside.

Would you mind sharing how you secured the 3rd fan on top of the ASIC heatsink? I'm looking to do something similar, but am unsure of the best way to more forward.