Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

[DouglasteR]

two of them yes as my docu explains. the other two are 43x 10gb breakout. also you do not want to use the built in DHCP server on these things

Thanks !

 

[who]

Hi everyone!
I believe this is my first time posting on this thread...
Been following this thread for maybe 2 years or so.

First of all I'll like to thank fohdeesha for sharing his knowledge and expertise!

Initially I've gotten myself a ICX6430-12P, running since mid-2022 and recently ICX6450-12P (likely replace or keep as spare) as well as a 7250-24P.

I've encounter a small license issue on the 6450 last night as i was upgrading it.

ICX6450-C12PD Router#show license
Index    Lic Mode        Lic Name               Lid/Serial No  Lic Type    Status     Lic Period    Lic Capacity
Stack unit 1:
1        Node Lock       ICX6450-PREM-LIC-SW    H4C<snip>PLN8    Normal      Active     Unlimited         1
2        Node Lock       ICX6450-10G-LIC-POD    H4C<snip>PLN8    Normal      Invalid    Unlimited         2

The 10G license status is invalid.
Has anyone experienced this? I'm guessing a lot of you here are not using the 6450-12P so it might have slipped through the cracks.

I've removed and added it back, still shows the same invalid status.
SHA256 hash (bc1160442a077d0be720be4a20464f8d9ce837543e2ee94f262f9a38dc8c1bd2) of the license (2-6450-ports.xml) is the same from the latest brocade-08-03-2024.zip and brocade-09-16-2022.zip

ICX6450-C12PD Router#sh version
  Copyright (c) 1996-2016 Brocade Communications Systems, Inc. All rights reserved.
    UNIT 1: compiled on Apr 23 2020 at 10:57:06 labeled as ICX64R08030u
                (9871112 bytes) from Primary ICX64R08030u.bin
        SW: Version 08.0.30uT313
  Boot-Monitor Image size = 786944, Version:10.1.05T310 (kxz10105)
  HW: Stackable ICX6450-C12-PD
==========================================================================
UNIT 1: SL 1: ICX6450C 12-port-PD Management Module
         Serial  #: 2ax5o2jk68e
         License: ICX6450_PREM_ROUTER_SOFT_PACKAGE   (LID: H4CKTH3PLN8)
         P-ENGINE  0: type DEF0, rev 01
==========================================================================
UNIT 1: SL 2: ICX6450C-Copper 2port 2G Module
==========================================================================
UNIT 1: SL 3: ICX6450C-Fiber 2port 2G Module
==========================================================================
  800 MHz ARM processor ARMv5TE, 400 MHz bus
65536 KB flash memory
  512 MB DRAM
STACKID 1  system uptime is 42 minute(s) 23 second(s)
The system started at 00:00:17 GMT+00 Thu Jan 01 1970

The system : started=cold start

Any inputs would be very much appreciated. I don't think I'll be using 10G soon but would like to keep my options open.

Does the 6450-C12-PD supports 10G in the first place?
If not, that might explain the invalid status.


UPDATE: Ops! My bad! Looks like there's no 10G ports on the 6450C.

 

[molnart]

I just acquired an ICX7250-48p. I've been trying to get connected to the console but I've run into an issue. I've assembled a couple of the serial cables for these and using the settings that it should be (9600 8N1), all I get is gibberish. The way the output is produced seems like it is a boot up sequence and if I let it go, it will echo back what I type like it has gotten to the shell but the output is unreadable. Any ideas on how I can get it working?

have you been able to resolve this? i have tried multiple baud rates or cables, but still just getting garbage on the console (or no output even with one particular cable). i am not even able to type into the console, i get complete different characters than i am sending

also i have an USB FTDI board that I have tried with a spliced miniUSB cable. would it be safe to try the board on 5V voltage?

 

[enanothebrave]

I picked up an ICX7250 from ebay and wanted to check if anyone has come across this. I reached out to the seller as I'm assuming its borked. Booting into serial console presents the following. Keyboard input is not accepted so I cannot press b to interrupt the boot. I've tried two console cables, one made from a sacrificial mini usb, punched down to rj45 and connected to a cisco cable, and then one mini usb to db9 connected to a db9 to usb serial cable. Both cables present the same console log without the ability to type. The switch was advertised as coming from a working environment. Been searching the forums for some time looking for a similar situation, most folks seem to have one of the boot partitions working and I don't see much about not being able to type into the console.


EDIT: Ebay seller issued a refund and said don't ship it back. I may try and futz with the serial port and connect wires directly onto the motherboard just incase there is an issue there. Will let this post sit for a bit to see if anyone has any other guidance first though.


Brocade Bootloader: 10.1.05T215 (Mar 19 2015 - 16:39:20)

Validate Shmoo parameters stored in flash ..... OK
Restoring Shmoo parameters from flash .....
Running simple memory test ..... OK

ICX7250-48 Copper (POE), PVT1
SYS CPLD VER: 0x6, Released Ver: 0

Enter 'b' to stop at boot monitor: 0
bootdelay: ===
Booting image from Primary

NAND read: device 0 offset 0x0, size 0x2000000
................................ 33554432 bytes read: OK
Wrong Image Format for bootm command
ERROR: can't get kernel image!
could not boot from primary, no valid image; trying to boot from secondary
BOOTING image from Secondary

NAND read: device 0 offset 0x4000000, size 0x2000000
....................................................Skipping bad block 0x05a00000
Skipping bad block 0x05b00000
............ 33554432 bytes read: OK
Wrong Image Format for bootm command
ERROR: can't get kernel image!
ICX7250-Boot>

 

[enanothebrave]

have you been able to resolve this? i have tried multiple baud rates or cables, but still just getting garbage on the console (or no output even with one particular cable). i am not even able to type into the console, i get complete different characters than i am sending

If you are getting garbage/gibberish on a serial connection that usually has to do with an incorrect baud rate, should be 9600 for these. You should at least get clear text, I'm having the same issue with not being able to type though.

 

[manki09]

Hey Guys.
I've been running a Brocade ICX6610 for a while now with no issue. I recently upgraded my internet from 500mbps/20mbps to 2gb/300mbps. I have attached my current network diagram. What I changed since the upgrade is the modem from a TP-Link to Xfinity's XB8 and I changed from a HP T730 to a Lenovo m90q gen2.

I'm having a bit of an issue with traffic flowing through the ICX6610.

On 1GB enabled devices -
When I have both interfaces (LAN/WAN) on OPNSense to 2.5gbe or 10gbe my internet download rate is only 500-600mbps. If I lower either the LAN or WAN interface down to 1gbe I get about 940mbps just fine. Switch it back to 2.5gbe or 10gbe and the speed will drop back down to 500-600mbps.

On 40GB enabled devices (R730 to Windows 10 VM). It's a vary similar issue to 1GB devices, however it's about 1100mbps, which is better but not the 2gbe. If I switch the same VM to a 1GB port I again get the same 500-600gbps. I have also switched the VM over to the same vlan as the router interface to eliminate any intervlan issues, and the speeds are still the same.

If I hook a PC w/ 2.5gbe directly to my router I can get a consistent 2.1gbe download rate. So I know its not OPNSense directly causing the speed drops.

What I have noticed is dropped packets on interfaces when I do speed test.

Egress queues:
Queue counters    Queued packets    Dropped Packets
    0                 512                   0
    1                   0                   0
    2                   0                   0
    3                   0                   0
    4                   0                   0
    5                 281                   0
    6                   0                   0
    7             1941191                 919

The OPNSense LAN side is currently connected to the switch with a 10GB RJ45 SFP+, I have also testing with a cheap 2.5gb/10gb switch between the router and brocade with 2.5 to the router and twinax to the brocade

Anyone have any ideas what could cause this and a fix?


Edit:
I forgot to mention. I loaded OPNSense onto completely different hardware and got the same result.
I also originally used OPNSense in ESXi but moved it to bare metal to troubleshoot.

 

[Haulien]

Thanks for the guides! They've been really helpful on setting up.

I have one question however, I did the licensing for a ICX7250, but noticed it said it's a trial for 45 days. Is this something I should be worried about?

 

[jode]

I'm having a bit of an issue with traffic flowing through the ICX6610.

What makes you think this is an issue with the ICX6610?

If I lower either the LAN or WAN interface down to 1gbe I get about 940mbps just fine.

A change in OpnSense changes the situation significantly. This points to an issue in OpnSense.

Anyone have any ideas what could cause this and a fix?

Is OpnSense managing the layer 3 routing in your network?

I assume OpnSense hw does not have enough single core performance to keep up with your network traffic and therefore becoming the bottleneck.

There are multiple threads explaining how to enable layer 3 routing in ICX6610. You can even find examples buried deep in this thread.

 

[Marc_]

Need a little help with a pair of ICX6610-48 I have coming. I want to stack them using the rear QSFP ports. My question is, will any transceiver work or are they brocade specific? I've got some old meraki stuff laying around (MA-CBL-40G50CM) and would like to recycle them if possible?

EDIT: Can confirm these work

 

[autoturk]

Anybody have any first-hand experience with the icx7450-24p? I'm seeing a lot of them on ebay and having 40gb + 10gb copper + 10gb SFP+ seems like a home run feature wise, but don't know much about noise + power consumption

 

[manki09]

What makes you think this is an issue with the ICX6610?



A change in OpnSense changes the situation significantly. This points to an issue in OpnSense.



Is OpnSense managing the layer 3 routing in your network?

I assume OpnSense hw does not have enough single core performance to keep up with your network traffic and therefore becoming the bottleneck.

There are multiple threads explaining how to enable layer 3 routing in ICX6610. You can even find examples buried deep in this thread.

Thank you for replying Jode.

The ICX6610 is doing all of the routing in my network. My OPNSense (172.16.0.1/29) is unaware of any VLANs and simply has static routes for my 2 Vlans directed to the Brocade (172.16.0.2). The brocade then has a default gateway to OPNSense.

The reason why I think it's the brocade is when I connect a 2.5 or greater device direct to OPNSense using a spare IP on my 172.16.0.0/29 network I consistently get my full internet speeds (~2100/350). If I connect the brocade back up so that the only thing between the router and device is the brocade and on the same vlan the speeds tank down to ~550/350 with the same device connected at 1GB. I do also have a server connected up at 40gbps and it gets about 1000-1200mbps.



When I check the core utilization with htop or top on OPNSense. It never shows any core anywhere near 100% and they are generally 50% or lower. I've also tested OPNSense on another device that is 30% faster single core and double the multicore speed. That different hardware got exactly the same result. No higher, no lower speed.

When I do a speedtest with OPNSense set to 1GB I get no dropped packets on the Brocade, but when I turn change it back to 2.5G/10G the brocade starts to drop packets. This happens on all devices 1GB or higher. Note: I haven't tested conected at 100mbps or lower.

I attached 3 pictures in different senerios. Every test I made sure nothing had any resources maxed out down to individual cores.

In my opinion is seems that the brocade is being overloaded.... some how... and is dropping packets. This could also be a misconfiguration issue but all I have it doing is very simple static routing, no where near what it can handle.

 

[manki09]

Need a little help with a pair of ICX6610-48 I have coming. I want to stack them using the rear QSFP ports. My question is, will any transceiver work or are they brocade specific? I've got some old meraki stuff laying around (MA-CBL-40G50CM) and would like to recycle them if possible?

My experience with twinax cable is that they generally more universal than Fiber SFPs. I would recommend seeing if your Meraki ones work since you already have them.

 

[fohdeesha]

Thanks for the guides! They've been really helpful on setting up.

I have one question however, I did the licensing for a ICX7250, but noticed it said it's a trial for 45 days. Is this something I should be worried about?

its not a trial it will last forever

 

[jode]

Hi @manki09 ,

thanks for clarifying your network configuration.

When I do a speedtest with OPNSense set to 1GB I get no dropped packets on the Brocade, but when I turn change it back to 2.5G/10G the brocade starts to drop packets.

This sounds like an issue with the connection between the OPNSense and the Brocade. Is there a way you can try different cables/transceivers?

 

[manki09]

Hi @manki09 ,

thanks for clarifying your network configuration.



This sounds like an issue with the connection between the OPNSense and the Brocade. Is there a way you can try different cables/transceivers?

Hello, @jode

That was another thought I had.

My original setup was running a SFP+ RJ45 on the brocade to the 1/2.5/5/10g intel x550-T2 on the OPNSense. So I would have a 10G link between the two.

During testing I threw in a cheap 2x 10G SFP+ / 4x 2.5G RJ45 switch, which I had tested standalone, to eliminate the RJ45 SFP. I connected TwinAX from Brocade to cheap switch, then 2.5gb Ethernet to OPNSense. All negotiated correctly or set manually. (Intel x550-T2 NBase-T (2.5/5) has to be set manually). But still the problem persisted with the same results.

Yesterday, I also decided to test another SFP+ port on the brocade and still the problem persisted.



I've been toying with the idea of getting a used Cisco 3850 off ebay but want to get this working correctly since I already have it and is the cheaper option.

 

[jode]

All negotiated correctly or set manually. (Intel x550-T2 NBase-T (2.5/5) has to be set manually). But still the problem persisted with the same results.

Can you try a different cable (cat 6a)? Is there a way you can test with iperf3? Can you try a different SFP+ port on the ICX?

 

[manki09]

Can you try a different cable (cat 6a)? Is there a way you can test with iperf3? Can you try a different SFP+ port on the ICX?

Just FYI on that intel NIC they disabled 2.5g and 5g from being advertised/negotiated to. I don't think there was a specific reason other than 2.5/5g devices weren't common when released

I have tried several different cables. The length of all the cables between the modem, router and brocade are less then 1 meter.

I have tried a couple different SFP ports on the ICX with no luck.

I will try some iperf tests and see where that takes me.

I was able to acquire the same model ICX6610 that my work had and was basically not in use and will be trying to load my config onto that to eliminate hardware issue.

 

[Rttg]

the brocade starts to drop packets

There's something about dropping packets that feels unlikely.

Looking at your config, is there a reason why you have ``flow-control`` disabled for every interface? That looks a bit odd to me after comparing it to my switch configs.

 

[manki09]

There's something about dropping packets that feels unlikely.

Looking at your config, is there a reason why you have ``flow-control`` disabled for every interface? That looks a bit odd to me after comparing it to my switch configs.

Hey @Rttg, thanks for the reply.
I only have flow-control disabled for testing purposes... I should have removed it before posting it. I had the problem before I disabled it.

 

[manki09]

Can you try a different cable (cat 6a)? Is there a way you can test with iperf3? Can you try a different SFP+ port on the ICX?

Swapped in the spare 6610 and got the same results.

I performed a few Iperf3 test.
I did notice a trend while responding to this and I've confirmed that this is not an OPNSense issue since I'm dropping packets to other devices besides OPNSense. Configuring the interface at 1GB was just a band-aid.


It seems to be dropping packets only when sending data from a higher speed interface to a lower. A couple examples below.


40GB to 10GB = dropped packets
40GB to 1GB = dropped packets

1GB to 10GB = no dropped packets
1GB to 40GB = no dropped packets
1GB to 1GB = no dropped packets

I also noticed dropped packets on the brocade interface the router is connected to while doing the upload portion during a internet speedtest. Which is limited at 350~.