Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

[BoGs]

If you are willing to give up that easily unfortunately this was not going to work out, there is no nice UI that you can use for these older switches. You will need to change some settings unplug the SFP plugs from your other switch and test it out. Unless you work in that field which most of us do not. It took me 2 months to switch from OPNSense to Mikrotik router as I wanted to route more then 2.5G and instead of building a new server I went for my largest link SFP+.

The reason why people are saying not to do side gig is so that you can learn, and ultimately success and being proud of what you achieve.

What you want to do out of the gate is more advanced then most, what you are looking for is to setup a transit VLAN between your switch and router, your switch becomes the router by setting up routable interfaces and you create a route on the switch for other vlan subnets to pass it to OPNSense and on the OPNSense to pass it to the switch. You will also have to pass the DHCP relay to wherever your server is.

I used this to read up on intervlan routing (InterVLAN Routing) or this (Inter-VLAN Routing Configuration - %%currentyear%% Step-by-Step Tutorial) and while I think its for Cisco you can lookup the keyword commands in the brocade command reference sheet. Keeping router on a stick is probably the easiest option as you can setup your OPNSense vlans on top of the interface and away you go with trunk ports. I would do that FIRST then potentially experimenting with _new_ vlans on different ports so you do not break the internet.

Good luck

 

[ramicio]

If you are willing to give up that easily unfortunately this was not going to work out, there is no nice UI that you can use for these older switches. You will need to change some settings unplug the SFP plugs from your other switch and test it out. Unless you work in that field which most of us do not. It took me 2 months to switch from OPNSense to Mikrotik router as I wanted to route more then 2.5G and instead of building a new server I went for my largest link SFP+.

The reason why people are saying not to do side gig is so that you can learn, and ultimately success and being proud of what you achieve.

What you want to do out of the gate is more advanced then most, what you are looking for is to setup a transit VLAN between your switch and router, your switch becomes the router by setting up routable interfaces and you create a route on the switch for other vlan subnets to pass it to OPNSense and on the OPNSense to pass it to the switch. You will also have to pass the DHCP relay to wherever your server is.

I used this to read up on intervlan routing (InterVLAN Routing) or this (Inter-VLAN Routing Configuration - %%currentyear%% Step-by-Step Tutorial) and while I think its for Cisco you can lookup the keyword commands in the brocade command reference sheet. Keeping router on a stick is probably the easiest option as you can setup your OPNSense vlans on top of the interface and away you go with trunk ports. I would do that FIRST then potentially experimenting with _new_ vlans on different ports so you do not break the internet.

Good luck

I cannot learn how to do this unless someone literally shows me how to do this. Documents and videos are of no help to me. I don't understand the stuff. I only understand by seeing how it is done. I cannot experiment with more ports. My OPNSense router is 1u and has only a dual-port SFP+ NIC. Impossible.

 

[DavidVermillion]

I cannot learn how to do this unless someone literally shows me how to do this. Documents and videos are of no help to me. I don't understand the stuff. I only understand by seeing how it is done. I cannot experiment with more ports. My OPNSense router is 1u and has only a dual-port SFP+ NIC. Impossible.

That's why no one wants to configure your stuff as a "side gig." A) The knowledge is fairly esoteric and B) people who refuse to learn and just want someone else to do it all for them or feed them step by step instructions usually end being combative, demanding, and want far more support.

Sorry, my dude, but nothing in your tone sounds like you'd be an exception to this. My spider-sense from back when I did freelance IT is tingling. Good luck, I hope you can find someone to do it for you.

 

[ramicio]

That's why no one wants to configure your stuff as a "side gig." A) The knowledge is fairly esoteric and B) people who refuse to learn and just want someone else to do it all for them or feed them step by step instructions usually end being combative, demanding, and want far more support.

Sorry, my dude, but nothing in your tone sounds like you'd be an exception to this. My spider-sense from back when I did freelance IT is tingling. Good luck, I hope you can find someone to do it for you.

Such a weird attitude, makes no logical sense. People are fine punching a clock to do that very same IT work, and no one there wants to know how the sausage is made. Meanwhile, someone who want the same help, want to know what they did, with a quick explainer, nope. Taboo. And it's not even esoteric. It's basic IT stuff. 1. What's combative? 2. What's demanding? 3. Where did I say I want support (and you're sounding like you're mocking me as if I'd want it for free, when I've said the opposite)? Not being combative and demanding. I don't understand how to do this, and posting generic articles for an idiot like me to read isn't helpful. The same effort spent to do that could just have been spent to show me literally how to do it. It's just industrial gatekeeping as far as I'm concerned.

 

[bwahaha]

@ramicio you may not see the behavior they were referring to, but we can.
You didn't ask for a quick explanation, you asked for a hand holding.

1) what combative? well, that post, as example....
2) What's demanding? well that post, as example...
3) whats.... that? well, that post for example.

We get it, something seemingly simple, should be a simple answer. Unfortunately, you were given solid information and you said rejected it.

We aren't on a Professional Support forum, we're a hobbyist learning forum. Many of us are, were, or will be professionals, so when "outside of scope of homelab" pops in, and demands professional support, for free (yes, you offered to pay), and becomes combative, it irks us. We've dealt with demanding customers, clients, users, and we come here to get away from them.

 

[Cheburashka]

Why do you want to run jumbo frames in 2024? They're almost entirely unnecessary for 10gb, have you done any throughput testing etc showing that you need them for some reason?

I guess I was trying to figure out why between my QNAP, the ICX SW and my W11 desktop, am I getting 5.6gig with iperf. I was just going through threads and trying to identify ways to improve the performance of the transfer.

I have to do this test from a linux live boot ISO instead of Windows to see if the issue is within the OS or not.

 

[hmw]

I guess I was trying to figure out why between my QNAP, the ICX SW and my W11 desktop, am I getting 5.6gig with iperf. I was just going through threads and trying to identify ways to improve the performance of the transfer.

I have to do this test from a linux live boot ISO instead of Windows to see if the issue is within the OS or not.

jumbo frames might help by 10% but unless you're tunneling or encapsulating other protocols, they're almost never the issue.

try running iperf in parallel mode and see if you can saturate 10g by running 2-3 iperf streams in parallel ( for example iperf3 -P 3 ). Windows is notorious for not being able to saturate 10G with just one connection/thread. The other problem is something in between that might cause impedance mismatch like a SFP to 10GBase-T transceiver etc

 

[86turbodsl]

can anyone help me understand what a proper stack setup on a 6610 is supposed to look like? I have a 6610 48p in the house, and a 6610 24p in the shop. I ran stack secure-setup, it has a connection, it's complaining about a link. I thought one of the fibers might be mixed up because i've swapped sides a few times trying to get the link up. It's currently showing

One port, the left as green on each end. I am using 1/2/6 and 1/2/7 (bottom ports) on the big one and 3/2/1 and 3/2/2 (bottom) on the other end. 1/2/6 and 3/2/1 are showing as green currently. Should i assume i have a dead fiber if i tried switching sides on the one end of one of the fibers. Or swap optics out?

Stack Details

Unit ID	Type	Role	Mac Address	Priority	State	Comment
1	S ICX6610-48P	active	748e.f8fb.f0a2	128	local	Ready
3	D ICX6610-24P	standby	cc4e.24c5.b6fc	0	remote	Ready

alone: standalone, D: dynamic config, S: static config


Stack Port Status

Unit ID	Stack-port1	Stack-port2
1	dn (1/2/1)	up (1/2/6)
3	up (3/2/1-3/2/2)	dn (3/2/6-3/2/7)

Stack Neighbors

Unit ID	Stack-port1	Stack-port2
1	none	3
3	1	none

 

[DouglasteR]

Hi there guys,

The time has come and i´m on the verge to buy the 24p POE+ version and setup it as my main switch/dhcp in my home.

But i have one question first, can i connect my ConnectX3 - QSFP+ directly to the switch and connect at 40Gbps ?

My use case is learning switching, power some ip cameras and common soho networking.

Thanks.

 

[BBergle]

Sometimes the issue is bad serial cable. Some of the Arista and Celestica switches I bought came with their own serial cables, and they work much better than old one I had.

I am actually using a serial cable from an Arista switch I got on ebay. The seller sent me a new switch and I am having the same issue. It has to be one of the cables/adapters I am using. I feel like im going insane. I have tried three different computers. M2 Macbook pro, Intel Macbook Pro and a windows PC and they all have the same issue but the windows computer has horizontal lines instead of question marks. I have new cables coming tomorrow but my current ones work fine for my Arista switch so I really don't know what's going on. All I can seem to get on my terminal screen are these characters ����

 

[kpfleming]

Hi there guys,

The time has come and i´m on the verge to buy the 24p POE+ version and setup it as my main switch/dhcp in my home.

But i have one question first, can i connect my ConnectX3 - QSFP+ directly to the switch and connect at 40Gbps ?

My use case is learning switching, power some ip cameras and common soho networking.

Thanks.

If by "the 24p POE+" version you mean the 7250-24P, then no, the high speed ports are 10Gbps maximum.

 

[DouglasteR]

I was talking about the ICX6610-24p !

My idea was using the back QSFP ports as 40G uplink !

 

[fohdeesha]

I was talking about the ICX6610-24p !

My idea was using the back QSFP ports as 40G uplink !

two of them yes as my docu explains. the other two are 43x 10gb breakout. also you do not want to use the built in DHCP server on these things

 

[DouglasteR]

two of them yes as my docu explains. the other two are 43x 10gb breakout. also you do not want to use the built in DHCP server on these things

Thanks !

 

[who]

Hi everyone!
I believe this is my first time posting on this thread...
Been following this thread for maybe 2 years or so.

First of all I'll like to thank fohdeesha for sharing his knowledge and expertise!

Initially I've gotten myself a ICX6430-12P, running since mid-2022 and recently ICX6450-12P (likely replace or keep as spare) as well as a 7250-24P.

I've encounter a small license issue on the 6450 last night as i was upgrading it.

ICX6450-C12PD Router#show license
Index    Lic Mode        Lic Name               Lid/Serial No  Lic Type    Status     Lic Period    Lic Capacity
Stack unit 1:
1        Node Lock       ICX6450-PREM-LIC-SW    H4C<snip>PLN8    Normal      Active     Unlimited         1
2        Node Lock       ICX6450-10G-LIC-POD    H4C<snip>PLN8    Normal      Invalid    Unlimited         2

The 10G license status is invalid.
Has anyone experienced this? I'm guessing a lot of you here are not using the 6450-12P so it might have slipped through the cracks.

I've removed and added it back, still shows the same invalid status.
SHA256 hash (bc1160442a077d0be720be4a20464f8d9ce837543e2ee94f262f9a38dc8c1bd2) of the license (2-6450-ports.xml) is the same from the latest brocade-08-03-2024.zip and brocade-09-16-2022.zip

ICX6450-C12PD Router#sh version
  Copyright (c) 1996-2016 Brocade Communications Systems, Inc. All rights reserved.
    UNIT 1: compiled on Apr 23 2020 at 10:57:06 labeled as ICX64R08030u
                (9871112 bytes) from Primary ICX64R08030u.bin
        SW: Version 08.0.30uT313
  Boot-Monitor Image size = 786944, Version:10.1.05T310 (kxz10105)
  HW: Stackable ICX6450-C12-PD
==========================================================================
UNIT 1: SL 1: ICX6450C 12-port-PD Management Module
         Serial  #: 2ax5o2jk68e
         License: ICX6450_PREM_ROUTER_SOFT_PACKAGE   (LID: H4CKTH3PLN8)
         P-ENGINE  0: type DEF0, rev 01
==========================================================================
UNIT 1: SL 2: ICX6450C-Copper 2port 2G Module
==========================================================================
UNIT 1: SL 3: ICX6450C-Fiber 2port 2G Module
==========================================================================
  800 MHz ARM processor ARMv5TE, 400 MHz bus
65536 KB flash memory
  512 MB DRAM
STACKID 1  system uptime is 42 minute(s) 23 second(s)
The system started at 00:00:17 GMT+00 Thu Jan 01 1970

The system : started=cold start

Any inputs would be very much appreciated. I don't think I'll be using 10G soon but would like to keep my options open.

Does the 6450-C12-PD supports 10G in the first place?
If not, that might explain the invalid status.


UPDATE: Ops! My bad! Looks like there's no 10G ports on the 6450C.

 

[molnart]

I just acquired an ICX7250-48p. I've been trying to get connected to the console but I've run into an issue. I've assembled a couple of the serial cables for these and using the settings that it should be (9600 8N1), all I get is gibberish. The way the output is produced seems like it is a boot up sequence and if I let it go, it will echo back what I type like it has gotten to the shell but the output is unreadable. Any ideas on how I can get it working?

have you been able to resolve this? i have tried multiple baud rates or cables, but still just getting garbage on the console (or no output even with one particular cable). i am not even able to type into the console, i get complete different characters than i am sending

also i have an USB FTDI board that I have tried with a spliced miniUSB cable. would it be safe to try the board on 5V voltage?

 

[enanothebrave]

I picked up an ICX7250 from ebay and wanted to check if anyone has come across this. I reached out to the seller as I'm assuming its borked. Booting into serial console presents the following. Keyboard input is not accepted so I cannot press b to interrupt the boot. I've tried two console cables, one made from a sacrificial mini usb, punched down to rj45 and connected to a cisco cable, and then one mini usb to db9 connected to a db9 to usb serial cable. Both cables present the same console log without the ability to type. The switch was advertised as coming from a working environment. Been searching the forums for some time looking for a similar situation, most folks seem to have one of the boot partitions working and I don't see much about not being able to type into the console.


EDIT: Ebay seller issued a refund and said don't ship it back. I may try and futz with the serial port and connect wires directly onto the motherboard just incase there is an issue there. Will let this post sit for a bit to see if anyone has any other guidance first though.


Brocade Bootloader: 10.1.05T215 (Mar 19 2015 - 16:39:20)

Validate Shmoo parameters stored in flash ..... OK
Restoring Shmoo parameters from flash .....
Running simple memory test ..... OK

ICX7250-48 Copper (POE), PVT1
SYS CPLD VER: 0x6, Released Ver: 0

Enter 'b' to stop at boot monitor: 0
bootdelay: ===
Booting image from Primary

NAND read: device 0 offset 0x0, size 0x2000000
................................ 33554432 bytes read: OK
Wrong Image Format for bootm command
ERROR: can't get kernel image!
could not boot from primary, no valid image; trying to boot from secondary
BOOTING image from Secondary

NAND read: device 0 offset 0x4000000, size 0x2000000
....................................................Skipping bad block 0x05a00000
Skipping bad block 0x05b00000
............ 33554432 bytes read: OK
Wrong Image Format for bootm command
ERROR: can't get kernel image!
ICX7250-Boot>

 

[enanothebrave]

have you been able to resolve this? i have tried multiple baud rates or cables, but still just getting garbage on the console (or no output even with one particular cable). i am not even able to type into the console, i get complete different characters than i am sending

If you are getting garbage/gibberish on a serial connection that usually has to do with an incorrect baud rate, should be 9600 for these. You should at least get clear text, I'm having the same issue with not being able to type though.

 

[manki09]

Hey Guys.
I've been running a Brocade ICX6610 for a while now with no issue. I recently upgraded my internet from 500mbps/20mbps to 2gb/300mbps. I have attached my current network diagram. What I changed since the upgrade is the modem from a TP-Link to Xfinity's XB8 and I changed from a HP T730 to a Lenovo m90q gen2.

I'm having a bit of an issue with traffic flowing through the ICX6610.

On 1GB enabled devices -
When I have both interfaces (LAN/WAN) on OPNSense to 2.5gbe or 10gbe my internet download rate is only 500-600mbps. If I lower either the LAN or WAN interface down to 1gbe I get about 940mbps just fine. Switch it back to 2.5gbe or 10gbe and the speed will drop back down to 500-600mbps.

On 40GB enabled devices (R730 to Windows 10 VM). It's a vary similar issue to 1GB devices, however it's about 1100mbps, which is better but not the 2gbe. If I switch the same VM to a 1GB port I again get the same 500-600gbps. I have also switched the VM over to the same vlan as the router interface to eliminate any intervlan issues, and the speeds are still the same.

If I hook a PC w/ 2.5gbe directly to my router I can get a consistent 2.1gbe download rate. So I know its not OPNSense directly causing the speed drops.

What I have noticed is dropped packets on interfaces when I do speed test.

Egress queues:
Queue counters    Queued packets    Dropped Packets
    0                 512                   0
    1                   0                   0
    2                   0                   0
    3                   0                   0
    4                   0                   0
    5                 281                   0
    6                   0                   0
    7             1941191                 919

The OPNSense LAN side is currently connected to the switch with a 10GB RJ45 SFP+, I have also testing with a cheap 2.5gb/10gb switch between the router and brocade with 2.5 to the router and twinax to the brocade

Anyone have any ideas what could cause this and a fix?


Edit:
I forgot to mention. I loaded OPNSense onto completely different hardware and got the same result.
I also originally used OPNSense in ESXi but moved it to bare metal to troubleshoot.

 

[Haulien]

Thanks for the guides! They've been really helpful on setting up.

I have one question however, I did the licensing for a ICX7250, but noticed it said it's a trial for 45 days. Is this something I should be worried about?