Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

[wittyet]

today I came across this post, and got excited. thanks to OP who provides well written info on each model.
I went to ebay, and wow. Is the inflation really that bad over past 5 years? I can barely see anything below $200, phew

 

[i386]

5 years? nope the last 3.5 years
and 24 port versions cost more than the 48 port versions

 

[Kahooli]

today I came across this post, and got excited. thanks to OP who provides well written info on each model.
I went to ebay, and wow. Is the inflation really that bad over past 5 years? I can barely see anything below $200, phew

Be patient. I ust got one a month ago for $75 shipped.

 

[freph91]

Is anyone aware of how to get an established ACL working properly? I've been testing with my desktop IP and have been unable to get it to work successfully. I've tried rebooting the hosts on both sides (using nginx as a test for http/https on a separate host machine), clearing counters, removing and re-applying the interface, but it seems to count all TCP connections as established and does nothing against the initial SYN packet.

Unit specs:

SSH@icx7250-24#sh ver
  Copyright (c) Ruckus Networks, Inc. All rights reserved.
    UNIT 1: compiled on May  6 2022 at 23:28:37 labeled as SPR08095g
      (33554432 bytes) from Primary SPR08095g.bin (UFI)
        SW: Version 08.0.95gT213
      Compressed Primary Boot Code size = 786944, Version:10.1.24T215 (spz10124)
       Compiled on Thu Apr 21 02:08:36 2022

  HW: Stackable ICX7250-24
==========================================================================
UNIT 1: SL 1: ICX7250-24 24-port Management Module
      Serial  #:<redacted>
      Software Package: ICX7250_L3_SOFT_PACKAGE   (LID: <redacted>)
      Current License: l3-prem-8X10G
      P-ASIC  0: type B344, rev 01  Chip BCM56344_A0
==========================================================================
UNIT 1: SL 2: ICX7250-SFP-Plus 8-port 80G Module
==========================================================================
1000 MHz ARM processor ARMv7 88 MHz bus
    8 MB boot flash memory
    2 GB code flash memory
    2 GB DRAM
STACKID 1  system uptime is 123 day(s) 17 hour(s) 46 minute(s) 39 second(s)
The system started at 06:25:07 GMT-05 Sun Jan 29 2023

The system : started=warm start   reloaded=by "reload"

SSH@icx7250-24#sh access-list accounting vlan 8 in detail
ACL Accounting Table
==========================================================================================================================================================
ACL Name: vlan-8-in
UnitID: 1  Region: 0  Filter Seq Num: 10     Filter Def: permit tcp any any established                               HitCnt: 50902630   ByteCnt: 6644853474
UnitID: 1  Region: 0  Filter Seq Num: 11     Filter Def: deny tcp host 192.168.8.238 any eq 443                       HitCnt: 0          ByteCnt: 0
UnitID: 1  Region: 0  Filter Seq Num: 20     Filter Def: permit tcp any any eq http                                   HitCnt: 0          ByteCnt: 0
UnitID: 1  Region: 0  Filter Seq Num: 30     Filter Def: permit tcp any any eq ssl                                    HitCnt: 0          ByteCnt: 0
UnitID: 1  Region: 0  Filter Seq Num: 40     Filter Def: permit udp any any eq ssl                                    HitCnt: 1281       ByteCnt: 230097
...
UnitID: 1  Region: 0  Filter Seq Num: 520    Filter Def: permit ip any any log                                        HitCnt: 120        ByteCnt: 25054
UnitID: 1  Region: 0  Filter Seq Num: 65001  Filter Def: deny ip any any                                              HitCnt: 0          ByteCnt: 0

Given the ACL layout and direction (applied in on VLAN8, which both hosts are in) I'd expect the SYN for http/https to hit seq 20 and seq 30 respectively. Note that seq 520 doesn't increment when doing this testing, so it's not making it down to the catch-all for logging. The explicit deny in seq 11 has no effect on initial SYN packets whatsoever from the test host. If I add a block above the established rule (cloning seq 11) it of course immediately blocks *all* https traffic from the test host since it gets dropped as soon as it reaches the switch port, but I'm at a bit of a loss here over the established rule. If it should be blocking SYNs and not packets that have ACK/RST then it seems like I either misunderstand how the switch logic works or there's a nasty bug here.

Edit: Hmm, seems like 'established' may actually be bugged on these switches: access-list 'established' not working properly in 09.0.10

 

[kpfleming]

I could never get 'established' to work properly, which was a big part of my motivation to move back to L2-only functionality on mine.

 

[Kahooli]

today I came across this post, and got excited. thanks to OP who provides well written info on each model.
I went to ebay, and wow. Is the inflation really that bad over past 5 years? I can barely see anything below $200, phew

Huge stock showed up in cali

Brocade ICX6610-48P-E 48 Ports Rack-Mountable Gigabit Ethernet Switch for sale online | eBay

Find many great new & used options and get the best deals for Brocade ICX6610-48P-E 48 Ports Rack-Mountable Gigabit Ethernet Switch at the best online prices at eBay! Free shipping for many products!

www.ebay.com

 

[eptesicus]

> conf t
> ip address x.x.x.x/24

That will be the management IP reachable via any of the ports with a default PVID. You just have to assign your laptop a static IP in the management range since the DHCP server won't be active by default.

Finally able to circle back on this. Doing just that, the command is invalid. 'ip' is valid, but the 'address' command is not.

 

[EngChiSTH]

today I came across this post, and got excited. thanks to OP who provides well written info on each model.
I went to ebay, and wow. Is the inflation really that bad over past 5 years? I can barely see anything below $200, phew

where are you located? US-IL here and if you are local, I have 6450-24 and 6450-48 I am not using (upgraded to 7250 as I wanted for fiber) . local pickup, price you consider reasonable, and it is yours as they are 'backup' for me anyway. I rather not deal with shipping as these are bulky/heavy. both models are non-P.

 

[wittyet]

where are you located? US-IL here and if you are local, I have 6450-24 and 6450-48 I am not using (upgraded to 7250 as I wanted for fiber) . local pickup, price you consider reasonable, and it is yours as they are 'backup' for me anyway. I rather not deal with shipping as these are bulky/heavy. both models are non-P.

I wish I lived in the US, but I am in Canada, and I agree enterprise switches are really heavy, I am currently playing with two HPE, I guess I am keeping it this way for now, thanks for the offer

 

[NablaSquaredG]

Just wanted to add my 2 cents on how much power ICX6450-24P (PoE models) draw when idle and with nothing but power connected.

I tested three ICX6450-24P (using a Shelly Plus 1PM), and all three consistently reported the same values:

25 W power draw during boot (when the two fans run at full speed)
22 W power draw after boot (when the two fans drop to low speed)
35 W power draw after a few seconds when the PoE power supply is initialized

So it seems the PoE power supply adds ~13 W on top of the idle power without PoE.

@fohdeesha I measure 25W with ICX6450-24 (non-P) in IDLE. Can you please correct this in the start post?

Edit, more power consumption:

ICX7450-48, 1 PSU, 1 Fan, 2x40G module (ICX7400-1x40GQ), 1x 4x10 module (ICX7400-4x10GF), 230V: 70W IDLE (no ports connected)
ICX7650-48ZP-E2, 1 PSU connected, 2 Fans, 230W: 120W IDLE (no ports connected)

 

[deeceesth]

one of my ICX725024P switches stopped supplying POE power. The switch still runs, just without POE to my APs. I had to put in an injector in between to get my APs back up. All other function appears to be the same.

where should I start with trying to fix this? Is there a separate POE PSU inside? or do I just need to replace the entire switch?

 

[Bluerai]

Hey everyone, I’m running out of ideas to try and I’d love some input from folks smarter than myself so here we go:

I have a 6450-48p that I have reset and followed the guides on, etc (all amazing by the way, and so is this thread). Using two of the suggested MikroTik copper SFP+ modules I can run a speed test to my computer and hit around 1850Mbps as expected. However when I run any sort of speed test on the standard switch ports, my speeds are roughly 400Mbps.

Am I mistakenly under the impression that I should be seeing closer to 940Mbps on the rest of the switch ports? I’ve messed with flow control, duplex settings, a ton of Google searching and I’m not getting very far. It looks like the switch doesn’t recognize the SFP+ modules and they show empty, and I can see that the phy device never initializes from the console. Yet it will tell me when I unplug the module and they’re clearly working.

Hopefully this isn’t functioning as expected, but I’m not sure what to try next. I’ve only tried with two of the SFP ports, unplugged a module and left one in, and still haven’t seen any changes.

Any ideas or direction for next steps would be appreciated!

Bumping this, anyone have any ideas?
I do have a donor switch that I couldn’t putty into so I’m thinking I could swap parts around if need be..

 

[NablaSquaredG]

one of my ICX725024P switches stopped supplying POE power. The switch still runs, just without POE to my APs. I had to put in an injector in between to get my APs back up. All other function appears to be the same.

where should I start with trying to fix this? Is there a separate POE PSU inside? or do I just need to replace the entire switch?

Maybe check show inline power detail?

 

[NablaSquaredG]

For reference, fans I've seen in switches (will be extended in the future)

ICX6450-24: 1 Fan
Foxconn PIA040H12L

40x40x28mm, 12V DC, 0.1A (0.16A printed on fan), Max Airflow = 0.33m³/min or 11.67CFM, Max Static Pressure = 11.9mm H2O / 0.47in H2O / 117Pa, 35.6db(A), 8500 RPM
Ball Bearing?
Tacho



ICX6450-24P: 2 Fans

Nidec W40S12BMA5-52, 40x40x28mm, 12V DC, 0.12A (0.14A printed on fan), Max Airflow = 0.31m³/min or 10.9CFM, Max Static Pressure = 101Pa or 0.41in H2O, 34.5dB(A), 8500 RPM, -10°C - +70°C, MTBF: 70,000h @ 40°C
Part Number W40S12BMA5-52 decoded:
W = Vane Axial, Closed Flange
40 = 40mm
S = 28mm thickness
12 = 12V
B = Ball Bearings
M = Medium Speed
A = Generation
5 = Blade count
-52 = Tachometer (no other options like PWM)


PSA: Replacing those fans with Delta EFB0412VHD-R00 will have the switch mark them as failed, because the R has a locked rotor sensor and NOT a Tachometer.


ICX6450-48: 1 Fan, Nidec W40S12BMA5-52 (see above)


ICX7250-48: 2 Fans, Foxconn PIA040H12P

40x40x28mm, 12V DC, 0.59A (0.81A printed on fan), Max Airflow = 0.746m³/min or 26.33CFM, Max Static Pressure = 51.82mm H2O / 2.04in H2O / 508Pa, 53.8db(A), 18000 RPM
Ball Bearing?
Tacho

ICX7250-48P: 3 Fans, Foxconn PIA040H12P

 

[deeceesth]

Maybe check show inline power detail?

strange output. it just looks like nothing is there. the switch definitely has poe and it was running with poe up until a month ago.
reboot didn't fix it.

SSH@ICX7250-24P Switch>show inline power detail


Power Supply Data On unit 1:
++++++++++++++++++



Power Supply Data:
++++++++++++++++++

power supply 2 is not present


POE Details Info. On Unit 1 :


General PoE Data:
+++++++++++++++++

Firmware
Version
----------------
00.0.0

Hardware
Version                                                           
----------------                                                 
                                                                  
                                                                  
Device HW version         :                                       
Device Temperature(deg-C) :                                       
Device Status             :                                       
                                                                  
                                                                  
                                                                  
Cumulative Port State Data:                                       
+++++++++++++++++++++++++++                                       
                                                                  
#Ports    #Ports     #Ports   #Ports    #Ports       #Ports     #Ports
Admin-On  Admin-Off  Oper-On  Oper-Off  Off-Denied   Off-No-PD  Off-Fault
-------------------------------------------------------------------------
                                                                  
                                                                  
                                                                  
Cumulative Port Power Data:                                       
+++++++++++++++++++++++++++                                       
                                                                  
#Ports  #Ports  #Ports        Power       Power                   
Pri: 1  Pri: 2  Pri: 3  Consumption  Allocation                   
-----------------------------------------------

 

[NablaSquaredG]

strange output. it just looks like nothing is there. the switch definitely has poe and it was running with poe up until a month ago.
reboot didn't fix it.

Hu that is bad.

I don't know whether the PoE board would still show up even if the 54V rail from the power supply is missing. Maybe a fuse on the PoE board is blown. Have you already opened the switch?

 

[deeceesth]

Hu that is bad.

I don't know whether the PoE board would still show up even if the 54V rail from the power supply is missing. Maybe a fuse on the PoE board is blown. Have you already opened the switch?

Not yet. I'm preparing my back up switch and once i swap it out I'm going to open it up. I'll report back once I find out more info.

 

[obsidian]

OK - considering the rear ports are not intended to be used for data ports, you won't find any information on this in the manuals etc. Using them as just data ports is easy enough, but splitting them up and using some for stacking and some for data ports requires some finangling to make the stacking code not try and take over all 4 ports.

However it's possible and stable across reboots etc. basically follow this, it might take you a couple times. This will enable stacking between two units just using the 40gbE qsfp ports, leaving 4x breakout QSFPs total available for data use

#unplug all stacking cables except for 1, going between port 1/2/1 on unit 1 and 1/2/1 on unit two
#don't plug anything else into unit 2 or you'll get a loop
#first you have to unstack all the units and remove any existing stack configuration:
enable
stack unconfigure all
write mem
reload
#both switches should reload, wait till they come back up

#when they're back up, on unit 1:
enable
conf t
stack unit 1
no stack-trunk 1/2/1 to 1/2/2
stack-port 1/2/1 1/2/6
#ignore the warning that pops up about not showing up in "show run"

#still running the below on unit 1!
stack unit 2
no stack-trunk 2/2/1 to 2/2/2
stack-port 2/2/1 2/2/6
#ignore the warning that pops up about not showing up in "show run"
exit
stack enable
write mem
exit
stack secure-setup

#When the setup is done, do "show run"
#stack unit 2 section probably has "stack-trunk 2/2/1 to 2/2/2" added back to it

#the second switch is currently rebooting to rejoin the stack, wait for it to do so
#once it comes back online, do the following to remove the stack unit 2 trunk:
#still on unit 1:
conf t
stack unit 2
no stack-trunk 2/2/1 to 2/2/2
#will probably reboot unit 2 again
#wait for it to come back online, then connect the second pair of 40gbE ports

Once you connect the second pair of 40gbe ports (1/2/6 on unit 1 to 1/2/6 on unit 2) you should be done. Check by running "show stack", you should see both units "ready" with the connection diagram showing both 40gbE ports linked up like below:

ICX1#sh stack
T=6m28.6: alone: standalone, D: dynamic cfg, S: static
ID   Type          Role    Mac Address    Pri State   Comment
1  S ICX6610-48P   active  cc4e.24b8.d9d0 128 local   Ready
2  S ICX6610-48P   standby cc4e.243e.aa74   0 remote  Ready

    active       standby
     +---+        +---+
-2/6| 1 |2/1--2/1| 2 |2/6-
|   +---+        +---+   |
|                        |
|------------------------|
Standby u2 - protocols ready, can failover
Current stack management MAC is cc4e.24b8.d9d0

If you have the same output, do a "write mem" then a "reload" to reload the whole stack. It should come back up fresh in the correct state and running "show stack" again should have the same output as before.

Your final stacking config should look like this:

stack unit 1
  module 1 icx6610-48p-poe-port-management-module
  module 2 icx6610-qsfp-10-port-160g-module
  module 3 icx6610-8-port-10g-dual-mode-module
  priority 128
  stack-port 1/2/1 1/2/6
stack unit 2
  module 1 icx6610-48p-poe-port-management-module
  module 2 icx6610-qsfp-10-port-160g-module
  module 3 icx6610-8-port-10g-dual-mode-module
  stack-port 2/2/1 2/2/6
stack enable

I know this is an older post, but was wondering if the opposite would be possible.

Instead of using port 1/2/1 (40G qsfp Port) for stacking could the could the 4x breakout QSFP port be used for trunking leaving the 40gbE qsfp ports for data use?

 

[NablaSquaredG]

Instead of using port 1/2/1 (40G qsfp Port) for stacking could the could the 4x breakout QSFP port be used for trunking leaving the 40gbE qsfp ports for data use?

That question has been asked many times by many people (including me)...

Sadly, the answer is: If you find a way to do this, please let us know
Right now, it seems like it isn't possible. But I don't think @fohdeesha himself has ever given a definite answer

 

[obsidian]

That question has been asked many times by many people (including me)...

Sadly, the answer is: If you find a way to do this, please let us know
Right now, it seems like it isn't possible. But I don't think @fohdeesha himself has ever given a definite answer

I was kinda hoping someone had figured it out.