Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

[TonyArrr]

At this point, I am looking at using Webmin to handle ISC-DHCP and creating a basic VLAN/subnet aware GUI for PowerDNS (or both)

Might be a a good time to start reading up on ISC-Kea, which is replacing ISC-DHCPd
I had been getting really into how to configure and admin ISC-DHCPd and then found out that is it actually deprecated in favour of Kea, so starting again with it before my network moves over.

 

[Nehalem501]

Hello,

I've recently acquired an ICX 6450-48 (non PoE version) and its (single) stock fan is very loud.
When reading this thread, I understand the fan should slow down after a while after the unit has finished booting, but mine seems stuck at full speed all the time without anything connected to the switch (I kept it on for over an hour once to check if the fan would slow down after some time).

I've already followed the guide to update the firmware to the latest version (08.0.30uT313).

The temperature displayed when running show chassis is always between 25° C and 40° C and for the fan it says Fan ok, speed (auto): [[1]]<->2

I don't know if it is supposed to work like that, but the fan blows air from inside the switch to outside the switch.

Is there something wrong with my unit, or should I just change the fan? (which one would be the quietest? I would prefer to keep some form of cooling rather than running it completely fanless, summers in France can get pretty hot and most homes don't have AC here)

Thanks for your help.

Yes, Fan MOSFET fried. Seems to be a common issues with those switches.

I’ve looked into this for the past few days and the issue looks to be more complicated than it seems.

I found the fan MOSFET, and the Op Amp whose output goes into the MOSFET. Both being quite close to the fan connector the PCB.
With a multimeter and an oscilloscope I measured the voltage and looked at the signal on all pins and everything is absolutely the same wether the fan speed is on [1] or [2]. At least it confirms that the fan runs at the max speed all the time (and I have indeed measured 12V going into the fan).

I’ve tried to see where the trace that goes into the input of the op amp leads. But it goes to a via and then the trace continues inone of the internal layers of the PCB, so next to impossible to find out without some schematics (which I couldn’t find on the internet).

For the time being I have disconnected the fan, and reassembled the unit. After keeping the switch on for a few hours, with some traffic going through it (watching a 4K Netflix stream) the temperatures were around 65°C, which seems quite high.

Which fan would you recommend that would provide ok cooling while being silent when running with a 12V input ? Thanks.

 

[zeroturnpete]

Thanks for all of the useful info, just picked up a 6610 (non poe) for 70 bucks. What is the best way to go about getting 10 gig licenses around the forum

 

[hmw]

Might be a a good time to start reading up on ISC-Kea, which is replacing ISC-DHCPd
I had been getting really into how to configure and admin ISC-DHCPd and then found out that is it actually deprecated in favour of Kea, so starting again with it before my network moves over.

Yeah I have been reading up into Kea vs DHCPd. On the face of it - Kea sounds like a great alternative. However, the reason you don't see widespread adoption of Kea amongst open source projects is because:

[1] most of the features that make Kea stand out over and above DHCPd like the API for subnets that allows adding and deleting subnets without restarting the Kea server - are hidden behind a paywall, with the cheapest option being $550 for a 1 year subscription + 5 years usage assuming you have < 1000 active leases

[2] You can run Kea with a 'memfile' backend but some of the advanced options need a SQL database like MySQL or PostgreSQL. And Kea doesn't do SQLite

Interestingly - on the OPNsense forums, they were floating Free Radius as a DHCP alternative, from ver 3.0.x it has a full DHCP stack and there's some mods to let it use SQLite (so that host leases etc are inside a database), might just have a look at that ...

 

[custom90gt]

Thanks for all of the useful info, just picked up a 6610 (non poe) for 70 bucks. What is the best way to go about getting 10 gig licenses around the forum

by reading the first post and following those instructions.

 

[TonyArrr]

are hidden behind a paywall, with the cheapest option being $550 for a 1 year subscription + 5 years usage assuming you have < 1000 active leases

Whoa, somehow I completely missed that. Very steer clear then!

 

[Mushishi]

https://www.fs.com/products/69905.html

Okay, cool, I see the "dual-compatibility solutions" allows me to pick different ends. But I was also hoping for some advice as to whether the oracle-branded intel card takes an Intel module or something else, and how I might discover that (short of "plug it in, it doesn't work"). Or, if the Brocade isn't restricted, I just go with an Intel-to-Intel connection?
I guess I should pick up some fiber too, to share shipping costs. And a few other things that might come up as I test things out. Ah, discovery!

I got one of the Oracle branded x520's some time ago also and while i can't remember if i had any problems with the optics i know i followed this post to unlock the card to accept all optics.

https://forums.servethehome.com/index.php?threads/patching-intel-x520-eeprom-to-unlock-all-sfp-transceivers.24634/

 

[wittyet]

today I came across this post, and got excited. thanks to OP who provides well written info on each model.
I went to ebay, and wow. Is the inflation really that bad over past 5 years? I can barely see anything below $200, phew

 

[i386]

5 years? nope the last 3.5 years
and 24 port versions cost more than the 48 port versions

 

[Kahooli]

today I came across this post, and got excited. thanks to OP who provides well written info on each model.
I went to ebay, and wow. Is the inflation really that bad over past 5 years? I can barely see anything below $200, phew

Be patient. I ust got one a month ago for $75 shipped.

 

[freph91]

Is anyone aware of how to get an established ACL working properly? I've been testing with my desktop IP and have been unable to get it to work successfully. I've tried rebooting the hosts on both sides (using nginx as a test for http/https on a separate host machine), clearing counters, removing and re-applying the interface, but it seems to count all TCP connections as established and does nothing against the initial SYN packet.

Unit specs:

SSH@icx7250-24#sh ver
  Copyright (c) Ruckus Networks, Inc. All rights reserved.
    UNIT 1: compiled on May  6 2022 at 23:28:37 labeled as SPR08095g
      (33554432 bytes) from Primary SPR08095g.bin (UFI)
        SW: Version 08.0.95gT213
      Compressed Primary Boot Code size = 786944, Version:10.1.24T215 (spz10124)
       Compiled on Thu Apr 21 02:08:36 2022

  HW: Stackable ICX7250-24
==========================================================================
UNIT 1: SL 1: ICX7250-24 24-port Management Module
      Serial  #:<redacted>
      Software Package: ICX7250_L3_SOFT_PACKAGE   (LID: <redacted>)
      Current License: l3-prem-8X10G
      P-ASIC  0: type B344, rev 01  Chip BCM56344_A0
==========================================================================
UNIT 1: SL 2: ICX7250-SFP-Plus 8-port 80G Module
==========================================================================
1000 MHz ARM processor ARMv7 88 MHz bus
    8 MB boot flash memory
    2 GB code flash memory
    2 GB DRAM
STACKID 1  system uptime is 123 day(s) 17 hour(s) 46 minute(s) 39 second(s)
The system started at 06:25:07 GMT-05 Sun Jan 29 2023

The system : started=warm start   reloaded=by "reload"

SSH@icx7250-24#sh access-list accounting vlan 8 in detail
ACL Accounting Table
==========================================================================================================================================================
ACL Name: vlan-8-in
UnitID: 1  Region: 0  Filter Seq Num: 10     Filter Def: permit tcp any any established                               HitCnt: 50902630   ByteCnt: 6644853474
UnitID: 1  Region: 0  Filter Seq Num: 11     Filter Def: deny tcp host 192.168.8.238 any eq 443                       HitCnt: 0          ByteCnt: 0
UnitID: 1  Region: 0  Filter Seq Num: 20     Filter Def: permit tcp any any eq http                                   HitCnt: 0          ByteCnt: 0
UnitID: 1  Region: 0  Filter Seq Num: 30     Filter Def: permit tcp any any eq ssl                                    HitCnt: 0          ByteCnt: 0
UnitID: 1  Region: 0  Filter Seq Num: 40     Filter Def: permit udp any any eq ssl                                    HitCnt: 1281       ByteCnt: 230097
...
UnitID: 1  Region: 0  Filter Seq Num: 520    Filter Def: permit ip any any log                                        HitCnt: 120        ByteCnt: 25054
UnitID: 1  Region: 0  Filter Seq Num: 65001  Filter Def: deny ip any any                                              HitCnt: 0          ByteCnt: 0

Given the ACL layout and direction (applied in on VLAN8, which both hosts are in) I'd expect the SYN for http/https to hit seq 20 and seq 30 respectively. Note that seq 520 doesn't increment when doing this testing, so it's not making it down to the catch-all for logging. The explicit deny in seq 11 has no effect on initial SYN packets whatsoever from the test host. If I add a block above the established rule (cloning seq 11) it of course immediately blocks *all* https traffic from the test host since it gets dropped as soon as it reaches the switch port, but I'm at a bit of a loss here over the established rule. If it should be blocking SYNs and not packets that have ACK/RST then it seems like I either misunderstand how the switch logic works or there's a nasty bug here.

Edit: Hmm, seems like 'established' may actually be bugged on these switches: access-list 'established' not working properly in 09.0.10

 

[kpfleming]

I could never get 'established' to work properly, which was a big part of my motivation to move back to L2-only functionality on mine.

 

[Kahooli]

today I came across this post, and got excited. thanks to OP who provides well written info on each model.
I went to ebay, and wow. Is the inflation really that bad over past 5 years? I can barely see anything below $200, phew

Huge stock showed up in cali

Brocade ICX6610-48P-E 48-port PoE+ Gigabit Ethernet Switch 8x 10GbE Dual PSU | eBay

2xPower Cord. We stand behind our products. The listed prices do not apply. Reset to default.

www.ebay.com

 

[eptesicus]

> conf t
> ip address x.x.x.x/24

That will be the management IP reachable via any of the ports with a default PVID. You just have to assign your laptop a static IP in the management range since the DHCP server won't be active by default.

Finally able to circle back on this. Doing just that, the command is invalid. 'ip' is valid, but the 'address' command is not.

 

[EngChiSTH]

today I came across this post, and got excited. thanks to OP who provides well written info on each model.
I went to ebay, and wow. Is the inflation really that bad over past 5 years? I can barely see anything below $200, phew

where are you located? US-IL here and if you are local, I have 6450-24 and 6450-48 I am not using (upgraded to 7250 as I wanted for fiber) . local pickup, price you consider reasonable, and it is yours as they are 'backup' for me anyway. I rather not deal with shipping as these are bulky/heavy. both models are non-P.

 

[wittyet]

where are you located? US-IL here and if you are local, I have 6450-24 and 6450-48 I am not using (upgraded to 7250 as I wanted for fiber) . local pickup, price you consider reasonable, and it is yours as they are 'backup' for me anyway. I rather not deal with shipping as these are bulky/heavy. both models are non-P.

I wish I lived in the US, but I am in Canada, and I agree enterprise switches are really heavy, I am currently playing with two HPE, I guess I am keeping it this way for now, thanks for the offer

 

[NablaSquaredG]

Just wanted to add my 2 cents on how much power ICX6450-24P (PoE models) draw when idle and with nothing but power connected.

I tested three ICX6450-24P (using a Shelly Plus 1PM), and all three consistently reported the same values:

25 W power draw during boot (when the two fans run at full speed)
22 W power draw after boot (when the two fans drop to low speed)
35 W power draw after a few seconds when the PoE power supply is initialized

So it seems the PoE power supply adds ~13 W on top of the idle power without PoE.

@fohdeesha I measure 25W with ICX6450-24 (non-P) in IDLE. Can you please correct this in the start post?

Edit, more power consumption:

ICX7450-48, 1 PSU, 1 Fan, 2x40G module (ICX7400-1x40GQ), 1x 4x10 module (ICX7400-4x10GF), 230V: 70W IDLE (no ports connected)
ICX7650-48ZP-E2, 1 PSU connected, 2 Fans, 230W: 120W IDLE (no ports connected)

 

[deeceesth]

one of my ICX725024P switches stopped supplying POE power. The switch still runs, just without POE to my APs. I had to put in an injector in between to get my APs back up. All other function appears to be the same.

where should I start with trying to fix this? Is there a separate POE PSU inside? or do I just need to replace the entire switch?

 

[Bluerai]

Hey everyone, I’m running out of ideas to try and I’d love some input from folks smarter than myself so here we go:

I have a 6450-48p that I have reset and followed the guides on, etc (all amazing by the way, and so is this thread). Using two of the suggested MikroTik copper SFP+ modules I can run a speed test to my computer and hit around 1850Mbps as expected. However when I run any sort of speed test on the standard switch ports, my speeds are roughly 400Mbps.

Am I mistakenly under the impression that I should be seeing closer to 940Mbps on the rest of the switch ports? I’ve messed with flow control, duplex settings, a ton of Google searching and I’m not getting very far. It looks like the switch doesn’t recognize the SFP+ modules and they show empty, and I can see that the phy device never initializes from the console. Yet it will tell me when I unplug the module and they’re clearly working.

Hopefully this isn’t functioning as expected, but I’m not sure what to try next. I’ve only tried with two of the SFP ports, unplugged a module and left one in, and still haven’t seen any changes.

Any ideas or direction for next steps would be appreciated!

Bumping this, anyone have any ideas?
I do have a donor switch that I couldn’t putty into so I’m thinking I could swap parts around if need be..

 

[NablaSquaredG]

one of my ICX725024P switches stopped supplying POE power. The switch still runs, just without POE to my APs. I had to put in an injector in between to get my APs back up. All other function appears to be the same.

where should I start with trying to fix this? Is there a separate POE PSU inside? or do I just need to replace the entire switch?

Maybe check show inline power detail?