Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

[noduck]

I got a ICX7250-24p, and followed all the configuration steps. Upon reload, the switch complains of a PoE failure: "Failed Devices Bitmap on PoE module:4, numDev:3". I have tested all 24 ports, and they all provide power, but for the final 8 ports "show poe" doesn't show any information, it just says "Non-PD".

I found an earlier mention of the same error (and same ports), but I was not able to see any physical damage on the board (also, the PoE board is a later rev). See reference below.

I am okay having only 16 usable PoE ports and disabling power on ports 17 through 24. Does anybody have any insight if it is safe to operate the switch with this fault?

PoE Info: PoE module 1 of Unit 1 on ports 1/1/1 to 1/1/24 detected. Initializing....
2000 log entries of PoE Event Trace Log Buffer is allocated on unit 1 for unit 1 
PoE Event Trace Logging enabled for unit 1
PoE Error: Failed Devices Bitmap on PoE module:4, numDev:3.
Resetting module in slot 1 again to recover from dev fault
PoE Info: Hard Resetting in slot 1....
PoE Info: Resetting module in slot 1....completed.
PoE Error: Failed Devices Bitmap on PoE module:4, numDev:3.
PoE Severe Error: Hardware Fault with ports 1/1/17 to 1/1/24. Remove PDs and then configure "no inline power" on these ports.
PoE Info: PoE module 1 of Unit 1 initialization is done.

I have an ICX7250-24P and ran through the updates and I think something is wrong with PoE on it. whenever I try to install the PoE firmware I get this message:

 

[Jason Antes]

40G QSFP+ Breakout Active Optical Cables (AOC) and 40G QSFP+ to SFP+ Breakout DAC Cables are a good place to start looking. ICX will take practically any module and FS can code it for Brocade anyway.

I've been looking at doing a very short AOC breakout to a patch panel.

I have NIB Brocade AOC ones still if anyone is interested. Everyone wanted the QSFP to SFP+ DACs I had but no love for AOC.

 

[infoMatt]

I am okay having only 16 usable PoE ports and disabling power on ports 17 through 24. Does anybody have any insight if it is safe to operate the switch with this fault?

Great question... the answer is, unfortunately, "who knows?"... I mean, teoretically there are no problems if and only if the damaged chips can still distinguish correctly a PD from a non-PD, otherwise there is a risk of having 50ish volts applied to say a computer...

 

[ptibeur]

Hi all

I finally received an ICX- 6450-48 and started to work on the cooling/silence topic. My desk room pretty silent, I can hear the hard drive rotation of WD RE drives, to give a benchmark of "quietness" of the room.

First weird thing I noticed when I plugged the switch, is that the single stock fan never seems to slow down after booting. Comparatively, I played with an ICX 7250-24P at work and on this model, after a few minutes, the 3 screaming fans eventually slow down to a more reasonnable level.

Is it normal, by design, on the 6450 ? Could it be a bug in software version 08.0.30kT311 ?
I measured 11.60 Volts at the fan plug, yet the switch claims to be on speed 1 / low speed. I wonder what would happen with higher speed 2

To replace the single noisy fan, I made a mistake ordering 3x 40mm fans... with only 2 wires Sunon EE40201S2-1000U-999, rated 7.7 CFM.
Switch says "fan failed", naturally, but still boots. At the end of the day, the fans are not so silent either, whether I plug the 3 fans, or only 1. Maybe because of the 6000 RPM rotation speed ?
I'll go to the hardware store get some tools (hole saw) to cut additionnal holes for the 2 new fans. If I can't make the switch way more silent than this, I will have to take more aggressive counter-measures to kill the noise

A few "show" commands before modifying the cooling :

ICX6450-48 Switch#show chassis
The stack unit 1 chassis info:

Power supply 1 (NA - AC - Regular) present, status ok
Power supply 2 not present

Fan ok, speed (auto): [[1]]<->2

Fan speed switching temperature thresholds:
1 -> 2 @ 69 deg-C
1 <- 2 @ 64 deg-C

Sensor B Temperature Readings:
Current temperature : 38.0 deg-C
Sensor A Temperature Readings:
Current temperature : 43.5 deg-C
Warning level.......: 66.0 deg-C
Shutdown level......: 76.0 deg-C
Boot Prom MAC : cc4e.2473.ad40
Management MAC: cc4e.2473.ad40
ICX6450-48 Switch#

ICX6450-48 Switch#dm fan-speed
All fans in Low speed (0xff)
ICX6450-48 Switch#

ICX6450-48 Switch#show version
Copyright (c) 1996-2016 Brocade Communications Systems, Inc. All rights reserved.
UNIT 1: compiled on Oct 18 2016 at 05:31:09 labeled as ICX64S08030k
(8526668 bytes) from Primary ICX64S08030k.bin
SW: Version 08.0.30kT311
Boot-Monitor Image size = 786944, Version:10.1.05T310 (kxz10105)
HW: Stackable ICX6450-48
==========================================================================
UNIT 1: SL 1: ICX6450-48 48-port Management Module
Serial #: BZU3229K0BK
License: BASE_SOFT_PACKAGE (LID: dbwIHHOmFdm)
P-ENGINE 0: type DEF0, rev 01
P-ENGINE 1: type DEF0, rev 01
==========================================================================
UNIT 1: SL 2: ICX6450-SFP-Plus 4port 40G Module
==========================================================================
800 MHz ARM processor ARMv5TE, 400 MHz bus
65536 KB flash memory
512 MB DRAM
STACKID 1 system uptime is 10 minute(s) 34 second(s)
The system : started=cold start

ICX6450-48 Switch#


Adventure goes on

 

[noduck]

Great question... the answer is, unfortunately, "who knows?"... I mean, teoretically there are no problems if and only if the damaged chips can still distinguish correctly a PD from a non-PD, otherwise there is a risk of having 50ish volts applied to say a computer...

I had not considered that. It seems to be a valid concern, even with "no inline power", the port still powers on a connected PoE device.

 

[noduck]

Looking for multi-gig connectivity for 7250, I tried some of the SFP+ that I had lying around.
-I am not able to get 10G using S+RJ10; it is recognized as SFP (not plus), so only 1G
-Wiitek SFP+ module (see review on STH) do work at 10G. Further, they also linked at 2.5G and 5G ("OEM")

ICX7250-24P Router#show media ethernet 1/2/3
Port   1/2/3: Type  : 1GE M-SX(SFP)
             Vendor: MikroTik           Version: 2.07
             Part# : S+RJ10             Serial#: 9C5C031B2A68 
ICX7250-24P Router#show media ethernet 1/2/8
Port   1/2/8: Type  : 10GE SR 300m (SFP+)
             Vendor: OEM                Version: 1 
             Part# : SFP-10G-T          Serial#: WAMZ01023020

 

[Frnot]

First weird thing I noticed when I plugged the switch, is that the single stock fan never seems to slow down after booting.

I have the same issue. I'm running firmware version 08.0.30tT313. If it's a software bug, it is not patched in later versions.

It may be of interest to you that I have operated my switch for a day with the fan completely unplugged. It is sandwiched between two (relatively low powered) machines in my rack. The highest temperature I have noticed is 66C on Sensor A. It's little warm, but about 10 degrees below the shutdown level.
I have yet to install any 10Gig modules. While I anticipate they could raise the temperature a non insignificant amount, I have some hopes that this switch can be ran indefinitely with no fan.

A possible alternative is to splice a 100 Ohm resistor into the fan cable. At ~4.5V the fan is still louder than I prefer (completely silent), but it is a big improvement over stock. I don't have any heat shrink handy or I would perform this mod and call it a day.

 

[ICXGURU]

Looking for multi-gig connectivity for 7250, I tried some of the SFP+ that I had lying around.
-I am not able to get 10G using S+RJ10; it is recognized as SFP (not plus), so only 1G
-Wiitek SFP+ module (see review on STH) do work at 10G. Further, they also linked at 2.5G and 5G ("OEM")

ICX7250-24P Router#show media ethernet 1/2/3
Port   1/2/3: Type  : 1GE M-SX(SFP)
             Vendor: MikroTik           Version: 2.07
             Part# : S+RJ10             Serial#: 9C5C031B2A68
ICX7250-24P Router#show media ethernet 1/2/8
Port   1/2/8: Type  : 10GE SR 300m (SFP+)
             Vendor: OEM                Version: 1
             Part# : SFP-10G-T          Serial#: WAMZ01023020

That is interesting about the Wiitek linking at 2.5 and 5Gig. What did the 7250 say the link speed was in the show interface? I would think it reported 10gig even if the other end was 2.5 or 5 since those ASICs are not multigig.

 

[ptibeur]

Thank you Frnot for confirming you also have this bug, it's too bad it was not fixed in a later version... or maybe the latest one ? I'll try it and post some news here.

I'm not fan (haha) of the idea of running the switch completely fanless, I would be afraid it would get really too hot over time and would die, or even worse (fire ?). A quiet air flow for a nearly idle switch sounds like a good trade-off to me.

The 100 ohm trick is nice ! I tested the fans on a breadboard before mounting them and they work at 5V and even at 3.3V. 4.5V should be nearly inaudible. Now I gotta check where I can find some resistors in my lost city haha.

About the 10G ports, I don't think they're gonna bring so much additional heat : they are at the opposite corner of the single fan. It would have been more logical design to put them closer to the fan evacuating the heat if they would generate a lot of heat.

 

[fohdeesha]

Thank you Frnot for confirming you also have this bug, it's too bad it was not fixed in a later version... or maybe the latest one ? I'll try it and post some news here.

I'm not fan (haha) of the idea of running the switch completely fanless, I would be afraid it would get really too hot over time and would die, or even worse (fire ?). A quiet air flow for a nearly idle switch sounds like a good trade-off to me.

The 100 ohm trick is nice ! I tested the fans on a breadboard before mounting them and they work at 5V and even at 3.3V. 4.5V should be nearly inaudible. Now I gotta check where I can find some resistors in my lost city haha.

About the 10G ports, I don't think they're gonna bring so much additional heat : they are at the opposite corner of the single fan. It would have been more logical design to put them closer to the fan evacuating the heat if they would generate a lot of heat.

it's not a bug, the PWM circuitry that controls the fan voltage is fried on your switch. That's why the OS says it's sending speed 1, but you measured a full 12v

 

[fohdeesha]

OK. I've been through the thread and can't find an answer. I apologize if I've missed it.
What breakout cables do you use for the QSFP+ ports if you are running singlemode fiber? I see fiber channel ones used, I see mention of 850nm, but can't find any mention of single mode breakout cables.

pick up an 40gbase-lr4 optic off of ebay for ~75 bucks, then a singlemode MTP/MPO to LC breakout harness

 

[Frnot]

The 100 ohm trick is nice ! I tested the fans on a breadboard before mounting them and they work at 5V and even at 3.3V. 4.5V should be nearly inaudible. Now I gotta check where I can find some resistors in my lost city haha.

Before you wire anything permanantly, you should make sure that the fans can start up at the voltage you choose. I was able to get the fan to run at about 3.5V, but it didn't have enough force to overcome the initial inertia when the fans were stopped. I could only get the fan to consistently start at 4.5V. I think this might be one of the reasons the fans usually start on 100% and are ramped down later.

About the 10G ports, I don't think they're gonna bring so much additional heat : they are at the opposite corner of the single fan. It would have been more logical design to put them closer to the fan evacuating the heat if they would generate a lot of heat.

Not all engineers think logically. (or there may be more important reasons for the placement) I would do some tests before you throw the switch in your rack permanently.

it's not a bug, the PWM circuitry that controls the fan voltage is fried on your switch. That's why the OS says it's sending speed 1, but you measured a full 12v

I'm having the same issue. Is it common for this circuitry to be fried? How does this happen? And why does the "dm fan-speed" say speed 1 but is sending the signal 0xff? Is that not 100%?

I don't know the full history of my switch, but I was led to believe that it was pulled from a rack in the enterprise. It hasn't been tinkered with by any homelabbers (except for me).

 

[ptibeur]

it's not a bug, the PWM circuitry that controls the fan voltage is fried on your switch. That's why the OS says it's sending speed 1, but you measured a full 12v

Oooh, that's interesting, is it a known issue ? Is it fixable or replaceable in some way ? I might be able to replace a faulty component with a soldering iron if it's more or less accessible

Before you wire anything permanantly, you should make sure that the fans can start up at the voltage you choose. I was able to get the fan to run at about 3.5V, but it didn't have enough force to overcome the initial inertia when the fans were stopped. I could only get the fan to consistently start at 4.5V. I think this might be one of the reasons the fans usually start on 100% and are ramped down later.



Not all engineers think logically. (or there may be more important reasons for the placement) I would do some tests before you throw the switch in your rack permanently.

Yes I tested on a breadboard, the fans can start from full-stop with 3.3V applied, thus there should not be a problem if I feed them 4.5V. My preference would go towards fixing the fan control module in priority, if feasible.

I'll keep an eye on the temperatures yes, I'm still waiting to get the 10 Gbps NICs, I already got the SFP+ SR optics though.

 

[noduck]

That is interesting about the Wiitek linking at 2.5 and 5Gig. What did the 7250 say the link speed was in the show interface? I would think it reported 10gig even if the other end was 2.5 or 5 since those ASICs are not multigig.

The other end is a MS510TX on a 2.5G port:

10GigabitEthernet1/2/8 is up, line protocol is up
  Port up for 1 minute(s) 9 second(s)
  Hardware is 10GigabitEthernet, address is xxx
  Configured speed 10Gbit, actual 10Gbit, configured duplex fdx, actual fdx

 

[ICXGURU]

The other end is a MS510TX on a 2.5G port:

10GigabitEthernet1/2/8 is up, line protocol is up
  Port up for 1 minute(s) 9 second(s)
  Hardware is 10GigabitEthernet, address is xxx
  Configured speed 10Gbit, actual 10Gbit, configured duplex fdx, actual fdx

The issue you may have with that is the 7250 thinks it can send 10Gbps of traffic out that interface and the transceiver will drop up to 50% or 75% of the traffic since the link cannot send the traffic load. Normally flow control would catch that but in this case the remote end is not receiving more than it can handle since the packets will be dropped at the source interface, so it will never send flow control pause to slow it down. That means it's up to TCP to fix the problem with resends and the problem gets even worse.

 

[noduck]

I have replaced the stock fans in 7250-24p with two "Sunon 40x40x20mm 3 Pin Fan MB40201VX-000U", and the noise is much improved. At speed 1 it is barely audible, but speed 2 is still very obvious (but definitely improved from the screeching stock fans). Furthermore, the fans cycle back and forth between 1 and 2, because of the sensor 1 temperature (MGMT THERMAL PLANE), which cycles between ~85C and ~95C.

Hopefully somebody can provide insight into the following:

1. Is it normal to have the sensor 1 temperature this high?
2. Are there better (more quiet) fans to consider?
3. Would it help to place a (larger) fan over the ASIC (not sure if there is enough clearance)?

This the final topic that I am investigating on this 7250-24p to determine if it is fit for my purpose. The other two topics are the PoE ports (failed board) and multi-Gig (success with Wiitek). Hopefully I can figure out if I should look for a fully functional 7250, or just return it.

Logs:

Sep 25 19:22:58:W:System:Stack unit 1 Fan speed changed automatically to 1 
Sep 25 19:21:53:W:System:Stack unit 1 Fan speed changed automatically to 2 
Sep 25 19:18:23:W:System:Stack unit 1 Fan speed changed automatically to 1 
Sep 25 19:17:17:W:System:Stack unit 1 Fan speed changed automatically to 2 
Sep 25 19:13:47:W:System:Stack unit 1 Fan speed changed automatically to 1 
Sep 25 19:12:42:W:System:Stack unit 1 Fan speed changed automatically to 2 
Sep 25 19:09:07:W:System:Stack unit 1 Fan speed changed automatically to 1

show chassis

Fan controlled temperature: 
        Rule 1/2 (MGMT THERMAL PLANE): 91.8 deg-C
        Rule 2/2 (AIR OUTLET NEAR PSU): 46.5 deg-C

Fan speed switching temperature thresholds:
        Rule 1/2 (MGMT THERMAL PLANE):
                Speed 1: NM<-----> 93       deg-C
                Speed 2:        82<----->105 deg-C (shutdown)
        Rule 2/2 (AIR OUTLET NEAR PSU):
                Speed 1: NM<-----> 58       deg-C
                Speed 2:        49<----->105 deg-C (shutdown)

 

[noduck]

The issue you may have with that is the 7250 thinks it can send 10Gbps of traffic out that interface and the transceiver will drop up to 50% or 75% of the traffic since the link cannot send the traffic load. Normally flow control would catch that but in this case the remote end is not receiving more than it can handle since the packets will be dropped at the source interface, so it will never send flow control pause to slow it down. That means it's up to TCP to fix the problem with resends and the problem gets even worse.

Hmm... sounds like a valid concern. I haven't looked at all the configuration options yet, but would it be possible to address the issue with some kind of rate limiting?

 

[infoMatt]

Is it normal to have the sensor 1 temperature this high?

No, it's definitely not normal unless you are in a 50+°C room...
This is the problem of the "quieter" fans, they can move a lot less air, or with a lot less pressure, both resulting in reduced airflow over the heatsink than the designed and higher temperatures/lower useful life expectancy for the component.
You have to decide if the downside balances with the lower noise or not.

 

[dswartz]

Switches came in a couple of hours ago. Had to fumble through a bit, due to needing to upgrade OS from 2016. All good now - 2 node linear stack up and running. Off to the races!

 

[Freebsd1976]

a simple question about os version on icx 7250-24p , stay 8080f or upgrade 8092d or 8095?