Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

[snoturtle]

Hi all

I am having an fan issue with my icx6610 and I can't figure out what is causing it.
Every 2-3 minutes the fans ramp up ~30 seconds, not to full blast but up higher than normal.
The highest temp I have seen listed is 41.5C which should be well below temp to start ramping up fans

Any idea what might be causing this?

 

[fohdeesha]

What temp are you looking at? The fan ramps are based on the "Fan controlled temperature:" you see in show chassis output

For instance on one of my stacks:

Fan controlled temperature: 71.5 deg-C

Fan speed switching temperature thresholds:
                Speed 1: NM<----->84       deg-C
                Speed 2:       79<-----> 87 deg-C (shutdown)

as you can see, it's about 13 degrees from ramping up

 

[snoturtle]

Power supply 1 (AC - Regular) present, status ok
        Model Number:   23-0000143-01
        Serial Number:  05N
        Firmware Ver:    A
Power supply 1 Fan Air Flow Direction:  Back to Front
Power supply 2 (AC - Regular) present, status ok
        Model Number:   23-0000143-01
        Serial Number:  01F
        Firmware Ver:    A
Power supply 2 Fan Air Flow Direction:  Back to Front

Fan 1 ok, speed (auto): [[1]]<->2
Fan 2 ok, speed (auto): [[1]]<->2

Fan controlled temperature: 55.5 deg-C

Fan speed switching temperature thresholds:
                Speed 1: NM<----->68       deg-C
                Speed 2:       63<-----> 73 deg-C (shutdown)

Fan 1 Air Flow Direction:  Back to Front
Fan 2 Air Flow Direction:  Back to Front
MAC 1 Temperature Readings:
        Current temperature : 43.0 deg-C
CPU Temperature Readings:
        Current temperature : 55.5 deg-C
sensor A Temperature Readings:
        Current temperature : 35.0 deg-C
sensor B Temperature Readings:
        Current temperature : 48.5 deg-C
sensor C Temperature Readings:
        Current temperature : 30.0 deg-C
sensor D Temperature Readings:
        Current temperature : 30.5 deg-C
stacking card Temperature Readings:
        Current temperature : 48.0 deg-C
        Warning level.......: 70.0 deg-C
        Shutdown level......: 73.0 deg-C

 

[fohdeesha]

are you on the latest firmware and bootloader?

Yours is stating it ramps up sooner than mine (63 degrees), probably because you have 2 fan trays. You're only 8 degrees away from that. I'm assuming it's getting to 63 and ramping up, then getting back down to 55 or so you see now

I would try and catch it right when it ramps up, then run "show chassis" - see what the fan controlled temp is at, and check the fan status section and make sure it says:

Fan 1 ok, speed (auto): 1<->[[2]]
Fan 2 ok, speed (auto): 1<->[[2]]

indicating it is indeed spinning them up to level 2. If it's ramping up and getting louder but it still thinks they're on speed 1, then you might have something weird going on

 

[snoturtle]

This is from when it started to ramp up.

Should be on the latest firmware but I do have an invalid license listed not sure if that is an issue.


When I purchased it the listing stated that Power Supply 2 Fan has failed in Unit 1, but I see no indication of that.

The stack unit 1 chassis info:

Power supply 1 (AC - Regular) present, status ok
        Model Number:   23-0000143-01
        Serial Number:  05N
        Firmware Ver:    A
Power supply 1 Fan Air Flow Direction:  Back to Front
Power supply 2 (AC - Regular) present, status ok
        Model Number:   23-0000143-01
        Serial Number:  01F
        Firmware Ver:    A
Power supply 2 Fan Air Flow Direction:  Back to Front

Fan 1 ok, speed (auto): [[1]]<->2
Fan 2 ok, speed (auto): [[1]]<->2

Fan controlled temperature: 58.0 deg-C

Fan speed switching temperature thresholds:
                Speed 1: NM<----->68       deg-C
                Speed 2:       63<-----> 73 deg-C (shutdown)

Fan 1 Air Flow Direction:  Back to Front
Fan 2 Air Flow Direction:  Back to Front
MAC 1 Temperature Readings:
        Current temperature : 44.0 deg-C
CPU Temperature Readings:
        Current temperature : 57.5 deg-C
sensor A Temperature Readings:
        Current temperature : 36.0 deg-C
sensor B Temperature Readings:
        Current temperature : 50.0 deg-C
sensor C Temperature Readings:
        Current temperature : 31.0 deg-C
sensor D Temperature Readings:
        Current temperature : 31.0 deg-C
stacking card Temperature Readings:
        Current temperature : 48.5 deg-C
        Warning level.......: 70.0 deg-C
        Shutdown level......: 73.0 deg-C
Boot Prom MAC : 748e.f893.15c8
Management MAC: 748e.f893.15c8

are you on the latest firmware and bootloader?

Yours is stating it ramps up sooner than mine (63 degrees), probably because you have 2 fan trays. You're only 8 degrees away from that. I'm assuming it's getting to 63 and ramping up, then getting back down to 55 or so you see now

I would try and catch it right when it ramps up, then run "show chassis" - see what the fan controlled temp is at, and check the fan status section and make sure it says:

Fan 1 ok, speed (auto): 1<->[[2]]
Fan 2 ok, speed (auto): 1<->[[2]]

indicating it is indeed spinning them up to level 2. If it's ramping up and getting louder but it still thinks they're on speed 1, then you might have something weird going on

 

[fohdeesha]

can you tell if it's the fan trays ramping up or the power supplies? it says the fan trays are still on speed 1, so it might be a power supply thing. The Rev A's were odd, they might have their own fan speed curves, wouldn't surprise me

I highly doubt invalid licenses would do it, but if you send me the output of "show license" and "show version" I can send you valid licenses (if I haven't already) and you can delete the invalid ones

 

[snoturtle]

I think you are onto something with the power supplies.

Pulled the rack out so I could get behind it and watch the fans.

When it is quiet the fans on the psu's are not moving, when it ramps up they both start turning.

Almost seems there is a separate setting for the psu than the chassis.

can you tell if it's the fan trays ramping up or the power supplies? it says the fan trays are still on speed 1, so it might be a power supply thing. The Rev A's were odd, they might have their own fan speed curves, wouldn't surprise me

I highly doubt invalid licenses would do it, but if you send me the output of "show license" and "show version" I can send you valid licenses (if I haven't already) and you can delete the invalid ones

 

[fohdeesha]

well that's odd, I've never seen the PSU fans stop completely. That must be a behavior that only happens with 2 fan trays. If you'd like, pull one of the fan trays out, power it off, then power it back up fresh again, see if the PSU's run continuously (but quietly) and stop ramping up in bursts

 

[fohdeesha]

A new firmware for this series was released, v8030sa - a very tiny bugfix update, addressing a single bug that only happens in FIPS security mode, which I'd imagine nobody here is using. Such a small release they didn't even increment the end letter, they just tacked an "a" on the end, never seen them do that:

I know some like to be on the bleeding edge latest firmware anyway, so I've added it to the guide.

also, an important message from paul simon:

 

[snoturtle]

Tried pulling each fan tray and swapping them around with only one installed and still see the same behavior on the psu fans.

I will just deal with it as my concern was that something was wrong but it seems this is normal behavior for the one I got.

Thanks for your help

well that's odd, I've never seen the PSU fans stop completely. That must be a behavior that only happens with 2 fan trays. If you'd like, pull one of the fan trays out, power it off, then power it back up fresh again, see if the PSU's run continuously (but quietly) and stop ramping up in bursts

 

[neb50]

ICX 6450-48P power consumption is 52W with no ports connected on a kill-a-watt meter.
I will update this post with PoE devices after I swap my switches.

I have the 6450-48P and it is also around 50W with 2 SFP+ modules installed and a couple 1G ports active. Adding POE devices seems to only increase by the power draw of the POE device. It is about 2x the power of the 24 port 6450 and about 2x the power of the TPLink 1700-28 which is another 24 port 1G and 4 port SFP+.

its definitely the brocade thats wired weird. I replaced the nidec in a 6450 with a sunon and had to switch the black and yellow leads. Works great now and is completely silent.

I have a fan or 2 in that are making some noise, can you let me know which fan you used as a replacement?

 

[roberth58]

I have the 6450-48P and it is also around 50W with 2 SFP+ modules installed and a couple 1G ports active. Adding POE devices seems to only increase by the power draw of the POE device. It is about 2x the power of the 24 port 6450 and about 2x the power of the TPLink 1700-28 which is another 24 port 1G and 4 port SFP+.



I have a fan or 2 in that are making some noise, can you let me know which fan you used as a replacement?

I used a Model KDE1204PKVX and a KDE1204PKV3.MS.AR.GN, 10cfm or 6 cfm. available on amazon. the 6450 is made for 2 fans so I drilled out the cover for the 2nd fan and mounted 2 of the 10cfm versions for the 2 in the rack, and put 2 of the 6cfm in the 6450 in my familyroom. doubling up on the fans is overkill but I had them so why not.

 

[sean]

I used a Model KDE1204PKVX and a KDE1204PKV3.MS.AR.GN, 10cfm or 6 cfm. available on amazon. the 6450 is made for 2 fans so I drilled out the cover for the 2nd fan and mounted 2 of the 10cfm versions for the 2 in the rack, and put 2 of the 6cfm in the 6450 in my familyroom. doubling up on the fans is overkill but I had them so why not.

I used the KDE1204PKVX as well. They're 10 CFM instead of the ~21 CFM for the fans with the switch. My 6450-48P has three spots for fans so I replaced all three. I too had to flip the black and yellow wires. The temps rose about 8-10 degrees Celsius but they're hovering at 45 °C, well short of the 56 °C needed for fan speed "2".

 

[Thomas H]

My PoE is not working. I tried plugging in some Hikvision PoE cameras and nada. No LED lights. Tried different ports and is the same. It tests fine on my Cisco SG300-28P switch. I have the 6450-48P model and followed the setup guide include PoE update firmware to completion. Is there something else I need to configure?

 

[neb50]

My PoE is not working. I tried plugging in some Hikvision PoE cameras and nada. No LED lights. Tried different ports and is the same. Tested fine on my Cisco SG300-28P switch. I have the 6450-48P model and followed the setup guide include PoE update firmware.

The POE is disabled by default and you need to enable it on the ports that you want/need it on. I just set mine to have all ports enabled at Class 2 or 3 levels and it seems to work fine.

Setting it to have all ports enabled at Class 4 without the external supply will give you messages on the console at boot about ports being disabled due to power budget, but I think they would eventually get auto-enabled once the switch figured out that there wasn't enough active ports on the ones that were initially enabled and the POE budget was not getting exceeded.

 

[Thomas H]

The POE is disabled by default and you need to enable it on the ports that you want/need it on. I just set mine to have all ports enabled at Class 2 or 3 levels and it seems to work fine.

Setting it to have all ports enabled at Class 4 without the external supply will give you messages on the console at boot about ports being disabled due to power budget, but I think they would eventually get auto-enabled once the switch figured out that there wasn't enough active ports on the ones that were initially enabled and the POE budget was not getting exceeded.

How do I enable PoE?

 

[neb50]

How do I enable PoE?

I just enabled the web management per the guide and then set it up using the web interface.

 

[mb300sd]

How do I enable PoE?

interface ethernet 1/1/x
inline power

 

[Thomas H]

I just enabled the web management per the guide and then set it up using the web interface.

interface ethernet 1/1/x
inline power

Got it working, thanks! Is it bad to enable PoE on all ports?

 

[Thomas H]

How do I enable https for web management?