Drag to reposition cover

Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

infoMatt

Active Member
Apr 16, 2019
222
100
43
I am okay having only 16 usable PoE ports and disabling power on ports 17 through 24. Does anybody have any insight if it is safe to operate the switch with this fault?
Great question... the answer is, unfortunately, "who knows?"... I mean, teoretically there are no problems if and only if the damaged chips can still distinguish correctly a PD from a non-PD, otherwise there is a risk of having 50ish volts applied to say a computer...
 

ptibeur

New Member
Sep 4, 2020
5
0
1
Hi all :)

I finally received an ICX- 6450-48 and started to work on the cooling/silence topic. My desk room pretty silent, I can hear the hard drive rotation of WD RE drives, to give a benchmark of "quietness" of the room.

First weird thing I noticed when I plugged the switch, is that the single stock fan never seems to slow down after booting. Comparatively, I played with an ICX 7250-24P at work and on this model, after a few minutes, the 3 screaming fans eventually slow down to a more reasonnable level.

Is it normal, by design, on the 6450 ? Could it be a bug in software version 08.0.30kT311 ?
I measured 11.60 Volts at the fan plug, yet the switch claims to be on speed 1 / low speed. I wonder what would happen with higher speed 2 :oops:

To replace the single noisy fan, I made a mistake ordering 3x 40mm fans... with only 2 wires :( Sunon EE40201S2-1000U-999, rated 7.7 CFM.
Switch says "fan failed", naturally, but still boots. At the end of the day, the fans are not so silent either, whether I plug the 3 fans, or only 1. Maybe because of the 6000 RPM rotation speed ?
I'll go to the hardware store get some tools (hole saw) to cut additionnal holes for the 2 new fans. If I can't make the switch way more silent than this, I will have to take more aggressive counter-measures to kill the noise :p

A few "show" commands before modifying the cooling :

ICX6450-48 Switch#show chassis
The stack unit 1 chassis info:

Power supply 1 (NA - AC - Regular) present, status ok
Power supply 2 not present

Fan ok, speed (auto): [[1]]<->2

Fan speed switching temperature thresholds:
1 -> 2 @ 69 deg-C
1 <- 2 @ 64 deg-C

Sensor B Temperature Readings:
Current temperature : 38.0 deg-C
Sensor A Temperature Readings:
Current temperature : 43.5 deg-C
Warning level.......: 66.0 deg-C
Shutdown level......: 76.0 deg-C
Boot Prom MAC : cc4e.2473.ad40
Management MAC: cc4e.2473.ad40
ICX6450-48 Switch#

ICX6450-48 Switch#dm fan-speed
All fans in Low speed (0xff)
ICX6450-48 Switch#

ICX6450-48 Switch#show version
Copyright (c) 1996-2016 Brocade Communications Systems, Inc. All rights reserved.
UNIT 1: compiled on Oct 18 2016 at 05:31:09 labeled as ICX64S08030k
(8526668 bytes) from Primary ICX64S08030k.bin
SW: Version 08.0.30kT311
Boot-Monitor Image size = 786944, Version:10.1.05T310 (kxz10105)
HW: Stackable ICX6450-48
==========================================================================
UNIT 1: SL 1: ICX6450-48 48-port Management Module
Serial #: BZU3229K0BK
License: BASE_SOFT_PACKAGE (LID: dbwIHHOmFdm)
P-ENGINE 0: type DEF0, rev 01
P-ENGINE 1: type DEF0, rev 01
==========================================================================
UNIT 1: SL 2: ICX6450-SFP-Plus 4port 40G Module
==========================================================================
800 MHz ARM processor ARMv5TE, 400 MHz bus
65536 KB flash memory
512 MB DRAM
STACKID 1 system uptime is 10 minute(s) 34 second(s)
The system : started=cold start

ICX6450-48 Switch#


Adventure goes on ;)
 

noduck

Member
Sep 12, 2020
41
13
8
Great question... the answer is, unfortunately, "who knows?"... I mean, teoretically there are no problems if and only if the damaged chips can still distinguish correctly a PD from a non-PD, otherwise there is a risk of having 50ish volts applied to say a computer...
I had not considered that. It seems to be a valid concern, even with "no inline power", the port still powers on a connected PoE device.
 

noduck

Member
Sep 12, 2020
41
13
8
Looking for multi-gig connectivity for 7250, I tried some of the SFP+ that I had lying around.
-I am not able to get 10G using S+RJ10; it is recognized as SFP (not plus), so only 1G
-Wiitek SFP+ module (see review on STH) do work at 10G. Further, they also linked at 2.5G and 5G ("OEM")

Code:
ICX7250-24P Router#show media ethernet 1/2/3
Port   1/2/3: Type  : 1GE M-SX(SFP)
             Vendor: MikroTik           Version: 2.07
             Part# : S+RJ10             Serial#: 9C5C031B2A68 
ICX7250-24P Router#show media ethernet 1/2/8
Port   1/2/8: Type  : 10GE SR 300m (SFP+)
             Vendor: OEM                Version: 1 
             Part# : SFP-10G-T          Serial#: WAMZ01023020
 

Frnot

New Member
Sep 23, 2020
7
1
3
First weird thing I noticed when I plugged the switch, is that the single stock fan never seems to slow down after booting.
I have the same issue. I'm running firmware version 08.0.30tT313. If it's a software bug, it is not patched in later versions.

It may be of interest to you that I have operated my switch for a day with the fan completely unplugged. It is sandwiched between two (relatively low powered) machines in my rack. The highest temperature I have noticed is 66C on Sensor A. It's little warm, but about 10 degrees below the shutdown level.
I have yet to install any 10Gig modules. While I anticipate they could raise the temperature a non insignificant amount, I have some hopes that this switch can be ran indefinitely with no fan.

A possible alternative is to splice a 100 Ohm resistor into the fan cable. At ~4.5V the fan is still louder than I prefer (completely silent), but it is a big improvement over stock. I don't have any heat shrink handy or I would perform this mod and call it a day.
 
  • Like
Reactions: ptibeur

ICXGURU

Member
Jun 22, 2020
37
16
8
Looking for multi-gig connectivity for 7250, I tried some of the SFP+ that I had lying around.
-I am not able to get 10G using S+RJ10; it is recognized as SFP (not plus), so only 1G
-Wiitek SFP+ module (see review on STH) do work at 10G. Further, they also linked at 2.5G and 5G ("OEM")

Code:
ICX7250-24P Router#show media ethernet 1/2/3
Port   1/2/3: Type  : 1GE M-SX(SFP)
             Vendor: MikroTik           Version: 2.07
             Part# : S+RJ10             Serial#: 9C5C031B2A68
ICX7250-24P Router#show media ethernet 1/2/8
Port   1/2/8: Type  : 10GE SR 300m (SFP+)
             Vendor: OEM                Version: 1
             Part# : SFP-10G-T          Serial#: WAMZ01023020
That is interesting about the Wiitek linking at 2.5 and 5Gig. What did the 7250 say the link speed was in the show interface? I would think it reported 10gig even if the other end was 2.5 or 5 since those ASICs are not multigig.
 

ptibeur

New Member
Sep 4, 2020
5
0
1
Thank you Frnot for confirming you also have this bug, it's too bad it was not fixed in a later version... or maybe the latest one ? I'll try it and post some news here.

I'm not fan (haha) of the idea of running the switch completely fanless, I would be afraid it would get really too hot over time and would die, or even worse (fire ?). A quiet air flow for a nearly idle switch sounds like a good trade-off to me.

The 100 ohm trick is nice ! I tested the fans on a breadboard before mounting them and they work at 5V and even at 3.3V. 4.5V should be nearly inaudible. Now I gotta check where I can find some resistors in my lost city haha.

About the 10G ports, I don't think they're gonna bring so much additional heat : they are at the opposite corner of the single fan. It would have been more logical design to put them closer to the fan evacuating the heat if they would generate a lot of heat.
 

fohdeesha

Kaini Industries
Nov 20, 2016
2,875
3,374
113
34
fohdeesha.com
Thank you Frnot for confirming you also have this bug, it's too bad it was not fixed in a later version... or maybe the latest one ? I'll try it and post some news here.

I'm not fan (haha) of the idea of running the switch completely fanless, I would be afraid it would get really too hot over time and would die, or even worse (fire ?). A quiet air flow for a nearly idle switch sounds like a good trade-off to me.

The 100 ohm trick is nice ! I tested the fans on a breadboard before mounting them and they work at 5V and even at 3.3V. 4.5V should be nearly inaudible. Now I gotta check where I can find some resistors in my lost city haha.

About the 10G ports, I don't think they're gonna bring so much additional heat : they are at the opposite corner of the single fan. It would have been more logical design to put them closer to the fan evacuating the heat if they would generate a lot of heat.

it's not a bug, the PWM circuitry that controls the fan voltage is fried on your switch. That's why the OS says it's sending speed 1, but you measured a full 12v
 

fohdeesha

Kaini Industries
Nov 20, 2016
2,875
3,374
113
34
fohdeesha.com
OK. I've been through the thread and can't find an answer. I apologize if I've missed it.
What breakout cables do you use for the QSFP+ ports if you are running singlemode fiber? I see fiber channel ones used, I see mention of 850nm, but can't find any mention of single mode breakout cables.
pick up an 40gbase-lr4 optic off of ebay for ~75 bucks, then a singlemode MTP/MPO to LC breakout harness
 

Frnot

New Member
Sep 23, 2020
7
1
3
The 100 ohm trick is nice ! I tested the fans on a breadboard before mounting them and they work at 5V and even at 3.3V. 4.5V should be nearly inaudible. Now I gotta check where I can find some resistors in my lost city haha.
Before you wire anything permanantly, you should make sure that the fans can start up at the voltage you choose. I was able to get the fan to run at about 3.5V, but it didn't have enough force to overcome the initial inertia when the fans were stopped. I could only get the fan to consistently start at 4.5V. I think this might be one of the reasons the fans usually start on 100% and are ramped down later.

About the 10G ports, I don't think they're gonna bring so much additional heat : they are at the opposite corner of the single fan. It would have been more logical design to put them closer to the fan evacuating the heat if they would generate a lot of heat.
Not all engineers think logically. (or there may be more important reasons for the placement) I would do some tests before you throw the switch in your rack permanently.

it's not a bug, the PWM circuitry that controls the fan voltage is fried on your switch. That's why the OS says it's sending speed 1, but you measured a full 12v
I'm having the same issue. Is it common for this circuitry to be fried? How does this happen? And why does the "dm fan-speed" say speed 1 but is sending the signal 0xff? Is that not 100%?

I don't know the full history of my switch, but I was led to believe that it was pulled from a rack in the enterprise. It hasn't been tinkered with by any homelabbers (except for me).
 

ptibeur

New Member
Sep 4, 2020
5
0
1
it's not a bug, the PWM circuitry that controls the fan voltage is fried on your switch. That's why the OS says it's sending speed 1, but you measured a full 12v
Oooh, that's interesting, is it a known issue ? Is it fixable or replaceable in some way ? I might be able to replace a faulty component with a soldering iron if it's more or less accessible :)


Before you wire anything permanantly, you should make sure that the fans can start up at the voltage you choose. I was able to get the fan to run at about 3.5V, but it didn't have enough force to overcome the initial inertia when the fans were stopped. I could only get the fan to consistently start at 4.5V. I think this might be one of the reasons the fans usually start on 100% and are ramped down later.



Not all engineers think logically. (or there may be more important reasons for the placement) I would do some tests before you throw the switch in your rack permanently.
Yes I tested on a breadboard, the fans can start from full-stop with 3.3V applied, thus there should not be a problem if I feed them 4.5V. My preference would go towards fixing the fan control module in priority, if feasible.

I'll keep an eye on the temperatures yes, I'm still waiting to get the 10 Gbps NICs, I already got the SFP+ SR optics though.
 

noduck

Member
Sep 12, 2020
41
13
8
That is interesting about the Wiitek linking at 2.5 and 5Gig. What did the 7250 say the link speed was in the show interface? I would think it reported 10gig even if the other end was 2.5 or 5 since those ASICs are not multigig.
The other end is a MS510TX on a 2.5G port:
Code:
10GigabitEthernet1/2/8 is up, line protocol is up
  Port up for 1 minute(s) 9 second(s)
  Hardware is 10GigabitEthernet, address is xxx
  Configured speed 10Gbit, actual 10Gbit, configured duplex fdx, actual fdx
 

ICXGURU

Member
Jun 22, 2020
37
16
8
The other end is a MS510TX on a 2.5G port:
Code:
10GigabitEthernet1/2/8 is up, line protocol is up
  Port up for 1 minute(s) 9 second(s)
  Hardware is 10GigabitEthernet, address is xxx
  Configured speed 10Gbit, actual 10Gbit, configured duplex fdx, actual fdx
The issue you may have with that is the 7250 thinks it can send 10Gbps of traffic out that interface and the transceiver will drop up to 50% or 75% of the traffic since the link cannot send the traffic load. Normally flow control would catch that but in this case the remote end is not receiving more than it can handle since the packets will be dropped at the source interface, so it will never send flow control pause to slow it down. That means it's up to TCP to fix the problem with resends and the problem gets even worse.
 

noduck

Member
Sep 12, 2020
41
13
8
I have replaced the stock fans in 7250-24p with two "Sunon 40x40x20mm 3 Pin Fan MB40201VX-000U", and the noise is much improved. At speed 1 it is barely audible, but speed 2 is still very obvious (but definitely improved from the screeching stock fans). Furthermore, the fans cycle back and forth between 1 and 2, because of the sensor 1 temperature (MGMT THERMAL PLANE), which cycles between ~85C and ~95C.

Hopefully somebody can provide insight into the following:
  1. Is it normal to have the sensor 1 temperature this high?
  2. Are there better (more quiet) fans to consider?
  3. Would it help to place a (larger) fan over the ASIC (not sure if there is enough clearance)?

This the final topic that I am investigating on this 7250-24p to determine if it is fit for my purpose. The other two topics are the PoE ports (failed board) and multi-Gig (success with Wiitek). Hopefully I can figure out if I should look for a fully functional 7250, or just return it.

Logs:
Code:
Sep 25 19:22:58:W:System:Stack unit 1 Fan speed changed automatically to 1 
Sep 25 19:21:53:W:System:Stack unit 1 Fan speed changed automatically to 2 
Sep 25 19:18:23:W:System:Stack unit 1 Fan speed changed automatically to 1 
Sep 25 19:17:17:W:System:Stack unit 1 Fan speed changed automatically to 2 
Sep 25 19:13:47:W:System:Stack unit 1 Fan speed changed automatically to 1 
Sep 25 19:12:42:W:System:Stack unit 1 Fan speed changed automatically to 2 
Sep 25 19:09:07:W:System:Stack unit 1 Fan speed changed automatically to 1
show chassis
Code:
Fan controlled temperature: 
        Rule 1/2 (MGMT THERMAL PLANE): 91.8 deg-C
        Rule 2/2 (AIR OUTLET NEAR PSU): 46.5 deg-C

Fan speed switching temperature thresholds:
        Rule 1/2 (MGMT THERMAL PLANE):
                Speed 1: NM<-----> 93       deg-C
                Speed 2:        82<----->105 deg-C (shutdown)
        Rule 2/2 (AIR OUTLET NEAR PSU):
                Speed 1: NM<-----> 58       deg-C
                Speed 2:        49<----->105 deg-C (shutdown)
 

noduck

Member
Sep 12, 2020
41
13
8
The issue you may have with that is the 7250 thinks it can send 10Gbps of traffic out that interface and the transceiver will drop up to 50% or 75% of the traffic since the link cannot send the traffic load. Normally flow control would catch that but in this case the remote end is not receiving more than it can handle since the packets will be dropped at the source interface, so it will never send flow control pause to slow it down. That means it's up to TCP to fix the problem with resends and the problem gets even worse.
Hmm... sounds like a valid concern. I haven't looked at all the configuration options yet, but would it be possible to address the issue with some kind of rate limiting?
 

infoMatt

Active Member
Apr 16, 2019
222
100
43
Is it normal to have the sensor 1 temperature this high?
No, it's definitely not normal unless you are in a 50+°C room...
This is the problem of the "quieter" fans, they can move a lot less air, or with a lot less pressure, both resulting in reduced airflow over the heatsink than the designed and higher temperatures/lower useful life expectancy for the component.
You have to decide if the downside balances with the lower noise or not.
 

dswartz

Active Member
Jul 14, 2011
611
79
28
Switches came in a couple of hours ago. Had to fumble through a bit, due to needing to upgrade OS from 2016. All good now - 2 node linear stack up and running. Off to the races!
 
  • Like
Reactions: ICXGURU

dodgy route

Member
Aug 12, 2020
48
66
18
Australia
So, I have successfully halved my temps on my ICX6610-48P.
About 10 or 11 pages back I identified the noise is too much and went on my way...

Had to wait for 2 Noctua 200mm black fans and a pulse generator to show up from Amazon US, everything else I had sitting around already.

Had some 3mm acrylic/perspex or whatever this is that came from an old telecom rack front door, the whole thing was destined to be recycled. A part of this door will be used to replace the lid of the ICX6610.
I didn't want to hack the case lid for my fan mods because... it is not necessary in any way, its pretty much fully reversible modification.
I cut out the shape according to the switch lid flipped and traced it out on the sheet, when it was cut I marked and drilled the holes precisely while the lid was on so the holes are exactly where they need to be. The holes were also countersunk for the original lid screws to be used, perfect.




Meantime my pulse generator arrived and I did 2 things.
1) Create an excel spreadsheet to record the Noctua 100mm fan RPM Hz, while testing the PWM output from a spare mobo/cpu setup.
I used my cheap digital multimeter that can measure temperature and frequency for this, you just need the positive probe on my model.
HzReported mobo RPMCalc RPM from Hz: (Hz * 60) / 2Duty %
29.1862873100
27.782583190
22.065066070
16.348548950
11.534334535

Based with this information and a similar fan model as the switch ones I added a formula into my spreadsheet to calculate RPM from Hz or Hz from RPM for playing with.
Here is the google sheet you can use to calculate whatever you need for fans
My pulse generator calculator and 200mm Noctua info Google Sheet

2) Used my multimeter to test out the pulse generator with my new found calculations from the excel spreadsheet information. I tested 750Hz and got 22500 RPM exactly on the motherboard, just as per my calculator :)




I tested a few different Hz on the mobo once I adjusted the pulse generator accordingly for same output, I used 666Hz for 20000 RPM but anything in this region appears to work fine for the switch, works perfectly, use the multimeter on the pulse generator to set the required Hz (ie RPM required for switch) - At this point I didn't actually measure the switch fans yet, but no matter... just before gutting the fans from the switch I actually measured the frequency and got 650Hz on all 3 fans, so 666 remained..

Next, with the pulse generator worked out, was to do the physical work for mounting the fans as well as wiring, piece of cake.
For the fans, I laid out the fans where I wanted them, covering PSU and the CPU's, they are ginormous so coverage was not difficult, nor placement, and used a scriber thing (anything thats sharp will work) to mark the holes to cut out.



I used a dremel 561 bit which is excellent for cutting this plastic sheet, made it very easy and controllable, even with my crappy cheap non dremel rip off.
The fan mounting holes were also scribed and drilled out to accept regular fan screws of 5mm, also countersunk from the bottom so they dont stick out from under the plastic sheet.


I had some anti noise strip things from years ago I used anywhere where fans contact the sheet and sheet contacts the case, its visible in one of the photos, its stuck not on the switch but on the sheet itself, use masking tape or something that draws on the plastic so you can see it from other side when putting the tape on.
Anyways fans mounted and that part was done, just before I attached the lid on with screws, I put some hot glue over all the fan screws so they are insulated from any possible contact with any switch electronics.


For the PSU to be cooled effectively, I removed the metal top lid and traced out the fan hole, I cut out this portion and kept it, its easily stuck back together when/if returning to stock, you will need it if you are putting on the metal lid on as things are going to short out likely without it..


Wiring was actually piece of cake but YMMW, but im no stranger to wiring and I have experience from more complex wiring jobs through my hobbies...
Anyways I found this instead of me describing it too much, from another thread on this forum, even tho i measured my own to confirm as I recommend anyways...


No stock switch wiring was harmed, I simply removed every part of it from the fan tray and the PSU, the whole rubbish thing is removed in one piece so if it ever gets passed on as stock, its just a simple return operation.

I had also found a very old and broken fan controller that used to be on a pci bracket, I managed to cut the PCB to just fit in the fan tray sideway to have adjustability from outside easily. The broken part was repaired, the regulator snapped off 2 or the 3 soldering pads, so emergency repairs to anchor it down and some wire to repair the electrical connection to the tracks seems to have done the job. Works great now.

Use the diagram above, but for a front to back flow switch you need the 3 grounds, pulse generator and fan controller connected together (and fans grounds if not using controller).
The 12V power is on the bottom 3rd pin from the left, connect that to the pulse generator and fan controller (and fan power if not using controller).
Next pin, bottom, 4th pin from the left and one above it in top row need to be connected to the tone generator output, as well as the power supply fan RPM input too.

My wiring is basically from old motherboard header pins, think reset and power switches, just remove the plastic bits and add a bit of heat shrink, they hold well, needed smaller wire pin for the psu fan header tho, so thats something to consider..


The single green wire going to the PSU, is for the PSU fan trickery


My fans are PWM but im not feeding them PWM, I used some scrap old 3pin pass through and normal connectors from other projects so I can remove lid easily with one connector easily accessed at the back of the switch, at the fan controller output.


All done, just put the lid on,


I forgot to take a pic of the lid screws in place, but they are and the thing is in the rack for testing over next week how it does, temps are 1/2'd since fan switch over.
I also no longer run the blanking plate on the second fan tray. May make a mesh cover to keep any possible larger creatures out of it before I wire it in to my network next week.


This thing is quiet as a mouse 1 meter away, so shouldnt disturb you if your head is next to it... let alone in a rack somewhere else :)
Speaking of racks, I hope no one has missed the fact my switch is now 2U, something to consider if you are vertically challenged...

The 200mm fans output a LOT of air, its good stuff, also my power draw at the wall is 112W once the switch is booted with only console cable.
Old fan config is 3x40mm 0.81A, new config is 2x200mm 0.08A and a lot more flow.

Here is the whole album link for teardown plus fan mods all pics I took. Includes videos of noise, tho was hard to capture noise after mods, my daughter was lightly playing in the background so its more background noise really then the switch itself, this was fans around 1/2 to 2/3rds speed on the controller..
No fan errors or anything weird going on, as the pulse generator is doing its thing, quite well too.

Here is a shot of temps after 3 hours of on time in the rack with much other heat/etc, keep in mind this is with now closer to 1/2 speed on the fan controller.
If I turn the fan controller up to 11, its still quiet - at 1m hardly audible, in a rack not a chance.
Code:
SSH@bigbeef>show chassis
The stack unit 1 chassis info:

Power supply 1 not present
Power supply 2 (AC - PoE) present, status ok
        Model Number:   23-0000142-02
        Serial Number:  WME
        Firmware Ver:    A
Power supply 2 Fan Air Flow Direction:  Front to Back

Fan 1 ok, speed (auto): [[1]]<->2
Fan 2 not present

Fan controlled temperature: 41.0 deg-C

Fan speed switching temperature thresholds:
                Speed 1: NM<----->84       deg-C
                Speed 2:       79<-----> 87 deg-C (shutdown)

Fan 1 Air Flow Direction:  Front to Back
MAC 1 Temperature Readings:
        Current temperature : 38.0 deg-C
MAC 2 Temperature Readings:
        Current temperature : 40.5 deg-C
CPU Temperature Readings:
        Current temperature : 39.5 deg-C
sensor A Temperature Readings:
        Current temperature : 27.5 deg-C
sensor B Temperature Readings:
        Current temperature : 29.5 deg-C
sensor C Temperature Readings:
        Current temperature : 29.5 deg-C
stacking card Temperature Readings:
        Current temperature : 41.0 deg-C
        Warning level.......: 84.0 deg-C
        Shutdown level......: 87.0 deg-C
Boot Prom MAC : cc4e.243b.a2bc
Management MAC: cc4e.243b.a2bc
Cheers and enjoy!
 
Last edited: