Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

[anomaly]

Hey guys, short term browser and first time posting,

Following a series of modifications and scares, I replaced the fans in my ICX 7250-48P initially with very quiet SUNON fans which turned out to be not thermally respectable and the bane of my existence... buzzing. I swapped them out for Mechatronics fans after this discovery but I find myself coming back to the buzzing. I've done digging and came across older posts but for the ICX 6450...




I tried hooking up some capacitors (100µF and 220µF) which initially proposed the solution but then quickly became a nightmare which started with a shutoff where it didn't boot again until waiting some time with the outlet turned off and then a boot loop every time the switch fully started up.

Fast forward now, I wanted to ask around if anyone had any solutions for the buzzing specifically on an ICX 7xxx. I may try the 10µF capacitor but I fear this may result in a boot loop like before or even worse... needing to buy another switch.

Putting capacitors around without understanding why/how/what is a reliable method to toast your equipment.

I seriously doubt a capacitor across GND and VIN of the fan connector will fix any buzzing. The buzz is a characteristic of the motor coil itself, most of the time, and also highly personal. Perceiving some frequencies higher than others has more to do with your ears than the switch (tinitus, for example).

The PCB is designed by experienced engineers, and it has all the bypass and filtering capacitors it could possibly need. I'm pretty sure if you hooked a scope to the fan VIN you would see nothing out of the norm. The PSUs are top notch. This is not some POS night shift design from a Chinese overworked engineer.

TL;DR Don't do this.

Swap the fans whole, or leave the originals. The only thing you can do is play with the PWM, which other people did already. You still need to provide a PWM signal reading that falls within spec of both the fan and the monitoring on the software side. I have no idea if this is done via GPIOs or in the ASIC or where, but I am 100% sure they spec'd it for the stock fans. If somebody reverse engineers the location and format of this, and can alter the thresholds and spec/PWM settings, then it will be possible to modify the fans with less effort.

Seems overkill to me, but that is the only true way to solve the issue, other than a "PCB in the middle" emulating the expected PWM and passing a different PWM ("translated") to the modified fan, which is the same done by the stock firmware (switch sees a tachometer reading of X for a PWM signal Z, the board in the middle translates Z to a PWM signal that the non stock fan can use within a threshold, reports back an adjusted tachometer).

 

[cyinite]

Putting capacitors around without understanding why/how/what is a reliable method to toast your equipment.

I seriously doubt a capacitor across GND and VIN of the fan connector will fix any buzzing. The buzz is a characteristic of the motor coil itself, most of the time, and also highly personal. Perceiving some frequencies higher than others has more to do with your ears than the switch (tinitus, for example).

The PCB is designed by experienced engineers, and it has all the bypass and filtering capacitors it could possibly need. I'm pretty sure if you hooked a scope to the fan VIN you would see nothing out of the norm. The PSUs are top notch. This is not some POS night shift design from a Chinese overworked engineer.

TL;DR Don't do this.

Swap the fans whole, or leave the originals. The only thing you can do is play with the PWM, which other people did already. You still need to provide a PWM signal reading that falls within spec of both the fan and the monitoring on the software side. I have no idea if this is done via GPIOs or in the ASIC or where, but I am 100% sure they spec'd it for the stock fans. If somebody reverse engineers the location and format of this, and can alter the thresholds and spec/PWM settings, then it will be possible to modify the fans with less effort.

Seems overkill to me, but that is the only true way to solve the issue, other than a "PCB in the middle" emulating the expected PWM and passing a different PWM ("translated") to the modified fan, which is the same done by the stock firmware (switch sees a tachometer reading of X for a PWM signal Z, the board in the middle translates Z to a PWM signal that the non stock fan can use within a threshold, reports back an adjusted tachometer).

The quality really shows from others posts messing with these switches and living the abuse. I checked for debug-info and the switch looks to be fully operational. Thanks for letting me know that an overkill solution will be needed here. My final solution might be buying another switch (or 2) anyway such as the ICX 7150-24 non-PoE and 7150-C12P to go fanless as I can suspect I'll run into the same issue buying an ICX 6450-24P. I also considered an ICX 7150-24P but the price ends up in the same ballpark...

 

[anomaly]

The quality really shows from others posts messing with these switches and living the abuse. I checked for debug-info and the switch looks to be fully operational. Thanks for letting me know that an overkill solution will be needed here. My final solution might be buying another switch (or 2) anyway such as the ICX 7150-24 non-PoE and 7150-C12P to go fanless as I can suspect I'll run into the same issue buying an ICX 6450-24P. I also considered an ICX 7150-24P but the price ends up in the same ballpark...

I own 7x50 series switches (including C12 and 48P) and 6450s (also including C12s, and 48P). The most quiet will be 6450-48P with Sunon fans, if you are obsessive with the noise. 7250s need at least one fan reported "good". It will make it moot to swap the other two (in the 48p version anyway).

I think time spent on engineering the hack is more than the ROI you get out of it, indeed.

Get the 7150-C12P so you have a much more recent system. 6450 is ancient in comparison (IMO). You won't regret it Just remember you are limited to 2 10G uplinks there. If you can soundproof the closet or rack for the 7250-48P, it's the best out of all the options IMO. With a light PoE load the stock fans never go into mode 2.

 

[sic0048]

At the end of the day, most enterprise switches are designed to be placed in environments where noise isn't a big concern.

If you expect to put the switch at your desk that you sit and work at, or somewhere else where the noise will bother people, then these enterprise switches may not be the best option. On the other hand, if you put them in a closet somewhere, the will generally be fine. I have several Brocade switches running in a central wiring closet in my house (it's actually under the stairs) and the noise is not a concern. I can hear them if the house is completely quiet and I'm sitting the an adjacent room, but even then it is barely audible. I'm probably the only one that "hears" it and only because I know what it is. Everyone else just hears a little background noise.

 

[anomaly]

Agreed. The longterm ROI of quieting things down is similar to using short depth racks: not worth it. I used a telco rack for quite some time at my first "home lab", and went through iterations of welding, cutting and adapting standard rails, figuring out cases for M/Bs to fit in the thing, etc. Ultimately, quality of life got a lot of better when I switched to a standard depth rack. With the "quiet operation" issue it's the same problem. Get quiet equipment from the get go, or place it all in a suitable location. The 7250 series is inaudible from a closed space until you stress out/max the PoE budget (something that should never happen in a normal household).

Most people aren't tuned to realize/hear background noise. This happens all the time with HVAC and exhausts. When was the last time anybody made a comment on those?

 

[anomaly]

On the subject of fan speeds (I could create a separate thread just for that):

The stack unit 1 chassis info:

Power supply 1 (AC - PoE) present, status ok
Power supply 2 not present
Power supply 3 not present

Fan 1 ok, speed (auto): 1<->[[2]]
Fan 2 ok, speed (auto): 1<->[[2]]
Fan 3 ok, speed (auto): 1<->[[2]]

Fan controlled temperature:
    Rule 1/2 (MGMT THERMAL PLANE): 64.5 deg-C
    Rule 2/2 (AIR OUTLET NEAR PSU): 37.0 deg-C

Fan speed switching temperature thresholds:
    Rule 1/2 (MGMT THERMAL PLANE):
        Speed 1: NM<-----> 95       deg-C
        Speed 2:        85<----->105 deg-C (shutdown)
    Rule 2/2 (AIR OUTLET NEAR PSU)***active***:
        Speed 1: NM<-----> 41       deg-C
        Speed 2:        34<----->105 deg-C (shutdown)

Fan 1 Air Flow Direction:  Front to Back
Fan 2 Air Flow Direction:  Front to Back                        
Fan 3 Air Flow Direction:  Front to Back
Slot 1 Current Temperature: 64.5 deg-C (Sensor 1), 37.5 deg-C (Sensor 2)
Slot 2 Current Temperature: NA
    Warning level.......: 100.0 deg-C
    Shutdown level......: 105.0 deg-C

The interesting thing here would be adjusting the temperature of the second rule (" Rule 2/2 (AIR OUTLET NEAR PSU)". In my case with the rack loaded it gets to ~37.5 and kicks in, but this is due to the fact that the switch is blowing out to the front (which has a glass door), instead of the back. Swapping the front with plates that are vented or adding a top exhaust to that side would likely quiet it down.

Again, if somebody knows where to look or already knows where the temperature thresholds can be adjusted, let me know

@fohdeesha maybe?

Edit: depending on where/what "sensor 1" is, it could also be swapped for something that gives a permanent or adjusted reading (not really a good idea...). Let's see if anyone comments. I'm honestly curious enough to load the firmware in IDA/Ghidra.

 

[techfan32]

Firstly, thanks to you all for the great information in this thread! I’ve learnt a lot from it.

My problem: I initially set up my Brocade 7250 following this guide and thread over a serial connection. This was from within a VM that I was engaging with using NoMachine NX. I used this to setup ssh access which is working fine.

I am now trying to access the serial console from another VM using the same cable as before. I am accessing the VM using the Proxmox web GUI console (ie. noVNC).

I am able to send commands to the switch and even able to login as verified by the logs seen via connecting with ssh. The issue is that my minicom session reports no responses from the switch, I am completely blind to what it is doing and am only able to send commands.

Does anyone have any advice for fixing this? The thing that seems most different from when it was working is that I’m using the Proxmox web console to access the source VM. But I don’t see why that would cause this problem?

Any help is much appreciated!

 

[madmoot]

Recently picked up a 6450-48P and promptly replaced the fans with 3 x Delta EFB0412VHD-F00. Lightly loaded it runs warm but never jumps to speed 2, and close to silent... so pretty happy overall. Today I managed to snag a 7250-24(non PoE) for a decent price and while the consensus seems to be that these things can be a challenge to keep cool AND quiet, i'm hoping someone who's had one of these can offer some advice - would love to keep it for those 8 SFP ports

From the photos, it looks like it's got a single rear fan. Planned to follow the RoachedCoach build - Sunon MF60101V3-1000U-A99 on the ASIC, and Mechatronics MR4020X12B1-RSR in the rear, but concerned that it may not move enough air to keep it at speed 1.

Any advice is greatly appreciated.

 

[anomaly]

Today I managed to snag a 7250-24(non PoE) for a decent price and while the consensus seems to be that these things can be a challenge to keep cool AND quiet, i'm hoping someone who's had one of these can offer some advice - would love to keep it for those 8 SFP ports

From the photos, it looks like it's got a single rear fan. Planned to follow the RoachedCoach build - Sunon MF60101V3-1000U-A99 on the ASIC, and Mechatronics MR4020X12B1-RSR in the rear, but concerned that it may not move enough air to keep it at speed 1.

@madmoot Could you take a DMM/multimeter and check the voltages in the VIN/power pin of the rear fan? Also show me the output of 'show chassis' while still at speed 1?

I will look for photos of what was done by @RoachedCoach . Localized cooling on the sensors/ASIC is also an option, even if not ideal. The problem with 7250 is that it needs at least one functional fan reported. That one fan will generate the noise it generates... in other words, the switch will be as loud as the loudest fan operating. This is why I did not bother swapping 2 of the fans with Sunon maglevs. I also don't believe they provide sufficient cooling at all.

Another option is ventilating directly away from the PSU with a 40x40 or smaller fan grabbing power and GND from any available nearby pins. You can also split one of the fan connectors (respecting the non standard pinout) to get VIN/GND and power other non PWM driven fans.

 

[jode]

My question is can i use Port 1 and 3 in the stacking configuration and then use port 2 and 4 for standard data access?

Check out the Stacking Guide (fastiron-08030b-switchstackingguide.pdf) from foodesha's package. p46ff explain stacking topologies for ICX6450.

The good news - what you want is supported. Let us know when you got it working.

 

[madmoot]

@madmoot Could you take a DMM/multimeter and check the voltages in the VIN/power pin of the rear fan? Also show me the output of 'show chassis' while still at speed 1?

I will look for photos of what was done by @RoachedCoach . Localized cooling on the sensors/ASIC is also an option, even if not ideal. The problem with 7250 is that it needs at least one functional fan reported. That one fan will generate the noise it generates... in other words, the switch will be as loud as the loudest fan operating. This is why I did not bother swapping 2 of the fans with Sunon maglevs. I also don't believe they provide sufficient cooling at all.

Another option is ventilating directly away from the PSU with a 40x40 or smaller fan grabbing power and GND from any available nearby pins. You can also split one of the fan connectors (respecting the non standard pinout) to get VIN/GND and power other non PWM driven fans.

I should have the switch in my hands in about a week - will check fan voltage then. Just trying to get some replacement cooling lined up to have on hand asap. This thing will be living in my home office about 8 ft away, otherwise I wouldn't really care about the noise level

 

[andrea87]

Finally I've bit the bullet and bought an used 7250-24p for my homelab.

Currently I've been running three switches (a mikrotik 24p non PoE, an 8 port 2.5g and an 8-port PoE). Too few 10G sfp+ uplinks, one switch (mikrotik) getting almost empty as I've been migrating some stuff to the 2.5G switch.

I plan to run this switch in a 9U network cabinet hidden in a cubpboard under the stairs, with 2 120mm fans pushing air out. I'd like to install quieter fans than the stock ones, I do have some delta EFB0412VHD-R00 at hand (40x40x20, 10CFM, 0.416 inchH2O at 9K rpm). Would those fit well as a replacement? I don't plan on having too much PoE power active (below 50W) but definitely some DAC cables running at 10G (3-4) and a couple 10gbase-t sfp+ modules. If needed I could also fit a 60x60x10 fan on the ASIC as many of you did.

Think I've gotten myself a good weekend project.

 

[madmoot]

Finally I've bit the bullet and bought an used 7250-24p for my homelab.

Currently I've been running three switches (a mikrotik 24p non PoE, an 8 port 2.5g and an 8-port PoE). Too few 10G sfp+ uplinks, one switch (mikrotik) getting almost empty as I've been migrating some stuff to the 2.5G switch.

I plan to run this switch in a 9U network cabinet hidden in a cubpboard under the stairs, with 2 120mm fans pushing air out. I'd like to install quieter fans than the stock ones, I do have some delta EFB0412VHD-R00 at hand (40x40x20, 10CFM, 0.416 inchH2O at 9K rpm). Would those fit well as a replacement? I don't plan on having too much PoE power active (below 50W) but definitely some DAC cables running at 10G (3-4) and a couple 10gbase-t sfp+ modules. If needed I could also fit a 60x60x10 fan on the ASIC as many of you did.

Think I've gotten myself a good weekend project.

Looking forward to hearing how your modding goes... still waiting on my 7250-24 (non-PoE) to arrive and planning similar mods.

 

[anomaly]

Voltage should be constant, I read @rootwyrm 's interposer firmware and it seems the PWM frequency is 25kHz in the 7450 series. @fohdeesha is not engaging the thread right now, but he probably knows all this / can easily test. If the frequency, etc, is all the same for both 6450 and 7250, then it will be quite easy to write a modified interposer that "translates" the speed mode 2 RPM to slower speeds (while reporting roughly the same to the mainboard).

@rootwyrm put some "overkill" design to work in his HW, I asked him if the BAV99 diodes are really needed, but it won't hurt to have them. The teensy is definitely overkill considering we have ESP32 or plain Atmega if you forego the Wifi, which I am personally not interested in at all (out of caution too).

His code is documented here: icx-fan10-mod/src/main.c at main · rootwyrm/icx-fan10-mod

For 7250/6450 mainboard/chassis fans you need a maximum of 3 digital/PWM GPIOs and 3 digital GPIOs for the tachometer, for each side (fan side and mainboard side) for the three fans, so, 12 GPIOs total, to do it properly. The convenience of using the ESP32 is cost and the architecture for multitasking is already there (FreeRTOS with esp-idf). Arduino IMO is crap, as far as the library/glue code is involved.

 

[Dave Corder]

So, I accidentally bought an ICX7150-C12P on eBay the other day. It was listed "AS-IS" and started at $29.99, so I threw in a $29.99 bid just for kicks, then forgot all about it. No one else wanted to take a chance on it, and I won. It showed up yesterday. There was a dent in the underside of the chassis, at the front near the SFP+ ports that was flexing the PCB. I disassembled it, straightened the chassis it back out as best I could, and put it together.

However, it does appear to be totally dead. No lights or anything when I plug it in, and nothing on the RJ45 console port. I checked the PSU and it is putting out 52V or so, so it doesn't appear to be that common issue.

So it seems like it's a system board issue. Anyone have any ideas on what I can look at or what I might be able to test or repair to get it functional again? I couldn't find a whole lot on system board failures, just PSU failures, when I searched the archives.

 

[andrea87]

I've got my ICX7250-24P I bought off eBay in Europe. I believe someone already did a fan swap to it, as I've found two delta FFB0412VHN spliced onto the original connectors.

I've tried running the quieter fans I've bought (EFB0412VHD-00), but the asic temp gets way too hot. The airflow at "slow" speed / 4.5V feels at least half of what the fast FFB fans do move. This is a show chassis run after 15 mins of power up, with no interfaces up besides one ethernet for management:

SSH@ICX7250-24P Router>show chassis
The stack unit 1 chassis info:

Power supply 1 (AC - PoE) present, status ok
Power supply 2 not present

Fan 1 ok, speed (auto): [[1]]<->2
Fan 2 ok, speed (auto): [[1]]<->2

Fan controlled temperature:
        Rule 1/2 (MGMT THERMAL PLANE): 85.6 deg-C
        Rule 2/2 (AIR OUTLET NEAR PSU): 39.5 deg-C

Fan speed switching temperature thresholds:
        Rule 1/2 (MGMT THERMAL PLANE):
                Speed 1: NM<-----> 93       deg-C
                Speed 2:        82<----->105 deg-C (shutdown)
        Rule 2/2 (AIR OUTLET NEAR PSU):
                Speed 1: NM<-----> 58       deg-C
                Speed 2:        49<----->105 deg-C (shutdown)

Fan 1 Air Flow Direction:  Front to Back
Fan 2 Air Flow Direction:  Front to Back
Slot 1 Current Temperature: 85.6 deg-C (Sensor 1), 39.5 deg-C (Sensor 2)
Slot 2 Current Temperature: NA
        Warning level.......: 100.0 deg-C
        Shutdown level......: 105.0 deg-C

I've tried adding a baffle made of thin cardboard to move more air through the ASIC's heatsink, but it only made the temperatures worse, hitting even 88°C on the main chip.

Moving back to the faster fans, the switch stabilized at 36-38°C on the PSU and 81-82°C on the main chip. Tomorrow I might try also with a 60x60x10 fan as some of you did on the asic. I'll be at best at 40W pulled out of the PoE ports, if I can get a balance between temps and noise it would be great.

 

[gruntus maximus]

Registered just to give thanks. First timer, got my 6450-48P based on the activity here - a great set of noob friendly recommendations backed up by an awesomely detailed guide+config documentation+package by fohdeesha. If only 10% of HW guides out there were half as good. Off to tinker!

 

[codyjd1]

Check out the Stacking Guide (fastiron-08030b-switchstackingguide.pdf) from foodesha's package. p46ff explain stacking topologies for ICX6450.

The good news - what you want is supported. Let us know when you got it working.

Awesome thanks for the reply! I will check it out and hope I can get it all configured when they arrive! Thanks again!

 

[NablaSquaredG]

However, it does appear to be totally dead. No lights or anything when I plug it in, and nothing on the RJ45 console port. I checked the PSU and it is putting out 52V or so, so it doesn't appear to be that common issue.

Maybe the onboard voltage regulators that step down 52V to 12V are dead

 

[andrea87]

Today I've managed to get a 60x60x10 fan, I've tried again running the switch with the smaller fans pushing air out to the back and the dedicated fan on the ASIC, and I've got much better temps. After 2 hours running on a table, with 25°C room ambient air, the ASIC was at 68°C while the PSU area at 40°C. The cooling feels adeguate, provided I don't push the PoE system too much. With these fans installed, my unit pulls 40W during boot and about 50W once it's loaded (with 3 gbe ports up).

Attached a few pics of the mod and also the switch itself.

I've configured a few vlans for all my systems, tomorrow if all goes well I'll put the switch into the network rack into it's definitive working place.