Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

[ThreeFN]

I went back and double checked my 6450 non-poe in my office and I completely forgot to swap pins on the Sunon in this switch when I installed it. Whoops, no wonder it was completely silent. High temp over the last month was only 62°C so no harm done but I swapped pins so the fan will actually function.

Is there any way to address the pwm noise at low speed on these? I'm willing to cobble together some simple electronics inside the case to quiet things back down, the pwm noise is actually really irritating in a quiet room.

Hackaday just refers to another guys post, but they've earned, and deserve the love.

Short version, a low pass filter tuned to the pwm freq, ie a resistor and capacitor. You'll waste some power in the resistor but oh well.
[SCIENCE]
However the fan itself is brushless so is already doing some 'pwm' of it's own to drive the coils, which is speed dependent and therefore isn't constant. That's a reason PWM to a fans power pin is a 'suboptimal solution' and 4 pin fans are a thing (4th pin being a command signal that the fan uses to adjust itself, rather than via incoming power manipulation).

Your incoming PWM of the power to the fan 'fights' with the frequency of the coils turning on and off in the brushless motor, and you get some 'beating' constructive/destructive interference. Basically, if the fan is in a position to require power to the one motor coil, and the incoming power PWM is 'on', all is good. If those are out of sync, no power is provided, fan slows down. The increased noise is probably times when the incoming pwm cycles in the middle of a coil activation, 'suddenly' providing or dropping power, jerking the motor/fan. Will it hurt the fan? probably not as much as eventual lubrication loss, and the fan probably has enough inertial and a low enough power/torque in the motor to smooth things out. It just ends up causing noise.
[/SCIENCE]

Trying to figure out how to balance the filter to the fan is probably a fools errand (given the fan has at least 2 speeds) but filtering the PWM should be easy. If you multimeter has a frequency counter feature (I think my $35 extech does) you can get the pwm freq from that, run a low pass filter calc on that, and size your cap/resistor pair. But the results to sound may vary and I'd be a bit surprised if there wasn't some filtering already on the mainboard. Hooking a oscilloscope to the fan header, it would be obvious if it was pure pwm or if there was some filtering, but I haven't had an o-scope to hand for a fair few years.

 

[arglebargle]

Hackaday just refers to another guys post, but they've earned, and deserve the love.

Short version, a low pass filter tuned to the pwm freq, ie a resistor and capacitor. You'll waste some power in the resistor but oh well.
...
Trying to figure out how to balance the filter to the fan is probably a fools errand (given the fan has at least 2 speeds) but filtering the PWM should be easy. If you multimeter has a frequency counter feature (I think my $35 extech does) you can get the pwm freq from that, run a low pass filter calc on that, and size your cap/resistor pair. But the results to sound may vary and I'd be a bit surprised if there wasn't some filtering already on the mainboard. Hooking a oscilloscope to the fan header, it would be obvious if it was pure pwm or if there was some filtering, but I haven't had an o-scope to hand for a fair few years.

Wow, thank you! This is a great breakdown. I'll spend some time reading about this and see if I can't get a friend to bring a scope by the next time he's on my side of the bay, he usually jumps at the chance to put a scope on electronics. I've got a meter with a frequency counter function but no scope unfortunately.

 

[fohdeesha]

Whoops, no wonder it was completely silent. High temp over the last month was only 62°C so no harm done but I swapped pins so the fan will actually function.

Do you guys believe me now that these things need basically zero cooling at homelab usage levels?

as for the PWM noise, without cobbling a bunch of shit together the easiest thing would probably be just tap off the raw 12V from the PSU and drive a fan with it, and size the fan so the full speed CFM is your ideal CFM/noise (eg a very small/slow fan)

 

[svtkobra7]

Do you guys believe me now that these things need basically zero cooling at homelab usage levels?

NO!

I was in my dc the other day (server closet) and pulled both servers (also turned off the ceiling exhaust) and I finally heard the ICX-6450 ...

At first I was like what is that ever so faint noise?
Then I realized that the 6450 does indeed have fans in it! (had never heard them before)
I LOL when ppl bitch about the fan noise on that switch.

 

[fohdeesha]

rear rack rails for 6610s for cheap if anyone has some droopy boys Brocade ICX6610-48P-PE Switch Rack Mount RMK Kit | eBay

 

[Dreece]

My 6610 now runs quiet. I put the other psu cable in. It's amazing.
So looks like it goes into turbo hairdryer mode to annoy the hell out of you until you give it what it desires, power redundancy. That wasn't in the manual, well the manual I read, it came with it for free, just one small page, titled, 'sold as seen, not for idiots'.

 

[ThreeFN]

To be fair to... myself... my 6450 is mounted top of a 4 post that holds 'all the tv stuff', in the living room next to the tv, and the rule of that rack is if it's louder than the PS3, it gets the axe or surgery.

Update on my rollout, I've got the 6450 deployed, uplink via S+RJ10 on single ~150ft run of CAT5e. Modules are probably running at 5gbps given 5e and distance. Debating if I want to shlep the rip-box (only other 10G nic server I have right now) just for the speed test...

Newest headache is now the XBOne. Apparently if it goes to standby and you have instant on / background updates turned on, it will yoyo the interface, both up/down as well as speed all over the place. There was a moment following a 'sh int br' of me saying "what in the world device do I have that's running 10M?!?!" Honestly I'd contemplate an orphaned 5port switch just to kill the log spam, but I'll be a good little engineer first and try and solve the problem.

 

[fohdeesha]

My 6610 now runs quiet. I put the other psu cable in. It's amazing.
So looks like it goes into turbo hairdryer mode to annoy the hell out of you until you give it what it desires, power redundancy. That wasn't in the manual, well the manual I read, it came with it for free, just one small page, titled, 'sold as seen, not for idiots'.

the proper manual (this thread) has mentioned that a few times it should only happen with rev A power supplies. Rev b and C should happily run at low speed with just 1 PSU active

 

[Dreece]

85 pages is wayyyyy too much data, maybe a nice little update to the front page where you've already posted great tips, of typical common questions+answers and numpty newbie mistakes

This is the output relative to fans:

Power supply 1 (AC - Regular) present, status ok
     Model Number:    23-0000144-01
    Serial Number:    0HL    
    Firmware Ver:      B
Power supply 1 Fan Air Flow Direction:  Front to Back
Power supply 2 (AC - Regular) present, status ok
     Model Number:    23-0000144-01
    Serial Number:    0CB    
    Firmware Ver:      B
Power supply 2 Fan Air Flow Direction:  Front to Back

Goes to full-blast when I take psu 2 offline, and then goes back to whisper mode when plugged back in.

 

[fohdeesha]

Does it instantly go to full blast, or take a little bit? That's definitely not normal, I have 2 chassis here in a lab with 1 rev B plugged in and can barely hear them

if it takes a little bit, it's probably warming up to fan speed 2 threshold without the extra airflow of the second PSU fan. In that case the room they're in is probably just warmer than usual or the fan tray is on the wrong side compared to the PSU (if you only have 1 fan tray)

 

[Dreece]

Instantly, well it rises to full under 2 seconds.

Do not know if this is relevant, it does state in the chassis output that Fan 2 is not installed, is that an optional fan kit?

 

[fohdeesha]

there's an optional second fan slot/tray on the back that's typically not needed. I would try powering off the switch, moving the fan tray you do have to the other slot, and powering it back on with 1 psu (the same single psu you tried before). I do remember a member having similar behavior with a single PSU when the fan tray was criss cross from it (or maybe next to it, can't remember)

 

[Dreece]

Ok, I will do that on Sunday. Unable to for now as the switch is now core to the home-side topology and currently running some long-toothed number crunching tasks.

When I do take it offline, is there anything else you advise I consider/look at to report back?

 

[arglebargle]

Do you guys believe me now that these things need basically zero cooling at homelab usage levels?

as for the PWM noise, without cobbling a bunch of shit together the easiest thing would probably be just tap off the raw 12V from the PSU and drive a fan with it, and size the fan so the full speed CFM is your ideal CFM/noise (eg a very small/slow fan)

Oh I never doubted you, I just thought better safe than sorry y'know. The only reason I care right now is that the 10Gb switch chip gets stupidly hot when it's in use and there's no airflow, the chips near the 1G ports seem totally fine passively cooled.

If I tap the 12v supply to drive a quiet fan at full speed is it safe to hook the tach line up to the header for failure alerting? I haven't touched fan control circuits since before everything switched from linear regulation to PWM so I don't really know if they're doing anything tricky.

If anyone else is going to do this a Sunon maglev V3 fan would be ideal at full speed, they're damned quiet and the switch really only needs the tiniest amount of airflow to drop temps by like 15-20C.

To be fair to... myself... my 6450 is mounted top of a 4 post that holds 'all the tv stuff', in the living room next to the tv, and the rule of that rack is if it's louder than the PS3, it gets the axe or surgery.

The PWM whine from the 6450 at low speed is the loudest thing in my office when the air filter isn't running. If it's more noticeable than my desktop it goes under the knife until it isn't

 

[fohdeesha]

If I tap the 12v supply to drive a quiet fan at full speed is it safe to hook the tach line up to the header for failure alerting?

Absolutely, at fan speed 2 (full speed) that's all it's doing anyway, getting the full 12v rail

 

[klui]

I just went through all 84 pages of this thread and like Vicious I got the S2500 and would like to check out these Brocades. I have my S2500 working in my lab-lab (not home-lab) with my home config with another FW and am anxious to see what kinds of response I'll get with my CX3s even though I have no need for the extra bandwidth. These switches are at a nice price point, with all the port types I'll need for the near future, compared to the Arista 7050Q/Qx available on eBay. Having worked with the Arista 7060CX2-32S for a project, I've gotten to like them quite a bit.

Thanks fohdeesha for your enthusiasm and generosity in sharing your knowledge for not only these Brocades but the Aristas and other topics in other threads.

 

[arglebargle]

I just went through all 84 pages of this thread and like Vicious I got the S2500 and would like to check out these Brocades. I have my S2500 working in my lab-lab (not home-lab) with my home config with another FW and am anxious to see what kinds of response I'll get with my CX3s even though I have no need for the extra bandwidth. These switches are at a nice price point, with all the port types I'll need for the near future, compared to the Arista 7050Q/Qx available on eBay. Having worked with the Arista 7060CX2-32S for a project, I've gotten to like them quite a bit.

Thanks fohdeesha for your enthusiasm and generosity in sharing your knowledge for not only these Brocades but the Aristas and other topics in other threads.

The prices on the dual 10Gb CX3s have started dropping in the last couple of months too. I need a couple more cards and it's tempting at this point to just switch to buying MCX312's and skip the need for the QSFP to SFP adapters. $25/2x40Gb VPI board is a damned nice price, but once you've added on $12*2 per card for the QSFP adapters you're right up into the same price range as the 2x10Gb cards anyway. I don't know if there's much difference in power draw but I'd imagine the 2x10Gb boards are at least a couple W lower than the 40/56Gb model.

I meant to post this last week when my switches arrived, after all of the horror stories of poorly packaged switches it was nice to have these arrive packed well: https://photos.app.goo.gl/PFPHKmW2z7WvG2628

 

[u238]

Say I had a dumb switch (my ISP provided ONT / router / switch device) and a stack of 2 ICX switches. Is there any way to create a two link failover type lagg setup between the dumb switch that doesn't support LACP and the switch stack?

 

[arglebargle]

Say I had a dumb switch (my ISP provided ONT / router / switch device) and a stack of 2 ICX switches. Is there any way to create a two link failover type lagg setup between the dumb switch that doesn't support LACP and the switch stack?

If you just want to add redundancy to your dumb L2 switches connection to the stack spanning-tree should do what you want. Turn on 802.1w for the vlan your dumb-switch traffic is interfacing with on the stack and plug your redundant links into different stack switches on that vlan and that should be it. If you lose the stack switch with the currently active link RSTP will re-learn the topology and unblock another link to take over.

You almost definitely want to use 802.1w here instead of OG STP, as I understand it it's a lot faster at handling newly plugged devices. Also, beware of "rstp" vs "802.1w" while configuring -- "rstp" is a Brocade proprietary early implementation of the spec, it might not play nicely with other rstp capable devices.

 

[Churchill]

My switch arrived but it did not come with a console/serial cable. Much hunting later and I found some on amazon and they will arrive on Monday. Then i'll flash the 6650 to the latest and go from there.

Overall: The sound is NOT that bad, but I don't have anything plugged into it just yet. No louder than my supermicro servers that I have running in my office and will be quiter once I close the doors to silence all the gear.