Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

[noise850]

(MB40201VX same static pressure but 10+ CFM instead of 6.someting).

I think that you can solve your problems by choosing one of these.

MB40201VX arrived today. Swapped it out and it's about the same noise level and feels about the same airflow as the stock fan. Overall it is a tad bit quieter since there was some sort of a bad bearing noise on the stock fan. However, the switch still isn't lowering the fan speed on boot. Just to make sure I put the original fan back in, booted it, and it also doesn't drop the fan speed on boot. The only way I can get the fan to slow down is with the dd set_pwrfan_low command. Is this correct, or is there something wrong with this switch?

Overall, I am happier with the MB40201VX. Today the room the switch is in got up to 90 F, which is about as hot as it ever gets (poor insulation, poor ventilation, and the sun sets on that side but I have to work with what I've got)

These were the max temp readings on the switch:

dd set_pwrfan_low

Fan ok, speed (auto): [[1]]<->2

Fan speed switching temperature thresholds:
        1 -> 2 @ 69 deg-C
        1 <- 2 @ 64 deg-C

Sensor B Temperature Readings:
        Current temperature : 50.0 deg-C
Sensor A Temperature Readings:
        Current temperature : 62.5 deg-C
        Warning level.......: 66.0 deg-C
        Shutdown level......: 76.0 deg-C

dd set_pwrfan_high

Fan ok, speed (auto): [[1]]<->2

Fan speed switching temperature thresholds:
        1 -> 2 @ 69 deg-C
        1 <- 2 @ 64 deg-C

Sensor B Temperature Readings:
        Current temperature : 47.5 deg-C
Sensor A Temperature Readings:
        Current temperature : 57.5 deg-C
        Warning level.......: 66.0 deg-C
        Shutdown level......: 76.0 deg-C

What do you think? Will the switch be okay with the fan set manually to low? Should I try re-flashing the firmware to see if that fixes the auto fan speed switching? If I use dd set_pwrfan_low will the switch override if it gets too hot or will it shutdown?

Thanks for the help!

 

[infoMatt]

However, the switch still isn't lowering the fan speed on boot. Just to make sure I put the original fan back in, booted it, and it also doesn't drop the fan speed on boot. The only way I can get the fan to slow down is with the dd set_pwrfan_low command. Is this correct, or is there something wrong with this switch?

No, definitely it should lower the fans. But I'm placing my bet on a software issue, because the fact that it can slow down the fans when forced with the "set_pwrfan_low" clears any suspect of a bad PWM circuitry and what not. Maybe once issued one of the two set_pwrfan commands it won't fall back to an auto speed regulation... I have to call @fohdeesha, I don't have an answer for this, sorry.

IMHO, if those are your maximum temps in the hottest days, I don't think that it hurts keeping it on the lower speed, at least until there's an answer to the previous question.. at least, it's easier on your ears . You have at least 13°C of headroom until the forced shutdown.

 

[noise850]

There are loud resonant frequencies happening with the MB40201VX that didn't happen with the MB40201V3 when set to low speed mode. Sounds like multiple frequencies mixed together, like a dial tone but pitched up a bit. You can't hear it in the high speed mode since it is masked by the loud noise of the fan but in low speed mode it's very noticeable. I have a very small office space and you can even hear this resonance down the hall with 2 other PCs turned on in the room. You can't hear the resonance with the brocade case lid off, only when it's on.

I'm beginning to think this just isn't going to work as I am now on my third replacement fan. I am not going for pure silence but if I can at least keep it around the same noise as the other desktops that would be great. Are there any other fans I can try that will keep this thing cool, even if manually set to high speed mode? I know a lot of people on here are happy with the 6450 so not sure why I am having such bad luck with this one.

 

[infoMatt]

if I can at least keep it around the same noise as the other desktops that would be great.

Honestly, a 1U box cannot be as quiet as a desktop PC using 120 or 140mm fans... The wider the fans the lower the noise they make moving the same amount of air.
Maybe you've never listened to a pizza-box server with 14 40x40x40mm (usually 7 groups of 2 stacked for static pressure) fans all revving up to the maximum, screaming at boot

 

[noise850]

I completely understand it is going to be louder than the desktops and we have about 30 pizza boxes at work so I know their sound really well. For a home office it's more of a quality vs quantity of noise issue. The MB40201V3 was pushing about as much air on high speed as the MB40201VX is on low speed, but this resonant frequency wasn't there. The sound of air flow is one thing, a solid tone is another. Even the groaning sound of the failing stock fan was more bearable than whatever weird interaction is happening now.

Also this is a non PoE model, so maybe having the one fan vs multiple on the PoE models is causing me issues?

I guess I can try the Noctua A4x20 and see how it works out. From reading over the thread seems some people have had good luck, others haven't.

 

[ljvb]

Honestly, a 1U box cannot be as quiet as a desktop PC using 120 or 140mm fans... The wider the fans the lower the noise they make moving the same amount of air.
Maybe you've never listened to a pizza-box server with 14 40x40x40mm (usually 7 groups of 2 stacked for static pressure) fans all revving up to the maximum, screaming at boot

This would be pretty much every HP server... I have 2 DL360pG8's and 2 DL380pG8's on top of 2 icx switches.. and it sounds like a jumbo jet on takeoff when the rack comes up after a power failure (my UPS can only run them for about 15 min)

I keep the rack in the garage, so the noise is less of an issue. The heat (and humidity when the garage door is open) is more of an issue.

Still not as bad as the HP 3000 blade chassis with 8 blades in it.... that was loud.. by loud standards

 

[Spearfoot]

Many thanks to @fohdeesha and the other posters here who have really made this a great thread.

You've made me lighten my wallet... but it's been well worth it.

Bought a Ruckus ICX 7150-C12P last month to replace the Cisco SG350-10P in my office. It works great!

This morning I negotiated a reasonable price -- less than $200 including shipping and taxes -- from an eBay seller for a beefier ICX 6610-48P-E unit w/ 2 power supplies. That model number specifies a single power supply, so the extra one must have been added on later? Anyway, if anyone's interested, the seller has 4 more identical units for sale:

Brocade ICX 6610 Icx6610-48-pe 48 Port Gigabit Poe Switch 4xavailable for sale online | eBay

Find many great new & used options and get the best deals for Brocade ICX 6610 Icx6610-48-pe 48 Port Gigabit Poe Switch 4xavailable at the best online prices at eBay! Free shipping for many products!

www.ebay.com

This bad boy will replace the Aruba S2500-48P out in my shop.

Now I'm getting quotes for pulling fiber from my office out to the shop, and to upgrade the CAT5 run from my office to the living room. The fun never $top$!

 

[ViciousXUSMC]

I replaced my S2500's with the Brocades and I am happy I did just a more natural configuration environment than the S2500 that was intended to use with a controller.

The 6450 is perfect for me with the fan mods as its feet away on my rack and I can still do voice over work without any noise, the 6610 is just there waiting for its day to shine as its way too loud to use in the same room.

Id like to re-home the S2500's but I think it would cost nearly as much to ship as what I could sell them for :/ and unfortunately I never did find brackets for my Brocade switches so I had to buy rack shelves and I am not super thrilled about that.

I have not had any heat issues with the fan mod, but the switch is not heavily loaded, maybe 12 or 14 Ethernet connections and only sending PoE to a single AP. Two SFP+ ports running fiber to my ESXi server and my desktop.

 

[Spearfoot]

I replaced my S2500's with the Brocades and I am happy I did just a more natural configuration environment than the S2500 that was intended to use with a controller.

The 6450 is perfect for me with the fan mods as its feet away on my rack and I can still do voice over work without any noise, the 6610 is just there waiting for its day to shine as its way too loud to use in the same room.

Id like to re-home the S2500's but I think it would cost nearly as much to ship as what I could sell them for :/ and unfortunately I never did find brackets for my Brocade switches so I had to buy rack shelves and I am not super thrilled about that.

I have not had any heat issues with the fan mod, but the switch is not heavily loaded, maybe 12 or 14 Ethernet connections and only sending PoE to a single AP. Two SFP+ ports running fiber to my ESXi server and my desktop.

Hello, @ViciousXUSMC -- I remember you well from the Aruba S-series thread; another thread that made me spend money! Ha Ha!

I'll be connecting 3 FreeNAS/ESXi AIO systems to the 6610, plus an AP and other miscellaneous stuff. Just the 3 AIOs use 5 x SFP+ and 13 x Gbe ports, counting IPMI. Truth is -- the Aruba would serve my needs just fine, and this ICX-6610 is vast overkill... but you know the old saying: "There's no kill, like overkill!"

 

[hmw]

Yeah talk about temptation. Read almost all of the thread and then caved in and got a 6610-48P with two PSUs for $140 and a 6610-24P with two Rev B PSUs for $170

Hopefully they should be here next week and I’ll ping @fohdeesha for licenses (you’re awesome man!)

I might just keep the 24 port variant and dispose of the 48 port one. The eBay seller told me he’d try to put REV B PSUs for the 48 port but I don’t have much hopes of that happening

 

[ViciousXUSMC]

Hello, @ViciousXUSMC -- I remember you well from the Aruba S-series thread; another thread that made me spend money! Ha Ha!

I'll be connecting 3 FreeNAS/ESXi AIO systems to the 6610, plus an AP and other miscellaneous stuff. Just the 3 AIOs use 5 x SFP+ and 13 x Gbe ports, counting IPMI. Truth is -- the Aruba would serve my needs just fine, and this ICX-6610 is vast overkill... but you know the old saying: "There's no kill, like overkill!"

Yes the S2500 served me well, but the 6450 was more efficient, had more power budget, and was actually quieter after the fan mods. I wanted to explore the CLI more and try to get more videos up so I was doing it just to do it more than out of any real need.

I got so busy IRL and lost a lot of footage I took of everything.

I much prefer configuration of the Brocade to the Aruba though as a Cisco guy it feels pretty natural.
Next thing for me to do is automation via PLINK so I can torture the kids by turning off access when they are bad with a single click of a button.

I was able to fabricate my own brackets for the Aruba switches out of some Cisco brackets that work perfectly, but no luck on the Brocade switches :/ that is probably my only sad thing about the new switches.

 

[ljvb]

Hello, @ViciousXUSMC -- I remember you well from the Aruba S-series thread; another thread that made me spend money! Ha Ha!

I'll be connecting 3 FreeNAS/ESXi AIO systems to the 6610, plus an AP and other miscellaneous stuff. Just the 3 AIOs use 5 x SFP+ and 13 x Gbe ports, counting IPMI. Truth is -- the Aruba would serve my needs just fine, and this ICX-6610 is vast overkill... but you know the old saying: "There's no kill, like overkill!"

Curious, if they are all in one setups, even 3, why are you using so many ports. I have 2 DL360ps with 2 port 10GB intel, and 4x1GB flexlom, and 2 DL380ps with the same. I'm only using 3 ports per machine (2 10GB ports and 1 1GB for backup connection to the switch), plus 2 ports for my ubiquiti setup ( each setup consists of 1 AP and 1 POE powered 8 port switch, AP gets power from the 8 port, the 8 port from the brocade, so only 2 ports there).

Rest of the ports (I have 2 24 port 6610s) are for drops from around the house, and eventually POE IP cameras.

 

[rosaage]

No, definitely it should lower the fans. But I'm placing my bet on a software issue, because the fact that it can slow down the fans when forced with the "set_pwrfan_low" clears any suspect of a bad PWM circuitry and what not. Maybe once issued one of the two set_pwrfan commands it won't fall back to an auto speed regulation... I have to call @fohdeesha, I don't have an answer for this, sorry.

IMHO, if those are your maximum temps in the hottest days, I don't think that it hurts keeping it on the lower speed, at least until there's an answer to the previous question.. at least, it's easier on your ears . You have at least 13°C of headroom until the forced shutdown.

I'm also having the same problem as noise850 with the switch stuck in high speed fan mode, using the command he used I can lower the fans. I tried a reload with the fans set to low, and at the same time as the "enter b to stop at boot" the fans changed to high and stays like this. I have a 6450-48 non p model just updated using the file from the guide (08030t). There is also some kind of whining noise running in low. It may be present at high too just non audible when the fans are a bit louder.
I haven't tried running it for more than about an hour yet but after that time it sits around 40C for sensor B and 50C for sensor A in low fan mode, so well below the threshold temperatures.

Any specific fans I should be on the lookout for if I want to replace it? one of the suppliers with good shipping (Distrelec) has over 30 different 12v 40mm fans just from sunon. Distrelec Link

 

[Spearfoot]

Curious, if they are all in one setups, even 3, why are you using so many ports. I have 2 DL360ps with 2 port 10GB intel, and 4x1GB flexlom, and 2 DL380ps with the same. I'm only using 3 ports per machine (2 10GB ports and 1 1GB for backup connection to the switch), plus 2 ports for my ubiquiti setup ( each setup consists of 1 AP and 1 POE powered 8 port switch, AP gets power from the 8 port, the 8 port from the brocade, so only 2 ports there).

Rest of the ports (I have 2 24 port 6610s) are for drops from around the house, and eventually POE IP cameras.

My earlier quote about how there's "No kill like overkill" applies here. I only use so many ports because of that mindset. Hey, I've got 'em -- why not use 'em?

Two of the ESXi systems have 4 x Gbe ports along with a dual-ported 10Gb Intel X520-DA2. So on these I dedicate two Gbe ports to the VMware management network and use two as offline standbys to one of the X520 ports for VM traffic. I use the remaining X520 port for a direct connection between the two FreeNAS instances for rsync and replication. So, adding in an IPMI port apiece, I end up using 10 of the Gbe ports just for these two guys.

The third ESXi box only has 2 x Gbe ports, so I can't set it up with that level of redundancy.

 

[infoMatt]

There is also some kind of whining noise running in low. It may be present at high too just non audible when the fans are a bit louder.

Mine has it as well, I think it's due to the low-ish frequency of the PWM modulation used to lower the fan speed. Maaayyyybe it's possible to make it go down by adding some capacitor to smooth it, but I don't want to advise anyone to add some random caps... I don't want to risk someone's equipment.

 

[rosaage]

Mine has it as well, I think it's due to the low-ish frequency of the PWM modulation used to lower the fan speed. Maaayyyybe it's possible to make it go down by adding some capacitor to smooth it, but I don't want to advise anyone to add some random caps... I don't want to risk someone's equipment.

Thank you! adding a 330µF cap removed the whining noise.

Here is a video where i power on the switch, test the cap, and measure the pvm frequency (~3kHz).

This video shows a different fan. Since this fan is more quiet and runs slower the whining is even worse. Here you can easily hear the difference with and without the cap. (Ignore the ugly soldering, my gas soldering iron ran out of gas)
The switch had some nice points for soldering, there is a ceramic smd cap together with a set of unpopulated pads on each of the fan headers.

 

[infoMatt]

Thank you! adding a 330µF cap removed the whining noise.

Well, it's kind of a beefy one for sure I'm glad it worked for you... I'll have to source some random caps that I have lying around. Mine is even worse (excluding the rattling noise at boot, maybe it will go away when the fan is tighten to the case), because being it a PoE model it has two fans.

BTW, is that an 122K? Nice find!

 

[noise850]

I'm also having the same problem as noise850 with the switch stuck in high speed fan mode...There is also some kind of whining noise running in low. It may be present at high too just non audible when the fans are a bit louder.

This is it! This is exactly the problem I'm having with my 6450-48 non PoE. The sound is amplified much more once the case lid is on and that noise is maddening.

I wish I had thought about the cap mod, but I already ordered a Noctua A4x20 PWM (they were out of FLX and I didn't care I've spent more on fans than on the switch). So far the Noctua sounds perfect at high speed mode and is keeping the switch at 49 deg on Sensor A, 60 deg on Sensor B at the hottest part of the day. I'll have to keep watch on it but if these temps stay this way I'm fine with this setup. If not, I'll try putting a cap and the Sunon back in.

And if anyone else needs the info, on the Noctua PWM fan you swap the Black and Green wires. Yellow and Blue stay the same.

 

[hmw]

My ICX-6610-24PE arrived - one of the corners where the right hand PSU is - was dented. Does anyone know where to get replacement screws?

 

[hmw]

Following the Fohdeesha guide on post #1, the s/w version for the ICX-6610 is 08030t. I noticed on Ruckus the latest firmware is 08030u. Tried to look at release notes and it says

Enhancements in FastIron 08.0.30u
FastIron 08.0.30u release contains defect fixes. There are no enhancements in this release.

Anyone upgraded to 08.0.30u? Does it have any known bug fixes other than the HDHomeRun issue?