Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

[Gerhen]

…

I wanted to ask first, what is best way to keep up on updates?
…
Next up- nearly silent - oh boy that is not my experience so far. This switch is easily the loudest component in my rack by 10 miles.

Your config shouldn’t be impacted when you upgrade.

I normally run the copy tftp system-manifest command and direct it to copy into the primary flash memory to start. If the new firmware is runnning stable, you can then copy primary to secondary. If it isn’t stable…you can always fall back and reload from secondary.

Firmware can be obtained from the ruckus support site you just need to register an account/email. It’s a good idea to register and read the documentation before you update esp. if you intend to move from the 08 to 09 firmware (some ppl run that branch).

Did you let the switch run to steady state operation? Initially at startup, the fans are loud (like wear hearing protection loud), after a few minutes the fans reach steady state and become less loud though I wouldn’t it in the same room if I was WFH.

 

[TeleFragger]

So I picked up an ICX7250 and went through all the initial Setup / Config / Licensing Guide - Thank you @fohdeesha for the excellent documentation to get things started, made things very easy.

Happy I didn't go for the ICX6610 even though I did want the 40GbE. I opted for 7250 since I wanted things to stay quiet and figured let me start out at 10GbE since that's what I have on my TrueNAS box (only issue there is it's onboard rj45 but that's a problem to figure out later). If the 6610 is even louder than this then dayum.

Appreciate the thread and any suggestions anyone wishes to provide.

I cant wait for my 7250 to show up. Did you go POE version?

for the rj45... 10gb... I am new too but from what my searching got me, you just need these to put into the sfp+ port

For Cisco SFP-10G-T, Ubiquiti UF-RJ45-10G Transceiver, SFP+ to RJ45 10GBase-T | eBay

For Cisco SFP-10G-T, Ubiquiti UF-RJ45-10G, Transceiver 10G SFP+ 10Gbase-T Module. The 10GBase-T copper transceiver is a high performance integrated duplex data link for bi-directional communication over copper cable.

www.ebay.com

I too wanted the 40gb ports but have no 40gb cards... i might have to keep my old brocade FastIron 648S to "toy with" but yes once I saw the LOUDNESS of the other, I chose the 7250... can always upgrade again.. lol in the future.

 

[NachoCDN]

for the rj45... 10gb... I am new too but from what my searching got me, you just need these to put into the sfp+ port

For Cisco SFP-10G-T, Ubiquiti UF-RJ45-10G Transceiver, SFP+ to RJ45 10GBase-T | eBay

For Cisco SFP-10G-T, Ubiquiti UF-RJ45-10G, Transceiver 10G SFP+ 10Gbase-T Module. The 10GBase-T copper transceiver is a high performance integrated duplex data link for bi-directional communication over copper cable.

www.ebay.com

i got the same one for my 6610 and it worked out of the box no issue.

 

[Koop]

for the rj45... 10gb... I am new too but from what my searching got me, you just need these to put into the sfp+ port

For Cisco SFP-10G-T, Ubiquiti UF-RJ45-10G Transceiver, SFP+ to RJ45 10GBase-T | eBay

For Cisco SFP-10G-T, Ubiquiti UF-RJ45-10G, Transceiver 10G SFP+ 10Gbase-T Module. The 10GBase-T copper transceiver is a high performance integrated duplex data link for bi-directional communication over copper cable.

www.ebay.com

I did go with the PoE version- because every other version was more expensive as far as I could find. Even those with less ports.

Word of caution from all I've ready these type of SFPs run super hot. Since my switch is already apparently too hot this may not be the best idea. Should work no problem but just something to keep in mind. No experience first hand with this- just that it seems to be pretty normal for these.

Firmware can be obtained from the ruckus support site you just need to register an account/email. It’s a good idea to register and read the documentation before you update esp. if you intend to move from the 08 to 09 firmware (some ppl run that branch).

Did you let the switch run to steady state operation? Initially at startup, the fans are loud (like wear hearing protection loud), after a few minutes the fans reach steady state and become less loud though I wouldn’t it in the same room if I was WFH.

Roger that on obtaining firmware and checking documentation first.

And yeah it was at steady operation- oh trust me there's a HUGE difference between start up and normal operation hah but even at the normal operation the sound level was pretty high. I have my rack nearby in my office and it's a bit much. Hoping there's more I can do to quiet it down.

 

[TeleFragger]

I did go with the PoE version- because every other version was more expensive as far as I could find. Even those with less ports.

yeah I went poe cuz i need 8 poe lol

how much did you pay? i am at $160 for mine.

 

[Koop]

yeah I went poe cuz i need 8 poe lol

how much did you pay? i am at $160 for mine.

$114 shipped

 

[TeleFragger]

$114 shipped

nice..beat me by $50!!!

 

[fohdeesha]

So I picked up an ICX7250 and went through all the initial Setup / Config / Licensing Guide - Thank you @fohdeesha for the excellent documentation to get things started, made things very easy. I did cisco courses about 17 years ago and man did it bring back memories. Plan to play with this switch and see what I can learn.

I wanted to ask first, what is best way to keep up on updates? I just happen to notice the webui posted by @Sealside and wow that actually looks nice. What is the best way to update while keeping my configuration? Sorry for such a noob question.

Next up- nearly silent - oh boy that is not my experience so far. This switch is easily the loudest component in my rack by 10 miles. Do you think there is an issue with the thermals in my unit that I can address? Or is it a matter of swapping the stock fans out? I did get the 48 port PoE version- it was a choice based on price (cheaper than any other option I could find). I found from a lot of googling @RoachedCoach 's post proclaiming success with the Sunon MF60101V3-1000U-A99 and Mechatronics MR4020X12B1-RSR fans. Where should I buy fans likes in in NA? Would others still recommend this path? Definitely want to do all I can to cool and quiet down my switch.

Happy I didn't go for the ICX6610 even though I did want the 40GbE. I opted for 7250 since I wanted things to stay quiet and figured let me start out at 10GbE since that's what I have on my TrueNAS box (only issue there is it's onboard rj45 but that's a problem to figure out later). If the 6610 is even louder than this then dayum.

Appreciate the thread and any suggestions anyone wishes to provide.

do the fans spin down and get significantly quieter after it finishes booting? Sounds like they may be stuck in full speed, it should not be that loud once it spins down

 

[Koop]

do the fans spin down and get significantly quieter after it finishes booting? Sounds like they may be stuck in full speed, it should not be that loud once it spins down

Yeah absolutely, but they're still pretty loud even after they spun down from initial boot. Not what I would describe as quiet. I assumed it's the temp of the switch and the fans were still automatically turned up higher perhaps to compensate? Any way I can show what I am seeing?

 

[Andydude]

Yeah absolutely, but they're still pretty loud even after they spun down from initial boot. Not what I would describe as quiet. I assumed it's the temp of the switch and the fans were still automatically turned up higher perhaps to compensate? Any way I can show what I am seeing?

I used the RoachCoach mod and it worked for me: https://forums.servethehome.com/ind...be-40gbe-switching.21107/page-159#post-256375

@deff_janiels also followed the mod and took a good pic here: https://forums.servethehome.com/ind...erful-10gbe-40gbe-switching.21107/post-331607

Bought three Mechatronics MR4020X12B1-RSR from Digikey to replace the three rear fans and a Sunon MF60101V3-1000U-A99 to go directly on the IC heatsink and it's nice and quiet and sits in a rack at around 60 degrees.

SSH@brocade-7250-core-switch#show chassis
The stack unit 1 chassis info:

Power supply 1 (AC - PoE) present, status ok
Power supply 2 not present
Power supply 3 not present

Fan 1 ok, speed (auto): [[1]]<->2
Fan 2 ok, speed (auto): [[1]]<->2
Fan 3 ok, speed (auto): [[1]]<->2

Fan controlled temperature:
        Rule 1/2 (MGMT THERMAL PLANE): 60.0 deg-C
        Rule 2/2 (AIR OUTLET NEAR PSU): 31.0 deg-C

Fan speed switching temperature thresholds:
        Rule 1/2 (MGMT THERMAL PLANE):
                Speed 1: NM<-----> 95       deg-C
                Speed 2:        85<----->105 deg-C (shutdown)
        Rule 2/2 (AIR OUTLET NEAR PSU):
                Speed 1: NM<-----> 41       deg-C
                Speed 2:        34<----->105 deg-C (shutdown)

Fan 1 Air Flow Direction:  Front to Back
Fan 2 Air Flow Direction:  Front to Back
Fan 3 Air Flow Direction:  Front to Back
Slot 1 Current Temperature: 60.0 deg-C (Sensor 1), 31.0 deg-C (Sensor 2)
        Warning level.......: 100.0 deg-C
        Shutdown level......: 105.0 deg-C
Boot Prom MAC : 609c.9f70.97a4
Management MAC: 609c.9f70.97a4

 

[frogtech]

So I picked up an ICX7250 and went through all the initial Setup / Config / Licensing Guide - Thank you @fohdeesha for the excellent documentation to get things started, made things very easy. I did cisco courses about 17 years ago and man did it bring back memories. Plan to play with this switch and see what I can learn.

I wanted to ask first, what is best way to keep up on updates? I just happen to notice the webui posted by @Sealside and wow that actually looks nice. What is the best way to update while keeping my configuration? Sorry for such a noob question.

Next up- nearly silent - oh boy that is not my experience so far. This switch is easily the loudest component in my rack by 10 miles. Do you think there is an issue with the thermals in my unit that I can address? Or is it a matter of swapping the stock fans out? I did get the 48 port PoE version- it was a choice based on price (cheaper than any other option I could find). I found from a lot of googling @RoachedCoach 's post proclaiming success with the Sunon MF60101V3-1000U-A99 and Mechatronics MR4020X12B1-RSR fans. Where should I buy fans likes in in NA? Would others still recommend this path? Definitely want to do all I can to cool and quiet down my switch.

Happy I didn't go for the ICX6610 even though I did want the 40GbE. I opted for 7250 since I wanted things to stay quiet and figured let me start out at 10GbE since that's what I have on my TrueNAS box (only issue there is it's onboard rj45 but that's a problem to figure out later). If the 6610 is even louder than this then dayum.

Appreciate the thread and any suggestions anyone wishes to provide.

I have not had much success in silencing ICX7250-24P switches, which are the 24 port PoE models that have 2 fans. I did some fan swaps, but I can't remember to which models, and it didn't really help much with noise and the temps are still kind of bad. One of my switches is 66c right now and the other is 59c. I added some shiddy lil 60mm slim fans on top of the main ASIC, but I don't think it contributes much besides being a nice cope.

I actually came to this thread seeing if anyone has finally yet settled on the "penultimate" fan for these switches that is a good compromise of both temps and noise.

I don't understand why it's so hard to find a good quality layer 3 switch that doesn't run at a million degrees with 747 fans. Those arista switches in the great deals forum seem tempting, but the unquestioningly better availability of documentation and software support for the ruckus products is just much nicer from QoL stand point, as opposed to having to "grift" around for firmware updates from other users or people who have support contracts

edit: i think i am actually going to try those arctic 15k fans and see how well they do. i'll report back. someone else in this thread seemed to indicate they were fine.

 

[Sealside]

I have not had much success in silencing ...

I don't understand why it's so hard to find a good quality layer 3 switch that doesn't run at a million degrees with 747 fans.

I have a juniper ex3300 24P running for several years with noctuas. It is dead silent appart from the regular white noise. You just pop them in and run.

What you can do on the 7250 is to run noctuas with an additional arduino/esp etc which will basically make the noctua to run full speed always. This is somewhat of a hack requiring additional hardware and software. (Same mod as for the 6610 more or less)

 

[Koop]

You know, looking at the fans on Digikey I see you can order all manner of passive heatsinks and such too. Is there any value to potentially using a different heatsink on the broadcom chip? copper? Maybe a larger one if I can get the pushpins in the right place. Maybe it doesn't matter... hmm.. I wonder.

 

[shremi]

Yeah absolutely, but they're still pretty loud even after they spun down from initial boot. Not what I would describe as quiet. I assumed it's the temp of the switch and the fans were still automatically turned up higher perhaps to compensate? Any way I can show what I am seeing?

I did replace the fans on my 6450 with 120mm fans on top of the lid , now this switch is dead silent , i might get some 7250s in a few weeks and try this method on them.

 

[NachoCDN]

the 192.168.1.1 has a route available as i have BGP routing setup between the pfsense box and the switch. as shown below:View attachment 35155

that's what is so perplexing about why this doesn't work. that being said, i do have a double nat situation going on as the pfsense router is behind another router.

i'd also like to add that the route back to say a client on the 10.5.0.0 seems to work as i'm able to ping 10.5.0.2 from the pfsense router at 192.168.1.1, so pinging in to the switch and the vlan 5 seems to work no problem.

@fohdeesha any ideas? if not that's cool as well!

 

[sic0048]

I don't understand why it's so hard to find a good quality layer 3 switch that doesn't run at a million degrees with 747 fans.

Because "business class" switches like these are intended to be installed in network rooms and server rooms where a low noise threshold is much less important than longevity through adequate cooling. They were never designed to be put in a quiet residential setting.

 

[frogtech]

Actually on one of my switches, one of the fans is having reported as failed, but all 3 of the fans inside the chassis are spinning.

I have an identical setup between two 7250-24Ps, two non-original fans and one of them is wired up to a splitter for a cheapo 60mm fan. Any reason fans would still spin but one header would report a failure?

 

[Koop]

I did replace the fans on my 6450 with 120mm fans on top of the lid , now this switch is dead silent , i might get some 7250s in a few weeks and try this method on them.

Yeah I ordered new fans from Digikey per previous recommendations as well as removed the heatsink off the broadcom chip which I assume is one of the main contributors to fans having to ramp up, hence why everyone is strapping fans to the heatsink.

I'm going to take some time to lap the heatsink and use some of my own thermal paste to see if that'll improve things. Replace the fans per previous mods recommendations and see what happens! Hopefully I'll get things to run cooler and the fans will ramp down on their own. If it's too loud for me even after that I'll be selling a modded switch shortly, hah.

Because "business class" switches like these are intended to be installed in network rooms and server rooms where a low noise threshold is much less important than longevity through adequate cooling. They were never designed to be put in a quiet residential setting.

Absolutely this. Regardless of any recommendations these things were made to be in a data center not my closet. Another thing to consider is the age of the equipment we're dealing with. Who knows how many hours and hours of operation these guys have been chugging away at. Thermal paste dries. Heatsinks fall off. Fans die. Components fail. It's all the nature of the beast when dealing with second hand and older- yet still very capable equipment for our homelab purposes.

Though I did take @fohdeesha's OP as face value. Hey you said nearly silent! haha. It's all good. I enjoy this homelab tinkering work. I don't mind tearing a switch apart just to see if I can make it run cooler and quieter. If I totally bork the thing it's all good. Was curious to see if anyone else had redone heatsink of these guys.



Just some photos of the chip and it's original thermal interface.

BTW here's an off the wall question. I see that that POE is delivered via a separate board and several headers. The headers are clearly labeled for what range of ports PoE is supplied to on each of them. Any idea what would happen if I were to say, leave some disconnect for a range of ports? What if I didn't attach it at all in a an attempt to make the switch non-PoE? Just curious if anyone ever experimented with doing something like that and what negative or possibly positive results may be?

I'm obviously just guessing here. I'm not a hardware expert.

 

[frogtech]

Yeah I ordered new fans from Digikey per previous recommendations as well as removed the heatsink off the broadcom chip which I assume is one of the main contributors to fans having to ramp up, hence why everyone is strapping fans to the heatsink.

I'm going to take some time to lap the heatsink and use some of my own thermal paste to see if that'll improve things. Replace the fans per previous mods recommendations and see what happens! Hopefully I'll get things to run cooler and the fans will ramp down on their own. If it's too loud for me even after that I'll be selling a modded switch shortly, hah.



Absolutely this. Regardless of any recommendations these things were made to be in a data center not my closet. Another thing to consider is the age of the equipment we're dealing with. Who knows how many hours and hours of operation these guys have been chugging away at. Thermal paste dries. Heatsinks fall off. Fans die. Components fail. It's all the nature of the beast when dealing with second hand and older- yet still very capable equipment for our homelab purposes.

Though I did take @fohdeesha's OP as face value. Hey you said nearly silent! haha. It's all good. I enjoy this homelab tinkering work. I don't mind tearing a switch apart just to see if I can make it run cooler and quieter. If I totally bork the thing it's all good. Was curious to see if anyone else had redone heatsink of these guys.

View attachment 35196View attachment 35197

Just some photos of the chip and it's original thermal interface.

BTW here's an off the wall question. I see that that POE is delivered via a separate board and several headers. The headers are clearly labeled for what range of ports PoE is supplied to on each of them. Any idea what would happen if I were to say, leave some disconnect for a range of ports? What if I didn't attach it at all in a an attempt to make the switch non-PoE? Just curious if anyone ever experimented with doing something like that and what negative or possibly positive results may be?

I'm obviously just guessing here. I'm not a hardware expert.

i literally just did a repaste on one of my switches last night, i was curious if it would make a difference. i would say, not really. i used arctic mx4 2019 edition. it was kind of annoying getting that thermal pad off but didn't take too much effort. i would probably apply heat and try to pull it off in one piece, because it is kind of fabric-like? it doesn't really come off like paste at all, it tears and rips. but again i probably wouldn't bother. it didnt make a difference temperature wise.

 

[Koop]

i literally just did a repaste on one of my switches last night, i was curious if it would make a difference. i would say, not really. i used arctic mx4 2019 edition. it was kind of annoying getting that thermal pad off but didn't take too much effort. i would probably apply heat and try to pull it off in one piece, because it is kind of fabric-like? it doesn't really come off like paste at all, it tears and rips.

Yeah why I said I'd lap it off. Literally sand it down a bit and get the aluminum heatsink as smooth as possible. Too bad on not seeing a difference. Will see how I fair.