Drag to reposition cover

Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

Jason Antes

Active Member
Feb 28, 2020
230
78
28
Twin Cities
just FYI BNA has been end of life/discontinued for quite some time so I probably wouldn't sink too much time into setting it up and finding licenses if you haven't already @CIR-Engineering
Yeah, it has. I happen to have it and the licenses so it was easy but it can be worth setting up and using given certain needs with older hardware even if you have to find a generous person. It certainly wouldn't be worth buying at this point. It was just another option for this person and I'm not up on if there are better options to get that single pane of glass currently. :)
 
  • Like
Reactions: fohdeesha

firworks

Member
May 7, 2021
38
29
18
ICX6610-48P arrived today from eBay. Followed the docs and it worked perfectly. It's all up and updated and connected. Thank you @fohdeesha for the guide. No way I would have ever figured out how to get this thing going on my own.

So far the only problem is the fan in one of the two supplies is much louder than the other. They're both Rev. A but I think it's actually just the fan going out or a bad bearing or something. I guess I'll just have to learn to live with it as the replacement supplies are incredibly expensive compared to the whole switch with two supplies and fan modules.

I'm still waiting for a few cards to arrive to try out 10/40gbe but for now it's working nicely at 1g.
 

CIR-Engineering

I am a functional adult?
Jan 14, 2021
153
61
28
49
Chicago USA
www.cir-engineering.com
it can stack a ton of different ways, with one link linearly or two in a ring, or two parallel, etc, but it will only stack with the same model (other 6450's). It had a limited stacking mode as well where it would stack with a 6610, but it just acted like a dumb L2 port extender and if the 6610 died, the 6450 would as well. long story short, if you want to use the stacking feature, you're gunna have to buy another 6450
Thanks for the clarification.
 

Sundar

New Member
Oct 31, 2018
19
3
3
Thanks fohdeesha, for this fantastic information.
I am thinking of buying a Brocade ICX-6450-24P.
I only want the basic switch functionality, along with the ability to use one or more 10G SFP+ ports for uplinking to another switch. I will not be using PoE.

Will I need special licenses for this?
 

ccie4526

Active Member
Jan 25, 2021
167
134
43
Thanks fohdeesha, for this fantastic information.
I am thinking of buying a Brocade ICX-6450-24P.
I only want the basic switch functionality, along with the ability to use one or more 10G SFP+ ports for uplinking to another switch. I will not be using PoE.

Will I need special licenses for this?
If you don't need the PoE, then look for the ICX6450-24. Save a few cents. You will only need the special licenses if your switch doesn't already have the 2-port 10G adder license.... it supports 2x 10G by default, the adder license enables the second two ports.

I needed the PoE capability on my switch, so picked up the 6450-24P.
 

Sundar

New Member
Oct 31, 2018
19
3
3
ccie4526: Thanks for your quick reply.
I will not be using PoE, but I only find the PoE versions for sale on eBay.
If I am not using PoE, can I disconnect both fans? Do the fans use standard 4-pin connectors?
If I want to enable the second 2 10G ports, how do I get the adder license?
 

Sundar

New Member
Oct 31, 2018
19
3
3
And I suppose the 10G uplink will work (at 10Gbps) with a different model of switch also - I have a Quanta LB9A uplink switch
 

etherbadger

New Member
Jun 20, 2021
4
0
1
Is this what the 10G port adder license looks like? so 2 base + 2 from the license?

SSH@ICX6450-24P>sh lic
Index Lic Mode Lic Name Lid/Serial No Lic Type Status Lic Period Lic Capacity
Stack unit 1:
1 Node Lock ICX6450-10G-LIC-POD dbtF<snip> Normal Active Unlimited 2
 

EvoDyn

New Member
Jun 23, 2021
18
12
3
Is this what the 10G port adder license looks like? so 2 base + 2 from the license?

SSH@ICX6450-24P>sh lic
Index Lic Mode Lic Name Lid/Serial No Lic Type Status Lic Period Lic Capacity
Stack unit 1:
1 Node Lock ICX6450-10G-LIC-POD dbtF<snip> Normal Active Unlimited 2
I just picked up the same one as you with the same output. I believe you will need a second license based on this example as a fully licensed 6450.
 
Last edited:
  • Like
Reactions: etherbadger

DMFDMinister

New Member
Oct 4, 2020
4
10
3
First post here - love this community already and lots of helpful posts on modding for less noise. Have to run a small lab for VMware certs out of my home office and replacing a stack of 3750s with a 7250-48p and 8x licensed 10Gb ports. Already that is quieter, but not enough for me to be happy yet. I've read all of the fan mod threads multiple times and essentially it seems that there's a comedy of issues with the fan mods leading to people using wood screws to wedge fans on top of the ASIC heatsink to cool it down after putting in slower fans - otherwise the system might ramp up to speed 2 indefinitely or might switch between speed 1 and speed 2 relentlessly since slower fans move less air and cause the ASIC to go to speed 2. Top this with the fact that it sounds like the board itself requires a minimum RPM to boot and you can either put in fans that meet this RPM or you can spoof the signal - which is also quite tedious.

Annoyed slightly by this issue, I started thinking about re-applying thermal paste - because I didn't quite understand how my switch could idle at 58C on the ASIC with 68-70F ambient temps when I literally only had a serial cable plugged into it and nothing else. I think the thing deterring people from doing this is that it's very difficult to do without damaging the switch. The heat sink is mounted with push pins that typically require you to have access to the other side of the board to get them out. Once the pins are in, they're not coming out without some serious persuasion.

So I tried to take the circuit board out only to find that it's literally riveted to the switch chassis and practically impossible. What followed was almost 60 minutes of me trying to get these things out without damaging the circuit board. I think I may have been successful overall. I had to remove everything down to the board to work in the space. Small warning- the fins arekind of edgy - I sliced my finger tips twice getting this out proceeded by dousing them in alcohol when I was cleaning the junk off the heatsink/asic.

View attachment 15993


And what I found is in my opinion utter stupidity. If you've applied thermal compound to a heat sink before, you know that the best practice is a very thin layer of compound to aid the transmission of heat. Too thick and you lose effectiveness. CPUs call for a compound the size of a grain of rice. What I found left me dumbfounded that this is how this equipment was designed. Perhaps I'm missing some of the finer details of enterprise switching...?

Pre-cleaning
View attachment 15994

Post Cleaning
View attachment 15995


Board after the fact - you can see a little scratching around the holes, but that should be fine given they're no circuitry there.
View attachment 15996

So I'm in a spot where I'd like to put a better heatsink on here - preferably one with a small fan integrated. The problem is that the heatsink literally doesn't have a part number and I cannot find any reference for the size / compatibility. I measure it to be approx 65mmx70mm 10mm fins and 2 push-pin mounts. There's nothing out there that fits the bill ... everything on digi and a few other sites are mostly square.

So I might just have to reapply paste and mount the heatsink. I think I have to replace the push-pins as I kind of mangled them getting them out - not really an issue as they're cheap I believe 3MM, and I can put slightly stronger springs on them this time around if I have to go that direction to improve contact with the ASIC.

Anyone know where I can get specs on this heatsink? I imagine re-applying good paste to the heatsink (like MX4 thermal compound or ceramique) could go a long ways here to improving temperatures given the previous situations.

I wanted to follow this posted up with an update. Yes, I need to clean out some dust.

I've been successfully running a modded 7250-48P for the last 9 months 24/7 in my closet and I'm finally getting pics for folks because I had to take down the lab due to a move. I didn't get exactly the temperature results that I wanted, however I did achieve the overall goal of being able to run this switch 5 feet away from my desk behind some accordion-style closet doors with very little disturbance. In hindsight, removing the heat sink probably wasn't the best idea, but I learned quite a bit along the way about heat dissipation and even ended up crimping my own molex power cables.

- Successfully maintained speed 1 at all times. I tested to see how hot I could let it get, so I opened my window on a hot day and it hit speed 2 one time when I hit ambient temps of between 78-80 degrees.

- ASIC Temps generally ranged between 62 and 68 - this is due the limitations I ran into with the heatsink size

- Ultra-Quiet Operations - I don't have a measurement for this, but my R620 running at 10% was louder


I found a company called Advanced Thermal Solutions (ATS) that builds heatsinks, heatsink attachments, and other cooling components. Their product MaxiGRIP and SuperGRIP can attach a heatsink or heatsink & fan to literally any ASIC without requiring pins or screws. It's pretty slick and has many different applications - so all you modders out there - check it out. The reason I had to lean in this direction is because I couldn't find anything on Digi-Key that would work with the pre-dilled holes in the mainboard for the stock heatsink (it's a very unique shape). So I had to find a solution that would fit within the standard 5mm clearance around the 35x35mm ASIC. It took a little bit of trial and error, but I was eventually able to find a combination that worked.

- 3x Mechatronics Fan 40x40x20 fans
- 1x ATS FanSINK 35x35x14.5mm
- 1x 30x30x10mm12vDC Fan
- 1x Crimping Tool
- 1x 10-pack Molex Female House 2510 Pins

The heat sink comes with some pretty decent phase change material on it. Wwhen you re-wire all the fans, you must remember that Brocade has a unique wiring methodology.

Black = Ground
Yellow = 12V
Green = Sensor

So when you re-wire these fans, be careful to match the wires with the right pins!


You can wire the ASIC fan into any one of the chassis fans and it works perfectly fine (only the chassis sensor reports to the mainboard as there is no sensor wire on the ASIC fan). As others have said, the sensor freaks out if your chassis fans are spinning less than a certain RPM (I forget exactly what threshold is, but somewhere in the 7-9K mark) and makes everything run in speed 2. So you need to have fans running fast enough to not trigger that. After much research, I settled on the chassis fans above. Also, I turned the fans to blow IN towards the ASIC (back to front) and this got me down another degree or two.

The FanSINK kit comes with a MaxiGRIP attachment. MaxiGRIP technically needs 7mm clearance on all sides of the ASIC and the actual engineered standard is 5mm, which Brocade follows on the 7250. SuperGRIP only needs 5mm, but SuperGRIP cannot handle FanSINK, so I bought and tested the MaxiGRIP and it's miraculously workable and just barely misses a couple of items on the mainboard you'll see in pictures.

Again, ATS MaxiGRIP and SuperGRIP ASIC attachments can work on almost ANY ASIC and come in varying sizes. Super handy.

Here's some pics of what it looked like. And to finish, I'm not sure I'd recommend this approach to anyone because taking that stock heatsink off was pretty dangerous and I could have slipped and screwed something up. I'm sure there was a right was to do it (like removing the mainboard first), but really, just find a good 40x40mm or 50x50mm fan and figure out how to toss it on top of that stock heatsink and wire it into the power for one of the fans like I did here is probably a lot easier and potentially more effective. But keep this fix in mind for your next project.


20210624_192423.jpg20210624_192447.jpg20210624_192506.jpg20210624_192534.jpg20210624_192559.jpg
 

rootpeer

Member
Oct 19, 2019
86
17
8
Hello!

I have an ICX6450. I have a fiber SFP ONU installed in one of the 10Gb ports.

On switch reboots, sometimes the ONU hangs and I have to remove it from the port and re-insert it.

Is there a way to power-cycle the SFP port so I can automate this?
 

EvoDyn

New Member
Jun 23, 2021
18
12
3
Is this what the 10G port adder license looks like? so 2 base + 2 from the license?

SSH@ICX6450-24P>sh lic
Index Lic Mode Lic Name Lid/Serial No Lic Type Status Lic Period Lic Capacity
Stack unit 1:
1 Node Lock ICX6450-10G-LIC-POD dbtF<snip> Normal Active Unlimited 2
I just picked up the same one as you with the same output. I believe you will need a second license based on this example as a fully licensed 6450.
Ignore what I said above. You are correct based on this and this post to only need a single license to have all four ports 10G on a ICX6450. To be fully licensed one would just need to add a ICX6450-PREM-LIC-SW license for advanced L3 features.

However, I am curious why there would be a second ICX6450-10G-LIC-POD license on that example.
 
  • Like
Reactions: etherbadger

etherbadger

New Member
Jun 20, 2021
4
0
1
Ignore what I said above. You are correct based on this and this post to only need a single license to have all four ports 10G on a ICX6450. To be fully licensed one would just need to add a ICX6450-PREM-LIC-SW license for advanced L3 features.

However, I am curious why there would be a second ICX6450-10G-LIC-POD license on that example.

My 3rd SFP+ port nic arrived today, so I can test it out and see if more than 2 come up at 10GB speeds.

I've got a small fleet of HP prodesk 600 SFF desktops for a proxmox cluster. been using 1gb speeds but want to try out ceph and gluster, so the 10gbe will be very nice.
 

etherbadger

New Member
Jun 20, 2021
4
0
1
My 3rd SFP+ port nic arrived today, so I can test it out and see if more than 2 come up at 10GB speeds.

seems to be working at 10G for 3 ports i have plugged in:

SSH@neuron>sh int br ethernet 1/2/1 to 1/2/4

Port Link State Dupl Speed Trunk Tag Pvid Pri MAC Name
1/2/1 Up Forward Full 10G None Yes 1 0 cc4e.2451.d7c0
1/2/2 Up Forward Full 10G None Yes 1 0 cc4e.2451.d7c0
1/2/3 Up Forward Full 10G None Yes 1 0 cc4e.2451.d7c0
1/2/4 Down None None None None Yes 1 0 cc4e.2451.d7c0
SSH@neuron>sh lic
Index Lic Mode Lic Name Lid/Serial No Lic Type Status Lic Period Lic Capacity
Stack unit 1:
1 Node Lock ICX6450-10G-LIC-POD <> Normal Active Unlimited 2
 

supacupa

New Member
Jun 2, 2021
15
2
3
I recently acquired one of these switches (ICX6610), but can only get two of the four QSFP+ ports on the back to work at all. They work at 40 gbits. I can get one of my QSFP+ cards to connect to one of the SFP+, but they only work at 1 gbit. What do I need to do to either make the the other two back ports to work or to make the front SFP ports work at 10 gbits?

I should note I updated the bootloader to grz10100

The command:
i2c write a 0 feedface00000200ffffffffffffffff 1
returned:
undefined device ID 10 i2c device write failed at 0, ERROR
Maybe I need to get an EEPROM for it?
 
Last edited:

LodeRunner

Active Member
Apr 27, 2019
553
235
43
Hello!

I have an ICX6450. I have a fiber SFP ONU installed in one of the 10Gb ports.

On switch reboots, sometimes the ONU hangs and I have to remove it from the port and re-insert it.

Is there a way to power-cycle the SFP port so I can automate this?
Does going into the port config and issuing a disable / enable reset it? It does that for PoE devices, but I haven't worked with one of those SFP ONU units, so I couldn't tell you if disable actually cuts power to the SFP itself.
 

tozmo

Active Member
Feb 1, 2017
147
105
43
75
Well, I got my 7450-32zp finally up and running the latest firmware. The 2.5gbe ports are sadly nonfunctional, I can't figure out how I could fix them as the switch has physical damage to the exterior. I have a cobbled USBmini-serial cable, and updated bootrom/firmware from the regular USB port on the switch. copy flash bootrom is no longer a supported command after a certain version, apparently.

If anyone is wondering, it's my first 1u device, and it is LOUD imo. The startup is nutso loud, but even when it settles to regular speed, it's still audible due to the high pitch multiple rooms away. There are three fan settings: loud, jet engine, and then 85c is shutdown. I don't have the skill to do what that one guy did with rigging his own fans and looping wires into his PSU to make it more silent.