Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

[CIR-Engineering]

Well, from reading this, it does look like the 6450 supports linear stacking...

https://usermanual.wiki/Ruckus/fastiron08020aswitchstackingguide.377537056/html#pf29

craigr

 

[CIR-Engineering]

So in your use case, what does stacking get you beyond a single management interface that properly configured STP doesn't?

And again, it doesn't matter if the code train you're supports stacking because stacking across different models is not supported. Additionally, Ruckus also notes that stacked switches MUST run the same firmware.

It gets me no benefit other than a single management interface. That seems cool and I just want to learn in the lab because I have never setup a stack before.

I think I have read a number of post where people have successfully stacked using different models? I'm on about page ~125 of this thread.

I did not know the switches had to be on the same firmware. This may kybosh the whole thing or maybe I could downgrade the 7150; haven't looked yet to see if it can be backdated far enough. May not be worth it anyway.

Thanks for your input,
craigr

 

[Jason Antes]

You could get the old Brocade Network advisor to put everything into 1 interface to manage. There are some with *licenses* out there. I run my VDX and ICX through BNA to manage them.

 

[sergi0]

With a new cable, everything worked fine. The right config for the serial cable is this one :
DTE Pinouts RTS (1), DTR(2), TXD (3), GND (4), GND (5), RXD (6), DSR (7), CTS (8)

S.

I'll try another cable. Do you have a precise reference in mind ? Something I can order via Amazon and test quickly ?
S.

 

[Tcalp]

I am working at configuring a Brocade 6450-48 , anyone around that could help with some IP routing / VLANs so that our address space will work ?

 

[fohdeesha]

Well, from reading this, it does look like the 6450 supports linear stacking...

https://usermanual.wiki/Ruckus/fastiron08020aswitchstackingguide.377537056/html#pf29

craigr

it can stack a ton of different ways, with one link linearly or two in a ring, or two parallel, etc, but it will only stack with the same model (other 6450's). It had a limited stacking mode as well where it would stack with a 6610, but it just acted like a dumb L2 port extender and if the 6610 died, the 6450 would as well. long story short, if you want to use the stacking feature, you're gunna have to buy another 6450

 

[fohdeesha]

You could get the old Brocade Network advisor to put everything into 1 interface to manage. There are some with *licenses* out there. I run my VDX and ICX through BNA to manage them.

just FYI BNA has been end of life/discontinued for quite some time so I probably wouldn't sink too much time into setting it up and finding licenses if you haven't already @CIR-Engineering

 

[Jason Antes]

just FYI BNA has been end of life/discontinued for quite some time so I probably wouldn't sink too much time into setting it up and finding licenses if you haven't already @CIR-Engineering

Yeah, it has. I happen to have it and the licenses so it was easy but it can be worth setting up and using given certain needs with older hardware even if you have to find a generous person. It certainly wouldn't be worth buying at this point. It was just another option for this person and I'm not up on if there are better options to get that single pane of glass currently.

 

[firworks]

ICX6610-48P arrived today from eBay. Followed the docs and it worked perfectly. It's all up and updated and connected. Thank you @fohdeesha for the guide. No way I would have ever figured out how to get this thing going on my own.

So far the only problem is the fan in one of the two supplies is much louder than the other. They're both Rev. A but I think it's actually just the fan going out or a bad bearing or something. I guess I'll just have to learn to live with it as the replacement supplies are incredibly expensive compared to the whole switch with two supplies and fan modules.

I'm still waiting for a few cards to arrive to try out 10/40gbe but for now it's working nicely at 1g.

 

[CIR-Engineering]

it can stack a ton of different ways, with one link linearly or two in a ring, or two parallel, etc, but it will only stack with the same model (other 6450's). It had a limited stacking mode as well where it would stack with a 6610, but it just acted like a dumb L2 port extender and if the 6610 died, the 6450 would as well. long story short, if you want to use the stacking feature, you're gunna have to buy another 6450

Thanks for the clarification.

 

[Sundar]

Thanks fohdeesha, for this fantastic information.
I am thinking of buying a Brocade ICX-6450-24P.
I only want the basic switch functionality, along with the ability to use one or more 10G SFP+ ports for uplinking to another switch. I will not be using PoE.

Will I need special licenses for this?

 

[ccie4526]

Thanks fohdeesha, for this fantastic information.
I am thinking of buying a Brocade ICX-6450-24P.
I only want the basic switch functionality, along with the ability to use one or more 10G SFP+ ports for uplinking to another switch. I will not be using PoE.

Will I need special licenses for this?

If you don't need the PoE, then look for the ICX6450-24. Save a few cents. You will only need the special licenses if your switch doesn't already have the 2-port 10G adder license.... it supports 2x 10G by default, the adder license enables the second two ports.

I needed the PoE capability on my switch, so picked up the 6450-24P.

 

[Sundar]

ccie4526: Thanks for your quick reply.
I will not be using PoE, but I only find the PoE versions for sale on eBay.
If I am not using PoE, can I disconnect both fans? Do the fans use standard 4-pin connectors?
If I want to enable the second 2 10G ports, how do I get the adder license?

 

[Sundar]

And I suppose the 10G uplink will work (at 10Gbps) with a different model of switch also - I have a Quanta LB9A uplink switch

 

[fohdeesha]

If I want to enable the second 2 10G ports, how do I get the adder license?

the second sentence in this entire thread

 

[etherbadger]

Is this what the 10G port adder license looks like? so 2 base + 2 from the license?

SSH@ICX6450-24P>sh lic
Index Lic Mode Lic Name Lid/Serial No Lic Type Status Lic Period Lic Capacity
Stack unit 1:
1 Node Lock ICX6450-10G-LIC-POD dbtF<snip> Normal Active Unlimited 2

 

[EvoDyn]

Is this what the 10G port adder license looks like? so 2 base + 2 from the license?

SSH@ICX6450-24P>sh lic
Index Lic Mode Lic Name Lid/Serial No Lic Type Status Lic Period Lic Capacity
Stack unit 1:
1 Node Lock ICX6450-10G-LIC-POD dbtF<snip> Normal Active Unlimited 2

I just picked up the same one as you with the same output. I believe you will need a second license based on this example as a fully licensed 6450.

 

[DMFDMinister]

First post here - love this community already and lots of helpful posts on modding for less noise. Have to run a small lab for VMware certs out of my home office and replacing a stack of 3750s with a 7250-48p and 8x licensed 10Gb ports. Already that is quieter, but not enough for me to be happy yet. I've read all of the fan mod threads multiple times and essentially it seems that there's a comedy of issues with the fan mods leading to people using wood screws to wedge fans on top of the ASIC heatsink to cool it down after putting in slower fans - otherwise the system might ramp up to speed 2 indefinitely or might switch between speed 1 and speed 2 relentlessly since slower fans move less air and cause the ASIC to go to speed 2. Top this with the fact that it sounds like the board itself requires a minimum RPM to boot and you can either put in fans that meet this RPM or you can spoof the signal - which is also quite tedious.

Annoyed slightly by this issue, I started thinking about re-applying thermal paste - because I didn't quite understand how my switch could idle at 58C on the ASIC with 68-70F ambient temps when I literally only had a serial cable plugged into it and nothing else. I think the thing deterring people from doing this is that it's very difficult to do without damaging the switch. The heat sink is mounted with push pins that typically require you to have access to the other side of the board to get them out. Once the pins are in, they're not coming out without some serious persuasion.

So I tried to take the circuit board out only to find that it's literally riveted to the switch chassis and practically impossible. What followed was almost 60 minutes of me trying to get these things out without damaging the circuit board. I think I may have been successful overall. I had to remove everything down to the board to work in the space. Small warning- the fins arekind of edgy - I sliced my finger tips twice getting this out proceeded by dousing them in alcohol when I was cleaning the junk off the heatsink/asic.

View attachment 15993


And what I found is in my opinion utter stupidity. If you've applied thermal compound to a heat sink before, you know that the best practice is a very thin layer of compound to aid the transmission of heat. Too thick and you lose effectiveness. CPUs call for a compound the size of a grain of rice. What I found left me dumbfounded that this is how this equipment was designed. Perhaps I'm missing some of the finer details of enterprise switching...?

Pre-cleaning
View attachment 15994

Post Cleaning
View attachment 15995


Board after the fact - you can see a little scratching around the holes, but that should be fine given they're no circuitry there.
View attachment 15996

So I'm in a spot where I'd like to put a better heatsink on here - preferably one with a small fan integrated. The problem is that the heatsink literally doesn't have a part number and I cannot find any reference for the size / compatibility. I measure it to be approx 65mmx70mm 10mm fins and 2 push-pin mounts. There's nothing out there that fits the bill ... everything on digi and a few other sites are mostly square.

So I might just have to reapply paste and mount the heatsink. I think I have to replace the push-pins as I kind of mangled them getting them out - not really an issue as they're cheap I believe 3MM, and I can put slightly stronger springs on them this time around if I have to go that direction to improve contact with the ASIC.

Anyone know where I can get specs on this heatsink? I imagine re-applying good paste to the heatsink (like MX4 thermal compound or ceramique) could go a long ways here to improving temperatures given the previous situations.

I wanted to follow this posted up with an update. Yes, I need to clean out some dust.

I've been successfully running a modded 7250-48P for the last 9 months 24/7 in my closet and I'm finally getting pics for folks because I had to take down the lab due to a move. I didn't get exactly the temperature results that I wanted, however I did achieve the overall goal of being able to run this switch 5 feet away from my desk behind some accordion-style closet doors with very little disturbance. In hindsight, removing the heat sink probably wasn't the best idea, but I learned quite a bit along the way about heat dissipation and even ended up crimping my own molex power cables.

- Successfully maintained speed 1 at all times. I tested to see how hot I could let it get, so I opened my window on a hot day and it hit speed 2 one time when I hit ambient temps of between 78-80 degrees.

- ASIC Temps generally ranged between 62 and 68 - this is due the limitations I ran into with the heatsink size

- Ultra-Quiet Operations - I don't have a measurement for this, but my R620 running at 10% was louder


I found a company called Advanced Thermal Solutions (ATS) that builds heatsinks, heatsink attachments, and other cooling components. Their product MaxiGRIP and SuperGRIP can attach a heatsink or heatsink & fan to literally any ASIC without requiring pins or screws. It's pretty slick and has many different applications - so all you modders out there - check it out. The reason I had to lean in this direction is because I couldn't find anything on Digi-Key that would work with the pre-dilled holes in the mainboard for the stock heatsink (it's a very unique shape). So I had to find a solution that would fit within the standard 5mm clearance around the 35x35mm ASIC. It took a little bit of trial and error, but I was eventually able to find a combination that worked.

- 3x Mechatronics Fan 40x40x20 fans
- 1x ATS FanSINK 35x35x14.5mm
- 1x 30x30x10mm12vDC Fan
- 1x Crimping Tool
- 1x 10-pack Molex Female House 2510 Pins

The heat sink comes with some pretty decent phase change material on it. Wwhen you re-wire all the fans, you must remember that Brocade has a unique wiring methodology.

Black = Ground
Yellow = 12V
Green = Sensor

So when you re-wire these fans, be careful to match the wires with the right pins!


You can wire the ASIC fan into any one of the chassis fans and it works perfectly fine (only the chassis sensor reports to the mainboard as there is no sensor wire on the ASIC fan). As others have said, the sensor freaks out if your chassis fans are spinning less than a certain RPM (I forget exactly what threshold is, but somewhere in the 7-9K mark) and makes everything run in speed 2. So you need to have fans running fast enough to not trigger that. After much research, I settled on the chassis fans above. Also, I turned the fans to blow IN towards the ASIC (back to front) and this got me down another degree or two.

The FanSINK kit comes with a MaxiGRIP attachment. MaxiGRIP technically needs 7mm clearance on all sides of the ASIC and the actual engineered standard is 5mm, which Brocade follows on the 7250. SuperGRIP only needs 5mm, but SuperGRIP cannot handle FanSINK, so I bought and tested the MaxiGRIP and it's miraculously workable and just barely misses a couple of items on the mainboard you'll see in pictures.

Again, ATS MaxiGRIP and SuperGRIP ASIC attachments can work on almost ANY ASIC and come in varying sizes. Super handy.

Here's some pics of what it looked like. And to finish, I'm not sure I'd recommend this approach to anyone because taking that stock heatsink off was pretty dangerous and I could have slipped and screwed something up. I'm sure there was a right was to do it (like removing the mainboard first), but really, just find a good 40x40mm or 50x50mm fan and figure out how to toss it on top of that stock heatsink and wire it into the power for one of the fans like I did here is probably a lot easier and potentially more effective. But keep this fix in mind for your next project.

 

[rootpeer]

Hello!

I have an ICX6450. I have a fiber SFP ONU installed in one of the 10Gb ports.

On switch reboots, sometimes the ONU hangs and I have to remove it from the port and re-insert it.

Is there a way to power-cycle the SFP port so I can automate this?

 

[Scarlet]

Posting this here for visibility, as I'm mainly trying to stack multiple 6450s.

Which fiber should I install in a new building?