Drag to reposition cover

Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

EvoDyn

New Member
Jun 23, 2021
2
2
3
Is this what the 10G port adder license looks like? so 2 base + 2 from the license?

SSH@ICX6450-24P>sh lic
Index Lic Mode Lic Name Lid/Serial No Lic Type Status Lic Period Lic Capacity
Stack unit 1:
1 Node Lock ICX6450-10G-LIC-POD dbtF<snip> Normal Active Unlimited 2
I just picked up the same one as you with the same output. I believe you will need a second license based on this example as a fully licensed 6450.
 
Last edited:
  • Like
Reactions: etherbadger

DMFDMinister

New Member
Oct 4, 2020
4
7
3
First post here - love this community already and lots of helpful posts on modding for less noise. Have to run a small lab for VMware certs out of my home office and replacing a stack of 3750s with a 7250-48p and 8x licensed 10Gb ports. Already that is quieter, but not enough for me to be happy yet. I've read all of the fan mod threads multiple times and essentially it seems that there's a comedy of issues with the fan mods leading to people using wood screws to wedge fans on top of the ASIC heatsink to cool it down after putting in slower fans - otherwise the system might ramp up to speed 2 indefinitely or might switch between speed 1 and speed 2 relentlessly since slower fans move less air and cause the ASIC to go to speed 2. Top this with the fact that it sounds like the board itself requires a minimum RPM to boot and you can either put in fans that meet this RPM or you can spoof the signal - which is also quite tedious.

Annoyed slightly by this issue, I started thinking about re-applying thermal paste - because I didn't quite understand how my switch could idle at 58C on the ASIC with 68-70F ambient temps when I literally only had a serial cable plugged into it and nothing else. I think the thing deterring people from doing this is that it's very difficult to do without damaging the switch. The heat sink is mounted with push pins that typically require you to have access to the other side of the board to get them out. Once the pins are in, they're not coming out without some serious persuasion.

So I tried to take the circuit board out only to find that it's literally riveted to the switch chassis and practically impossible. What followed was almost 60 minutes of me trying to get these things out without damaging the circuit board. I think I may have been successful overall. I had to remove everything down to the board to work in the space. Small warning- the fins arekind of edgy - I sliced my finger tips twice getting this out proceeded by dousing them in alcohol when I was cleaning the junk off the heatsink/asic.

View attachment 15993


And what I found is in my opinion utter stupidity. If you've applied thermal compound to a heat sink before, you know that the best practice is a very thin layer of compound to aid the transmission of heat. Too thick and you lose effectiveness. CPUs call for a compound the size of a grain of rice. What I found left me dumbfounded that this is how this equipment was designed. Perhaps I'm missing some of the finer details of enterprise switching...?

Pre-cleaning
View attachment 15994

Post Cleaning
View attachment 15995


Board after the fact - you can see a little scratching around the holes, but that should be fine given they're no circuitry there.
View attachment 15996

So I'm in a spot where I'd like to put a better heatsink on here - preferably one with a small fan integrated. The problem is that the heatsink literally doesn't have a part number and I cannot find any reference for the size / compatibility. I measure it to be approx 65mmx70mm 10mm fins and 2 push-pin mounts. There's nothing out there that fits the bill ... everything on digi and a few other sites are mostly square.

So I might just have to reapply paste and mount the heatsink. I think I have to replace the push-pins as I kind of mangled them getting them out - not really an issue as they're cheap I believe 3MM, and I can put slightly stronger springs on them this time around if I have to go that direction to improve contact with the ASIC.

Anyone know where I can get specs on this heatsink? I imagine re-applying good paste to the heatsink (like MX4 thermal compound or ceramique) could go a long ways here to improving temperatures given the previous situations.

I wanted to follow this posted up with an update. Yes, I need to clean out some dust.

I've been successfully running a modded 7250-48P for the last 9 months 24/7 in my closet and I'm finally getting pics for folks because I had to take down the lab due to a move. I didn't get exactly the temperature results that I wanted, however I did achieve the overall goal of being able to run this switch 5 feet away from my desk behind some accordion-style closet doors with very little disturbance. In hindsight, removing the heat sink probably wasn't the best idea, but I learned quite a bit along the way about heat dissipation and even ended up crimping my own molex power cables.

- Successfully maintained speed 1 at all times. I tested to see how hot I could let it get, so I opened my window on a hot day and it hit speed 2 one time when I hit ambient temps of between 78-80 degrees.

- ASIC Temps generally ranged between 62 and 68 - this is due the limitations I ran into with the heatsink size

- Ultra-Quiet Operations - I don't have a measurement for this, but my R620 running at 10% was louder


I found a company called Advanced Thermal Solutions (ATS) that builds heatsinks, heatsink attachments, and other cooling components. Their product MaxiGRIP and SuperGRIP can attach a heatsink or heatsink & fan to literally any ASIC without requiring pins or screws. It's pretty slick and has many different applications - so all you modders out there - check it out. The reason I had to lean in this direction is because I couldn't find anything on Digi-Key that would work with the pre-dilled holes in the mainboard for the stock heatsink (it's a very unique shape). So I had to find a solution that would fit within the standard 5mm clearance around the 35x35mm ASIC. It took a little bit of trial and error, but I was eventually able to find a combination that worked.

- 3x Mechatronics Fan 40x40x20 fans
- 1x ATS FanSINK 35x35x14.5mm
- 1x 30x30x10mm12vDC Fan
- 1x Crimping Tool
- 1x 10-pack Molex Female House 2510 Pins

The heat sink comes with some pretty decent phase change material on it. Wwhen you re-wire all the fans, you must remember that Brocade has a unique wiring methodology.

Black = Ground
Yellow = 12V
Green = Sensor

So when you re-wire these fans, be careful to match the wires with the right pins!


You can wire the ASIC fan into any one of the chassis fans and it works perfectly fine (only the chassis sensor reports to the mainboard as there is no sensor wire on the ASIC fan). As others have said, the sensor freaks out if your chassis fans are spinning less than a certain RPM (I forget exactly what threshold is, but somewhere in the 7-9K mark) and makes everything run in speed 2. So you need to have fans running fast enough to not trigger that. After much research, I settled on the chassis fans above. Also, I turned the fans to blow IN towards the ASIC (back to front) and this got me down another degree or two.

The FanSINK kit comes with a MaxiGRIP attachment. MaxiGRIP technically needs 7mm clearance on all sides of the ASIC and the actual engineered standard is 5mm, which Brocade follows on the 7250. SuperGRIP only needs 5mm, but SuperGRIP cannot handle FanSINK, so I bought and tested the MaxiGRIP and it's miraculously workable and just barely misses a couple of items on the mainboard you'll see in pictures.

Again, ATS MaxiGRIP and SuperGRIP ASIC attachments can work on almost ANY ASIC and come in varying sizes. Super handy.

Here's some pics of what it looked like. And to finish, I'm not sure I'd recommend this approach to anyone because taking that stock heatsink off was pretty dangerous and I could have slipped and screwed something up. I'm sure there was a right was to do it (like removing the mainboard first), but really, just find a good 40x40mm or 50x50mm fan and figure out how to toss it on top of that stock heatsink and wire it into the power for one of the fans like I did here is probably a lot easier and potentially more effective. But keep this fix in mind for your next project.


20210624_192423.jpg20210624_192447.jpg20210624_192506.jpg20210624_192534.jpg20210624_192559.jpg
 

rootpeer

Member
Oct 19, 2019
58
13
8
Hello!

I have an ICX6450. I have a fiber SFP ONU installed in one of the 10Gb ports.

On switch reboots, sometimes the ONU hangs and I have to remove it from the port and re-insert it.

Is there a way to power-cycle the SFP port so I can automate this?
 

EvoDyn

New Member
Jun 23, 2021
2
2
3
Is this what the 10G port adder license looks like? so 2 base + 2 from the license?

SSH@ICX6450-24P>sh lic
Index Lic Mode Lic Name Lid/Serial No Lic Type Status Lic Period Lic Capacity
Stack unit 1:
1 Node Lock ICX6450-10G-LIC-POD dbtF<snip> Normal Active Unlimited 2
I just picked up the same one as you with the same output. I believe you will need a second license based on this example as a fully licensed 6450.
Ignore what I said above. You are correct based on this and this post to only need a single license to have all four ports 10G on a ICX6450. To be fully licensed one would just need to add a ICX6450-PREM-LIC-SW license for advanced L3 features.

However, I am curious why there would be a second ICX6450-10G-LIC-POD license on that example.
 
  • Like
Reactions: etherbadger

etherbadger

New Member
Jun 20, 2021
3
0
1
Ignore what I said above. You are correct based on this and this post to only need a single license to have all four ports 10G on a ICX6450. To be fully licensed one would just need to add a ICX6450-PREM-LIC-SW license for advanced L3 features.

However, I am curious why there would be a second ICX6450-10G-LIC-POD license on that example.

My 3rd SFP+ port nic arrived today, so I can test it out and see if more than 2 come up at 10GB speeds.

I've got a small fleet of HP prodesk 600 SFF desktops for a proxmox cluster. been using 1gb speeds but want to try out ceph and gluster, so the 10gbe will be very nice.
 

etherbadger

New Member
Jun 20, 2021
3
0
1
My 3rd SFP+ port nic arrived today, so I can test it out and see if more than 2 come up at 10GB speeds.

seems to be working at 10G for 3 ports i have plugged in:

SSH@neuron>sh int br ethernet 1/2/1 to 1/2/4

Port Link State Dupl Speed Trunk Tag Pvid Pri MAC Name
1/2/1 Up Forward Full 10G None Yes 1 0 cc4e.2451.d7c0
1/2/2 Up Forward Full 10G None Yes 1 0 cc4e.2451.d7c0
1/2/3 Up Forward Full 10G None Yes 1 0 cc4e.2451.d7c0
1/2/4 Down None None None None Yes 1 0 cc4e.2451.d7c0
SSH@neuron>sh lic
Index Lic Mode Lic Name Lid/Serial No Lic Type Status Lic Period Lic Capacity
Stack unit 1:
1 Node Lock ICX6450-10G-LIC-POD <> Normal Active Unlimited 2
 

supacupa

New Member
Jun 2, 2021
15
2
3
I recently acquired one of these switches (ICX6610), but can only get two of the four QSFP+ ports on the back to work at all. They work at 40 gbits. I can get one of my QSFP+ cards to connect to one of the SFP+, but they only work at 1 gbit. What do I need to do to either make the the other two back ports to work or to make the front SFP ports work at 10 gbits?

I should note I updated the bootloader to grz10100

The command:
i2c write a 0 feedface00000200ffffffffffffffff 1
returned:
undefined device ID 10 i2c device write failed at 0, ERROR
Maybe I need to get an EEPROM for it?
 
Last edited:

LodeRunner

Active Member
Apr 27, 2019
175
74
28
Hello!

I have an ICX6450. I have a fiber SFP ONU installed in one of the 10Gb ports.

On switch reboots, sometimes the ONU hangs and I have to remove it from the port and re-insert it.

Is there a way to power-cycle the SFP port so I can automate this?
Does going into the port config and issuing a disable / enable reset it? It does that for PoE devices, but I haven't worked with one of those SFP ONU units, so I couldn't tell you if disable actually cuts power to the SFP itself.
 

tozmo

Member
Feb 1, 2017
72
42
18
71
Well, I got my 7450-32zp finally up and running the latest firmware. The 2.5gbe ports are sadly nonfunctional, I can't figure out how I could fix them as the switch has physical damage to the exterior. I have a cobbled USBmini-serial cable, and updated bootrom/firmware from the regular USB port on the switch. copy flash bootrom is no longer a supported command after a certain version, apparently.

If anyone is wondering, it's my first 1u device, and it is LOUD imo. The startup is nutso loud, but even when it settles to regular speed, it's still audible due to the high pitch multiple rooms away. There are three fan settings: loud, jet engine, and then 85c is shutdown. I don't have the skill to do what that one guy did with rigging his own fans and looping wires into his PSU to make it more silent.
 

Dave Corder

Active Member
Dec 21, 2015
160
77
28
39
I recently acquired one of these switches (ICX6610), but can only get two of the four QSFP+ ports on the back to work at all. They work at 40 gbits. I can get one of my QSFP+ cards to connect to one of the SFP+, but they only work at 1 gbit. What do I need to do to either make the the other two back ports to work or to make the front SFP ports work at 10 gbits?

I should note I updated the bootloader to grz10100

The command:
i2c write a 0 feedface00000200ffffffffffffffff 1
returned:
undefined device ID 10 i2c device write failed at 0, ERROR
Maybe I need to get an EEPROM for it?
Two of the QSFP+ ports on the back are 40 Gbps ports, the other two are 4x 10 Gbps breakout ports (that cannot be used as a single 40 Gbps port).
 

SuperMiguel

New Member
Jun 17, 2021
8
0
1
Which sfp+ Cable you recommended for the Brocade ICX6450? I need about 3ft of cabling also which nic?

Also is the OP still providing free licenses? I need to unlock the 10 g ports on the icx6450 if i end up getting one…

last question;) are there any switches with more than 4 10g ports that is not as loud as the icx6610?? I have one and it is pretty loud. Are the noise levels between regular 24 and 48poe the same?
 
Last edited:

epicurean

Active Member
Sep 29, 2014
705
51
28
The network in my home is suddenly very slow, especially when viewing the IP cameras. I used to use wireguard on my phone and can see all my ip cameras almost instantly, but now a few cameras come on and off and very sluggish. Plex also sluggish with nvidia shield. Intermittenly network connection also goes off. I have rebooted my switches but does not seem to make a difference.
I am not sure where to start troubleshooting. A dedicated PFsense router is connected to a ICX6450 48 POE switch, which is also connectedt o a unifi 16 POE switch. Half of my IP cams are in the unifi switch (which I intend to migrate to the ICX6450), and the other half in the ICX6450. 3 x R610 ruckus APs are connected to the ICX6450 , as is a single ruckus H510 as well.
The IP cameras in the ICX6450 are on vlan 80, whilst those in the unifi switch is on the main vlan 1 - in the process of being moved to vlan 80.

Any help appreciated. much thanks!
 

supacupa

New Member
Jun 2, 2021
15
2
3
Two of the QSFP+ ports on the back are 40 Gbps ports, the other two are 4x 10 Gbps breakout ports (that cannot be used as a single 40 Gbps port).
How does that work? I plug a 40 Gbps cable and it runs at 10g? or there are 4x10 Gbps connections?
 

DavidRa

Infrastructure Architect
Aug 3, 2015
297
134
43
Central Coast of NSW
www.pdconsec.net
How does that work? I plug a 40 Gbps cable and it runs at 10g? or there are 4x10 Gbps connections?
You would need a breakout cable - a DAC (which has the QSFP module on one end and it splits out to four 10Gbps SFP+ modules). One example is this one 40G QSFP+ Breakout Active Optical Cables (AOC) - not a recommendation, just first I found as an example.
 

supacupa

New Member
Jun 2, 2021
15
2
3
You would need a breakout cable - a DAC (which has the QSFP module on one end and it splits out to four 10Gbps SFP+ modules). One example is this one 40G QSFP+ Breakout Active Optical Cables (AOC) - not a recommendation, just first I found as an example.
OK, so it's just the reverse of what's up front. I think I understand. I have a breakout cable. Now I need to figure out how to unlock 10Gbit speeds.
 

rootpeer

Member
Oct 19, 2019
58
13
8
Does going into the port config and issuing a disable / enable reset it? It does that for PoE devices, but I haven't worked with one of those SFP ONU units, so I couldn't tell you if disable actually cuts power to the SFP itself.
No unfortunately it does not. It just disables the networking.
 

mmx

New Member
Dec 18, 2015
9
9
3
Hi all, longtime lurker, first time poster (in this thread, any way).

Thanks to @fohdeesha and the immense amount of knowledge found in this thread by numerous contributors, I bought 4x 6610s last year to replace the crappy Cisco SG200/SG220s we have at work. I've been taking my time with the migration as it's my first time working with such powerful devices and I want to make sure I've got everything covered. My goal is to offload inter-VLAN routing from pfSense onto the 6610s (via a transit VLAN).

I want to share a small discovery I made today in hopes of making these switches friendlier to newcomers like me.

One of my biggest challenges was managing ACLs through the CLI. I'm used to centrally managing firewall rules through a web panel, so having a non-visual interface takes some getting used to. Some of my concerns & questions were:
  1. How can I add/modify/remove a single ACL entry in an access-list?
  2. How can I reorder ACLs in an access-list?
  3. How can I write ACLs more efficiently? Is there an equivalent to pfSense's IP/Port aliases?
This is where Brocade Network Advisor comes in. Now I know it's EOL, but it's still quite a valuable tool for someone like me as it answers all the questions above. BNA is probably nothing new to the experienced out there, yet it's barely mentioned in this thread apart from @Jason Antes bringing it up in April 2021 and last week. I think it deserves some recognition even if it's mostly archaic and superseded by Ruckus.

One of the greatest features in BNA is the fact that you can create Networks, Network Groups, Services and Service Groups. These are basically an alternative to pfSense's aliases, and they're extremely useful when writing ACLs for several domain networks. You can even include Groups in Groups, equivalent to referencing an alias inside another alias in pfSense. Just this feature alone avoids having to repeat yourself, thus avoiding mistakes when writing ACLs for dozens of networks with similar rules. I've read through the documentation provided by Ruckus, and there's no way to replicate this functionality through the CLI (as far as I can tell).

For example, I want to create a single Service Group for all Active Directory Domain Controller ports. Here's a few screenshots to showcase the process. Service ports are protocol-specific, however you will still have to create separate ACLs for TCP and UDP. The final screenshot will show you how every individual entry gets created automagically with only 1-2 entries created in BNA.

For anyone interested, the version I have found online is 14.2.12 (IP only, no SAN support) and it doesn't require a paid licence. I've deployed this particular version without issue.

I hope this helps someone out there!
 
Last edited:

gregsachs

Active Member
Aug 14, 2018
426
125
43
Which sfp+ Cable you recommended for the Brocade ICX6450? I need about 3ft of cabling also which nic?

Also is the OP still providing free licenses? I need to unlock the 10 g ports on the icx6450 if i end up getting one…

last question;) are there any switches with more than 4 10g ports that is not as loud as the icx6610?? I have one and it is pretty loud. Are the noise levels between regular 24 and 48poe the same?
For a 3 ft run I'd use a dac, more robust. Stay at 5m or less, the 6450 doesn't like 10m dacs