Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

mattaw

Member
Jul 30, 2018
51
15
8
Nice purchase! Join the lucky club as I got 3x for $160 icx6610 48P's.

Does anyone have a scrap mainboard? I need two replacement push pins for the main routing IC heatsinks. I guess they failed with heat and age.

IMG_20181109_223624.jpg

I'm also getting a poe PD error on port 48. It powers fine, but won't sense current etc. nor light the poe led. Swapped the Poe board into another switch and the problem followed the board. Just scrubbed the board as it was dirty but all the components look ok to me. Thoughts?
 
Last edited:
  • Like
Reactions: fohdeesha

fohdeesha

Kaini Industries
Nov 20, 2016
1,959
1,783
113
29
fohdeesha.com
never seen those pins fail, crazy. Wonder if it got knocked around? I have a junk ICX6650, it might have the same heatsinks/pins from the looks of it, I'll take a look.

Wanna do me a huge favor, if the heatsink is free now, can you move it to the side and get the text/model number of the ASIC? They've kept it incredibly secret - I know it's a Marvell Cheetah 5, which I'm pretty sure was a custom build just for Brocade, but it would be nice to have some verification/ actual P/N

in my experience, PD errors that follow the board on a specific port mean that port took a pretty nasty indirect or direct static hit (usually by a PoE device like a camera that was outdoors). No way to fix that I know of
 

fohdeesha

Kaini Industries
Nov 20, 2016
1,959
1,783
113
29
fohdeesha.com
no luck on the 6650 heatsink, it's metal bolts to the chassis. There's pushpins for the mgmt CPU next to it, but they look much smaller than what you need. You'll probably just have to do what I did for some old blade heatsinks, go to the hardware store and find some matching nuts, bolts and washers. stick the bolt up through the bottom of the PCB so you don't have to worry about clearance underneath (any extra bolt length will just be sticking up through the heatsink) just don't over tighten them, they barely need to be snug

https://i.imgur.com/meDUqfv.jpg
 

fohdeesha

Kaini Industries
Nov 20, 2016
1,959
1,783
113
29
fohdeesha.com
If anyone is interested: I have reverse airflow (Intake) PSU's (2) and fan trays (2) I'm looking to trade for the regular exhaust models. Pretty sure they'd all fit in a medium or large USPS flat rate box, so maybe $12 shipping for each of us

EDIT: trade has been made, no longer have them
 
Last edited:

mattaw

Member
Jul 30, 2018
51
15
8
Thanks for the info, EOS on the port sounds very likely. I should look up the chip that supplies the power and see what the filter design is.

As for the heatsinks they seem glued onto the ics. It may just be ancient thermal grease? Have you tried freezing them off - it was an old trick for GeForce cards.
 

Juggie

New Member
Nov 3, 2018
27
8
3
Are these brocades able to do sfp (not sfp+) in 2.5gbit? The local fiber provider here uses sfp gpon at 2.5. it works with their crappy modem and also with a ES-16-XG from ubiquiti which has recently added it. Folks have tried with other gear including Intel sfp cards but they all seem to be limited to 1gig on sfp.
 

fohdeesha

Kaini Industries
Nov 20, 2016
1,959
1,783
113
29
fohdeesha.com
Thanks for the info, EOS on the port sounds very likely. I should look up the chip that supplies the power and see what the filter design is.

As for the heatsinks they seem glued onto the ics. It may just be ancient thermal grease? Have you tried freezing them off - it was an old trick for GeForce cards.
I would think it's just caked up thermal compound - I can't imagine why they would use thermal adhesive AND retaining pins. Can you try to gently rotate the heatsink to break it free? (don't pull up). I've never personally attempted removing them, since the retaining pins are one time use (once you compress them to get them out of the hole, they never quite work right again)
 

fohdeesha

Kaini Industries
Nov 20, 2016
1,959
1,783
113
29
fohdeesha.com
Are these brocades able to do sfp (not sfp+) in 2.5gbit? The local fiber provider here uses sfp gpon at 2.5. it works with their crappy modem and also with a ES-16-XG from ubiquiti which has recently added it. Folks have tried with other gear including Intel sfp cards but they all seem to be limited to 1gig on sfp.
nope, it will link at 1gbe
 

mattaw

Member
Jul 30, 2018
51
15
8
I would think it's just caked up thermal compound - I can't imagine why they would use thermal adhesive AND retaining pins. Can you try to gently rotate the heatsink to break it free? (don't pull up). I've never personally attempted removing them, since the retaining pins are one time use (once you compress them to get them out of the hole, they never quite work right again)
Please accept a weeks delay - I have a conference
 

maes

Member
Nov 11, 2018
67
30
18
Longtime lurker here, finally decided to take the jump and go for an ICX6450-24 for the home lab. It'll be a nice boost for the NAS that's running over direct-attach 10Gbe right now.

I'm having the worst of lucks finding one with rackmount tabs on, that will ship outside of the USA and is still affordable, but found a few tab-less ones in the $150-$170 range.
Anyone would have the p/n for the accessory kit or compatible replacement rackmount tabs?
 

maes

Member
Nov 11, 2018
67
30
18
it seems one is listed under icx6450-rmk, not a great price though Brocade Ruckus ICX6450-RMK 2 Post Rackmount Kit + Full Accessory Kit | eBay
Thanks, I actually found it at two local stores for a good bit lower, especially with shipping taken into account. The eBay link is asking for ~$75 for international shipping.

Channel Electronics - Buy Mfgs Part-#:ICX6400-RMK online in Canada
Ruckus 2 Post - network device mounting kit - ICX6400-RMK - Rack Mounting Accessories - CDW.CA

I'll see when I get the actual switch and confirm it runs. In the meantime, back to modding my NSC-800!

Oh do you still have any of those 10gbe licenses by any chance?
 
Last edited:

Cutha

Member
Sep 24, 2016
68
7
8
46
Canada
Thanks for all the fantastic information @fohdeesha ! It saved me from buying a pair of MicroTik's.

My kids computers are in the basement and when I seriously started down this rabbit hole my server rack was was a rigged up 2 post with 2x4's for rear supports and a couple of comparatively quiet servers and a switch. It is now a full size rack with multiple Supermicro servers and an Infiniband switch. The heat and noise proved to much and everything is being relocated. I was trying to figure out how to bridge between the workstations and the server without sacrificing bandwidth when I saw this post and I have now purchased 2* "Brocade ICX 6610-48P 48-port 1 GbE RJ45 PoE+ Ethernet Switch ICX6610-48P".

I plan on using a "Mellanox FDR 4X Infiniband& 40G Ethernet QSFP+ 15M cable MC2207310-015" to connect the two ICX6610's. Will that work?

I could use a bit of help figuring out what to purchase to finish this up. The workstations had 1GB Cat6 connecting them to the regular network and 40GB IB connecting them to the storage on the servers. I have a bunch of Mellanox MHQH29B-XTR (flashed for RDMA) and varied lengths of QDR QSFP DAC's. What is going to be the cheapest route to connect 6 workstations to the 10GB and 40GB ports? I think the Mellanox cards I have are IB only.

The cable run for the workstations to the switch are between 15' and 30'. I am assuming that I will need to purchase new cabling and cards. My current thought is buy 4 cheap Mellanox EN cards and use a QSFP to 4x10GB SFP+ cable and for the other two workstations get 2 Mellanox 40GBE cards and use my existing QSFP DAC cables. Does this make sense? Is there a cheaper way? Any suggestion for NIC part numbers to search for?

Thanks again for all the great information.
 
  • Like
Reactions: fohdeesha

PnoT

Active Member
Mar 1, 2015
608
140
43
Texas
I got my ICX6610-48P in today and it showed up with 2 PSU and 2 FANs that are reverse airflow! It's time to start setting this thing up.
 
Last edited:
  • Like
Reactions: fohdeesha

Juggie

New Member
Nov 3, 2018
27
8
3
How many folks am I dogging it out with on Ebay trying to get a hold of a 6610/6450-48P? :) Seems to be slim pickings at the moment at least those that are good deals.
 

Skud

Active Member
Jan 3, 2012
110
47
28
If you really want to hack something up (I am not responsible if you fry the serial port):

Got my 7250-24P today and initially I powered it up without the console plugged in. This unit takes way longer to boot than the 6610 and I thought it was DOA since the fans weren't coming down.

I built a mini-usb to RJ45 so I could use my standard Cisco cable and then I saw that it was in fact booting, it just sits at "Starting Kernel" for a long time before moving along..

Here is the pinout for the contraption I made. Seems to be working well.
Code:
Mini-USB           Keystone (568B)
========            ===============
1 (VCC)            N/C
2 (UART RX)        6 (Grn)
3 (UART TX)        3 (Grn/Wht)
4 (Reserved)       N/C
5 (GND)            4 (Blue)
Off to tinker!!

Thanks!!
Riley
 

koisama

New Member
Oct 22, 2018
16
2
3
Is $300 a reasonable price for non-poe 6610-24P with two power supplies and fans?
Offered $250, seller won't go below $300.