Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

[svtkobra7]

this is me hiding my jealousy

Just take one from this guy's book ...

 

[mattaw]

Nice purchase! Join the lucky club as I got 3x for $160 icx6610 48P's.

Does anyone have a scrap mainboard? I need two replacement push pins for the main routing IC heatsinks. I guess they failed with heat and age.



I'm also getting a poe PD error on port 48. It powers fine, but won't sense current etc. nor light the poe led. Swapped the Poe board into another switch and the problem followed the board. Just scrubbed the board as it was dirty but all the components look ok to me. Thoughts?

 

[fohdeesha]

never seen those pins fail, crazy. Wonder if it got knocked around? I have a junk ICX6650, it might have the same heatsinks/pins from the looks of it, I'll take a look.

Wanna do me a huge favor, if the heatsink is free now, can you move it to the side and get the text/model number of the ASIC? They've kept it incredibly secret - I know it's a Marvell Cheetah 5, which I'm pretty sure was a custom build just for Brocade, but it would be nice to have some verification/ actual P/N

in my experience, PD errors that follow the board on a specific port mean that port took a pretty nasty indirect or direct static hit (usually by a PoE device like a camera that was outdoors). No way to fix that I know of

 

[fohdeesha]

no luck on the 6650 heatsink, it's metal bolts to the chassis. There's pushpins for the mgmt CPU next to it, but they look much smaller than what you need. You'll probably just have to do what I did for some old blade heatsinks, go to the hardware store and find some matching nuts, bolts and washers. stick the bolt up through the bottom of the PCB so you don't have to worry about clearance underneath (any extra bolt length will just be sticking up through the heatsink) just don't over tighten them, they barely need to be snug

https://i.imgur.com/meDUqfv.jpg

 

[fohdeesha]

If anyone is interested: I have reverse airflow (Intake) PSU's (2) and fan trays (2) I'm looking to trade for the regular exhaust models. Pretty sure they'd all fit in a medium or large USPS flat rate box, so maybe $12 shipping for each of us

EDIT: trade has been made, no longer have them

 

[mattaw]

Thanks for the info, EOS on the port sounds very likely. I should look up the chip that supplies the power and see what the filter design is.

As for the heatsinks they seem glued onto the ics. It may just be ancient thermal grease? Have you tried freezing them off - it was an old trick for GeForce cards.

 

[Juggie]

Are these brocades able to do sfp (not sfp+) in 2.5gbit? The local fiber provider here uses sfp gpon at 2.5. it works with their crappy modem and also with a ES-16-XG from ubiquiti which has recently added it. Folks have tried with other gear including Intel sfp cards but they all seem to be limited to 1gig on sfp.

 

[fohdeesha]

Thanks for the info, EOS on the port sounds very likely. I should look up the chip that supplies the power and see what the filter design is.

As for the heatsinks they seem glued onto the ics. It may just be ancient thermal grease? Have you tried freezing them off - it was an old trick for GeForce cards.

I would think it's just caked up thermal compound - I can't imagine why they would use thermal adhesive AND retaining pins. Can you try to gently rotate the heatsink to break it free? (don't pull up). I've never personally attempted removing them, since the retaining pins are one time use (once you compress them to get them out of the hole, they never quite work right again)

 

[fohdeesha]

Are these brocades able to do sfp (not sfp+) in 2.5gbit? The local fiber provider here uses sfp gpon at 2.5. it works with their crappy modem and also with a ES-16-XG from ubiquiti which has recently added it. Folks have tried with other gear including Intel sfp cards but they all seem to be limited to 1gig on sfp.

nope, it will link at 1gbe

 

[Juggie]

nope, it will link at 1gbe

K so no 2.5 or 5 gbit ethernet in these.

 

[mattaw]

I would think it's just caked up thermal compound - I can't imagine why they would use thermal adhesive AND retaining pins. Can you try to gently rotate the heatsink to break it free? (don't pull up). I've never personally attempted removing them, since the retaining pins are one time use (once you compress them to get them out of the hole, they never quite work right again)

Please accept a weeks delay - I have a conference

 

[maes]

Longtime lurker here, finally decided to take the jump and go for an ICX6450-24 for the home lab. It'll be a nice boost for the NAS that's running over direct-attach 10Gbe right now.

I'm having the worst of lucks finding one with rackmount tabs on, that will ship outside of the USA and is still affordable, but found a few tab-less ones in the $150-$170 range.
Anyone would have the p/n for the accessory kit or compatible replacement rackmount tabs?

 

[mattaw]

Anyone would have the p/n for the accessory kit or compatible replacement rackmount tabs?

Luckily brocade part numbers are sane: ICX6400-RMK

 

[fohdeesha]

it seems one is listed under icx6450-rmk, not a great price though Brocade Ruckus ICX6450-RMK 2 Post Rackmount Kit + Full Accessory Kit | eBay

 

[maes]

it seems one is listed under icx6450-rmk, not a great price though Brocade Ruckus ICX6450-RMK 2 Post Rackmount Kit + Full Accessory Kit | eBay

Thanks, I actually found it at two local stores for a good bit lower, especially with shipping taken into account. The eBay link is asking for ~$75 for international shipping.

Channel Electronics - Buy Mfgs Part-#:ICX6400-RMK online in Canada
Ruckus 2 Post - network device mounting kit - ICX6400-RMK - Rack Mounting Accessories - CDW.CA

I'll see when I get the actual switch and confirm it runs. In the meantime, back to modding my NSC-800!

Oh do you still have any of those 10gbe licenses by any chance?

 

[Cutha]

Thanks for all the fantastic information @fohdeesha ! It saved me from buying a pair of MicroTik's.

My kids computers are in the basement and when I seriously started down this rabbit hole my server rack was was a rigged up 2 post with 2x4's for rear supports and a couple of comparatively quiet servers and a switch. It is now a full size rack with multiple Supermicro servers and an Infiniband switch. The heat and noise proved to much and everything is being relocated. I was trying to figure out how to bridge between the workstations and the server without sacrificing bandwidth when I saw this post and I have now purchased 2* "Brocade ICX 6610-48P 48-port 1 GbE RJ45 PoE+ Ethernet Switch ICX6610-48P".

I plan on using a "Mellanox FDR 4X Infiniband& 40G Ethernet QSFP+ 15M cable MC2207310-015" to connect the two ICX6610's. Will that work?

I could use a bit of help figuring out what to purchase to finish this up. The workstations had 1GB Cat6 connecting them to the regular network and 40GB IB connecting them to the storage on the servers. I have a bunch of Mellanox MHQH29B-XTR (flashed for RDMA) and varied lengths of QDR QSFP DAC's. What is going to be the cheapest route to connect 6 workstations to the 10GB and 40GB ports? I think the Mellanox cards I have are IB only.

The cable run for the workstations to the switch are between 15' and 30'. I am assuming that I will need to purchase new cabling and cards. My current thought is buy 4 cheap Mellanox EN cards and use a QSFP to 4x10GB SFP+ cable and for the other two workstations get 2 Mellanox 40GBE cards and use my existing QSFP DAC cables. Does this make sense? Is there a cheaper way? Any suggestion for NIC part numbers to search for?

Thanks again for all the great information.

 

[PnoT]

I got my ICX6610-48P in today and it showed up with 2 PSU and 2 FANs that are reverse airflow! It's time to start setting this thing up.

 

[Juggie]

How many folks am I dogging it out with on Ebay trying to get a hold of a 6610/6450-48P? Seems to be slim pickings at the moment at least those that are good deals.

 

[Skud]

If you really want to hack something up (I am not responsible if you fry the serial port):

Got my 7250-24P today and initially I powered it up without the console plugged in. This unit takes way longer to boot than the 6610 and I thought it was DOA since the fans weren't coming down.

I built a mini-usb to RJ45 so I could use my standard Cisco cable and then I saw that it was in fact booting, it just sits at "Starting Kernel" for a long time before moving along..

Here is the pinout for the contraption I made. Seems to be working well.

Mini-USB           Keystone (568B)
========            ===============
1 (VCC)            N/C
2 (UART RX)        6 (Grn)
3 (UART TX)        3 (Grn/Wht)
4 (Reserved)       N/C
5 (GND)            4 (Blue)

Off to tinker!!

Thanks!!
Riley

 

[koisama]

Is $300 a reasonable price for non-poe 6610-24P with two power supplies and fans?
Offered $250, seller won't go below $300.