Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

[fohdeesha]

And I thought I was OCD

 

[svtkobra7]

And I thought I was OCD

I was joking ... gosh ... maybe I can only see in grey-scale and therefore it was readily apparent? OK, yes I guess I do notice minor details like that (let's forget about the slight differences in cropping too ), but $hit like title misalignment etc that repeats on each slide in a .pptx deck annoys the hell out of me ... to the extent I have to leave the meeting and go puke in the bathroom.

Maybe a wee bit ocd - is that a bad thing?

 

[fohdeesha]

Maybe a week bit ocd - is that a bad thing?

In our profession, probably a positive!

 

[svtkobra7]

In our profession, probably a positive!

Your profession, I wish it were mine.

Crap, I looked at the datasheets when I was tired and it is the 24P ICX7250 that would fit. Argh.

EDIT = well at least I could swap in a 7250 in place of the 6450 and have 4 more 10G ports for the future (4/4 on 6450 used ATM), but really would have loved to use the CX-3's 40G ports ...

 

[Matt G]

I remember someone earlier in this thread complaining about the cost of brocade transceivers with shipping to EU. If you want a couple of them this one comes out to about $9 including shipping and import taxes with BO.
8x Brocade 10G-SFPP-SR 57-0000075-01 XBR-000180 Ethernet SR SFP+ >200pcs

I'm building my server and network at the moment, mostly shooting from the hip, and since I decided to use icx6610's to upgrade my network I figured I'd post some cables and cards I've ordered to see if you guys think any of it will work together. The server is a Dell r720xd with baremetal Freenas.

HP 649281-B21 Infiniband 10/40GB Dual Port Adapter Low Profile Bracket #HP
Intend to try my hands at flashing this.

MELLANOX CX314A BCBT MCX314A CONNECTX 3 EN 40gb NETWORK ADAPTER #MELLANOX
Bought this before I knew you could buy them cheaper and flash them.

Brocade 1020 BR-1020 Dual Port 10Gb PCI-E
For a random windows 10 machine somewhere in the house.

Cables!

Brocade 40G-QSFP-QSFP-C-0101 1.0m QSFP-QSFP Kabel (SAS)
For switch to server and/or stacking.

Mellanox 834972-B27 100Gbs Optical cable 20m #Mellanox
Switch to Win10 machine with mellanox card.

Mellanox MC220731V-030 #MellanoxTechnologies
Switch to Win10 machine with mellanox card... if 20m isn't enough.

And then there's the transceivers mentioned above, I bought a buttload of those. I hope I got the links working correctly.

So, did I get anything right?

 

[fohdeesha]

8x Brocade 10G-SFPP-SR 57-0000075-01 XBR-000180 Ethernet SR SFP+ >200pcs

these are at least correct for 10gbps over fiber

HP 649281-B21 Infiniband 10/40GB Dual Port Adapter Low Profile Bracket #HP
Intend to try my hands at flashing this.

just use the guide linked in the first post in this thread, easy peazy: https://forums.servethehome.com/ind...net-dual-port-qsfp-adapter.20525/#post-198015

MELLANOX CX314A BCBT MCX314A CONNECTX 3 EN 40gb NETWORK ADAPTER #MELLANOX
Bought this before I knew you could buy them cheaper and flash them.

it'll work great, just make sure you still upgrade the firmware

Brocade 1020 BR-1020 Dual Port 10Gb PCI-E
For a random windows 10 machine somewhere in the house.

just buy a mellanox connectx2 or connectx3 like everyone else, these brocade cards are old, the drivers not so good, and are extremely picky when it comes to optics/DACs

Brocade 40G-QSFP-QSFP-C-0101 1.0m QSFP-QSFP Kabel (SAS)
For switch to server and/or stacking.

that's the correct cable you want for 40gbE QSFP+

Mellanox 834972-B27 100Gbs Optical cable 20m #Mellanox
Switch to Win10 machine with mellanox card.

This is a QSFP28 cable for 100gbE, it will not work with your 40gbE QSFP+ ports, let alone the 10gbE connections on the win10 machine. It's also an active cable, meaning it has optics built in to each end.

Mellanox MC220731V-030 #MellanoxTechnologies
Switch to Win10 machine with mellanox card... if 20m isn't enough.

This was a 40gbE (or 56gbps infiniband) QSFP+ cable, again not what you need for the 10gbE connection to the win10 box. It was also an active cable.

You have optics for the switch 10gbE SFP+ slots, so you don't want an active cable, you just need multimode fiber with LC connectors. Grade OM1, OM2, or OM3 will all work fine. OM2 tends to be the same price as OM1 so I go with OM2 even for really short connections typically. Just search like "om2 lc multimode 30m" and you should get a bunch of results 30 meters long: NEW TYCO 30M LC-LC 50/125 OM2 DUPLEX MM MULTIMODE OFNP FIBER OPTIC PATCH CABLE | eBay

 

[Matt G]

This is a QSFP28 cable for 100gbE, it will not work with your 40gbE QSFP+ ports, let alone the 10gbE connections on the win10 machine. It's also an active cable, meaning it has optics built in to each end.


This was a 40gbE (or 56gbps infiniband) QSFP+ cable, again not what you need for the 10gbE connection to the win10 box. It was also an active cable.

I got those after reading that they would be compatible with 40gb/s ethernet. The thread was talking about using them with Mellanox MCX354 in ethernet mode so I figured I could buy a similar card for one of my Win10 machines. I guess I'm gonna run out of 40 gbe ports on the switch real quick though. I guess most if not all of the desktops will have 10gbe, just wanted to have the option to run 40gbe to one atleast.

The br1020 was ordered solely on the suggestion from the buyers guide for freenas here on STH. I've gone through more of the forums now and know that the mellanox cards have way better support.

Thanks for the run down of all the crap I've ordered!

 

[fohdeesha]

if they're QSFP28 they *should* be backwards compatible with QSFP+, but last time I tossed an EDR active fiber cable in an ICX6610 I couldn't get it to link up. Granted that could have been the other end. Either way, no reason to spend more on a 100gbE active fiber cable when you can typically get QSFP+ much cheaper

the freenas buyers guide is probably quite old, back when freenas did not support mellanox

and keep in mind the ICX6610 only has 2x 40gbE ports, the other two QSFP+ ports are breakout only

 

[Matt G]

and keep in mind the ICX6610 only has 2x 40gbE ports, the other two QSFP+ ports are breakout only

That's pretty much the only thing I knew for certain. You've written some top notch guides/specs in this thread that are easy to understand. I think two stacked 6610s with the server on 40gbe and the rest on 10gbe will be quite sufficient for me... I did however save an ebay search for arista 7050qx incase of... emergency?

Oh, and the EDR cable was kinda cheap, $35 with shipping for a 20m incl transceivers with BO (66' for those stuck on imperial units). Cheapest qsfp cables I find at 20m are $28 plus $54 shipping. While I'm still rambling on and taking up space in this thread I might aswell ask if I can use the same cable for stacking two icx6610s and switch to server?

 

[fohdeesha]

if you can get the EDR cables to link up on the brocades then yes they can be used for stacking. I tried an active EDR cable a few years ago and couldn't get it to link, granted this was several firmware versions back when the rear ports still behaved oddly when used for non-stacking scenarios. you'll just have to give it a go and report back

 

[capn13]

Well after perusing this somewhat longish thread and being in need of some networking gear for our new house, I finally bit the bullet and got one 6450-48P. I admit I was a little bit concerned when I first powered it on and all fans ramped to max

That one will go on the 1st floor and feed some cams via POE. Fibre back down to the core switch on the ground floor (core switch model still TBD!).

Not sure if this is how they ship but the seller had erased the config and provided a newish firmware. This was not quite as new as @fohdeesha - so will be following his guide this weekend- but the unit came with two of the four SFP+ 10G ports licensed.

Fun

 

[Matt G]

Do you mean stack the 6610 with the 6450 so they behave like one switch? It's possible, but the 6450 goes into an "l2 only" mode kicking all l3 traffic over to the 6610, so you don't get redundancy (if your 6610 dies, the 6450 can no longer do l3). The "mixed stacking" between these two was kind of an afterthought. That said it does work, and if all you're looking for is the ease of turning them both into a single device to manage with one interface/cli, it will definitely do that

long story short you enable stacking, set the stacking ports on the 6610 to one of the front 10gbe ports, then run the stacking setup command. You need to read fastiron-08030b-switchstackingguide.pdf from the firmware zip, specifically the Mixed Stacking section starting on page 177

Sweet jesus somebody plz end my ebay account... I bought another switch now.

Figured the 6450-48p would work in the living room and be wife compatible. I cut a hole in the wall and stuffed an IKEA kitchen shelf with frosted glass doors for electronics that need IR and other electronics on top of that on the other side of the wall. It's an old house with pretty much no sound isolation, when the gpu fan in the computer spins up to max you hear it through the wall. Currently there's two netgear switches that I figured the 6450 would replace.

I read through the chapter on mixed stacking in the manual and I can't find any mention of using the 40gb breakout ports to stack 6450s on? Since I don't have the equipment yet I can't just "try it".

Could I use one of these in the 6610 and run a fiber to the 6450 for stacking?
HPE QSFP/SFP+ Adapter Kit 40GbE QSFP+ auf 10GbE SFP+ 655874-B21 655902-001 +++

Could I use a breakout cable and stack multiple 6450s?
Are there any options for breakout cables that are 30m (100') and split close to the qsfp transceiver?

 

[fohdeesha]

mixed 6450/6610 stacking only works using the front 10gb ports

 

[BoredSysadmin]

First of all - A HUGE thanks to @fohdeesha for providing such massive and dummy proof guides and steps super easy to follow. (Have a beer on me)

Got my ICX6610-24p (1 ps/1 fan) in the mail on Monday. I got it for an absolutely ridiculously low price of $135 [offer]+$40 s/h.
Finally had a chance today to run thru the update with flawless results with the f/w update and licensing.
Things I personally stumbled for a second (but re-reading the guide found the solution) - was for getting into bootloader it said "press b to stop at the boot monitor" and that prompt is really really quick. A second was PoE firmware, which does take about 10 mins as mentioned. I just kept running "show log" till it was after 90 percent complete entry and then PoE firmware actually was loaded. First time around PoE firmware wasn't fully loaded and got corrupted, but that doesn't affect other functions.

At this point, I am a bit less scared of the switch as the CLI is very similar to IOS which I kinda feel somewhat ok with.
Told my good friend from Israel that I've bought 16 port 10 gig switch for this kind of money and his jaw was nowhere to be found for a while.
Next to wire my freenas and my vmware hosts. fun stuff!

 

[SuperServer]

So power surge hit seemed to of taken a hit to my server and the icx6610 came home to a down network and the icx spinning violently. Not picking up address from web GUI telnet ssh and not recognizing it on the router either tried to serial connect to it but couldn’t get access to it. Gonna try a different pc tonight.

Basically boots up as normal however none of the 48 1gb ports light up at all. The 10g blink for a bit then stop I assume after the standard boot up time and then go solid the management port blinks with activity but comes up unresponsive on my surface.

What’s the over under this guy is shot or just the power supply needs to be replaced?

 

[arglebargle]

This site is terrible for my wallet.

6610-24P ordered and on the way. 3 10Gb ports after linking my two 6450s was nowhere near enough bandwidth.

 

[capn13]

This site is terrible for my wallet.

6610-24P ordered and on the way. 3 10Gb ports after linking my two 6450s was nowhere near enough bandwidth.

You're telling me.

But fear not you're not alone: ordered two 7250-24's earlier so the house can now be re-wired/fibred

Goodbye cheapo switches, hellow bandwidth!!

 

[seatrope]

Hi all, and thanks @fohdeesha again for all of your mentoring!

Question for anyone who’s opened their 6450. I have a 6450-48P which I’m not using any POE devices with. Want to swap fans but I can’t figure out how to open the case. A search of this thread didn’t yield anything relevant (apologies if I missed something). I removed all screws and the top cover is loose except for the area towards the back which seems riveted to the frame below with two rivets on each side. I tried to attach a photo but for some reason there’s an upload error.

Did you guys just force it open? Seems to be bending the frame backward when I try to lever it open.

 

[arglebargle]

Did you guys just force it open? Seems to be bending the frame backward when I try to lever it open.

You don't have to force it, it'll come off freely when all screws are removed. There are a *lot* of screws holding the cover on though and I remember some of them being in unexpected places. Give me a few minutes to move everything off of mine and pull it out and I'll detail screw locations.

 

[seatrope]

Seem to have removed all the screws I can see. Did another close inspection. What I can see is that there are 4 rivet type things along the back edge of the side panels (2 each side) which are preventing me from lifting off the top. Still can’t upload a picture ...