Drag to reposition cover

Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

Emdee

New Member
Feb 28, 2019
5
0
1
the card looks good, but that cable is an SFP+ breakout cable, when it sounds like you want to do 40gb direct to that mellanox card in your dell, in which case you need qsfp to qsfp like this NEW Brocade 58-0000033-01 4x10GE QSFP Passive Cable 1M | eBay
I want to connect them to the SFP+ ports, the QSFP's on the back of my 6610's have been taken by the stacking cables (I know I don't need it, but that is not the point :p).

From what I understood reading through the thread I can split the 40Gb into 4 separate 10Gb connections? I've never worked with qsfp before, so I'm not really sure how this stuff works.

Are you looking for connect-x3 cards just in the netherlands?
If you don't mind to wait you can get them cheaper from the states: HP 544QSFP Infiniband 10/40GB FDR/EN PCIe Dual Port HCA 656089-001 649281-B21 | eBay
Indeed, initially I was afraid it might break during shipping, but I guess I can buy three of them and it'll still be cheaper.
 

fohdeesha

Kaini Industries
Nov 20, 2016
2,117
1,982
113
30
fohdeesha.com
I want to connect them to the SFP+ ports, the QSFP's on the back of my 6610's have been taken by the stacking cables (I know I don't need it, but that is not the point :p).

From what I understood reading through the thread I can split the 40Gb into 4 separate 10Gb connections? I've never worked with qsfp before, so I'm not really sure how this stuff works.



Indeed, initially I was afraid it might break during shipping, but I guess I can buy three of them and it'll still be cheaper.
You can indeed breakout the switch qsfp ports to 4x 10gbe, but you can't do the same from the nic - nics do not support qsfp breakout, at least none I've heard of. You can use a qsfp > sfp+ adapter (search eBay for 655874-B21) and get a single 10gb connection out of the nic though
 
  • Like
Reactions: Emdee

Emdee

New Member
Feb 28, 2019
5
0
1
You can indeed breakout the switch qsfp ports to 4x 10gbe, but you can't do the same from the nic - nics do not support qsfp breakout, at least none I've heard of. You can use a qsfp > sfp+ adapter (search eBay for 655874-B21) and get a single 10gb connection out of the nic though
Ahhh, that makes sense.
I was wondering how the breakout would work in the OS.

In that case should I instead go for a double 10Gb SFP+ nic (Mellanox MCX312A-XCBT?) and two normal SFP+ DAC's cables (https://www.fs.com/de-en/products/36673.html)?
 

tjk

Active Member
Mar 3, 2013
285
50
28
www.servercentral.com
Anyone have a link to 1g modules that will work in the 6650 & 77xx line? I still have some older stuff that I need to connect with 1Gb copper links.

Thanks!
 

fohdeesha

Kaini Industries
Nov 20, 2016
2,117
1,982
113
30
fohdeesha.com
Ahhh, that makes sense.
I was wondering how the breakout would work in the OS.

In that case should I instead go for a double 10Gb SFP+ nic (Mellanox MCX312A-XCBT?) and two normal SFP+ DAC's cables (https://www.fs.com/de-en/products/36673.html)?
it's up to you, but I am a fan of going with the 40gb OEM cards (flashed with latest mellanox fw) and the qsfp > sfp+ adapter - it comes out to the same price total, and that way if you ever want to move to 40gbE, you already have a 40gbE nic installed and configured ready to go
 
  • Like
Reactions: tommybackeast

fohdeesha

Kaini Industries
Nov 20, 2016
2,117
1,982
113
30
fohdeesha.com
Anyone have a link to 1g modules that will work in the 6650 & 77xx line? I still have some older stuff that I need to connect with 1Gb copper links.

Thanks!
any of them will work, have used a lot of HP and cisco modules off ebay with zero issues - one of the great things about these brocades is how little they care about modules
 

tjk

Active Member
Mar 3, 2013
285
50
28
www.servercentral.com
cleaned up original post, added icx7150, icx7250, and icx6650 to it so all choices are in one post, with pictures for easy beef selection
When you have a few mins, think you can post a working config for MCT between 2x 6650's? Pretty please? :)

Also, possible to MCT > 3 or 4 of these together?
 

Discountsocks

New Member
Aug 16, 2018
8
7
3
Last edited:
  • Like
Reactions: tjk

packmule

New Member
Mar 21, 2019
5
1
3
I bought my first 6610-48P today! Thank you Fohdeesha for putting this information together. It's hard to beat a dirt cheap access switch that has the feature set to function in a collapsed core, distribution or datacenter role. These 6610s are swiss army knives for home lab networks! I thought about buying two, but the rest of my network needs attention too.

I'm upgrading to gigabit Internet soon and my existing home network is trash. They say a mechanic's car is the last to get fixed... well the same can be said for network guys and our home networks... or at least mine! My wired network currently consists of three MoCA adapters connecting my three coax drops. My dd-wrt routers are placed for maximal wife acceptance, which ironically makes them unacceptable when wifi doesn't work on the deck or corner rooms of the house! :D
 
  • Like
Reactions: fohdeesha

fohdeesha

Kaini Industries
Nov 20, 2016
2,117
1,982
113
30
fohdeesha.com
When you have a few mins, think you can post a working config for MCT between 2x 6650's? Pretty please? :)

Also, possible to MCT > 3 or 4 of these together?
I don't have any configs, never used MCT in production. There's a whole MCT section in the l2 guide though, page 157 of fastiron-08030mb-l2guide.pdf from the zip

keep in mind unlike stacking, MCT syncs l2 states only, l3 you're on your own. eg each switch will run its own BGP session separately just like if you didn't have MCT, and you'll have to make everything aware of each switches BGP session separately etc, depending on the setup you might have to get creative: Layer 3 routing on Cumulus Linux MLAG
 

tjk

Active Member
Mar 3, 2013
285
50
28
www.servercentral.com
I don't have any configs, never used MCT in production. There's a whole MCT section in the l2 guide though, page 157 of fastiron-08030mb-l2guide.pdf from the zip

keep in mind unlike stacking, MCT syncs l2 states only, l3 you're on your own. eg each switch will run its own BGP session separately just like if you didn't have MCT, and you'll have to make everything aware of each switches BGP session separately etc, depending on the setup you might have to get creative: Layer 3 routing on Cumulus Linux MLAG
Thanks, I use these purely as L2 devices for hypervisors and storage only.
 

arglebargle

H̸̖̅ȩ̸̐l̷̦͋l̴̰̈ỏ̶̱ ̸̢͋W̵͖̌ò̴͚r̴͇̀l̵̼͗d̷͕̈
Jul 15, 2018
656
234
43
So I finally took the plunge. I've been faffing about with my 6450s for the past month, they've been hung off my existing infrastructure with pis setup as serial adapter hosts so I can make breaking changes while I learn how to use them without having to walk all over the house to reset switches after locking myself out.

Anyway, last night everything finally clicked, so I spent a couple of hours at 4am ripping and tearing on my network infrastructure while the girlfriend and housemates were asleep. Everything's working great now, the brocades are a joy to use and once I pull the rest of the fiber I'll have 10GbE throughout the house.

And now I've got alerts setup for the next cheap 6610 that pops up, help!

A couple of questions:

What should I do about spanning tree? I have a pretty simple home network consisting of two 6450s, a couple of non Brocade managed switches and a couple of Proxmox machines that I'm going to be running openvswitch and dpdk on. Should I just turn on single spanning tree 802.1w globally on both switches and call it good or is there a better option that I don't know about? I honestly don't know much about RSTP other than it's a good idea to have it enabled in case someone not-me gets stupid with ethernet cables.

Is it possible to group ports and name them? I'd like to make some ease of use groupings for the 1g ports to save time/make it easier to grok what I'm working on. Something like "top-row" "bottom-row" or "top-left" (for the even number ports between 1/1/1 and 1/1/11.) Is that possible?
 
Last edited:

Lambdariver

New Member
Mar 24, 2019
20
14
3
Thanks for the epic guide! Just picked up an ICX6650 to tinker with and replace my netgear prosumer switch. What were you doing with the JTAG interface? Dumping the ROM?

Nothing like real systems for making mistakes and learning with. Hoping this gives me some street cred with the network engineers and help me stave off the label "paper architect" for one more year.
 
  • Like
Reactions: Sleyk

747builder

Active Member
Dec 17, 2017
105
57
28
Anyone have a link to 1g modules that will work in the 6650 & 77xx line? I still have some older stuff that I need to connect with 1Gb copper links.

Thanks!
my 6650 came with a couple of handfuls of 1G Copper
approved optics xbr-000190-a (1G copper)

I've used everything from juniper to cisco optics in mine. YMMV
 
  • Like
Reactions: Sleyk

Sleyk

Well-Known Member
Mar 25, 2016
1,108
491
83
Stamford, CT