Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)
- Thread starter fohdeesha
- Start date
anymore requests or can I disappear for like 9 years now
Wait who are you again?
(and pls don't => need mo ports already (I bought "the lil' guy") and will need you to answer my rudimentary questions)
I want to connect them to the SFP+ ports, the QSFP's on the back of my 6610's have been taken by the stacking cables (I know I don't need it, but that is not the point
).From what I understood reading through the thread I can split the 40Gb into 4 separate 10Gb connections? I've never worked with qsfp before, so I'm not really sure how this stuff works.
Indeed, initially I was afraid it might break during shipping, but I guess I can buy three of them and it'll still be cheaper.
You can indeed breakout the switch qsfp ports to 4x 10gbe, but you can't do the same from the nic - nics do not support qsfp breakout, at least none I've heard of. You can use a qsfp > sfp+ adapter (search eBay for 655874-B21) and get a single 10gb connection out of the nic thoughI want to connect them to the SFP+ ports, the QSFP's on the back of my 6610's have been taken by the stacking cables (I know I don't need it, but that is not the point
From what I understood reading through the thread I can split the 40Gb into 4 separate 10Gb connections? I've never worked with qsfp before, so I'm not really sure how this stuff works.
Indeed, initially I was afraid it might break during shipping, but I guess I can buy three of them and it'll still be cheaper.
Ahhh, that makes sense.
I was wondering how the breakout would work in the OS.
In that case should I instead go for a double 10Gb SFP+ nic (Mellanox MCX312A-XCBT?) and two normal SFP+ DAC's cables (https://www.fs.com/de-en/products/36673.html)?
Anyone have a link to 1g modules that will work in the 6650 & 77xx line? I still have some older stuff that I need to connect with 1Gb copper links.
Thanks!
Thanks!
it's up to you, but I am a fan of going with the 40gb OEM cards (flashed with latest mellanox fw) and the qsfp > sfp+ adapter - it comes out to the same price total, and that way if you ever want to move to 40gbE, you already have a 40gbE nic installed and configured ready to go
cleaned up original post, added icx7150, icx7250, and icx6650 to it so all choices are in one post, with pictures for easy beef selection
When you have a few mins, think you can post a working config for MCT between 2x 6650's? Pretty please?
Also, possible to MCT > 3 or 4 of these together?
When you have a few mins, think you can post a working config for MCT between 2x 6650's? Pretty please?
Also, possible to MCT > 3 or 4 of these together?
Doesn't this work?
http://www.netadmin.us/docs/MCT_Config_Guide.pdf
For multiple MCT, MCT is not stacking. You can have 2 switches in an MCT group to the best of my knowledge:
MCT configuration considerations
Last edited:
I bought my first 6610-48P today! Thank you Fohdeesha for putting this information together. It's hard to beat a dirt cheap access switch that has the feature set to function in a collapsed core, distribution or datacenter role. These 6610s are swiss army knives for home lab networks! I thought about buying two, but the rest of my network needs attention too.
I'm upgrading to gigabit Internet soon and my existing home network is trash. They say a mechanic's car is the last to get fixed... well the same can be said for network guys and our home networks... or at least mine! My wired network currently consists of three MoCA adapters connecting my three coax drops. My dd-wrt routers are placed for maximal wife acceptance, which ironically makes them unacceptable when wifi doesn't work on the deck or corner rooms of the house!
I'm upgrading to gigabit Internet soon and my existing home network is trash. They say a mechanic's car is the last to get fixed... well the same can be said for network guys and our home networks... or at least mine! My wired network currently consists of three MoCA adapters connecting my three coax drops. My dd-wrt routers are placed for maximal wife acceptance, which ironically makes them unacceptable when wifi doesn't work on the deck or corner rooms of the house!
I don't have any configs, never used MCT in production. There's a whole MCT section in the l2 guide though, page 157 of fastiron-08030mb-l2guide.pdf from the zipWhen you have a few mins, think you can post a working config for MCT between 2x 6650's? Pretty please?
Also, possible to MCT > 3 or 4 of these together?
keep in mind unlike stacking, MCT syncs l2 states only, l3 you're on your own. eg each switch will run its own BGP session separately just like if you didn't have MCT, and you'll have to make everything aware of each switches BGP session separately etc, depending on the setup you might have to get creative: Layer 3 routing on Cumulus Linux MLAG
So I finally took the plunge. I've been faffing about with my 6450s for the past month, they've been hung off my existing infrastructure with pis setup as serial adapter hosts so I can make breaking changes while I learn how to use them without having to walk all over the house to reset switches after locking myself out.
Anyway, last night everything finally clicked, so I spent a couple of hours at 4am ripping and tearing on my network infrastructure while the girlfriend and housemates were asleep. Everything's working great now, the brocades are a joy to use and once I pull the rest of the fiber I'll have 10GbE throughout the house.
And now I've got alerts setup for the next cheap 6610 that pops up, help!
A couple of questions:
What should I do about spanning tree? I have a pretty simple home network consisting of two 6450s, a couple of non Brocade managed switches and a couple of Proxmox machines that I'm going to be running openvswitch and dpdk on. Should I just turn on single spanning tree 802.1w globally on both switches and call it good or is there a better option that I don't know about? I honestly don't know much about RSTP other than it's a good idea to have it enabled in case someone not-me gets stupid with ethernet cables.
Is it possible to group ports and name them? I'd like to make some ease of use groupings for the 1g ports to save time/make it easier to grok what I'm working on. Something like "top-row" "bottom-row" or "top-left" (for the even number ports between 1/1/1 and 1/1/11.) Is that possible?
Anyway, last night everything finally clicked, so I spent a couple of hours at 4am ripping and tearing on my network infrastructure while the girlfriend and housemates were asleep. Everything's working great now, the brocades are a joy to use and once I pull the rest of the fiber I'll have 10GbE throughout the house.
And now I've got alerts setup for the next cheap 6610 that pops up, help!
A couple of questions:
What should I do about spanning tree? I have a pretty simple home network consisting of two 6450s, a couple of non Brocade managed switches and a couple of Proxmox machines that I'm going to be running openvswitch and dpdk on. Should I just turn on single spanning tree 802.1w globally on both switches and call it good or is there a better option that I don't know about? I honestly don't know much about RSTP other than it's a good idea to have it enabled in case someone not-me gets stupid with ethernet cables.
Is it possible to group ports and name them? I'd like to make some ease of use groupings for the 1g ports to save time/make it easier to grok what I'm working on. Something like "top-row" "bottom-row" or "top-left" (for the even number ports between 1/1/1 and 1/1/11.) Is that possible?
Last edited:
Thanks for the epic guide! Just picked up an ICX6650 to tinker with and replace my netgear prosumer switch. What were you doing with the JTAG interface? Dumping the ROM?
Nothing like real systems for making mistakes and learning with. Hoping this gives me some street cred with the network engineers and help me stave off the label "paper architect" for one more year.
Nothing like real systems for making mistakes and learning with. Hoping this gives me some street cred with the network engineers and help me stave off the label "paper architect" for one more year.
Just a heads up, if anyone is looking for a few 40Gb cards, This seller will accept $15 Best Offer per card.
I just picked up 4 x 40Gb SUN nics 4 x $15 = $60.
SUN 7046442 DUAL 40GB/SEC QDR INFINIBAND CARD 46655503046 | eBay
Also this seller was posted somewhere here before as accepting best offer of $20 per card, but they will also accept $15 per card as well:
HP 656089-001 Infiband 4 X 10/40GB DP adapter 544QSFP 4872503154148 | eBay
I just picked up 4 x 40Gb SUN nics 4 x $15 = $60.
SUN 7046442 DUAL 40GB/SEC QDR INFINIBAND CARD 46655503046 | eBay
Also this seller was posted somewhere here before as accepting best offer of $20 per card, but they will also accept $15 per card as well:
HP 656089-001 Infiband 4 X 10/40GB DP adapter 544QSFP 4872503154148 | eBay