Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

Discussion in 'Networking' started by fohdeesha, Jul 12, 2018.

  1. fohdeesha

    fohdeesha Kaini Industries

    Joined:
    Nov 20, 2016
    Messages:
    1,109
    Likes Received:
    854
    #1541
  2. fohdeesha

    fohdeesha Kaini Industries

    Joined:
    Nov 20, 2016
    Messages:
    1,109
    Likes Received:
    854
    anymore requests or can I disappear for like 9 years now
     
    #1542
    mathiastro, tjk, itronin and 2 others like this.
  3. svtkobra7

    svtkobra7 Active Member

    Joined:
    Jan 2, 2017
    Messages:
    306
    Likes Received:
    61
    Wait who are you again?

    (and pls don't => need mo ports already (I bought "the lil' guy") and will need you to answer my rudimentary questions)
     
    #1543
    fohdeesha likes this.
  4. Emdee

    Emdee New Member

    Joined:
    Feb 28, 2019
    Messages:
    5
    Likes Received:
    0
    I want to connect them to the SFP+ ports, the QSFP's on the back of my 6610's have been taken by the stacking cables (I know I don't need it, but that is not the point :p).

    From what I understood reading through the thread I can split the 40Gb into 4 separate 10Gb connections? I've never worked with qsfp before, so I'm not really sure how this stuff works.

    Indeed, initially I was afraid it might break during shipping, but I guess I can buy three of them and it'll still be cheaper.
     
    #1544
  5. fohdeesha

    fohdeesha Kaini Industries

    Joined:
    Nov 20, 2016
    Messages:
    1,109
    Likes Received:
    854
    You can indeed breakout the switch qsfp ports to 4x 10gbe, but you can't do the same from the nic - nics do not support qsfp breakout, at least none I've heard of. You can use a qsfp > sfp+ adapter (search eBay for 655874-B21) and get a single 10gb connection out of the nic though
     
    #1545
    Emdee likes this.
  6. Emdee

    Emdee New Member

    Joined:
    Feb 28, 2019
    Messages:
    5
    Likes Received:
    0
    Ahhh, that makes sense.
    I was wondering how the breakout would work in the OS.

    In that case should I instead go for a double 10Gb SFP+ nic (Mellanox MCX312A-XCBT?) and two normal SFP+ DAC's cables (https://www.fs.com/de-en/products/36673.html)?
     
    #1546
  7. tjk

    tjk Active Member

    Joined:
    Mar 3, 2013
    Messages:
    220
    Likes Received:
    30
    Anyone have a link to 1g modules that will work in the 6650 & 77xx line? I still have some older stuff that I need to connect with 1Gb copper links.

    Thanks!
     
    #1547
  8. fohdeesha

    fohdeesha Kaini Industries

    Joined:
    Nov 20, 2016
    Messages:
    1,109
    Likes Received:
    854
    it's up to you, but I am a fan of going with the 40gb OEM cards (flashed with latest mellanox fw) and the qsfp > sfp+ adapter - it comes out to the same price total, and that way if you ever want to move to 40gbE, you already have a 40gbE nic installed and configured ready to go
     
    #1548
    tommybackeast likes this.
  9. fohdeesha

    fohdeesha Kaini Industries

    Joined:
    Nov 20, 2016
    Messages:
    1,109
    Likes Received:
    854
    any of them will work, have used a lot of HP and cisco modules off ebay with zero issues - one of the great things about these brocades is how little they care about modules
     
    #1549
  10. fohdeesha

    fohdeesha Kaini Industries

    Joined:
    Nov 20, 2016
    Messages:
    1,109
    Likes Received:
    854
    cleaned up original post, added icx7150, icx7250, and icx6650 to it so all choices are in one post, with pictures for easy beef selection
     
    #1550
    tommybackeast, yukaia and Ouraing like this.
  11. tjk

    tjk Active Member

    Joined:
    Mar 3, 2013
    Messages:
    220
    Likes Received:
    30
    When you have a few mins, think you can post a working config for MCT between 2x 6650's? Pretty please? :)

    Also, possible to MCT > 3 or 4 of these together?
     
    #1551
  12. Discountsocks

    Discountsocks New Member

    Joined:
    Aug 16, 2018
    Messages:
    7
    Likes Received:
    3
    #1552
    Last edited: Mar 22, 2019
    tjk likes this.
  13. packmule

    packmule New Member

    Joined:
    Mar 21, 2019
    Messages:
    3
    Likes Received:
    1
    I bought my first 6610-48P today! Thank you Fohdeesha for putting this information together. It's hard to beat a dirt cheap access switch that has the feature set to function in a collapsed core, distribution or datacenter role. These 6610s are swiss army knives for home lab networks! I thought about buying two, but the rest of my network needs attention too.

    I'm upgrading to gigabit Internet soon and my existing home network is trash. They say a mechanic's car is the last to get fixed... well the same can be said for network guys and our home networks... or at least mine! My wired network currently consists of three MoCA adapters connecting my three coax drops. My dd-wrt routers are placed for maximal wife acceptance, which ironically makes them unacceptable when wifi doesn't work on the deck or corner rooms of the house! :D
     
    #1553
    fohdeesha likes this.
  14. fohdeesha

    fohdeesha Kaini Industries

    Joined:
    Nov 20, 2016
    Messages:
    1,109
    Likes Received:
    854
    I don't have any configs, never used MCT in production. There's a whole MCT section in the l2 guide though, page 157 of fastiron-08030mb-l2guide.pdf from the zip

    keep in mind unlike stacking, MCT syncs l2 states only, l3 you're on your own. eg each switch will run its own BGP session separately just like if you didn't have MCT, and you'll have to make everything aware of each switches BGP session separately etc, depending on the setup you might have to get creative: Layer 3 routing on Cumulus Linux MLAG
     
    #1554
  15. tjk

    tjk Active Member

    Joined:
    Mar 3, 2013
    Messages:
    220
    Likes Received:
    30
    Thanks, I use these purely as L2 devices for hypervisors and storage only.
     
    #1555
  16. arglebargle

    arglebargle H̸̖̅ȩ̸̐l̷̦͋l̴̰̈ỏ̶̱ ̸̢͋W̵͖̌ò̴͚r̴͇̀l̵̼͗d̷͕̈

    Joined:
    Jul 15, 2018
    Messages:
    518
    Likes Received:
    154
    So I finally took the plunge. I've been faffing about with my 6450s for the past month, they've been hung off my existing infrastructure with pis setup as serial adapter hosts so I can make breaking changes while I learn how to use them without having to walk all over the house to reset switches after locking myself out.

    Anyway, last night everything finally clicked, so I spent a couple of hours at 4am ripping and tearing on my network infrastructure while the girlfriend and housemates were asleep. Everything's working great now, the brocades are a joy to use and once I pull the rest of the fiber I'll have 10GbE throughout the house.

    And now I've got alerts setup for the next cheap 6610 that pops up, help!

    A couple of questions:

    What should I do about spanning tree? I have a pretty simple home network consisting of two 6450s, a couple of non Brocade managed switches and a couple of Proxmox machines that I'm going to be running openvswitch and dpdk on. Should I just turn on single spanning tree 802.1w globally on both switches and call it good or is there a better option that I don't know about? I honestly don't know much about RSTP other than it's a good idea to have it enabled in case someone not-me gets stupid with ethernet cables.

    Is it possible to group ports and name them? I'd like to make some ease of use groupings for the 1g ports to save time/make it easier to grok what I'm working on. Something like "top-row" "bottom-row" or "top-left" (for the even number ports between 1/1/1 and 1/1/11.) Is that possible?
     
    #1556
    Last edited: Mar 24, 2019
  17. Lambdariver

    Lambdariver New Member

    Joined:
    Mar 24, 2019
    Messages:
    5
    Likes Received:
    2
    Thanks for the epic guide! Just picked up an ICX6650 to tinker with and replace my netgear prosumer switch. What were you doing with the JTAG interface? Dumping the ROM?

    Nothing like real systems for making mistakes and learning with. Hoping this gives me some street cred with the network engineers and help me stave off the label "paper architect" for one more year.
     
    #1557
    Sleyk likes this.
  18. fohdeesha

    fohdeesha Kaini Industries

    Joined:
    Nov 20, 2016
    Messages:
    1,109
    Likes Received:
    854
    dumping live RAM contents
     
    #1558
    Sleyk likes this.
  19. 747builder

    747builder Member

    Joined:
    Dec 17, 2017
    Messages:
    74
    Likes Received:
    20
    my 6650 came with a couple of handfuls of 1G Copper
    approved optics xbr-000190-a (1G copper)

    I've used everything from juniper to cisco optics in mine. YMMV
     
    #1559
    Sleyk likes this.
  20. Sleyk

    Sleyk Active Member

    Joined:
    Mar 25, 2016
    Messages:
    685
    Likes Received:
    142
    #1560
Similar Threads: Brocade Series
Forum Title Date
Networking Has anyone got a Mellanox ConnectX-5 100G card connected to a Brocade ICX 40G QSFP to QSFP? Apr 5, 2019
Networking [Solved] Brocade ICX6610-48P PoE Errors Apr 2, 2019
Networking Anyone done a Brocade 6610 fan mod? Apr 1, 2019
Networking Brocade 6650 Mar 9, 2019
Networking Brocade ICX7400-1X40GQ vs ICX7000-1X40GQ Feb 13, 2019

Share This Page