Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

Discussion in 'Networking' started by fohdeesha, Jul 12, 2018.

  1. fohdeesha

    fohdeesha Kaini Industries

    Joined:
    Nov 20, 2016
    Messages:
    1,489
    Likes Received:
    1,253
    they are referencing mixing optics slots like sfp/qsfp with built in copper ports. The QSFP and SFP ports are both "fiber ports". As long as the ports are optics slots (qsfp, sfp, etc) you can mix module types no problem. Ran a lag between an SR optic module and an RJ45 10GBASE-T module without issue:


    Code:
    SSH@GG-CORE-N1>sh lag test2
    Total number of LAGs:          2
    Total number of deployed LAGs: 2
    Total number of trunks created:2 (118 available)
    LACP System Priority / ID:     1 / 748e.f8f7.8498
    LACP Long timeout:             120, default: 120
    LACP Short timeout:            3, default: 3
    === LAG "test2" ID 1 (dynamic Deployed) ===
    Deployment: HW Trunk ID 2
    Port       Link    State   Dupl Speed Trunk Tag Pvid Pri MAC             Name
    1/3/2      Up      Forward Full 10G   1     No  1    0   748e.f8f7.8498
    2/3/2      Up      Forward Full 10G   1     No  1    0   748e.f8f7.8498
    Port       [Sys P] [Port P] [ Key ] [Act][Tio][Agg][Syn][Col][Dis][Def][Exp][Ope]
    1/3/2           1        1   20001   Yes   L   Agg  Syn  Col  Dis  No   No   Ope
    2/3/2           1        1   20001   Yes   L   Agg  Syn  Col  Dis  No   No   Ope

    1/3/2 is an SR optics module, 2/3/2 is a mikrotik copper rj45 module
     
    #941
    juey likes this.
  2. gigogig

    gigogig New Member

    Joined:
    Oct 29, 2018
    Messages:
    14
    Likes Received:
    2
    i just got 2 6450-24p's in and trying to update their firmware both are having problems,
    unit one is bootlooping with:

    | | | | | __ ) ___ ___ | |_ | | | |___| _ \ / _ \ / _ \| __| | |_| | - Pastebin.com

    "after rebooting to secondary boot partition

    | | | | | __ ) ___ ___ | |_ | | | |___| _ \ / _ \ / _ \| __| | |_| |__ - Pastebin.com



    after booting from secondary partition unit 1 is up, but as its stuck in router not switch mode i don't know how to properly update it to switch mode with new firmware?







    _____________________________________________________
    and unit two was throwing :

    For port 1/2/1 4.0x9004 = 0x100

    For port 1/2/2 4.0x9003 = 0x0
    For port 1/2/2 4.0x9004 = 0x100

    For port 1/2/3 4.0x9003 = 0x0
    For port 1/2/3 4.0x9004 = 0x100

    For port 1/2/4 4.0x9003 = 0x0
    For port 1/2/4 4.0x9004 = 0x100

    scratch unit 2, firmware fixed it from doing that constantly, so unit 2 look all good now
     
    #942
    Last edited: Nov 2, 2018
  3. fohdeesha

    fohdeesha Kaini Industries

    Joined:
    Nov 20, 2016
    Messages:
    1,489
    Likes Received:
    1,253
    that is dev/testing firmware (especially the bootloader), I've seen this come on ebay units (supposedly from brocade hq) a couple times, it's fixable, you definitely don't want to run that software and OS

    On both units, can you follow the guide to flash new bootloader and OS? it should NOT have the big ascii u-boot logo during boot. If that doesn't work, there's a more "hardcore" way of overwriting the dev software from u-boot, I would probably have to do that remotely over google hangouts or teamviewer as it's too much to write out

    Either way, once you're fully flashed to the normal bootloader, poe firmware, and OS, can you please run the following commands in the bootloader (hit b during boot to get into it), and paste the results on Pastebin.com - #1 paste tool since 2002! ?

    printenv
    printenv bootcmd
    printenv defbootcmd
    printenv extra_bootargs

    Most of the ICX's I've seen come like this also had dev/chassis polling/test mode flags set in u-boot, this will tell me if there are, then I can tell you how to get rid of them. please do not skip this step. If it's quicker for you, just PM me teamviewer info and I can do all of it including the flashing, this dev software can be tricky to replace
     
    #943
  4. gigogig

    gigogig New Member

    Joined:
    Oct 29, 2018
    Messages:
    14
    Likes Received:
    2
    unit 2 took the firmwares without any fuss, the were constantly outputting odd data before the update but after the firmware upgrade, they seem good, here is there boot up, Bootloader Version: 10.1.05T310 (Mar 19 2015 - 16:39:59) Model ID: 1.0.0. - Pastebin.com and env Bootloader Version: 10.1.05T310 (Mar 19 2015 - 16:39:59) Model ID: 1.0.0. - Pastebin.com

    unit one on the other hand , seemed to be in working order until i tried to update the firmware, that's when i got stuck in the reboot loop here | | | | | __ ) ___ ___ | |_ | | | |___| _ \ / _ \ / _ \| __| | |_| | - Pastebin.com

    and here is unit one env Bootloader Version: 10.1.05T310 (Mar 19 2015 - 16:39:59) Model ID: 1.0. - Pastebin.com

    what should i do to unit one? retry the flash?

    edit i retried the flash, this is what i got
    UNIT 1: compiled on Nov 18 2011 at 14:51:58 labeled as ICXR07400b1 - Pastebin.com
     
    #944
    Last edited: Nov 2, 2018
  5. fohdeesha

    fohdeesha Kaini Industries

    Joined:
    Nov 20, 2016
    Messages:
    1,489
    Likes Received:
    1,253
    unit 2 looks good - make sure to update it's PoE firmware as well if you haven't

    unit 1 definitely still has a dev bootloader which is why it can't boot. I can fix this for you, but I need remote access (teamviewer is usually easiest). it involves low level NAND flash writing commands that if screwed up will brick it, so it's too risky for me to write them out and then you try to follow them
     
    #945
  6. realtomatoes

    realtomatoes Active Member

    Joined:
    Oct 3, 2016
    Messages:
    243
    Likes Received:
    31
    got me one of those HP 649281-B21 and a qsfp cable recently to connect my 2 nodes directly for VM storage traffic.
    am now looking at plugging the other port to the 6450 which am still shopping for.
    newb question: i am gonna need something like the MAM1Q00A-QSA to connect the HP nic to the 6450, right?
     
    #946
  7. fohdeesha

    fohdeesha Kaini Industries

    Joined:
    Nov 20, 2016
    Messages:
    1,489
    Likes Received:
    1,253
    one of these GENUINE 655902-001 655874-B21 HP MELLANOX QSFP to SFP ADAPTER 886111656585 | eBay

    with an sfp+ optic like (brand does not matter) Brocade 10GB 57-0000075-01 10G-SFPP-SR 10Gbase-sr/sw 850nm SFP+ Transceiver >500 689719088136 | eBay

    (unless you plan on using a DAC, never tried an SFP+ DAC with a qsfp to sfp adapter, but it should probably work)

    follow this post to get the proper firmware on the NIC https://forums.servethehome.com/ind...net-dual-port-qsfp-adapter.20525/#post-198015
     
    #947
    tommybackeast likes this.
  8. realtomatoes

    realtomatoes Active Member

    Joined:
    Oct 3, 2016
    Messages:
    243
    Likes Received:
    31
    thanks, for this and all the knowledge you've shared. it is greatly appreciated.
     
    #948
    fohdeesha likes this.
  9. fohdeesha

    fohdeesha Kaini Industries

    Joined:
    Nov 20, 2016
    Messages:
    1,489
    Likes Received:
    1,253
    thanks to @gigogig 's patience I was able to copy out the internal-use only dev bootloader before overwriting it with stock - been wanting to do that for a while but haven't found another ICX with such a bootloader all year. Seems to have a ton of custom routines and commands, programming voltages and fans etc, will load it on one of my bench 6450's sometime soon and see if there's anything useful
     
    #949
  10. mattaw

    mattaw Member

    Joined:
    Jul 30, 2018
    Messages:
    40
    Likes Received:
    13
    @fohdeesha , do I need the brocade serial cable? Googling around I have found the pinout but I wasn't sure if it was necessary with the dedicated management port. Maybe there is a cheap alternative?
     
    #950
  11. fohdeesha

    fohdeesha Kaini Industries

    Joined:
    Nov 20, 2016
    Messages:
    1,489
    Likes Received:
    1,253
    For what switch? the ICX6xxx series use the industry standard RJ45 serial - https://www.amazon.com/Console-Essential-Accesory-Ubiquity-Switches/dp/B01AFNBC3K

    You'll need serial access to reset the switch to factory defaults and clear any old passwords it comes with, you can't do that from the management ethernet port (especially if the past config never set it up with an IP anyway)
     
    #951
    mattaw likes this.
  12. fohdeesha

    fohdeesha Kaini Industries

    Joined:
    Nov 20, 2016
    Messages:
    1,489
    Likes Received:
    1,253
    So...who bought the 3x 7150's and 7250?
     
    #952
  13. anlin

    anlin New Member

    Joined:
    Dec 8, 2016
    Messages:
    29
    Likes Received:
    6
    Just pulled the trigger on a 7250-24P, can't wait for it to arrive! Do you know if the 7250 is also brand agnostic when it comes to SFP+ modules? Cisco 10-2415-03 from China are the cheapest ones available to me at about $7. The brocade modules you linked to above cost $40 each after shiping and customs charges :(
     
    #953
  14. fohdeesha

    fohdeesha Kaini Industries

    Joined:
    Nov 20, 2016
    Messages:
    1,489
    Likes Received:
    1,253
    yup the 7 series is identical to the 6 series in those regards, no locks of any kind
     
    #954
  15. mattaw

    mattaw Member

    Joined:
    Jul 30, 2018
    Messages:
    40
    Likes Received:
    13
    Of the 3x icx6610's coming there are 4 psus but only 2x fans out of a possible 6. The cost of new fan units is >$175 each?!?
     
    #955
  16. fohdeesha

    fohdeesha Kaini Industries

    Joined:
    Nov 20, 2016
    Messages:
    1,489
    Likes Received:
    1,253
    well you're only getting 1 working unit, so you actually have a spare fan tray :p (looking on the bright side)

    fan trays pop up around 20 or 30 dollars on ebay, you just have to be patient (really patient, sometimes)
     
    #956
  17. fohdeesha

    fohdeesha Kaini Industries

    Joined:
    Nov 20, 2016
    Messages:
    1,489
    Likes Received:
    1,253
    I'll see if I can get an icx7xxx update guide and firmware dl up before all of your guys switches arrive
     
    #957
  18. Skud

    Skud Member

    Joined:
    Jan 3, 2012
    Messages:
    71
    Likes Received:
    18
    I grabbed one 7150 and one 7250-24P.

    Gonna see how the 7250 compares to the 6610-24P I just received. I may use the 7250 instead if the noise and power is a lot less. I don’t really have a use for the QSFPs anyways.

    Not that my 6610 is loud. Is there much of a difference in noise between the rev b and c PSUs of the 6610?

    Riley
     
    #958
    fohdeesha likes this.
  19. mattaw

    mattaw Member

    Joined:
    Jul 30, 2018
    Messages:
    40
    Likes Received:
    13
    I'll keep my eyes open. So you don't need two fan trays per switch then? That's a relief.
     
    #959
  20. fohdeesha

    fohdeesha Kaini Industries

    Joined:
    Nov 20, 2016
    Messages:
    1,489
    Likes Received:
    1,253
    nope, 99% of them were sold with 1, the fan tray is already redundant with the PSU fans (it will run with zero fan trays, it will just complain and tell you to replace the failed tray ASAP)

    zero difference between rev b and rev c sound wise
     
    #960
    mattaw likes this.
Similar Threads: Brocade Series
Forum Title Date
Networking Mikrotik compatible with Brocade 57-0000075-01 transceivers? Dec 17, 2019
Networking [SOLVED] Help Needed - Brocade ICX 6450 + Ruckus R720 Nov 25, 2019
Networking Brocade ICX 6610 - what does dhcp-client enable do? Nov 19, 2019
Networking MikroTik CSS326-24G-2S+RM vs Brocade ICX6450-24 Nov 1, 2019
Networking Brocade ICX6450-24P vs Aruba S2500-24P for Homelab Sep 22, 2019

Share This Page