Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

Discussion in 'Networking' started by fohdeesha, Jul 12, 2018.

  1. spali

    spali Member

    Joined:
    Nov 4, 2018
    Messages:
    32
    Likes Received:
    3
    Does this also work if I use the 40g's as uplink to a 40g switch?
    My plan is to uplink two 6610 to my arista 40 g "core" switch.
    But if I understand it right, this forces me to not stack or stack with the breakout ports. I wonder how I could force the main traffic to the core switch over the 40g uplinks in a "breakout" stack.
    Any ideas or suggestions?
     
    #1881
  2. Terry Wallace

    Terry Wallace PsyOps SysOp

    Joined:
    Aug 13, 2018
    Messages:
    114
    Likes Received:
    62
    I’d take 2 of the 40 gig ports on each 6610 and make them a stack
    Then I’d take the other 2 ports off the 6610s and make an mlag to your arista. That way you get failover stack connections.
     
    #1882
  3. spali

    spali Member

    Joined:
    Nov 4, 2018
    Messages:
    32
    Likes Received:
    3
    I'm new to mlag, but aren't the links then limited to 8x 10g per switch?
     
    #1883
  4. Joao Paulo Mello

    Joao Paulo Mello New Member

    Joined:
    Jan 16, 2019
    Messages:
    14
    Likes Received:
    1
    Guys. Finally I got my hands on my ICX -6610-48p and it could not be different being a Cisco guy I'm having some trouble to configure the 2 rear stacking ports.
    I want to use as a regular port to connect to my windows 10 box but no joy.
    I'm using the fllowing hardware and cables:
    NIC: MCX354A-FCBT 649281-B21 656089-001 VPI FDR 40GbE Mellanox from HP 544QSFP MCX354A-FCBT 649281-B21 656089-001 VPI FDR 40GbE Mellanox OEMFirmware | eBay
    Cable: Netapp 112-00178 5M 16ft QSFP-QSFP External SAS Cable X6559-R6 from NetApp X 6559 R 6 SAS Cable 5 M QSFP for sale online | eBay

    Windows 10 recognized the NIC right away but no link and the ICX6610 says the same:

    #show interfaces ethernet 1/2/1
    40GigabitEthernet1/2/1 is down, line protocol is down
    Port down for 1 hours 13 minutes 24 seconds
    Hardware is 40GigabitEthernet, address is cc4e.24f8.ef2c (bia cc4e.24f8.ef5d)
    Interface type is 40Gig Fiber
    Configured speed 40Gbit, actual unknown, configured duplex fdx, actual unknown
    Configured mdi mode AUTO, actual unknown
    Member of L2 VLAN ID 1, port is untagged, port state is BLOCKING
    BPDU guard is Disabled, ROOT protect is Disabled, Designated protect is Disabled
    Link Error Dampening is Disabled
    STP configured to ON, priority is level0, mac-learning is enabled
    Flow Control is enabled
    Mirror disabled, Monitor disabled
    Mac-notification is disabled
    Not member of any active trunks
    Not member of any configured trunks
    No port name


    The same happens with the 1/2/6.
    The device came factory reset so it was pretty clean, that's my running config, pretty short:

    #show running-config
    Current configuration:
    !
    ver 08.0.10mT7f3
    !
    stack unit 1
    module 1 icx6610-48p-poe-port-management-module
    module 2 icx6610-qsfp-10-port-160g-module
    module 3 icx6610-8-port-10g-dual-mode-module
    stack-trunk 1/2/1 to 1/2/2
    stack-trunk 1/2/6 to 1/2/7
    !
    !
    !
    !
    vlan 1 name DEFAULT-VLAN by port
    router-interface ve 1
    !
    !
    !
    !
    aaa authentication web-server default local
    aaa authentication enable default local
    aaa authentication login default local
    enable telnet authentication
    !
    username root password .....
    !
    !
    !
    !
    !
    !
    !
    !
    !
    interface management 1
    ip address 192.168.1.1 255.255.255.0
    !
    interface ve 1
    ip address 10.10.240.250 255.255.255.0
    !
    !
    !
    !
    !
    !
    !
    !
    !
    end




    Any help would be greatly appreciated.

     
    #1884
  5. itronin

    itronin Active Member

    Joined:
    Nov 24, 2018
    Messages:
    214
    Likes Received:
    112
    You are missing a couple of things. I'm assuming you are running a single switch and will be doing no stacking at all. I'm assuming your vlan is untagged, it is vlan 1 and your are using 1/2/1 for your data port.

    1) you need to disable the stack and reboot the switch
    conf t
    disable stack
    exit
    write mem
    reload


    2) you need to add the 40Gbe port to your vlan
    conf t
    vlan 1
    untagged ethe 1/2/1
    # don't forget to add any other ports (tagged or untagged) to vlan 1.
    exit
    exit
     
    #1885
  6. Bell System

    Bell System New Member

    Joined:
    May 26, 2019
    Messages:
    4
    Likes Received:
    1
    As promised, I wanted to update everyone on my 6450-24P switches from eBay seller "G-electronic."

    I ordered two switches and unfortunately, both PoE modules are bad. They just keep printing the following over and over again:

    Code:
    PoE Error: Device 0 failed to start on PoE module.
    PoE Error: Device 1 failed to start on PoE module.
    Resetting module in slot 1 again to recover from dev fault
    PoE Info: Hard Resetting in slot 1....
    PoE Info: Resetting module in slot 1....completed.
    Oddly, some ports do provide PoE power, but doing a
    Code:
    sh inline power
    indicates that the switch doesn't even detect a connected PoE device.

    One switch even has at least one port that is totally dead.

    Seems like this guy is either reselling the same bad PoE switches hoping no one will notice, or he has a huge lot full of damaged switches. In any case, mine are going back, and I recommend staying away from this seller if you're interested in getting a switch with working PoE.
     
    #1886
    fohdeesha likes this.
  7. Joao Paulo Mello

    Joao Paulo Mello New Member

    Joined:
    Jan 16, 2019
    Messages:
    14
    Likes Received:
    1
    Thanks for you reply.

    I did but I had to "stack disable" instead of "disable stack".
    The port is on vlan 1 since it's default vlan and is already untagged
    #show interfaces ethernet 1/2/1
    40GigabitEthernet1/2/1 is down, line protocol is down
    Port down for 6 minutes 58 seconds
    Hardware is 40GigabitEthernet, address is cc4e.24f8.ef2c (bia cc4e.24f8.ef5d)
    Interface type is 40Gig Fiber
    Configured speed 40Gbit, actual unknown, configured duplex fdx, actual unknown
    Configured mdi mode AUTO, actual unknown
    Member of L2 VLAN ID 1, port is untagged, port state is BLOCKING
    BPDU guard is Disabled, ROOT protect is Disabled, Designated protect is Disabled
    Link Error Dampening is Disabled
    STP configured to ON, priority is level0, mac-learning is enabled
    Flow Control is enabled
    Mirror disabled, Monitor disabled
    Mac-notification is disabled
    Not member of any active trunks
    Not member of any configured trunks
    No port name
    MTU 1500 bytes, encapsulation ethernet
    300 second input rate: 0 bits/sec, 0 packets/sec, 0.00% utilization
    300 second output rate: 0 bits/sec, 0 packets/sec, 0.00% utilization
    0 packets input, 0 bytes, 0 no buffer
    Received 0 broadcasts, 0 multicasts, 0 unicasts
    0 input errors, 0 CRC, 0 frame, 0 ignored
    0 runts, 0 giants
    0 packets output, 0 bytes, 0 underruns
    Transmitted 0 broadcasts, 0 multicasts, 0 unicasts
    0 output errors, 0 collisions
    Relay Agent Information option: Disabled

    Egress queues:
    Queue counters Queued packets Dropped Packets
    0 0 0
    1 0 0
    2 0 0
    3 0 0
    4 0 0
    5 0 0
    6 0 0
    7 0 0


    The problem remains, no link.
    I read something mentioning that I need to have the latest firmware running in order to use the stack ports as data port.
    That's my show version:

    #show ver
    Copyright (c) 1996-2014 Brocade Communications Systems, Inc. All rights reserved.
    UNIT 1: compiled on Jul 08 2015 at 16:29:14 labeled as FCXR08010m
    (9716466 bytes) from Primary FCXR08010m.bin
    SW: Version 08.0.10mT7f3
    Boot-Monitor Image size = 370695, Version:10.1.00T7f5 (grz10100)
    HW: Stackable ICX6610-48-HPOE
    ==========================================================================
    UNIT 1: SL 1: ICX6610-48P POE 48-port Management Module
    Serial #: xxxxxxxxxxxxx
    License: ICX6610_ADV_ROUTER_SOFT_PACKAGE (LID: xxxxxxxxxxxxxxxxxxxx)
    P-ENGINE 0: type E02B, rev 01
    P-ENGINE 1: type E02B, rev 01
    ==========================================================================
    UNIT 1: SL 2: ICX6610-QSFP 10-port 160G Module
    ==========================================================================
    UNIT 1: SL 3: ICX6610-8-port Dual Mode(SFP/SFP+) Module
    ==========================================================================
    800 MHz Power PC processor 8544E (version 0021/0023) 400 MHz bus
    65536 KB flash memory
    512 MB DRAM
    STACKID 1 system uptime is 13 minutes 7 seconds
    The system : started=warm start reloaded=by "reload"
     
    #1887
    Last edited: Jun 4, 2019
  8. Joao Paulo Mello

    Joao Paulo Mello New Member

    Joined:
    Jan 16, 2019
    Messages:
    14
    Likes Received:
    1
    Solved!!!
    Going back to the past one more time I noticed the @arglebargle post: https://forums.servethehome.com/ind...gbe-40gbe-switching.21107/page-15#post-198883

    That solved for me :).
    Thanks @arglebargle, thanks everyone!

    Edit:
    I'm back with one more question: My unit came with 2 PSU's and just one fan tray. Does make sense to assume that installing another fan tray the fans will calm down a bit? The PSU's fan spin down after boot and I can live with that but if there's any way to slow down those boys a little that would be great. Have anyone had such experience?
     
    #1888
    Last edited: Jun 4, 2019
  9. fohdeesha

    fohdeesha Kaini Industries

    Joined:
    Nov 20, 2016
    Messages:
    1,482
    Likes Received:
    1,249
    #1889
    Joao Paulo Mello likes this.
  10. fohdeesha

    fohdeesha Kaini Industries

    Joined:
    Nov 20, 2016
    Messages:
    1,482
    Likes Received:
    1,249
    if your PSUs are already spinning down, adding a second fan tray will not make them spin down further
     
    #1890
    Joao Paulo Mello likes this.
  11. Joao Paulo Mello

    Joao Paulo Mello New Member

    Joined:
    Jan 16, 2019
    Messages:
    14
    Likes Received:
    1
    Will do.

    Damn! But I can live with the current noise level when the PSU's spins down...I have other stuff on the rack, some of them modded to be quiet and some couldn't be modded. So it's just another addition to the choir.

    I was just testing the 40g adapter on Windows, in fact it will be used at my NAS (Ubuntu), the default protocol mode on linux is Infiniband as well? Do I'll need to install the driver and set to Ethernet somewhere?
     
    #1891
  12. bkvamme

    bkvamme New Member

    Joined:
    Sep 23, 2018
    Messages:
    6
    Likes Received:
    1
    Hi,
    Might be a long shot, but does anyone know the ECCN and CCATS number for the ICX-6610-48P? My freight forwarder has been unable to get the numbers.

    Based on what I see online, the ECCN should be: 5A002, but I am unable to find the CCATS number.

    Edit: OK, this is getting a bit ridiculous. Contacted Ruckus support, and the rep stated that I since I purchased the switch through an unauthorised vendor (eBay) I would need to purchase a support contract for them to provide me with this number. Wth. Ubiquiti and Cisco are listing these online.
     
    #1892
    Last edited: Jun 5, 2019
  13. fohdeesha

    fohdeesha Kaini Industries

    Joined:
    Nov 20, 2016
    Messages:
    1,482
    Likes Received:
    1,249

    its still the same link in the original thread in bold
     
    #1893
    dwright1542 likes this.
  14. fohdeesha

    fohdeesha Kaini Industries

    Joined:
    Nov 20, 2016
    Messages:
    1,482
    Likes Received:
    1,249
    was digging around in @britinpdx 's ICX7250, it came with some dev software, was unbricking it for him and came across a hidden environment variable: setenv diag_mode_enable=1

    Enabling this from the bootloader by running:

    setenv diag_mode_enable 1
    saveenv
    reset

    makes it reset into the bootloader again, except now it will be full featured u-boot with everything available, all the i2c raw read and write commands etc, all the stuff that's hidden in the "normal" bootloader prompt.

    to get out of it and go back to booting normally just run:

    setenv diag_mode_enable
    saveenv
    reset

    Should work on any brocade switch running a u-boot based bootloader (NOT the 6610)
     
    #1894
    capn13, tommybackeast and arglebargle like this.
  15. sean

    sean Member

    Joined:
    Sep 26, 2013
    Messages:
    60
    Likes Received:
    29
    I checked their source and it appears in the patch file for the 6650 and 7250 but not the 6450 or 7650.
     
    #1895
    fohdeesha and arglebargle like this.
  16. cuco

    cuco Member

    Joined:
    Feb 13, 2018
    Messages:
    44
    Likes Received:
    2
    Hey,
    does anyone know a good place to buy cheap rackmount ears for the 6610?
    Cant find anything on ebay
    Thanks
     
    #1896
  17. HomelabN0rth

    HomelabN0rth New Member

    Joined:
    Jan 30, 2019
    Messages:
    3
    Likes Received:
    2
    Hey all, thanks for all the info here. It made me jump on the ICX7250 train via ebay but I F*ed up and ordered from G-Electronics and got a switch with corrupted images on both the primary and secondary. I can get into ICX7250-Boot> but can't figure out how to do anything else. This is what happens:

    ICX7250-Boot> tftpboot
    ERROR: Ethernet external port not connected
    at bcmiproc_eth.c:124/bcmiproc_eth_open()
    could not boot from tftp, trying to boot from primary
    Error: The header crc ffffffff and the calculated crc 4007851a mismatch
    Fatal: Firmware integrity checksum failed
    Booting image from Primary
    no current device selected
    Wrong Image Format for bootm command
    ERROR: can't get kernel image!
    could not boot from primary, no valid image; trying to boot from secondary
    Error: The header crc ffffffff and the calculated crc 4007851a mismatch
    Fatal: Firmware integrity checksum failed
    Booting image from Secondary
    no current device selected
    Wrong Image Format for bootm command
    ERROR: can't get kernel image!
    Wrong Image Format for tftpboot command
    ERROR: can't get kernel image!
    ICX7250-Boot>

    When the switch boots, it boots to a DIAG> prompt. From there, I can type linux_shell and get the switch to reboot, and if I press B during the beginning of that process, I can get to this ICX7250-Boot> prompt. Any suggestions how I can get from this boot menu to upload the new firmware to the device? Or am I just totally borked?
     
    #1897
  18. fohdeesha

    fohdeesha Kaini Industries

    Joined:
    Nov 20, 2016
    Messages:
    1,482
    Likes Received:
    1,249
    yours came with the exact same diag bootloader and broadcom test image that @britinpdx did (also from g-electronic). install teamviewer and send me credentials, I can fix it. you'll need a putty window up connected to the switch over serial and the management port plugged into a network with a tftp server

    we can definitively rule out g-electronic doing ANY testing, what a liar
     
    #1898
    tommybackeast and HomelabN0rth like this.
  19. HomelabN0rth

    HomelabN0rth New Member

    Joined:
    Jan 30, 2019
    Messages:
    3
    Likes Received:
    2
    Actually, I got it figured out.
    I had to setenv the ip settings in the Boot>, used TFTP then to mount a new boot image. Restarted the switch. After that, did the same using update_primary and update_ secondary to update the main image and afterwards was back in business. What a pain in the everything tho.
     
    #1899
    fohdeesha likes this.
  20. fohdeesha

    fohdeesha Kaini Industries

    Joined:
    Nov 20, 2016
    Messages:
    1,482
    Likes Received:
    1,249
    damn, I was hoping to extract those development images before wiping them :p
     
    #1900
    HomelabN0rth likes this.
Similar Threads: Brocade Series
Forum Title Date
Networking Mikrotik compatible with Brocade 57-0000075-01 transceivers? Dec 17, 2019
Networking [SOLVED] Help Needed - Brocade ICX 6450 + Ruckus R720 Nov 25, 2019
Networking Brocade ICX 6610 - what does dhcp-client enable do? Nov 19, 2019
Networking MikroTik CSS326-24G-2S+RM vs Brocade ICX6450-24 Nov 1, 2019
Networking Brocade ICX6450-24P vs Aruba S2500-24P for Homelab Sep 22, 2019

Share This Page