Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

Discussion in 'Networking' started by fohdeesha, Jul 12, 2018.

  1. AllenB

    AllenB New Member

    Joined:
    Oct 16, 2018
    Messages:
    2
    Likes Received:
    2
    Got my ICX6610-48P fully nine days ago, literally *on the day* of moving into a new house. Now finally spending some quality time in the (still very messy, stuff on the floor) home office getting set up. What a great piece of used kit!

    My main desktop is Ubuntu Linux 18.04 and as some of you probably have, I ran into issues with the old SSH server on the switch. For anyone who's not familiar with getting around it, here's what I did to generate and install a public key, then connect to the switch:

    On Linux: (with IP 192.168.1.10/24)
    cd ~/.ssh
    ssh-keygen -t rsa
    ssh-keygen -e -f id_rsa.pub >id_rsa_ssh2.pub # convert to SSH2 format, demanded by the switch
    cp id_rsa_ssh2.pub /tftpboot

    On ICX6610: (with IP 192.168.1.99/24)
    ip ssh pub-key-file tftp 192.168.1.10 id_rsa_ssh2.pub
    ip ssh key-authentication yes

    On Linux:
    ssh -oKexAlgorithms=+diffie-hellman-group1-sha1 -i ~/.ssh/id_rsa 192.168.1.99
    # I set up an alias like this in .bashrc
    alias icx='ssh -oKexAlgorithms=+diffie-hellman-group1-sha1 -i ~/.ssh/id_rsa 192.168.1.99'

    Hope this comes in handy for someone.

    Cheers,
    Allen
     
    #961
    fohdeesha and vkfu like this.
  2. sean

    sean Member

    Joined:
    Sep 26, 2013
    Messages:
    60
    Likes Received:
    29
    #962
    Last edited: Nov 3, 2018
    AllenB likes this.
  3. fohdeesha

    fohdeesha Kaini Industries

    Joined:
    Nov 20, 2016
    Messages:
    1,464
    Likes Received:
    1,221
    #963
  4. Skud

    Skud Member

    Joined:
    Jan 3, 2012
    Messages:
    70
    Likes Received:
    15
    Ooooo... thanks for the heads up. That’s pretty annoying they did that. I’ve done some quick looking for a pinout but haven’t seen anything.

    I have a bunch of mini-USBs I could hack and put an 8P8C/RJ45 keystone there to make it work with my standard console cable.

    Riley
     
    #964
  5. fohdeesha

    fohdeesha Kaini Industries

    Joined:
    Nov 20, 2016
    Messages:
    1,464
    Likes Received:
    1,221
    If you really want to hack something up (I am not responsible if you fry the serial port):

    [​IMG]
     
    #965
  6. Skud

    Skud Member

    Joined:
    Jan 3, 2012
    Messages:
    70
    Likes Received:
    15
    Awesome! Thanks.
     
    #966
  7. anlin

    anlin New Member

    Joined:
    Dec 8, 2016
    Messages:
    29
    Likes Received:
    6
    @fohdeesha Good to know, even though it is annoying to have to buy yet another proprietary cable. How essential to the setup process is having a console cable? Are pins 1 and 4 meant to be left unconnected or pulled to the ground?

    @Skud please let me know if you get console access working with a hacked mini usb cable. I don't see a reason why it wouldn't work, but it's always nice to get confirmation.
     
    #967
    Last edited: Nov 4, 2018
  8. Skud

    Skud Member

    Joined:
    Jan 3, 2012
    Messages:
    70
    Likes Received:
    15
    Will do. I won’t get mine until later next week.

    I would say they are to be left unconnected. Pin 1 on a standard mini USB is VCC so I suspect they left this disconnected on purpose. Probably because there are a lot of people who would assume that this is like the Cisco USB ports where it’s actually a built in serial converter. They would plug in a standard usb cable and put 5v on pin 1.

    Pin 4 is the ID pin on a standard mini USB. Not used in this case. I’m going to leave it disconnected.
     
    #968
  9. anlin

    anlin New Member

    Joined:
    Dec 8, 2016
    Messages:
    29
    Likes Received:
    6
    This was my thinking as well. It's unfortunate Brocade didn't take a page from Cisco's book and just stick an FTDI/USB-UART chip in there, would have made things much easier. My switch probably won't arrive for at least another week and a half, so plenty of time to figure out what to do about the console cable.
     
    #969
  10. dwright1542

    dwright1542 Active Member

    Joined:
    Dec 26, 2015
    Messages:
    357
    Likes Received:
    68
    I had a hunch that STH people were bidding on this, so I stayed away. Glad my hunch was right.
     
    #970
  11. fohdeesha

    fohdeesha Kaini Industries

    Joined:
    Nov 20, 2016
    Messages:
    1,464
    Likes Received:
    1,221
    100% essential

    strangely the rest of the ICX7xxx series have exactly this, even the little 7150 (USB-C on the front, which is an embedded serial adapter). Not sure why they omitted the 7250
     
    #971
    Last edited: Mar 21, 2019
  12. fohdeesha

    fohdeesha Kaini Industries

    Joined:
    Nov 20, 2016
    Messages:
    1,464
    Likes Received:
    1,221
    #972
    tommybackeast likes this.
  13. TrumanHW

    TrumanHW New Member

    Joined:
    Sep 16, 2018
    Messages:
    23
    Likes Received:
    0
    How they managed to get so few CFM per dB while managing to keep the fans 8400 rpm.
    (That's what IP is for though.)

    Buy a QSFP+ switch that's successfully had OEM fans replaced with quiet, aftermarket fans?
     
    #973
  14. N1h1l1ty

    N1h1l1ty New Member

    Joined:
    Oct 21, 2018
    Messages:
    25
    Likes Received:
    1
    I'm assuming

    M:9 L:0 - chow_qsfp_read, qsfp 2, error in seting up mux
    ------------------------------------------------------------------
    M:9 L:0 - link_40G_4x10G_get_media: qsfp 2, port 1/2/2 error in reading qsfp
    chow_40G_4x10G_get_media: error in reading qsfp 1/2/2
    ------------------------------------------------------------------
    M:9 L:0 - chow_qsfp_read, qsfp 3, error in seting up mux
    ------------------------------------------------------------------
    M:9 L:0 - link_40G_4x10G_get_media: qsfp 3, port 1/2/7 error in reading qsfp
    chow_40G_4x10G_get_media: error in reading qsfp 1/2/7


    is a safe bet that the QSFP to SFP+ DAC breakout I got is no bueno? Any possible way of troubleshooting this further?

    EDIT: Upon booting looks like I'm seeing activity on the links and they do show up as up in the switch. Initially when I plugged them in the switch was on and I got nothing, so I reloaded. I guess might be a case of it not liking the modules being hotplugged?
     
    #974
  15. fohdeesha

    fohdeesha Kaini Industries

    Joined:
    Nov 20, 2016
    Messages:
    1,464
    Likes Received:
    1,221
    hotplugging is definitely supported, are you on the latest software? Which breakout cable is it? never seen those errors on the qsfp ports before

    Unrelated: got the ICX6450 dev bootloader running on one of my switches: hastebin
     
    #975
  16. N1h1l1ty

    N1h1l1ty New Member

    Joined:
    Oct 21, 2018
    Messages:
    25
    Likes Received:
    1
    Yep, on ver 08.0.30 straight from the docs page - looking up the serial for these QSFP breakouts, looks like its this guy here:

    5 Meter QSFP+ 4-SFP+ DAC Splitter Cable

    They seem to be running nicely now, I managed to get connectivity to the switch, but I'll need to test it hotplugging
     
    #976
  17. N1h1l1ty

    N1h1l1ty New Member

    Joined:
    Oct 21, 2018
    Messages:
    25
    Likes Received:
    1
    So definitely does not like hotplugging - getting nothing after replugging the DAC on the client end - same error upon reloading the switch

    EDIT: Tinkered further, and after another reload its now playing nice with hotplugging on the client end. Not *exactly* sure what the deal is - I did have to manually enable the interface again, though I'm guessing this is more my user error than the switch doing something silly. That being said, its reporting the correct config and state for 1/2/7 so it seems to be ... fine?
     
    #977
    Last edited: Nov 4, 2018
  18. fohdeesha

    fohdeesha Kaini Industries

    Joined:
    Nov 20, 2016
    Messages:
    1,464
    Likes Received:
    1,221
    that's...odd. everyone else in the thread (including myself) have been hotplugging qsfp no problem (in fact, my zfs store would never come online if it couldn't bring up the qsfp interface well after the switch boots)

    do you have optics or another dac you could try?
     
    #978
  19. N1h1l1ty

    N1h1l1ty New Member

    Joined:
    Oct 21, 2018
    Messages:
    25
    Likes Received:
    1
    Unfortunately I don't have another QSFP breakout to test out but if it proves a problem I may pick up one up to test out. Front panel SFP+ ports seem to be fine with Cisco branded optics however.

    EDIT: I suppose I should try cold booting the switch and see what happens - Let me do that and report in.

    EDIT2: Cold start - same error, but link is brought online automatically, and hotplugging does seem to work though it takes *forever*

    EDIT3: Well, things did not go as well as I had hoped. It appears that something is not playing nice with the QSFP breakouts and I'm getting perfect bandwidth on some links but gigabit speeds and crazy jitter on others. I need to do more testing but it's definitely not a duplex mismatch as I double checked all the clients, and I can't think of anything that would cause that on the switch - I checked with the clients hooked up to my 3750E via 10G and everything was hunky dory, so I'm going to operate under the suspicion that these QSFP breakouts are just not playing nice with the ICX6610 - Ordered another QSFP breakout from amazon set up for Cisco, should be here in the afternoon and I can hopefully use that to troubleshoot further. If that one doesn't play nice, I probably screwed up somewhere along the way when flashing the firmware, though everything else seems to work fine. Worst comes to worst I'll return these and the amazon one and go buy the Dell ones you suggested in the first place @fohdeesha

    Unfortunately I get the niggling suspicion that me cheaping out and buying these instead of the Dell QSFP breakout is the cause of the issue
     
    #979
    Last edited: Nov 5, 2018
  20. N1h1l1ty

    N1h1l1ty New Member

    Joined:
    Oct 21, 2018
    Messages:
    25
    Likes Received:
    1
    So reporting in - I've got another QSFP cable from Amazon, this time Cisco, and its throwing the same error. So it does not look like the cables are the cause of the issue - Any ideas what I could try? Possibly re-flashing firmware?

    EDIT: Re-flashed firmware, have not removed the stack units yet but it is no longer throwing an error on reload.
     
    #980
    Last edited: Nov 5, 2018
Similar Threads: Brocade Series
Forum Title Date
Networking Mikrotik compatible with Brocade 57-0000075-01 transceivers? Dec 17, 2019
Networking [SOLVED] Help Needed - Brocade ICX 6450 + Ruckus R720 Nov 25, 2019
Networking Brocade ICX 6610 - what does dhcp-client enable do? Nov 19, 2019
Networking MikroTik CSS326-24G-2S+RM vs Brocade ICX6450-24 Nov 1, 2019
Networking Brocade ICX6450-24P vs Aruba S2500-24P for Homelab Sep 22, 2019

Share This Page