Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

Discussion in 'Networking' started by fohdeesha, Jul 12, 2018.

  1. Switchback028

    Switchback028 New Member

    Joined:
    May 18, 2018
    Messages:
    22
    Likes Received:
    2
    Just an icx 7250-48 (non POE). I needed POE and more 10g so I went a different route. Still a great switch at an awesome price point.
     
    #2061
    tommybackeast likes this.
  2. Emanuele

    Emanuele New Member

    Joined:
    Mar 1, 2019
    Messages:
    16
    Likes Received:
    1
    Hi all,

    I have (finally!) turned on the 2x ICX 6610 that I bought 3 month ago.
    They have the following licenses:

    [​IMG]

    Does it means that they have already all unlocked?

    EDIT:
    Also, I'm trying to configure the 2x 40G port for the stack, in order to use the other breackout ports to 10G connections, but I'm getting this error:
    [​IMG]
    I have already removed the stack trunk:

    enable
    conf t
    stack unit 1
    no stack-trunk 1/2/1 to 1/2/2
    no stack-trunk 1/2/6 to 1/2/7
    exit
    write mem

    EDIT2:

    nevermind, the @fohdeesha stacking guide is OP, it did the trick
     
    #2062
    Last edited: Jul 20, 2019
  3. infoMatt

    infoMatt New Member

    Joined:
    Apr 16, 2019
    Messages:
    20
    Likes Received:
    8
    Exactly... at best it would identify the medium as a twinax direct connect, it will show it's serial number, but it won't measure power as there's nothing that can do it inside the cable...

    Take a look at this photo I've found with a quick search:
    [​IMG]
    As you can see, there are just a couple of resistor/capacitors on the board, and a small EEPROM on the other side that contains the "metadata" of the cable (vendor, model, serial#, ...) but there's nothing in line with the signal lines.
     
    #2063
    tommybackeast likes this.
  4. fohdeesha

    fohdeesha Kaini Industries

    Joined:
    Nov 20, 2016
    Messages:
    1,228
    Likes Received:
    987

    you have the 10g port license, but not the advanced routing license
     
    #2064
    tommybackeast likes this.
  5. Raice

    Raice New Member

    Joined:
    Jul 12, 2017
    Messages:
    29
    Likes Received:
    3
    Does anybody know if these switches support PFC and RDMA? Is it possible to create RoCE lossless network for storage using it?
     
    #2065
  6. klui

    klui New Member

    Joined:
    Feb 3, 2019
    Messages:
    29
    Likes Received:
    14
    #2066
  7. Anubis

    Anubis New Member

    Joined:
    Jul 14, 2019
    Messages:
    2
    Likes Received:
    0
    So, thanks to this thread I now have two 6450-24p sitting on my desk, waiting to be configured :)
    I've gone through the guide and upgraded them both, and I'm at the step where I intend to stack the two switches.
    As I understand it, one cable should be enough, but does it matter if that cable is from port 1 -> port 3 or can it be from port 1 -> port 1?
     
    #2067
  8. fohdeesha

    fohdeesha Kaini Industries

    Joined:
    Nov 20, 2016
    Messages:
    1,228
    Likes Received:
    987
    the port to port doesn't matter, however if I remember right, the default config only has ports 1 and 2 set for stacking. (or it could be 1 and 3, I can't remember, it'll be one of the first lines in "show run"). If you want to use a port that's not specified there you'll have to change the stack ports
     
    #2068
    Anubis likes this.
  9. Anubis

    Anubis New Member

    Joined:
    Jul 14, 2019
    Messages:
    2
    Likes Received:
    0
    "show run" displayed ports 1 and 3 as stack ports, so I connected port 1 ->port 1, created the stack and ran "no stack-port 1/2/3" and "no stack-port 2/2/3". I'm guessing that should make port 3 available as a regular 10G port?
     
    #2069
  10. infoMatt

    infoMatt New Member

    Joined:
    Apr 16, 2019
    Messages:
    20
    Likes Received:
    8
    Just a quick message to say that, after reading all messages of this thread, and thanks to all the information that @fohdeesha and all other valuable user have provided, I've toss the coin and I am an owner of an ICX6450-24P! :):p

    That thing is really well made! I haven't got much time to tinker with, but I've managed to update it to the latest release (thanks to the guide linked on the first page) and apply a very basic config, just to keep it usable.
    To me though it seems a bit loud, louder than say a new-ish Cisco 2960x, even with the fans spun down; to have a reference, I can hear it from 2 rooms in the quiet of the night in a all-brick Euro house (with the doors opened, btw).
    I need to try and oil the fans, maybe they're simply dry (EDIT: done -- no appreciable change in noise profile); otherwise it doesn't matter, I'd buy two new fans, not the end of the world.

    After about two hours, a "show chassis" shows that it was at 46°C with the room at 28-29, so no heat concerns... with the hand, I could barely feel any heat exhausted from the fans, in comparison with the room air around it.

    Now I have to study the Brocade syntax, and start thinking about the network that I want to build... maybe I'll tear the config apart and restart from scratch a couple of times as I change my mind :D we'll see!

    For me, we've found the "unicorn switch", that's for sure.
     
    #2070
    Last edited: Jul 25, 2019
    eleitl likes this.
  11. AngryLatinChanting

    AngryLatinChanting New Member

    Joined:
    Jul 14, 2019
    Messages:
    5
    Likes Received:
    0
    Aww man, just discovered this fantastic thread yesterday... 6610-48P is definitely my next hot item to get. Long time lurker, but I tend to stay in my lanes unless I am buying something out of the norm. I kinda regret that now, because this thread saved me from getting the Microtik crs309-1g-8s+in and being forever salty about the L3 speeds.

    But looks like I showed up 1 month late to the party, since the licensing favours @fohdeesha was doing for the community had to (of course) be taken advantage of and thus ruined for the rest of us. Nor do I want to pay $100+ more to sellers on principle alone, but even without the price increase this switch is pushing close to $500 CAD all in. Considered the 7250 just to avoid dealing with this licensing crap, but they cost just as much or more, and I'd take the hit to much needed 10g connectivity by not having the qsfp+ breakout :(

    Guess I'm going to have to be extra patient and wait a few months for a good deal to pop up. I just hope the main wave of old stock isn't coming to a close. Prices have definitely gone up since the start of this thread!
     
    #2071
  12. infoMatt

    infoMatt New Member

    Joined:
    Apr 16, 2019
    Messages:
    20
    Likes Received:
    8
    Please read the thread 'till the end... you've just missed the latest news about the licensing for the community.

    Well... being patient and snap the deals when they come up it's always a good strategy, unless you specifically need the switch right now.

    I've waited just over two leeeeengthy months as I've missed the opportunity to make an offer to a ICX 6450-24P, the only one as far as I could tell sold by an EU ebayer at a right price (~175 EUR + ship.), as it went out of stock.. just some weeks ago the seller added some other to the auction, and so it is now on my desk :p
     
    #2072
    Last edited: Jul 28, 2019
  13. AngryLatinChanting

    AngryLatinChanting New Member

    Joined:
    Jul 14, 2019
    Messages:
    5
    Likes Received:
    0
    Oof, just went back and saw the post on page 100. I didn't miss it, just read it out of order apparently as I was probably bouncing around the thread last night before I decided to go in order today. Is this what you were referring to or something else? Apologies if I'm missing something obvious.

    One more question for either @fohdeesha or anyone else that knows, but do these licenses only enable the 10g SPF+ functionality or also the advanced routing capability?
     
    #2073
  14. kroy

    kroy New Member

    Joined:
    May 17, 2019
    Messages:
    23
    Likes Received:
    6
    @AngryLatinChanting there are two. One enables the PoD (ports) and one enables advanced routing.
     
    #2074
  15. Dawg10

    Dawg10 Associate

    Joined:
    Dec 24, 2016
    Messages:
    195
    Likes Received:
    90
    @AngryLatinChanting Where in Canada? AB here; local would be nice.

    If you're interested, I'm planning on selling a 6610-24P, two Mellanox CX353A 40Gbe cards and two Mellanox FDR 40Gbe cables, all working together under ESXi 6.7. There's also a 6450-24 I no longer use.

    I'm now set up with the servers on 10G DAC into a 7250-24, 10G fiber over to a 6450-48P supporting the 1G environment.

    FYI & fair warning: 40G works but requires a lot of tuning to get the speeds up (as in: max out the performance/ power draw of all attached devices. Last winter it kept the garage warm but it's too much in the summer).
     
    #2075
  16. AngryLatinChanting

    AngryLatinChanting New Member

    Joined:
    Jul 14, 2019
    Messages:
    5
    Likes Received:
    0
    Toronto but curious; PM'ed. :)
     
    #2076
  17. infoMatt

    infoMatt New Member

    Joined:
    Apr 16, 2019
    Messages:
    20
    Likes Received:
    8
    Yep, I was referring to that post ;) @fohdeesha has decided to continue giving licences to the forum members (thank you!), so you're not screwed up ;)

    There are two types of licences, one to enable the PoD (Port on Demand) at 10G, and another one to enable advanced routing.
     
    #2077
    Last edited: Jul 28, 2019
    fohdeesha likes this.
  18. eleitl

    eleitl New Member

    Joined:
    Jul 2, 2019
    Messages:
    4
    Likes Received:
    0
    Verily, it boots:

    # picocom /dev/ttyS0
    picocom v1.7

    port is : /dev/ttyS0
    flowcontrol : none
    baudrate is : 9600
    parity is : none
    databits are : 8
    escape is : C-a
    local echo is : no
    noinit is : no
    noreset is : no
    nolock is : no
    send_cmd is : sz -vv
    receive_cmd is : rz -vv
    imap is :
    omap is :
    emap is : crcrlf,delbs,

    Terminal ready
    $
    ICX Boot Code Version 7.3.02 (grz07302)
    Enter 'a' to stop at memory test
    Enter 'b' to stop at boot monitor
    BOOT INFO: load monitor from boot flash, cksum = 1fef
    BOOT INFO: verify flash files........
    BOOT INFO: load image from primary copy...

    platform type = 8
    PCIE-1 LTSSM status: 22
    PCIE Switch status: 0
    .....................
    .....
    Starting Main Task ...CPSS DxCh Version: cpss3.4 release
    Pre Parsing Config Data ...

    Parsing Config Data ...
    stacking is enable. optical monitoring for stacking ports 1/2/1, 1/2/6 is not available.


    SYSLOG: <14>0d00h00m09s:ESC-SW1-BldgC System: Cold start

    SYSLOG: <15>0d00h00m09s:ESC-SW1-BldgC DHCPC: starting dhcp client service on 65 port(s)
    Copyright (c) 1996-2011 Brocade Communications Systems, Inc.
    UNIT 1: compiled on Apr 25 2012 at 16:45:34 labeled as FCXR07300c
    (6811631 bytes) from Primary fcxr07300c.bin
    SW: Version 07.3.00cT7f3
    Boot-Monitor Image size = 370555, Version:07.3.02T7f5 (grz07302)
    HW: Stackable ICX6610-24
    ==============================
    ============================================
    UNIT 1: SL 1: ICX6610-24 24-port Management Module
    Serial #: xxxxxxxxxxx
    License: ICX6610_PREM_ROUTER_SOFT_PACKAGE (LID: xxxxxxxxxxx)
    P-ENGINE 0: type E02B, rev 01
    ==========================================================================
    UNIT 1: SL 2: ICX6610-QSFP 10-port 160G Module
    ==========================================================================
    UNIT 1: SL 3: ICX6610-8-port Dual Mode(SFP/SFP+) Module
    ==========================================================================
    800 MHz Power PC processor 8544E (version 0021/0023) 400 MHz bus
    65536 KB flash memory
    512 MB DRAM
    STACKID 1 system uptime is 9 seconds
    The system : started=cold start
    My stack unit ID = 1, bootup role = active

    ........................
    ESC-SW1-BldgC>
    Power supply 1 detected.
    Power supply 1 is up.

    SYSLOG: <14>0d00h01m33s:ESC-SW1-BldgC System: Stack unit 1 Power supply 1 is up

    Power supply 2 detected.
    Power supply 2 is up.

    SYSLOG: <14>0d00h01m33s:ESC-SW1-BldgC System: Stack unit 1 Power supply 2 is up

    SYSLOG: <12>0d00h01m40s:ESC-SW1-BldgC NTP: Server 10.20.144.11 failed to respond

    SYSLOG: <12>0d00h01m55s:ESC-SW1-BldgC NTP: Server 10.20.144.11 failed to respond

    SYSLOG: <12>0d00h02m10s:ESC-SW1-BldgC NTP: Server 10.20.144.11 failed to respond

    SYSLOG: <12>0d00h02m25s:ESC-SW1-BldgC NTP: Server 10.20.144.11 failed to respond

    SYSLOG: <12>0d00h02m40s:ESC-SW1-BldgC NTP: Server 10.20.144.11 failed to respond

    SYSLOG: <15>0d00h02m52s:ESC-SW1-BldgC DHCPC: Stopped DHCP Client service

    SYSLOG: <12>0d00h02m55s:ESC-SW1-BldgC NTP: Server 10.20.144.11 failed to respond

    SYSLOG: <12>0d00h03m10s:ESC-SW1-BldgC NTP: Server 10.20.144.11 failed to respond
     
    #2078
    Last edited: Jul 29, 2019
  19. explosivelobster

    explosivelobster New Member

    Joined:
    Jul 28, 2018
    Messages:
    24
    Likes Received:
    2
    Sorry if this has already been asked, but is it possible to reverse the physical mounting (and therefore airflow direction) of the fans in the ICX6610 to convert an -I to an -E variant and vice versa?
     
    #2079
  20. infoMatt

    infoMatt New Member

    Joined:
    Apr 16, 2019
    Messages:
    20
    Likes Received:
    8
    Preface: I don't own that model, so take this with a grain of salt...
    Reversing airflow is as difficult as reversing the fan inside it's tray... but logically the switch will continue to believe that the flow is still acting accordingly to the fan/PSU identifier, and it still refuses to boot if you mix a believed-E fan with an actual -I PSU, as it thinks that you're crossing flows, impairing cooling capabilities.

    If you do open the tray to reverse the fan, you'll have to do it also inside the PSU, and stick with just that components mounted on the chassis... the moment you'll add a stock fan/PSU tray at best you'll cross flow, at worse the switch will shut itself down because it thinks you're crossing air flows.
     
    #2080
Similar Threads: Brocade Series
Forum Title Date
Networking Guest VLAN on Brocade 6450 Jul 18, 2019
Networking Has anyone got a Mellanox ConnectX-5 100G card connected to a Brocade ICX 40G QSFP to QSFP? Apr 5, 2019
Networking [Solved] Brocade ICX6610-48P PoE Errors Apr 2, 2019
Networking Anyone done a Brocade 6610 fan mod? Apr 1, 2019
Networking Brocade 6650 Mar 9, 2019

Share This Page