Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

Discussion in 'Networking' started by fohdeesha, Jul 12, 2018.

  1. fohdeesha

    fohdeesha Kaini Industries

    Joined:
    Nov 20, 2016
    Messages:
    1,391
    Likes Received:
    1,118
    that is incredibly odd - I don't know how reflashing firmware would have fixed it (unless it was a different version). The OS image is checksummed entirely at boot so if there was anything corrupt with it, it would refuse to boot (or fall back to secondary OS slot)
    when you say remove stack units, do you mean the lines in the config? Remove those, reload switch, and let me know if the behavior returns - that's the only difference between the old setup that was failing and the new one that isn't, as far as I can tell
     
    #981
  2. N1h1l1ty

    N1h1l1ty New Member

    Joined:
    Oct 21, 2018
    Messages:
    25
    Likes Received:
    1
    I've removed the stack units in the config, booted it (with the old DAC cables that I initially thought were the root cause) and it has not thrown any error:



    Very odd indeed. I can't think of anything that I had done that would have thrown those errors in the first place... Now to test out if the throughput issue is still there!
     

    Attached Files:

    #982
  3. N1h1l1ty

    N1h1l1ty New Member

    Joined:
    Oct 21, 2018
    Messages:
    25
    Likes Received:
    1
    @fohdeesha - OK I've managed to recreate the error again - Initially I thought it might be related to aaa authentication settings but it seems that its VLAN related - when I have ports 1/2/7 and 1/2/2 assigned to any vlans, be they tagged or untagged, the error crops up again
     
    #983
  4. fohdeesha

    fohdeesha Kaini Industries

    Joined:
    Nov 20, 2016
    Messages:
    1,391
    Likes Received:
    1,118
  5. N1h1l1ty

    N1h1l1ty New Member

    Joined:
    Oct 21, 2018
    Messages:
    25
    Likes Received:
    1
    Anything else I should try at this point to try and narrow it down?

    With no VLAN tagging throughput looks normal and im no longer seeing the crazy jitter / packet out of order messages on iperf.

    EDIT: Yep final confirmation here - looks like if the ports are used as access ports, everything works fine. However, as soon as I introduced tagging, the interfaces shut down and upon reloading the switch it is once again throwing the error.

    I've tried this now on a range of firmware, from 8030sa to 8030r and 8020
     
    #985
    Last edited: Nov 5, 2018
  6. fohdeesha

    fohdeesha Kaini Industries

    Joined:
    Nov 20, 2016
    Messages:
    1,391
    Likes Received:
    1,118
    that honestly sounds like bad hardware/bad 40g stack card, never seen anything like that before. Can you post your config?

    And you're sure it's not an issue with the vlan handling on the end devices, right?
     
    #986
  7. N1h1l1ty

    N1h1l1ty New Member

    Joined:
    Oct 21, 2018
    Messages:
    25
    Likes Received:
    1
    Definitely not an issue of vlan handling on the end devices as I've checked them before with the exact same configuration on my Cisco 3750E with no throughput issues - furthermore I just rechecked the config with ports 1/2/2 through 1/2/5 and 1/2/7 through 1/2/10 untagged on a vlan and had no issues achieving full 10g throughput.

    Config:

    Current configuration:
    !
    ver 08.0.20T7f3
    !
    stack unit 1
    module 1 icx6610-24p-poe-port-management-module
    module 2 icx6610-qsfp-10-port-160g-module
    module 3 icx6610-8-port-10g-dual-mode-module
    !
    global-stp
    !
    !
    !
    vlan 1 name DEFAULT-VLAN by port
    spanning-tree
    !
    vlan 10 name Management by port
    tagged ethe 1/1/24
    untagged ethe 1/1/1 to 1/1/6
    spanning-tree
    !
    vlan 20 name Hosts by port
    tagged ethe 1/1/24
    untagged ethe 1/2/2 to 1/2/5 ethe 1/2/7 to 1/2/10
    spanning-tree
    !
    !
    !
    !
    !
    ip dhcp-client disable
    !
    !
    !
    !
    !
    !
    !
    !
    !
    !
    !
    !
    !
    !
    !
    !
    !
    !
    end

    VLANs:

    PORT-VLAN 1, Name DEFAULT-VLAN, Priority level0, Spanning tree On
    Untagged Ports: (U1/M1) 7 8 9 10 11 12 13 14 15 16 17 18
    Untagged Ports: (U1/M1) 19 20 21 22 23
    Untagged Ports: (U1/M2) 1 6
    Untagged Ports: (U1/M3) 1 2 3 4 5 6 7 8
    Tagged Ports: None
    Uplink Ports: None
    DualMode Ports: None
    Mac-Vlan Ports: None
    Monitoring: Disabled
    PORT-VLAN 10, Name Management, Priority level0, Spanning tree On
    Untagged Ports: (U1/M1) 1 2 3 4 5 6
    Tagged Ports: (U1/M1) 24
    Uplink Ports: None
    DualMode Ports: None
    Mac-Vlan Ports: None
    Monitoring: Disabled
    PORT-VLAN 20, Name Hosts, Priority level0, Spanning tree On
    Untagged Ports: (U1/M2) 2 3 4 5 7 8 9 10
    Tagged Ports: (U1/M1) 24
    Uplink Ports: None
    DualMode Ports: None
    Mac-Vlan Ports: None
    Monitoring: Disabled
     
    #987
  8. N1h1l1ty

    N1h1l1ty New Member

    Joined:
    Oct 21, 2018
    Messages:
    25
    Likes Received:
    1
    Let me configure tagged again and post that
     
    #988
  9. N1h1l1ty

    N1h1l1ty New Member

    Joined:
    Oct 21, 2018
    Messages:
    25
    Likes Received:
    1
    Yep, as expected, we're right back to this:

    Parsing Config Data ...
    ------------------------------------------------------------------
    M:9 L:0 - chow_qsfp_read, qsfp 2, error in seting up mux
    ------------------------------------------------------------------
    M:9 L:0 - link_40G_4x10G_get_media: qsfp 2, port 1/2/2 error in reading qsfp
    chow_40G_4x10G_get_media: error in reading qsfp 1/2/2
    ------------------------------------------------------------------
    M:9 L:0 - chow_qsfp_read, qsfp 2, error in seting up mux
    ------------------------------------------------------------------
    M:9 L:0 - link_40G_4x10G_get_media: qsfp 2, port 1/2/3 error in reading qsfp
    chow_40G_4x10G_get_media: error in reading qsfp 1/2/3
    ------------------------------------------------------------------
    M:9 L:0 - chow_qsfp_read, qsfp 2, error in seting up mux
    ------------------------------------------------------------------
    M:9 L:0 - link_40G_4x10G_get_media: qsfp 2, port 1/2/4 error in reading qsfp
    chow_40G_4x10G_get_media: error in reading qsfp 1/2/4
    ------------------------------------------------------------------
    M:9 L:0 - chow_qsfp_read, qsfp 2, error in seting up mux
    ------------------------------------------------------------------
    M:9 L:0 - link_40G_4x10G_get_media: qsfp 2, port 1/2/5 error in reading qsfp
    chow_40G_4x10G_get_media: error in reading qsfp 1/2/5

    (only have 1 QSFP breakout cable attached at the moment, but I'm betting it would do the same for the second cable)

    Worth noting I only have port 1/2/2 and 1/2/3 plugged in.

    ICX6610-24P Router>show run
    Current configuration:
    !
    ver 08.0.20T7f3
    !
    stack unit 1
    module 1 icx6610-24p-poe-port-management-module
    module 2 icx6610-qsfp-10-port-160g-module
    module 3 icx6610-8-port-10g-dual-mode-module
    !
    global-stp
    !
    !
    !
    vlan 1 name DEFAULT-VLAN by port
    router-interface ve 1
    spanning-tree
    !
    vlan 10 name Management by port
    tagged ethe 1/1/24
    untagged ethe 1/1/1 to 1/1/6
    spanning-tree
    !
    vlan 20 name Hosts by port
    tagged ethe 1/1/24 ethe 1/2/2 to 1/2/5 ethe 1/2/7 to 1/2/10
    spanning-tree
    !
    !
    !
    !
    !
    ip dhcp-client disable
    !
    !
    !
    !
    !
    !
    !
    !
    !
    !
    interface ve 1
    ip address 192.168.0.55 255.255.255.0
    !
    !
    !
    !
    !
    !
    !
    !
    !
    end

    Is current running config.
     
    #989
  10. fohdeesha

    fohdeesha Kaini Industries

    Joined:
    Nov 20, 2016
    Messages:
    1,391
    Likes Received:
    1,118
    that is really bizarre. just out of curiosity, can you remove any refs to stp and reload the switch, and check for same behavior?
     
    #990
  11. N1h1l1ty

    N1h1l1ty New Member

    Joined:
    Oct 21, 2018
    Messages:
    25
    Likes Received:
    1
    Had that same thought actually, will try to do that now - that being said, it gets even *more* bizarre. I swear I am not making this up but in the meantime I've continued to troubleshoot, and I'm now seeing correct throughput across both servers with tagged vlans. Now, it could have been configuration error on my end because I completely rebuilt the client configurations in Windows Server 16 from scratch again, but what I *did* notice is that the behavior where 1/2/2 through 1/2/5 and 1/2/7 through 1/2/7 will shut down if I make any tagged vlan changes to them, and also shut down and will not come back up after being added to dynamic LAGs persists, despite me no longer seeing jitter / loss and limited bandwidth across a single link - I paired 1/2/2 - 1/2/3 and 1/2/4 - 1/2/5 and when I deployed each LAG, both ports would shut down requiring me to reload the switch.
     
    #991
  12. fohdeesha

    fohdeesha Kaini Industries

    Joined:
    Nov 20, 2016
    Messages:
    1,391
    Likes Received:
    1,118
    I think barring an act of god, you have a defective switch. the "M:9 L:0 - chow_qsfp_read, qsfp 2, error" are low level ASIC messages (Chow is the codename for this particular model of Marvell Prestera silicon). You should never see error messages from the ASIC itself unless something has gone horribly wrong

    Aside from everyone else in the thread, I know I've personally used the rear breakout ports tagged in a ton of vlans at once while also in LAGs (even LAG's involving breakout ports from 2 different stack members). Did so to serve a LAN party of 150+ people without issue: Fohdeesha/reboot-lan
     
    #992
  13. N1h1l1ty

    N1h1l1ty New Member

    Joined:
    Oct 21, 2018
    Messages:
    25
    Likes Received:
    1
    Yeah unfortunately that may just be the case. I'm going to test the 10gbe SFP+ modules tomorrow or the day after since I don't have any additional optics handy to get them under proper load. That being said, if everything else works I'll probably hang on to it and get another one down the line - And for what its worth, I'm seeing the breakout ports function more or less as expected right now - 10 gigabit throughput with correct LAG configuration end to end. I'll try and do some more testing in the morning as I've only got them tagged with 1 VLAN right now - I had them tagged with 3 before and that's where I was seeing issues...

    EDIT: Meant front SFP+ modules
     
    #993
  14. mattaw

    mattaw Member

    Joined:
    Jul 30, 2018
    Messages:
    40
    Likes Received:
    13
    So, having purchased 3x ICX6610-48P as-is, expecting only one to work, I find that they all seem to work fine. I have used @fohdeesha 's excellent guide and updated the bootloader, firmware and preliminarily tested a few gigabit ports on the front.

    I wonder if there is a built in self-test routine I can run somewhere? I am assuming there are a bunch of checks it runs on boot? I would love to give the switches a full test. I have ordered an active poe splitter as that is the only thing I think would enable a proper test of the POE functions.

    Also one of the systems identifies itself as STACKID 2 even after factory set-defaults. Is this normal?
     
    #994
  15. svtkobra7

    svtkobra7 Active Member

    Joined:
    Jan 2, 2017
    Messages:
    316
    Likes Received:
    64
    Received my ICX6450-24 yesterday and thanks to @fohdeesha 's excellent documentation was able to get it up and running in short order.

    In an attempt to contribute to the community, I figured it was relevant to call out a certain ebay listing for anyone looking to purchase (note I'm in no way associated with the seller): (ICX6450-24) BROCADE ICX 6450-24 SWITCH 24 PORTS MANAGED DESKTOP RACK/WALL-MOUNT 783555168134 | eBay

    Before you gawk at the $170 price, note that I'm the "1 sold" (2 available) and the seller quickly accepted my "best offer" via ebay message for $150 (and then changed listing format to accept BOs). Personally I thought $150 was a tad high until I opened up my new toy, then I thought it was a very good one:
    1. Listing = "gently used," but it is in absolutely mint condition (no visible sign of it ever being touched);
    2. Original box (opened however);
    3. Accessory kit with all sealed bags (rack ears, screws for rack ears, console cable, rack screws, and rack nuts);
    4. The power cable still has the original twist tie on it (o/c it could have been used with another cable).
    I figure just the console cable may be worth $10 to someone (note it is serial not USB though) and good luck finding rack ears separately a decent price.

    So, as I said just a small attempt to give back for anyone who is currently looking for a ICX6450-24 and wants to now exactly what they are getting. Considering my offer @ $150 was accepted without hesitation, I'd probably go lower initially.
     
    #995
    tommybackeast and fohdeesha like this.
  16. fohdeesha

    fohdeesha Kaini Industries

    Joined:
    Nov 20, 2016
    Messages:
    1,391
    Likes Received:
    1,118
    on boot it does pretty thorough checking of RAM, the ASIC(s), the i2c bus, etc. There's a dedicated test routine hidden in the bootloader if I remember right but it involves a snake test (daisychaining every single port to the next one)

    If you ran factory set-default in the old bootloader it shipped with, it's possible it was old enough it didn't remove stacking config as part of it's reset routine. You can try running it again now that it's on the latest BL

    or from the OS enable level you can try running "stack unconfigure clean" (if its not stack master, you need to do stack unconfigure me)

    if that STILL doesn't work, hard mode:
    over serial (will not work over telnet/ssh)
    #at any OS level:
    #press ctrl+y, let go, then press m, then hit enter
    #at the new console, show files with:
    dir
    #delete all the config files
    del stacking.boot
    del startup-config
    del startup-config.backup
    del startup-config.old
    #reboot
    reset
     
    #996
    Last edited: Nov 17, 2018
    svtkobra7 likes this.
  17. svtkobra7

    svtkobra7 Active Member

    Joined:
    Jan 2, 2017
    Messages:
    316
    Likes Received:
    64
    And here I've been thinking all these years a "snake test" was the ultimate test of mobile phone mastery ...

    [​IMG]
     
    #997
    mattaw and fohdeesha like this.
  18. fohdeesha

    fohdeesha Kaini Industries

    Joined:
    Nov 20, 2016
    Messages:
    1,391
    Likes Received:
    1,118
    7150 and 7250 guides/firmware up: FCX / ICX6610 - Fohdeesha Docs
    (on left hand side)
    Also gave it a makeover (changed the color)

    as usual if something is confusing or doesn't work let me know

    PS: when I made this thread I did not think we'd make it to 5 pages, and now we're at page 50 three or four months later. Crazy! Hope everyone is enjoying their gear - I know I felt like a kid in a candyshop when I found these models and how well they work for the price, glad to share that with others
     
    #998
    Last edited: Nov 8, 2018
  19. kapone

    kapone Active Member

    Joined:
    May 23, 2015
    Messages:
    616
    Likes Received:
    247
    So..uh..a funny thing happened.

    A 6610-24P popped up on ebay, it had minor bending of two of the SFP ports, and had NO power supplies or fans, just the chassis. I looked at it and said, what a waste...but then...the 24P does have a lower power draw than the 48 port version, and I don't care about the ethernet ports as much, I use may be 10 of them (the IPMI network is on it's own switch). I care about the 10g/40g ports and that's the same between the two versions.

    So...I put in a offer of $40 (including shipping) just for the heck of it. My thinking was, I have enough power supplies and fan trays for these switches...

    And...the seller accepted.

    That's with shipping...I can't ship one of these for less than $20 anywhere, and I have a significant discount from Fedex.

    All I can say is...:p:p:p
     
    #999
  20. fohdeesha

    fohdeesha Kaini Industries

    Joined:
    Nov 20, 2016
    Messages:
    1,391
    Likes Received:
    1,118
    this is me hiding my jealousy :D
     
    #1000
    svtkobra7 and kapone like this.
Similar Threads: Brocade Series
Forum Title Date
Networking [SOLVED] Help Needed - Brocade ICX 6450 + Ruckus R720 Nov 25, 2019
Networking Brocade ICX 6610 - what does dhcp-client enable do? Nov 19, 2019
Networking MikroTik CSS326-24G-2S+RM vs Brocade ICX6450-24 Nov 1, 2019
Networking Brocade ICX6450-24P vs Aruba S2500-24P for Homelab Sep 22, 2019
Networking Brocade VDX 6720 - what do I need to know? Sep 16, 2019

Share This Page