Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

Discussion in 'Networking' started by fohdeesha, Jul 12, 2018.

  1. maes

    maes Member

    Joined:
    Nov 11, 2018
    Messages:
    43
    Likes Received:
    24
    I have said ears and decent CAD knowledge, I'll see what I can draft over the next few days.

    Since it's a waterjet, I figure countersinks won't be possible? I'll skip them if it's preferred.

    Any preferred file format? Inches or mm?
     
    #1961
    tommybackeast, klui and cesmith9999 like this.
  2. DavidB

    DavidB Member

    Joined:
    Aug 31, 2018
    Messages:
    38
    Likes Received:
    11
    Doing great work there, I have a ICX6450 but it came without rack ears, so looking forward to some ears that don't cost as much as the whole unit on ebay
     
    #1962
  3. mb300sd

    mb300sd Active Member

    Joined:
    Aug 1, 2016
    Messages:
    190
    Likes Received:
    65
    No countersinks unfortunately. mm preferred. No need to fully model it, just a flat part with the offset where the bend needs to go is perfect (so I don't need to reflatten it for cutting).
     
    #1963
  4. klui

    klui Member

    Joined:
    Feb 3, 2019
    Messages:
    35
    Likes Received:
    16
    Not sure about other switches, the 6610s 2-post ears don't use countersunk screws.
     
    #1964
  5. ewer0012

    ewer0012 Member

    Joined:
    Feb 10, 2019
    Messages:
    56
    Likes Received:
    19
    Well shit... Thank you for helping me with my 6450, I really appreciate it. It's sad that it's always the few that ruin a good thing for the many.
     
    #1965
  6. maes

    maes Member

    Joined:
    Nov 11, 2018
    Messages:
    43
    Likes Received:
    24
    ICX6450 rack ears modeled and PM sent to @mb300sd
     
    #1966
    tommybackeast, ewer0012 and fohdeesha like this.
  7. FRPII

    FRPII Member

    Joined:
    Dec 2, 2015
    Messages:
    54
    Likes Received:
    17
    I have a 3 axis CNC at my disposal if that helps...
     
    #1967
    tommybackeast and ewer0012 like this.
  8. maes

    maes Member

    Joined:
    Nov 11, 2018
    Messages:
    43
    Likes Received:
    24
    A 3-axis CNC would be hilariously overkill; the ears are basically a thick piece of bent sheet metal with holes. A waterjet cutter and bending brake should be plenty enough.

    Still, if you're interested, here's the 'bent' version of the part; rack ear
     
    #1968
    tommybackeast and MikeWebb like this.
  9. FRPII

    FRPII Member

    Joined:
    Dec 2, 2015
    Messages:
    54
    Likes Received:
    17
    Overkill? How do you like my brackets out of bar stock? Imgur
     
    #1969
    maes, MikeWebb, Hrast and 2 others like this.
  10. mb300sd

    mb300sd Active Member

    Joined:
    Aug 1, 2016
    Messages:
    190
    Likes Received:
    65
    Ordered some material, should have some ready to go next week for those interested.
     
    #1970
    maes likes this.
  11. TheCodeLife

    TheCodeLife New Member

    Joined:
    Mar 29, 2019
    Messages:
    3
    Likes Received:
    3
    Just want to thank you fohdeesha for all you have done here! I was able to get enough SFP+ ports this way to run fiber throughout my house. Such a tremendous help and I'm sad to see this occur. I have no intention of selling my switch at this point, but is there any chance you could put up enough of the guide for removing all of the configuration for the licenses you helped us with so we can set our switches back to default prior to selling? The switch I purchased was already partially licensed and I would want to set it back to that state before selling. I'm sure a number of us on this site had partially licensed switches and would revert back to that state when selling.
     
    #1971
    maes and Hrast like this.
  12. Dave Corder

    Dave Corder Member

    Joined:
    Dec 21, 2015
    Messages:
    71
    Likes Received:
    23
    I'm a little confused on something.

    On the 6610-48p, can all four QSFP ports do a single 40gb link, or just 1/2/1 and 1/2/6 (with the other two being 4x10gb breakouts only)?
     
    #1972
  13. fohdeesha

    fohdeesha Kaini Industries

    Joined:
    Nov 20, 2016
    Messages:
    1,341
    Likes Received:
    1,079
     
    #1973
    tommybackeast likes this.
  14. Dave Corder

    Dave Corder Member

    Joined:
    Dec 21, 2015
    Messages:
    71
    Likes Received:
    23
    Thanks, I was having a brain fart.
     
    #1974
    fohdeesha likes this.
  15. Dave Corder

    Dave Corder Member

    Joined:
    Dec 21, 2015
    Messages:
    71
    Likes Received:
    23
    Ok, I'm having some trouble with one of my 40gb ports on my 6610-48P.

    I picked up a pair of these cables to use with my FreeNAS system and my Proxmox hypervisor: HP 498385-B23 3M 4X DDR/QDR QSFP IB CU CABLE *New Bulk* - 503815-002

    I am using Mellanox ConnectX-3 cards in each system (technically "HP 656089-001 Infiband 4 X 10/40GB DP adapter 544QSFP"), both flashed to the official Mellanox firmware and set to Ethernet mode).

    Port 1/3/7 is currently connected to port 1 on the Proxmox system with a QSFP->SFP+ adapter and a SFP+ 10gb SR optic. Port 1/2/6 is connected to port 1 on the FreeNAS system. I'm attempting to connect the Proxmox system to 1/2/1.

    Port 1/2/6 works fine. I duplicated (I think) my port config from 1/3/7 to port 1/2/1. Both the switch and Promox detect that the cable is plugged in, but I don't get a link.

    I swapped cables (knowing the FreeNAS cable works fine), no change.

    I swapped ports (FreeNAS to 1/2/1 and Proxmox to 1/2/6) and still 1/2/1 did not work.

    Right now I've got 1/2/1 connected to port 2 on the Proxmox system while I troubleshoot. It doesn't work there, either.

    Anyone have any ideas? Did I miss a step in my configuration? Are there some diagnostics I can run on the switch to test the port? Or do I just have a dead port?

    Code:
    SSH@sw-core-01>show config
    !
    Startup-config data location is flash memory
    !
    Startup configuration:
    !
    ver 08.0.30saT7f3
    !
    stack unit 1
      module 1 icx6610-48p-poe-port-management-module
      module 2 icx6610-qsfp-10-port-160g-module
      module 3 icx6610-8-port-10g-dual-mode-module
      stack-trunk 1/2/1 to 1/2/2
      stack-trunk 1/2/6 to 1/2/7
    stack disable
    !
    !
    !
    !
    vlan 1 name DEFAULT-VLAN by port
     router-interface ve 1
    !
    vlan 2 by port
    !
    vlan 10 name LAN by port
     tagged ethe 1/1/45 to 1/1/48 ethe 1/2/1 ethe 1/2/6 ethe 1/3/1 to 1/3/8
     untagged ethe 1/1/13 to 1/1/24
     router-interface ve 10
    !
    vlan 20 by port
     tagged ethe 1/1/45 to 1/1/48 ethe 1/2/1 ethe 1/3/1 to 1/3/6 ethe 1/3/8
    !
    vlan 30 by port
     tagged ethe 1/1/45 to 1/1/48 ethe 1/2/1 ethe 1/3/1 to 1/3/6 ethe 1/3/8
    !
    vlan 40 by port
     tagged ethe 1/1/45 to 1/1/48 ethe 1/2/1 ethe 1/3/1 to 1/3/6 ethe 1/3/8
    !
    vlan 99 name storage by port
     tagged ethe 1/2/1 ethe 1/2/6 ethe 1/3/1 ethe 1/3/7 to 1/3/8
    !
    !
    !
    !
    !
    aaa authentication web-server default local
    aaa authentication enable default local
    aaa authentication login default local
    hostname sw-core-01
    ip dhcp-client disable
    !
    username root password .....
    !
    !
    !
    !
    !
    !
    !
    !
    !
    interface ethernet 1/1/45
     dual-mode
    !
    interface ethernet 1/1/46
     dual-mode
    !
    interface ethernet 1/1/47
     port-name uap-pro-01
     dual-mode
    !
    interface ethernet 1/1/48
     port-name uap-pro-02
     dual-mode
     inline power
    !
    interface ethernet 1/2/1
     port-name proxmox-01-40gb
     dual-mode
    !
    interface ethernet 1/2/6
     port-name lcars-40gb
     dual-mode  10
    !
    interface ethernet 1/3/1
     port-name pfsense
     speed-duplex 10G-full
    !
    interface ethernet 1/3/2
     speed-duplex 10G-full
    !
    interface ethernet 1/3/3
     speed-duplex 10G-full
    !
    interface ethernet 1/3/4
     speed-duplex 10G-full
    !
    interface ethernet 1/3/5
     speed-duplex 10G-full
    !
    interface ethernet 1/3/6
     speed-duplex 10G-full
    !
    interface ethernet 1/3/7
     port-name lcars-10gb
     dual-mode  10
     speed-duplex 10G-full
    !
    interface ethernet 1/3/8
     port-name proxmox-01-10gb
     dual-mode
     speed-duplex 10G-full
    !
    interface ve 1
     ip address 10.0.1.2 255.255.255.0
    !
    interface ve 10
     ip address 10.0.10.2 255.255.255.0
    !
    !
    !
    !
    !
    !
    !
    !
    !
    end

    Code:
    root@proxmox-01:~# ethtool enp5s0d1
    Settings for enp5s0d1:
        Supported ports: [ FIBRE ]
        Supported link modes:   1000baseKX/Full
                                10000baseKX4/Full
                                10000baseKR/Full
                                40000baseCR4/Full
                                40000baseSR4/Full
                                56000baseCR4/Full
                                56000baseSR4/Full
        Supported pause frame use: Symmetric Receive-only
        Supports auto-negotiation: Yes
        Advertised link modes:  1000baseKX/Full
                                10000baseKX4/Full
                                10000baseKR/Full
                                40000baseCR4/Full
                                40000baseSR4/Full
        Advertised pause frame use: Symmetric
        Advertised auto-negotiation: Yes
        Speed: 40000Mb/s
        Duplex: Full
        Port: Direct Attach Copper
        PHYAD: 0
        Transceiver: internal
        Auto-negotiation: off
        Supports Wake-on: d
        Wake-on: d
        Current message level: 0x00000014 (20)
                       link ifdown
        Link detected: no
    

    Code:
    SSH@sw-core-01>show interface eth 1/2/1
    40GigabitEthernet1/2/1 is up, line protocol is up
      Port up for 1 minute(s) 14 second(s)
      Hardware is 40GigabitEthernet, address is 748e.f834.bc80 (bia 748e.f834.bcb1)
      Interface type is 40Gig Fiber
      Configured speed 40Gbit, actual 40Gbit, configured duplex fdx, actual fdx
      Configured mdi mode AUTO, actual none
      Member of 6 L2 VLANs, port is dual mode in Vlan 1, port state is FORWARDING
      BPDU guard is Disabled, ROOT protect is Disabled, Designated protect is Disabled
      Link Error Dampening is Disabled
      STP configured to ON, priority is level0, mac-learning is enabled
      Openflow is Disabled, Openflow Hybrid mode is Disabled,  Flow Control is enabled
      Mirror disabled, Monitor disabled
      Mac-notification is disabled
      Not member of any active trunks
      Not member of any configured trunks
      Port name is proxmox-01-40gb
      MTU 1500 bytes, encapsulation ethernet
      300 second input rate: 0 bits/sec, 0 packets/sec, 0.00% utilization
      300 second output rate: 0 bits/sec, 0 packets/sec, 0.00% utilization
      10568 packets input, 1170346 bytes, 0 no buffer
      Received 7580 broadcasts, 2751 multicasts, 237 unicasts
      0 input errors, 0 CRC, 0 frame, 0 ignored
      0 runts, 0 giants
      0 packets output, 0 bytes, 0 underruns
      Transmitted 0 broadcasts, 0 multicasts, 0 unicasts
      0 output errors, 0 collisions
      Relay Agent Information option: Disabled
    
    Egress queues:
    Queue counters    Queued packets    Dropped Packets
        0                1138           204491026
        1                   0                   0
        2                   0                   0
        3                   0                   0
        4                   0                   0
        5                  13                   0
        6                   0                   0
        7                   0                   0
    
     
    #1975
  16. fohdeesha

    fohdeesha Kaini Industries

    Joined:
    Nov 20, 2016
    Messages:
    1,341
    Likes Received:
    1,079
    try this, sometimes the rear 40gb ports can be weird the first time they link, esp if you haven't removed the stack trunks


    enable
    conf t
    stack unit 1
    no stack-trunk 1/2/1 to 1/2/2
    no stack-trunk 1/2/6 to 1/2/7
    exit
    write mem
    exit
    reload
     
    #1976
  17. DavidB

    DavidB Member

    Joined:
    Aug 31, 2018
    Messages:
    38
    Likes Received:
    11
    definitely interested. Right now it's resting on top of an old supermicro case and I may or may not have dropped it twice as a result of pulling out a DAC cable too enthusiastic
     
    #1977
  18. Dave Corder

    Dave Corder Member

    Joined:
    Dec 21, 2015
    Messages:
    71
    Likes Received:
    23
    No luck with these commands. Also tried removing and reinserting the cable a few times on the switch side. I also inspected the inside of the port with a flashlight and it looked OK to me, but it's at the bottom of my rack and I was crouched on the floor, so I can't be too certain. I'm going to try blowing it out with some compressed air (as soon as I find some), just to make sure there's no dust bunnies in there or something.

    It looks like QSFP SR optics are pretty cheap, too, so I'm going to order a pair and give those a try and see if I have any better luck than with the DAC.

    I'm several months past the purchase date past on eBay with this switch, so I doubt I have any recourse with the seller, but if I can't get the port to work I'll at least reach out to them and see...
     
    #1978
  19. Dave Corder

    Dave Corder Member

    Joined:
    Dec 21, 2015
    Messages:
    71
    Likes Received:
    23
    #1979
  20. Dave Corder

    Dave Corder Member

    Joined:
    Dec 21, 2015
    Messages:
    71
    Likes Received:
    23
    On the off chance that this is an issue on the Mellanox ConnectX-3 card side (both were flashed with the latest firmware and set to ETH mode), I flashed them following fohdeesha's guide and this is my current status of the card in the two systems:

    Code:
    root@proxmox-01:~/mft-4.12.0-105-x86_64-deb# mlxconfig -d /dev/mst/mt4099_pci_cr0 query
    
    Device #1:
    ----------
    
    Device type:    ConnectX3
    Device:         /dev/mst/mt4099_pci_cr0
    
    Configurations:                              Next Boot
    fl         SRIOV_EN                            False(0)
             NUM_OF_VFS                          8
             LINK_TYPE_P1                        ETH(2)
             LINK_TYPE_P2                        ETH(2)
             LOG_BAR_SIZE                        3
             BOOT_PKEY_P1                        0
             BOOT_PKEY_P2                        0
             BOOT_OPTION_ROM_EN_P1               False(0)
             BOOT_VLAN_EN_P1                     False(0)
             BOOT_RETRY_CNT_P1                   0
             LEGACY_BOOT_PROTOCOL_P1             None(0)
             BOOT_VLAN_P1                        1
             BOOT_OPTION_ROM_EN_P2               False(0)
             BOOT_VLAN_EN_P2                     False(0)
             BOOT_RETRY_CNT_P2                   0
             LEGACY_BOOT_PROTOCOL_P2             None(0)
             BOOT_VLAN_P2                        1
             IP_VER_P1                           IPv4(0)
             IP_VER_P2                           IPv4(0)
             CQ_TIMESTAMP                        True(1)
    root@proxmox-01:~/mft-4.12.0-105-x86_64-deb# flint -d /dev/mst/mt4099_pci_cr0 query
    Image type:            FS2
    FW Version:            2.42.5000
    FW Release Date:       5.9.2017
    Product Version:       02.42.50.00
    Device ID:             4099
    Description:           Node             Port1            Port2            Sys image
    GUIDs:                 24be05ffffa03cc0 24be05ffffa03cc1 24be05ffffa03cc2 24be05ffffa03cc3
    MACs:                                       24be05a03cc1     24be05a03cc2
    VSD:
    PSID:                  MT_1090120019
    

    Code:
    ➜  mft-4.12.0-105 sudo mlxconfig -d pci0:3:0:0 query
    
    Device #1:
    ----------
    
    Device type:    ConnectX3
    Device:         pci0:3:0:0
    
    Configurations:                              Next Boot
             SRIOV_EN                            False(0)
             NUM_OF_VFS                          8
             LINK_TYPE_P1                        ETH(2)
             LINK_TYPE_P2                        ETH(2)
             LOG_BAR_SIZE                        3
             BOOT_PKEY_P1                        0
             BOOT_PKEY_P2                        0
             BOOT_OPTION_ROM_EN_P1               False(0)
             BOOT_VLAN_EN_P1                     False(0)
             BOOT_RETRY_CNT_P1                   0
             LEGACY_BOOT_PROTOCOL_P1             None(0)
             BOOT_VLAN_P1                        1
             BOOT_OPTION_ROM_EN_P2               False(0)
             BOOT_VLAN_EN_P2                     False(0)
             BOOT_RETRY_CNT_P2                   0
             LEGACY_BOOT_PROTOCOL_P2             None(0)
             BOOT_VLAN_P2                        1
             IP_VER_P1                           IPv4(0)
             IP_VER_P2                           IPv4(0)
             CQ_TIMESTAMP                        True(1)
    ➜  mft-4.12.0-105 sudo flint -d pci0:3:0:0 query
    Image type:            FS2
    FW Version:            2.42.5000
    FW Release Date:       5.9.2017
    Product Version:       02.42.50.00
    Device ID:             4099
    Description:           Node             Port1            Port2            Sys image
    GUIDs:                 0002c9030040f720 0002c9030040f721 0002c9030040f722 0002c9030040f723
    MACs:                                       0002c940f720     0002c940f721
    VSD:
    PSID:                  MT_1090120019
    
     
    #1980
    Last edited: Jun 22, 2019
Similar Threads: Brocade Series
Forum Title Date
Networking MikroTik CSS326-24G-2S+RM vs Brocade ICX6450-24 Nov 1, 2019
Networking Brocade ICX6450-24P vs Aruba S2500-24P for Homelab Sep 22, 2019
Networking Brocade VDX 6720 - what do I need to know? Sep 16, 2019
Networking anyone have Brocade CER 2024C latest firmware Sep 14, 2019
Networking Guest VLAN on Brocade 6450 Jul 18, 2019

Share This Page