Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

Discussion in 'Networking' started by fohdeesha, Jul 12, 2018.

  1. MaxCFM

    MaxCFM Harder, Better, Faster, Stronger!

    Joined:
    Apr 11, 2013
    Messages:
    30
    Likes Received:
    2
    Well it was a risk with the seller I used as a few here have had bad luck with them, but I needed more than one switch at sub $200 per. Still need 4 more but those can wait a bit.

    That doesn't sound like much in terms of power draw on the POE side wonder if it was a marginal VRM and just gave up and took something else with it?

    Good luck with the new one @Zombielinux
     
    #2321
  2. Zombielinux

    Zombielinux New Member

    Joined:
    Jun 14, 2019
    Messages:
    8
    Likes Received:
    2
    Good lord thats a lot of ports. It definitely wasn't the poe side that died. It just stopped switching.

    Its not a voltage issue I can tell. Might be some little chip defect.

    Good luck with yours too.
     
    #2322
    MaxCFM likes this.
  3. MaxCFM

    MaxCFM Harder, Better, Faster, Stronger!

    Joined:
    Apr 11, 2013
    Messages:
    30
    Likes Received:
    2
    Yeah trying something diff with the lan setup. I have a few ports sitting around at the moment...
    Hmm can't seem to upload an image...
     
    #2323
  4. fohdeesha

    fohdeesha Kaini Industries

    Joined:
    Nov 20, 2016
    Messages:
    1,346
    Likes Received:
    1,084
    Thank you! that should be all I need, the management PPC chip and everything else was quite easy to identify
     
    #2324
    MaxCFM likes this.
  5. |DSI|

    |DSI| New Member

    Joined:
    Jan 7, 2016
    Messages:
    5
    Likes Received:
    1
    Thanks to this thread (=fohdeesha), I've managed to buy 3 ICX6450-24P units, which will be replacing my current mixed setup of access switches.
    Being located in EU it wasn't that cheap (180 EUR/ea inc. postage and taxes), but judging from early experience it was well worth.

    I'm now already looking for a pair of 6610-24P to replace my ageing core switches and upgrade to 40G, but it seems that they are hard to find in EU.
     
    #2325
  6. DRAGONKZ

    DRAGONKZ Member

    Joined:
    Apr 9, 2018
    Messages:
    61
    Likes Received:
    3
    Want to sell some of the heatsink pins? :)

    My heatsink with a broken one either appears to be using a tim pad, or some form of epoxy as it won’t budge (without a lot of force)

    Did your heatsinks just pull off easily once the pins were out?
     
    #2326
  7. zanechua

    zanechua Member

    Joined:
    May 6, 2016
    Messages:
    54
    Likes Received:
    7
    #2327
  8. Zombielinux

    Zombielinux New Member

    Joined:
    Jun 14, 2019
    Messages:
    8
    Likes Received:
    2
    Sure! Send me a PM and let me know where they're going. They kinda did, the trick with heatsinks like those is to twist rather than pry, you're less likely to ruin the chips underneath.
     
    #2328
  9. Todd F

    Todd F New Member

    Joined:
    Sep 25, 2019
    Messages:
    3
    Likes Received:
    3
    First off, I'd like to thank all those traveling this road before me. Lots of useful information. Additionally, shout out to Fohdeesha for the amazing resource and help.

    I wanted to add some information about getting the console to work on an ICX7250. I did not own a 'Cisco console cable', e.g. USB -> RJ45 serial. I had a couple RS232 to USB adapters I'd used with microcontrollers, etc. Those did not work. All I got was screens of gibberish text no matter what terminal settings I used.

    What worked was:
    - 'Cisco console cable' - just a generic one linked many pages ago.
    - Mini-USB to 5-pin motherboard adapter. Basically just a cable that already is pinned out to a connector.
    - RJ45 breakout board
    - Header pins to solder to RJ45 breakout.

    Simply repined the USB connector to match the RJ45 console cable et voila, working console!

    I have a 24 port model, so I should probably just cut a hole in the front panel and install my breakout there.
     
    #2329
    fohdeesha likes this.
  10. Smbaker

    Smbaker New Member

    Joined:
    Oct 9, 2019
    Messages:
    12
    Likes Received:
    5
    Can an icx6450 do 5gbe with a copper module in the sfp+ slot?

    Thanks,
    Scott
     
    #2330
  11. DRAGONKZ

    DRAGONKZ Member

    Joined:
    Apr 9, 2018
    Messages:
    61
    Likes Received:
    3
    I’ve tracked down some “new old stock” 4 post mounting kits for the 6610 in Australia, if anyone in AUS is after some.

    They aren’t necessarily cheap though!
     
    #2331
  12. fohdeesha

    fohdeesha Kaini Industries

    Joined:
    Nov 20, 2016
    Messages:
    1,346
    Likes Received:
    1,084
    no
     
    #2332
  13. juey

    juey Member

    Joined:
    Oct 1, 2018
    Messages:
    37
    Likes Received:
    8
    1.) Is it possible to stack the 6610 using one of the 40G ports and one breakout port ?
    tried that before, but i'm only getting the stack-trunk work with one 40G and one of the 10G breakout ports (e.g. 1/2/1 and 1/2/2, not 1/2/1 & 1/2/2 to 1/2/5).

    2.) What does the led "MS" mean ? On one of my stack members, its green, and led on the other one is orange.
    nvmd. i read the hardware installation guide, its the stacking role, green -> master, orange -> standby

    3.) My 2nd psu from my newly arrived switch is revving up and down, i tried to switch the psus but nothing changed. the 2nd fan tray (the left one, looking from behind) is also not working (no green light), is my board damaged ?
    new6610inside3.jpg
    Its the left side, left psu revving up and down, left fan not working, anything damaged ? Can it be debugged ?

    Here is the show chassis cmd (i deleted the mac addresses and X'd out the serials), stack unit 1 is my perfectly working 24p-e, stack unit 2 the new 48-e, i've plugged the second fan tray in the 24p-e, so currently the 48-e is only running one fan tray:
    Code:
    SSH@sw-core#sh chassis
    The stack unit 1 chassis info:
    
    Power supply 1 (AC - PoE) present, status ok
            Model Number:   23-0000142-02
            Serial Number:  XXX
            Firmware Ver:    A
    Power supply 1 Fan Air Flow Direction:  Front to Back
    Power supply 2 not present
    
    Fan 1 ok, speed (auto): [[1]]<->2
    Fan 2 ok, speed (auto): [[1]]<->2
    
    Fan controlled temperature: 46.0 deg-C
    
    Fan speed switching temperature thresholds:
                    Speed 1: NM<----->78       deg-C
                    Speed 2:       73<-----> 87 deg-C (shutdown)
    
    Fan 1 Air Flow Direction:  Front to Back
    Fan 2 Air Flow Direction:  Front to Back
    MAC 1 Temperature Readings:
            Current temperature : 41.5 deg-C
    CPU Temperature Readings:
            Current temperature : 40.0 deg-C
    sensor A Temperature Readings:
            Current temperature : 26.5 deg-C
    sensor B Temperature Readings:
            Current temperature : 39.0 deg-C
    sensor C Temperature Readings:
            Current temperature : 20.0 deg-C
    sensor D Temperature Readings:
            Current temperature : 18.0 deg-C
    stacking card Temperature Readings:
            Current temperature : 46.0 deg-C
            Warning level.......: 84.0 deg-C
            Shutdown level......: 87.0 deg-C
    Boot Prom MAC :
    Management MAC:
    
    The stack unit 2 chassis info:
    
    Power supply 1 (AC - Regular) present, status ok
            Model Number:   23-0000144-01
            Serial Number:  XXX
            Firmware Ver:    B
    Power supply 1 Fan Air Flow Direction:  Front to Back
    Power supply 2 (AC - Regular) present, status ok
            Model Number:   23-0000144-01
            Serial Number:  XXX
            Firmware Ver:    B
    Power supply 2 Fan Air Flow Direction:  Front to Back
    
    Fan 1 ok, speed (auto): [[1]]<->2
    Fan 2 not present
    
    Fan controlled temperature: 58.0 deg-C
    
    Fan speed switching temperature thresholds:
                    Speed 1: NM<----->80       deg-C
                    Speed 2:       75<-----> 85 deg-C (shutdown)
    
    Fan 1 Air Flow Direction:Front to Back
    MAC 1 Temperature Readings:
            Current temperature : 44.5 deg-C
    MAC 2 Temperature Readings:
            Current temperature : 58.0 deg-C
    CPU Temperature Readings:
            Current temperature : 57.5 deg-C
    sensor A Temperature Readings:
            Current temperature : 50.5 deg-C
    sensor B Temperature Readings:
            Current temperature : 48.5 deg-C
    sensor C Temperature Readings:
            Current temperature : 37.5 deg-C
    stacking card Temperature Readings:
            Current temperature : 55.0 deg-C
            Warning level.......: 81.0 deg-C
            Shutdown level......: 85.0 deg-C
    
     
    #2333
    Last edited: Oct 10, 2019
  14. fohdeesha

    fohdeesha Kaini Industries

    Joined:
    Nov 20, 2016
    Messages:
    1,346
    Likes Received:
    1,084
    yes you can stack with the 40gbe port and breakout port, that's exactly how I have mine running at home. You only need to specify the first breakout port (1/2/2) and it automatically uses the entire 4 ports of that breakout. Your config should look like this:

    Code:
    Current configuration:
    !
    ver 08.0.30tT7f3
    !
    stack unit 1
      module 1 icx6610-48p-poe-port-management-module
      module 2 icx6610-qsfp-10-port-160g-module
      module 3 icx6610-8-port-10g-dual-mode-module
      priority 128
      stack-trunk 1/2/1 to 1/2/2
      stack-port 1/2/1
    stack unit 2
      module 1 icx6610-48p-poe-port-management-module
      module 2 icx6610-qsfp-10-port-160g-module
      module 3 icx6610-8-port-10g-dual-mode-module
      stack-trunk 2/2/1 to 2/2/2
      stack-port 2/2/1
    
    To verify it's using all 5 links (1 40gbe and 4 breakouts) run "show stack con", you should see 5 links total (the diagram won't show it, because they probably did not have this kind of custom stack connection in mind when making those diagrams):

    Code:
    Probing the topology. Please wait ...
    telnet@ICX1>
        standby      active
         +---+        +---+
         | 2 |2/1==2/1| 1 |
         +---+        +---+
    
    trunk probe results: 1 links
    Link 1: u1 -- u2, num=5
      1: 1/2/1 (T0) <---> 2/2/1 (T0)
      2: 1/2/2 (T0) <---> 2/2/2 (T0)
      3: 1/2/3 (T0) <---> 2/2/3 (T0)
      4: 1/2/4 (T0) <---> 2/2/4 (T0)
      5: 1/2/5 (T0) <---> 2/2/5 (T0)
    CPU to CPU packets are fine between 2 units.
    
    no clue about your fan/hardware situation
     
    #2334
    juey likes this.
  15. Smbaker

    Smbaker New Member

    Joined:
    Oct 9, 2019
    Messages:
    12
    Likes Received:
    5
    Hmm, that's kind of what I was afraid the answer was going to be. I had visions of potentially using some QNA-UC5G1T USB3 5gbpe adapters, but sounds like that's not going to work. Wish that QNAP adapter had an SFP+ instead of copper and was physical at 10gbe even if it could still only sustain about 4gbe throughput on the usb side.

    Going to have to decide whether I really want a 6450 or if I should go with something like a Mikrotik, which is a less capable switch and costs more money (new), but I think it can do the 1/2.5/5/10 copper SFPs.
     
    #2335
  16. infoMatt

    infoMatt Member

    Joined:
    Apr 16, 2019
    Messages:
    39
    Likes Received:
    18
    What NAS do you have? Most of the times, you can get pretty much the maximum speed of the 2-3 bay models by LAGGing the two ethernet interfaces... The only scenario where 2Gbps of bandwidth could potentially not max out the drives is a 3+ bay with a RAID0 (so you'll have about 150-180MB/s per drive of burst write speed, assuming you are using "pro" drives @7200 rpm or more), but I'd bet that those processors cannot sustain a 2+Gbps stream to them via Eth on PCI... let alone via an USB adapter with the resource overhead of the USB bus...
     
    #2336
    fohdeesha likes this.
  17. Smbaker

    Smbaker New Member

    Joined:
    Oct 9, 2019
    Messages:
    12
    Likes Received:
    5
    Synology DS1019+. If I'd have spent a hundred bucks more and bought the DXS1618+ instead, it would have come with a PCIE slot and I could have thrown a 10G NIC in it, but I didn't... Live and learn. Running RAID6 on five WD RED NAS drives. Also has a SSD read cache in it. Using this to serve NFS, iSCSI to ESXi on a couple NUCs as well as SMB to my desktop. Performance to the NUCs is fine, but I would like to optimize performance to my desktop.

    There is a guy who ported the driver for that QNAP 5Gbe USB3 adapter to the Synology, and tried it on a DS918+ (one less bay than mine) and was reporting ~ 434MB/s sustained transfers to and from it. I don't recall if he said what his CPU utilization was, but it sounded like at ~ 434MB/s it was limited by the USB3 link speed, not CPU. These were iperf numbers, so not sure what happens when you throw in physical disk performance. Then you have to figure how much of the working set fits into the NAS's RAM, and how much can be served by the SSD read cache. (Could do SSD write cache as well on this NAS, but too many people report write cache errors crashing their volume for me to be comfortable with that)

    Once the 6450 arrives, I am going to try to utilize the NAS's two 1G interfaces, either bonding or SMB multichannel.
     
    #2337
  18. Freebsd1976

    Freebsd1976 Member

    Joined:
    Feb 23, 2018
    Messages:
    97
    Likes Received:
    7
    some sellers list icx7750-26q as is (no fan try no psu) and return is not acceptable , price 160usd, want give it a try.
    but my question how to got fan tray and psu , what the price will be?
    edit:
    find another seller list new icx7750 also new but no fan trays no psu , only 110 usd
     
    #2338
    Last edited: Oct 11, 2019
  19. ewer0012

    ewer0012 Member

    Joined:
    Feb 10, 2019
    Messages:
    56
    Likes Received:
    19
    Howdy,

    I just picked up a 6610 off of eBay from a random "Make Offer" of $100. Had 2 PSUs and 1 fan. It seems to be booting up fine into the boot monitor to update stuff, but when I follow the instructions Fohdeesha has to update it through TFTP and then go to boot it up, I get:

    Code:
    Monitor>ip address 10.0.1.73
      IP address = 10.0.1.73
      IP subnet mask = 255.255.255.0
    Monitor>copy tftp flash 10.0.1.19 grz10100.bin boot
    Loading image from Tftp server.................................................................Done
    Programming boot flash, please wait..
    Erasing....
    Writing
    Done
    Monitor>copy tftp flash 10.0.1.19 FCXR08030t.bin primary
    .......................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................Done
    .Monitor>factory set-default
    This command will remove configuration and keys detail.
    Do you want to continue? (Y/N) Y
    Done.
    Monitor>reset
    $
    ICX Boot Code Version 10.1.00 (grz10100)
    Enter 'a' to stop at memory test
    Enter 'b' to stop at boot monitor
    BOOT INFO: load monitor from boot flash, cksum = 71f1
    BOOT INFO: verify flash files........
    BOOT INFO: load image from primary copy...
    
    platform type = 12
    PCIE-1 LTSSM status: 22
    PCIE Switch status: 0
    ..............................
    Firmware integrity checksum passed
    .......
    Starting Main Task .Applying factory defaults..
    INFO: startup config data is not available, try to read from backup
    INFO: startup config data in the backup area is not available
    ..CPSS DxCh Version: cpss3.4p1 release
    And then it stalls there. :(

    Any help is appreciated. Hopefully, I didn't receive a dud.
     
    #2339
  20. Wolfstar

    Wolfstar Member

    Joined:
    Nov 28, 2015
    Messages:
    56
    Likes Received:
    24
    Was it booting before you did the firmware flashing? Have you tried booting from the secondary flash?
     
    #2340
Similar Threads: Brocade Series
Forum Title Date
Networking MikroTik CSS326-24G-2S+RM vs Brocade ICX6450-24 Nov 1, 2019
Networking Brocade ICX6450-24P vs Aruba S2500-24P for Homelab Sep 22, 2019
Networking Brocade VDX 6720 - what do I need to know? Sep 16, 2019
Networking anyone have Brocade CER 2024C latest firmware Sep 14, 2019
Networking Guest VLAN on Brocade 6450 Jul 18, 2019

Share This Page