Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

Discussion in 'Networking' started by fohdeesha, Jul 12, 2018.

  1. Churchill

    Churchill Admiral

    Joined:
    Jan 6, 2016
    Messages:
    756
    Likes Received:
    169

    I am doing an rsync between two systems to replicate the data.

    These are both 2620 Xeon V2 so relatively new. I'd love to know what kind of kernel tweaking i can do.

    The 40GB ports are all in their own VLAN seperated out from any network and traffic. fully isolated. I have 10GB ports I can hookup as well.
     
    #2241
  2. infoMatt

    infoMatt New Member

    Joined:
    Apr 16, 2019
    Messages:
    27
    Likes Received:
    11
    You could test if the bottleneck is in the switch making a direct connection between the two systems... if both NICs are at 40Gbe...
     
    #2242
  3. Churchill

    Churchill Admiral

    Joined:
    Jan 6, 2016
    Messages:
    756
    Likes Received:
    169

    I did this with iperf with 10 threads and wasn't able to hit 40Gb speed
     
    #2243
  4. Whitecitadel

    Whitecitadel New Member

    Joined:
    Feb 13, 2018
    Messages:
    1
    Likes Received:
    0
    Looks like the guide needs an update as 6650 is no longer anywhere close to $300!

    Is there anything between 6610 and 6650 with more QSFP ports than 4?

    Churchill - try more iPerf threads to saturate the link?
     
    #2244
    Last edited: Sep 12, 2019
  5. pir2

    pir2 New Member

    Joined:
    Aug 3, 2011
    Messages:
    4
    Likes Received:
    0
    Might be helpful to others...

    The Mikrotik S+RJ10 does not work with the ICX-7150-C12P. It shows up as an SFP Module intsead of SFP+. However, FS.com ones seem to work.

    SW version: 08080d

    Code:
    show media ethernet 1/3/1
    Port   1/3/1: Type  : 1GE M-SX(SFP)
                 Vendor: MikroTik           Version: 2.11
                 Part# : S+RJ10             Serial#:
    icx(config)#show media ethernet 1/3/2
    Port   1/3/2: Type  : 10GE Passive Twinax  1m (SFP+)
                 Vendor: OEM                Version: D
                 Part# : SFP-10G-T          Serial#: 
     
    #2245
  6. juey

    juey Member

    Joined:
    Oct 1, 2018
    Messages:
    31
    Likes Received:
    7
    Is it somehow possible to grab transceiver stats like rx power, temps,... unlike by the CLI? The SNMP MIB does not contain any entries for that, it seems that this feature is absent in all fastiron powered devices.
     
    #2246
  7. yukaia

    yukaia New Member

    Joined:
    Aug 8, 2017
    Messages:
    4
    Likes Received:
    0
    You have to run Brocade optics to get those metrics.
     
    #2247
  8. BoGs

    BoGs New Member

    Joined:
    Feb 18, 2019
    Messages:
    2
    Likes Received:
    0
    I was looking on Ebay for some of these switches and prices sure have gone up ;) I guess I will keep my nortel and arista :)
     
    #2248
  9. fohdeesha

    fohdeesha Kaini Industries

    Joined:
    Nov 20, 2016
    Messages:
    1,272
    Likes Received:
    1,017
    the MIB absolutely contains entries for that bro

    [​IMG]
     
    #2249
  10. juey

    juey Member

    Joined:
    Oct 1, 2018
    Messages:
    31
    Likes Received:
    7
    I know, i have Brocade optics and i do get transceiver stats via CLI.

    Ok, i did not express myself well, the MIB contains the objects, but they are not available on the ICX6610/6450:

    Code:
    juey@influxdb:~# snmpwalk -v2c -c public <IP> 1.3.6.1.4.1.1991.1.1.3.3.6.1.1
    FOUNDRY-SN-SWITCH-GROUP-MIB::snIfOpticalMonitoringTemperature = No Such Object available on this agent at this OID
    Found on page 259 from ipmib-aug2016-reference.pdf
     
    #2250
  11. fohdeesha

    fohdeesha Kaini Industries

    Joined:
    Nov 20, 2016
    Messages:
    1,272
    Likes Received:
    1,017
    it definitely works, at least on the icx7xxx series stuff I have access to test. there's a new table that takes into account lanes as well in case of multilane optics like 40g etc which is the table you need to use, the old one was removed:

    [​IMG]

    everything in that table works to pull info off of normal 10gbe optics:

    [​IMG]

    I guess it's possible it was never added to the 6xxx series, not sure
     
    #2251
  12. blinkenlights

    blinkenlights New Member

    Joined:
    May 24, 2019
    Messages:
    4
    Likes Received:
    1
    First, many thanks to @fohdeesha for all of his online and offline help. I've been playing around with Brocade's ICX7xxx switches for the past month and arrived at some conclusions that may be helpful for other members.

    My servers and network gear are located behind a door in an unfinished, unconditioned 250 sq. ft. utility area of the basement.. not ideal, but not unlike most homelabs. I purchased an ICX7250-48 (non-PoE) in decent condition a few weeks ago, but was disappointed by the thermal characteristics and stock fan noise. About a week later, I purchased a late-model ICX7450-48 (non-PoE) data center denizen and migrated my home network.

    Here are my findings for people deciding between the ICX7250 and ICX7450:
    1) The main ASIC in the ICX7250 (BCM56344) runs hot, especially in a room without an AC unit/direct vent. On cold boot at 22 deg-C (72 deg-F) ambient, the temperature sits around 55 deg-C and quickly rises to the low 70's (158+ deg-F). My ICX7450 (BCM56548) sits around 48 deg-C fully loaded and the fan noise is nearly the same to my old ears.

    The ICX7250 temperatures were not unexpected by Brocade, though: warning at 100.0 deg-C and shuts down at 105.0 deg-C. I chickened out around 80 deg-C and fully disassembled the switch, cleaned it up, and applied a quality ceramic thermal paste to the main ASIC. I then swapped the original Foxconn fans for the infamous "blew itself off the table" "sliced open my finger" Delta ultra-high speed fans (FFB0412SHN-F00). Yes, these fans move a ton of air at full voltage.. but the switch typically runs them around 30% of their maximum speed. The switch is much quieter - downright tolerable at 56dB a few feet away - and idles at 68 deg-C under the same environmental conditions. Still planning to post it for sale, as I don't need two core switches for home.

    Long story short, if you can tolerate fan noise and/or have a way to isolate the noise, the ICX7450 (and ICX6610, I imagine) are better performers for the money.

    Regarding the ICX7450:
    2) Find a switch with Rev. C power supplies - buy them separately if the price makes sense. I don't have hands-on experience with the older power supplies, but trust when people say they are much louder.
    3) You might want to pick up spare parts if you are in this for the long haul. I plan to buy one cold spare Rev C. power supply and one fan.

    General advice, aka everything of this world has a price:
    4) Unless you plan to use 24 or 48 ports with PoE, go for the non-PoE version. Heat generated by the additional circuit board(s) and larger power supplies will not be helping you. If I need to run PoE for a security camera or two, I can afford to buy an injector.
    5) Nothing unnecessary in the slots generating more heat or blocking airflow. If you don't need multiple QSFP ports for stacking, don't install them! In my case, I have a single ICX7400-4x10GC module up front but nothing in the rear slots. When I remove the module, the temperature drops from 48 deg-C to around 40 deg-C.

    Hope all of this helps someone out there thinking of buying one of these impressive switches.
     
    #2252
    fohdeesha likes this.
  13. Necrotyr

    Necrotyr Member

    Joined:
    Jun 25, 2017
    Messages:
    105
    Likes Received:
    13
    Does any one know why Brocade though 5mm holes in their rack ears were a good idea?

    I have to drill out the holes to make my M6 bolts and cagenuts fit...
     
    #2253
  14. itronin

    itronin Active Member

    Joined:
    Nov 24, 2018
    Messages:
    113
    Likes Received:
    42
    I'm guessing to sell more 4 post kits? (jk)
    ICX6610-RMK-4P
    I'm not sure why all relay and square hole racks and can't be one thread type. though I guess with square hole you can probably get cage nuts with 10-32 or 12-24 and avoid drilling out the ears...

    Thanks for the info on this - I had not tried m6 on ears and *was* thinking about consolidating all my gear into 1 four poster. I guess if I do I'll need a mix of cage nuts - probably paint 'em red or something so I know by looking at them what they are.
     
    #2254
  15. Necrotyr

    Necrotyr Member

    Joined:
    Jun 25, 2017
    Messages:
    105
    Likes Received:
    13
    M5 bolts and cagenuts would do the trick, but I don't have any of those, so I went and found my round file and voilá!

    Bear in mind, that I'm speaking about the ICX6450 ;)
     
    #2255
  16. magi

    magi New Member

    Joined:
    Aug 22, 2019
    Messages:
    2
    Likes Received:
    0
    Anyone doing a PSU fan switch? I disassembled one of my PSUs and the fan is FFB0412UHN. Wonder if there are recommended alternatives to it?
     
    #2256
  17. Terry Wallace

    Terry Wallace PsyOps SysOp

    Joined:
    Aug 13, 2018
    Messages:
    74
    Likes Received:
    30
    If no one else is going to say it. I will.

    They weren’t expecting someone with nuts your size ?


    :)
     
    #2257
  18. fohdeesha

    fohdeesha Kaini Industries

    Joined:
    Nov 20, 2016
    Messages:
    1,272
    Likes Received:
    1,017
    what switch, 6610? it monitors the rpm of the psu fans, if you replace it needs to be roughly the same rpm ~23k full speed, or it will refuse to boot. only other way is to build or buy something to trick the RPM signal the PSUs see from the fan. read: pain in the ass
     
    #2258
    magi likes this.
  19. magi

    magi New Member

    Joined:
    Aug 22, 2019
    Messages:
    2
    Likes Received:
    0
    Can try put a small PPM generator in so can trick the monitor. Anyway it has two PSUs so can work on one and still run the router.

    Will post if succeeded ;o
     
    #2259
  20. juey

    juey Member

    Joined:
    Oct 1, 2018
    Messages:
    31
    Likes Received:
    7
    What side effects do i have to face, if the uplink connection of stacking members will exceed the 100m limit ? Losing stack members, lost packets ? I want to stack one member using a 140m multimode cable, otherwise i just can stack the cores and use regular LAGs.
     
    #2260
Similar Threads: Brocade Series
Forum Title Date
Networking Brocade ICX6450-24P vs Aruba S2500-24P for Homelab Sep 22, 2019
Networking Brocade VDX 6720 - what do I need to know? Sep 16, 2019
Networking anyone have Brocade CER 2024C latest firmware Sep 14, 2019
Networking Guest VLAN on Brocade 6450 Jul 18, 2019
Networking Has anyone got a Mellanox ConnectX-5 100G card connected to a Brocade ICX 40G QSFP to QSFP? Apr 5, 2019

Share This Page