Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

Discussion in 'Networking' started by fohdeesha, Jul 12, 2018.

  1. maes

    maes Member

    Joined:
    Nov 11, 2018
    Messages:
    41
    Likes Received:
    24
    I'll check tomorrow if the generic set fits and let you know asap if you're interested.
     
    #1081
  2. Dave Corder

    Dave Corder Member

    Joined:
    Dec 21, 2015
    Messages:
    70
    Likes Received:
    21
    Much appreciated. No rush, though...this switch is going to be the core switch for the next house anyway (planning to move early next year).
     
    #1082
  3. PnoT

    PnoT Active Member

    Joined:
    Mar 1, 2015
    Messages:
    577
    Likes Received:
    118
    A little bit of an update after my last switch came in and the POE board was bad. The seller shipped me another one...


    [​IMG]

    It seems that a switch curse has been put on me =/
     
    #1083
    Last edited: Nov 21, 2018
  4. fohdeesha

    fohdeesha Kaini Industries

    Joined:
    Nov 20, 2016
    Messages:
    1,286
    Likes Received:
    1,026
    maybe obvious, but just in case: have you plugged that PSU into the wall? if it has no power, it'll say exactly that
     
    #1084
    maes likes this.
  5. Dave Corder

    Dave Corder Member

    Joined:
    Dec 21, 2015
    Messages:
    70
    Likes Received:
    21
    It's been a long couple of days at work troubleshooting database performance gremlins, so I'll gladly take obvious suggestions at this point.

    And, as it turns out, the great fohdessha is correct...I didn't have the power cord seated firmly in that PSU. The switch is happy now:

    Code:
    SSH@sw-core-03>show chassis
    The stack unit 1 chassis info:
    
    Power supply 1 (AC - PoE) present, status ok
        Model Number:    23-0000142-02
        Serial Number:    CC4
        Firmware Ver:     A
    Power supply 1 Fan Air Flow Direction:  Front to Back
    Power supply 2 (AC - PoE) present, status ok
        Model Number:    23-0000142-02
        Serial Number:    64K
        Firmware Ver:     A
    Power supply 2 Fan Air Flow Direction:  Front to Back
    
     
    #1085
    fohdeesha, mattaw and maes like this.
  6. mattaw

    mattaw Member

    Joined:
    Jul 30, 2018
    Messages:
    40
    Likes Received:
    13
    I'm so sorry. Do you have the original with failed poe or did you ship it back? Swapping the Poe board takes about 20 mins if you still have it ...
     
    #1086
  7. fohdeesha

    fohdeesha Kaini Industries

    Joined:
    Nov 20, 2016
    Messages:
    1,286
    Likes Received:
    1,026
    @PnoT jeez you have not had good luck with these things! I believe via pm he did indeed get the PoE board swapped over :)
     
    #1087
  8. u238

    u238 Member

    Joined:
    Aug 11, 2018
    Messages:
    38
    Likes Received:
    9
    Can anyone give me any insight on multicast? I have an experimental proxmox cluster that relies on multicast, and it seems to work fine in the default vlan. On a new vlan I created, multicast didn't work at all until I enabled multicast active and multicast version 3. Am I doing this right or should I be choosing other settings?
     
    #1088
  9. mattaw

    mattaw Member

    Joined:
    Jul 30, 2018
    Messages:
    40
    Likes Received:
    13
    Racked my icx6610, disabled dual mode and moved to vlans only. Benchmarked by ZFS array from my Windows 10 desktop via 40gbps ethernet. This is cache testing only, btw, not spinning disk performance.

    None too shabby. Next phases are kernel buffer tuning, IRQ binding and windows 10 buffer tuning to get those numbers up. I hope my PCIe 2.0 slots don't damage the throughput though.

    upload_2018-11-23_19-0-37.png
     
    #1089
    lambda and fohdeesha like this.
  10. mattaw

    mattaw Member

    Joined:
    Jul 30, 2018
    Messages:
    40
    Likes Received:
    13
    Basic networking irq affinity set and tcp tuning. 1.75GBs write and 2.65GBs read! Biggest improvement was banning numa node 0 of interrupts, and then pinning the mlx4 irqs to node 0. Leave irqbalance running for the rest of the system. 2018-11-23 AT irqbalance sysctl MCSx4 VLANx1.PNG
     
    #1090
    Last edited: Nov 25, 2018
    lambda, maes and mathiastro like this.
  11. maes

    maes Member

    Joined:
    Nov 11, 2018
    Messages:
    41
    Likes Received:
    24
    Hm, after testing, the Noctua NF-A4x20 runs so slow at the 'low speed' setting that the 6450-24 throws a 'fan failed' error.
    The 2nd fan header RPM doesn't seem to be monitored at all but its driving voltage varies in sync with the 1st fan header.

    Even hardwired for 12V, it's still slow enough to throw fan failed errors. Any quiet fan recommendations that don't throw that error message, or can it be casually ignored?
     
    #1091
  12. lambda

    lambda New Member

    Joined:
    Oct 27, 2018
    Messages:
    18
    Likes Received:
    6
    I just ordered two of these for my ICX6450-24P and still waiting them to arrive:

    SUNON 4020 KDE1204PKVX MS.AF.GN FAN DC12V 1.4W 3Pin Super Silence | eBay

    https://www.alliedelec.com/m/d/efd14bdd843dd6f08d20002435c34e64.pdf
     
    #1092
    maes likes this.
  13. PnoT

    PnoT Active Member

    Joined:
    Mar 1, 2015
    Messages:
    577
    Likes Received:
    118
    Be lucky you don't have an A PSU on the 6610! :eek:

    I'm looking into converting the Air Flow of the fan around and it looks pretty simple except for the 4 rubber fastening tubes. Testing a few methods out right now on how to get those back in after flipping the fan around.
     
    #1093
  14. mattaw

    mattaw Member

    Joined:
    Jul 30, 2018
    Messages:
    40
    Likes Received:
    13
    @fohdeesha - tried lifting a heatsink, and got no-where. Even though they have pins, the heatsinks are glued down with probably the same glue as the smaller ones used throughout the design.

    I have not tried freezing them off, not sure whether to go that far with a working switch.
     
    #1094
  15. fohdeesha

    fohdeesha Kaini Industries

    Joined:
    Nov 20, 2016
    Messages:
    1,286
    Likes Received:
    1,026
    it can be casually ignored, it's just not seeing the minimum RPM it expects from the stock fans
     
    #1095
    maes likes this.
  16. fohdeesha

    fohdeesha Kaini Industries

    Joined:
    Nov 20, 2016
    Messages:
    1,286
    Likes Received:
    1,026
    no worries at all, it's not very important. thanks for trying though! I agree, not worth it on a working switch :p
     
    #1096
  17. maes

    maes Member

    Joined:
    Nov 11, 2018
    Messages:
    41
    Likes Received:
    24
    It would take me some time to develop it since I haven't coded in just about a decade (I'm beyond rusty), but it just came to mind that with a voltage regulator, a voltage divider, an attiny4/5/9/10 and maybe a transistor, it should be possible to make a 'tachometer adjuster' to fool a switch into thinking it has a fan running at the speed it expects.

    Even the tiny SOT23 6-pin package should be enough;

    fan power goes to the voltage regulator, dropped to 3.3V to power the attiny
    fan power also goes to the voltage divider, the output of which goes to an ADC pin. This lets the attiny know when the ICX switch expects the fan to be running at full speed (85oorpm)
    fan tach wire goes to an input on the attiny
    ouput on the attiny goes to the fan header tach pin

    (overall pins used: VCC, GND, ADC, Tach in, Tach out.)

    On boot, when the fan is running at full speed (attiny sees 'higher' voltage on ADC pin)
    short delay for fan to reach full speed
    use 16-bit timer to count 'ticks' between two tach pulses from the fan
    repeat 7x, average result (power-of-2 samples makes it easy, just bitshift instead of dividing)
    this gives the microcontroller the 'reference' full speed (or rotation period, to be more accurate)
    from there, calculate the difference between the 'reference' full speed and the 'expected' full speed. This multiplier is important.​
    After boot and initialization:
    Maybe once a second, repeat count of 'ticks' between tach pulses from the fan
    Use said count and the multiplier obtained during the initialization phase to generate fake 'higher rpm' tach pulse train
    output to router tach sensor​

    This would still allow the switch to throw an alert if the fan actually stops. It's not a tach offset but a multiplier.
     
    #1097
  18. lambda

    lambda New Member

    Joined:
    Oct 27, 2018
    Messages:
    18
    Likes Received:
    6
    The performance test done by Brocade indicates that the ICX6610-48P is capable of sustaining 100% load with all the 10g ports fully saturated without dropping any frame. Combining the test result from Brocade and @fohdecsha's actual observation, I think it is pretty save to assume that ICX6610-48p is capable of handling RoCE v1 at lease in a non-mission critical setting. The test result is on page 9 of the following Brocade document kindly provided by @fohdecsha.

    Architecture Brief
     
    #1098
    Last edited: Nov 25, 2018
  19. Scrubs

    Scrubs New Member

    Joined:
    Nov 20, 2018
    Messages:
    1
    Likes Received:
    0
    I've been following this thread for awhile now and I was finally able to sell off some of my old networking gear and submitted some offers on some Brocade gear today. So far I've scored an ICX 6450-48 and a 6430-48p (needed a POE switch, no need for giant uplinks) for $220 shipped. I've got an offer out on a 6610-24 for around $110. Planning on using the 6610 for my TOR and to play with SDN and the rest for port distribution (my homelab is a little excessive).

    I just wanted to say thanks to all the people that contributed to this thread. I have/had a mismash of Cisco, Dell, and Ubiquiti gear. I'm looking forward to being on a single network stack (well, plus my CRS317). It makes it especially easy when STH has a great community surrounding it. Thanks everyone!
     
    #1099
  20. Jannis Jacobsen

    Jannis Jacobsen Active Member

    Joined:
    Mar 19, 2016
    Messages:
    304
    Likes Received:
    49
    Just got an icx6610-24 for $150 on buy it now on ebay.
    only 1 psu and 1 fan module.
    Hopefully the noise will be bearable... :)

    Anyone got a 250w psu and fan module for sale? :)
    Is the correct psu model rps15-e, or are there other models too?

    -jannis
     
    #1100
    Last edited: Nov 25, 2018
Similar Threads: Brocade Series
Forum Title Date
Networking Brocade ICX6450-24P vs Aruba S2500-24P for Homelab Sep 22, 2019
Networking Brocade VDX 6720 - what do I need to know? Sep 16, 2019
Networking anyone have Brocade CER 2024C latest firmware Sep 14, 2019
Networking Guest VLAN on Brocade 6450 Jul 18, 2019
Networking Has anyone got a Mellanox ConnectX-5 100G card connected to a Brocade ICX 40G QSFP to QSFP? Apr 5, 2019

Share This Page