Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

Discussion in 'Networking' started by fohdeesha, Jul 12, 2018.

  1. Necrotyr

    Necrotyr Member

    Joined:
    Jun 25, 2017
    Messages:
    110
    Likes Received:
    19
    Sounds like a nice vacation ;)
     
    #2301
    fohdeesha likes this.
  2. BobTB

    BobTB New Member

    Joined:
    Jul 19, 2019
    Messages:
    16
    Likes Received:
    3
    I think my problem with 1G M-LX(SFP transciever not getting the link up at all, has something to do with the port setting. I can not set 1GB Auto or Slave or whatever, as I get the:

    "Only Speed-duplex 10g-full or 1000-full-master are allowed on this port"

    How can I set one of the ICX-6610 SPF+ ports to act as 1GB port with all the options for me to try.
     
    #2302
  3. infoMatt

    infoMatt Member

    Joined:
    Apr 16, 2019
    Messages:
    39
    Likes Received:
    18
    The option "1000-full-master" is the correct one, as I have tried it on my 6450 with a Cisco 1G SX, and the link went up...
     
    #2303
  4. DRAGONKZ

    DRAGONKZ Member

    Joined:
    Apr 9, 2018
    Messages:
    61
    Likes Received:
    3
    2.5 months after I bought a 6610-48P it’s finally arrived from the USA (shipped to AUS) but it’s been damaged in transit and arrived with a bent rack mount ear and broken heat sink retainer pin :(

    (I could hear a slight rattle inside and opened it up to find the pin broken on one of the larger 2 heat sinks)

    Everything else luckily seems ok so far!

    No idea how a single retainer pin has snapped, but if someone has a dead unit and would be willing to send me a heat sink retainer pin I’d be happy to pay a reasonable amount!

    Same goes if someone wants to sell a set of 2 post or 4 post rack mounts :)

    Cheers
     
    #2304
  5. BobTB

    BobTB New Member

    Joined:
    Jul 19, 2019
    Messages:
    16
    Likes Received:
    3
    It does not work for me. Is there any way to "debug" the port connectivity, does anyone know how?
     
    #2305
  6. e97

    e97 Active Member

    Joined:
    Jun 3, 2015
    Messages:
    216
    Likes Received:
    74
    You bet. My main concern is getting the thing quiet!



    Good info. I'm aware there are other "hot" components in this pizza box :) IR camera is a quick way to get a heat map of what needs cooling.

    Water cooling the components in a switch isnt revolutionary [WARNING: some parts may induce severe cringe ] :




    Theres also immersion cooling at HPC facilities.

    A modern 12V water cooling pump rarely needs to be run at full load, I run mine at 30-40% and it using <5W

    I use DDC clones:

    s-l300-pwm-ddc.jpg
    or D5 clones:

    s-l1000-sc600.jpg

    If the generic ebay blocks dont work, I can mill my own with the HAAS CNC at the makerspace a couple towns over

    haas-cnc.jpg

    All good things to be aware of! ..but I dont think they are relevant in this case:

    I'm comparing an identical system, in an identical environment with two different PSUs, after multiple measurements.

    The loads measured are between:

    idle: 90W-130W
    load: 325W-350W

    So there's no vignette error at the extremes of the measurement range. I'm comparing idle of psu1 vs idle psu2 and the same for load.

    My reference is Keysight InfiniiVision oscilloscope:

    aglient-scope.jpg

    which shows my Kill-a-Watt is within 1~3%.

    The opensource energy meter is also 0.5% accurate - all the energy meters use the same design, non invasive current transformer and opto-coupled voltage sensor. The more expensive ones also add temp and humidity as you mentioned. The open source one includes it as an optional addon.

    If you have a NIST facility or university nearby, they'll calibrate for you if you ask nicely :)
    If you want a certificate from NIST you'll have to pay $$

    Thanks for the info! I got a USB protocol analyzer and it'll be a nice goal to practice RE. If its simple, record and playback and bit bang with Pi GPIOs if its slow enough. If all else fails, I'll replace only the fans.

    It looks like its SMbus, a documented protocol and theres a python library for the Pi! Fingers crossed theres nothing encrypted or proprietary..


    The main goal here is silence, increasing efficiency/decreasing power consumption is an optional bonus.
     
    #2306
    Last edited: Oct 1, 2019
  7. fohdeesha

    fohdeesha Kaini Industries

    Joined:
    Nov 20, 2016
    Messages:
    1,345
    Likes Received:
    1,080
    Within 1-3% of what? Voltage? Current? Phase angle? The error you'll get is the product of all of these errors integrated - also note an oscilloscope with an 8-bit ADC front end is going to give a less accurate measurement for raw voltage (or current, if you're reading voltage from a CT) than a $20 multimeter off of amazon, which is also why the "DVM" function in a lot of scopes (at least ones without 10-bit or higher front ends) only reads out to 3 digits - it's just not what they are designed to do. Scopes are great, but they are the last thing on my entire workbench I would use as a reference for this (unless I was calibrating the phase angle error for a specific CT or something, which most of the higher end energy meters will let you program in, if you can manage to measure it)

    if the opensource energy meter you speak of claims 0.5% base accuracy, that would be a much better place to start. Note that that is also a function of (a slave to, really) the accuracy of the CT's you use with it, and most of the CTs I see people use with these off ebay are very poor accuracy, especially at the lower ends of the full scale rating. I've found Continental Control Systems CT's to have less than 1% error while still being affordable, however they don't show up for sale very often
     
    #2307
    tommybackeast and yukaia like this.
  8. e97

    e97 Active Member

    Joined:
    Jun 3, 2015
    Messages:
    216
    Likes Received:
    74
    Watts! I've been talking about power but its good to be explicit.

    If the probes are "dumb" probes that straight up use the ADC, then they're a huge rip off for how much they cost!

    The probes are calibrated for voltage (~1mV) and current (0.4% 50mA - 5A) and approved for certification testing.. i'll switch them with an amazon multimeter and see if anyone complains :p

    Re: the CTs, two sets of copper wire wound around a conductor. The eBay ones are built the same way with the nearly the same materials as the expensive ones, The difference is post mfg testing, calibration and certification. If the CT outputs mV (vs mA) add in the quality of the passives, like you said compounding errors. There are techniques to reduce error such as Kalman filtering using measurements over time or multiple "noisy" sensors.

    Re: the open source meter, it must be calibrated. With a good reference you can get to revenue grade (0.5%). If you're building one get the mA CTs and use 1% passives for the voltage divider with a 16-bit ADC. That'll get you 0.5% with a good reference.

    A certain energywall v1 (look up neurio) uses the same mfg as the eBay CTs. the difference being testing and calibration to meet revenue grade vs selling off the assembly line after basic testing.

    No matter how good you are, its currently very very very hard to manufacture the EXACT same CT that wouldn't require only calibrating a reference design due to composition of material, lengths, injection over molding, impedance, and more fun things to think about.

    At least until we get atomic level replicators :D

    edit: here's my power meter for my test systems:

    power-meter.png

    Since the systems are on 24/7, every 100W I shave off = $170/yr I dont pay for electricity

    I calibrated it with the incandescent light bulb method:
    get various watt light bulbs, 20W, 40W, 60W, 100W, 200W, etc..
    measure
    plot expected vs actual
    fit a curve
    adjust meter.

    Its <2% vs what the energy company reports
     
    #2308
    Last edited: Oct 1, 2019
    Stephan likes this.
  9. DRAGONKZ

    DRAGONKZ Member

    Joined:
    Apr 9, 2018
    Messages:
    61
    Likes Received:
    3
    I’ve managed to salvage the mangled rack ear!

    I still need a heat sink pin if someone has a dead unit they are willing to take one from? (I’m happy to pay!)

    thanks
     
    #2309
  10. fohdeesha

    fohdeesha Kaini Industries

    Joined:
    Nov 20, 2016
    Messages:
    1,345
    Likes Received:
    1,080
    I seriously doubt you'll ever find the retention pin, they're also "one use" so when they are removed they tend to be worn down so they don't retain as well. What I've done for issues like this in the past (motherboard chipset heatsinks) is just go to the hardware store and find an appropriate sized nut and bolt - if the PCB doesn't have a nice grounded ring around where the screw will contact (eg metal contacting the PCB looks like it might cause damage), just pick up a plastic washer as well. stick the screw upside down so the head is underneath the pcb, then if the screw has extra length (it will) it'll just be sticking up in the air on the heatsink side and not impacting the bottom of the case. Just don't over tighten it! barely a turn past finger tight
     
    #2310
    T_Minus likes this.
  11. Necrotyr

    Necrotyr Member

    Joined:
    Jun 25, 2017
    Messages:
    110
    Likes Received:
    19
    Alternatively, take two zip ties that fit through the hole and use the locks as stoppers.
     
    #2311
    T_Minus and fohdeesha like this.
  12. fohdeesha

    fohdeesha Kaini Industries

    Joined:
    Nov 20, 2016
    Messages:
    1,345
    Likes Received:
    1,080
    I think I might like that even better
     
    #2312
  13. fohdeesha

    fohdeesha Kaini Industries

    Joined:
    Nov 20, 2016
    Messages:
    1,345
    Likes Received:
    1,080
    oh, while you're in there and have the heatsink (kind of) off, can you grab a high res pic of the top of the ASIC? (or otherwise get me the part number). I'm pretty sure through PCI vendor IDs and such I've got it narrowed down to Marvell Cheetah 5, but not sure if that's the 98CX8297 or what
     
    #2313
    MaxCFM likes this.
  14. MaxCFM

    MaxCFM Harder, Better, Faster, Stronger!

    Joined:
    Apr 11, 2013
    Messages:
    30
    Likes Received:
    2
    Well late to the party but got me some 6610's for our upcoming LAN party's and a 2nd project. Working though getting them cleaned and updated. Thanks to everyone before me for all the info and fohdeesha for letting the cat out of the bag and sticking with it.
     
    #2314
    Last edited: Oct 2, 2019
    fohdeesha and cesmith9999 like this.
  15. fohdeesha

    fohdeesha Kaini Industries

    Joined:
    Nov 20, 2016
    Messages:
    1,345
    Likes Received:
    1,080
    if you need licenses just let me know
     
    #2315
    MaxCFM and tommybackeast like this.
  16. Zombielinux

    Zombielinux New Member

    Joined:
    Jun 14, 2019
    Messages:
    8
    Likes Received:
    2
    So I think my 6610 just completely died on me. Was running just fine one minute, then went down. Havn't been able to get it back up since. No data from serial console (but not sure it ever worked). Cracked open the case, had two leds near the silver heatsink on solid, then another led about an inch towards the right and back about 1/4" blinking slowly (like every 15s). Fans going full bore.
     
    #2316
  17. juey

    juey Member

    Joined:
    Oct 1, 2018
    Messages:
    37
    Likes Received:
    8
    #2317
    fohdeesha likes this.
  18. Zombielinux

    Zombielinux New Member

    Joined:
    Jun 14, 2019
    Messages:
    8
    Likes Received:
    2
    @fohdeesha Since mine is officially dead (new powersupply didn't fix it, still shows no sign of life other than raging fans)

    Have an image https://i.imgur.com/xPiiEMk.jpg
    Any other chips you need a high res shot of? I think imgur is downscaling them, and until the replacement switch is in on monday, I won't have a place to host the full res ones.
     
    #2318
    Last edited: Oct 4, 2019
  19. MaxCFM

    MaxCFM Harder, Better, Faster, Stronger!

    Joined:
    Apr 11, 2013
    Messages:
    30
    Likes Received:
    2
    Well 2 of 6 6610's I got have bad POE boards, doing the ebay return dance. @Zombielinux was yours a POE version? and what all did you have it running if you don't mind sharing?
     
    #2319
  20. Zombielinux

    Zombielinux New Member

    Joined:
    Jun 14, 2019
    Messages:
    8
    Likes Received:
    2
    @MaxCFM yea, it was the 48port PoE model.

    I had it running 3 unifi WAPS, 2 1440p cameras and 1 4k camera on PoE

    Beyond that I had some desktops, a few raspis, TVs, a NAS on the 40G, and servers each with 2x bonded 10G.

    Bummer about the bad boards though.
     
    #2320
    MaxCFM likes this.
Similar Threads: Brocade Series
Forum Title Date
Networking MikroTik CSS326-24G-2S+RM vs Brocade ICX6450-24 Nov 1, 2019
Networking Brocade ICX6450-24P vs Aruba S2500-24P for Homelab Sep 22, 2019
Networking Brocade VDX 6720 - what do I need to know? Sep 16, 2019
Networking anyone have Brocade CER 2024C latest firmware Sep 14, 2019
Networking Guest VLAN on Brocade 6450 Jul 18, 2019

Share This Page