Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

Discussion in 'Networking' started by fohdeesha, Jul 12, 2018.

  1. Wolfstar

    Wolfstar Member

    Joined:
    Nov 28, 2015
    Messages:
    71
    Likes Received:
    41
    That's actually quite probable. Mine has no ring to it and there isn't even much of the 40mm fan whine really. Same switch - ICX6450-24P.
     
    #2281
  2. e97

    e97 Active Member

    Joined:
    Jun 3, 2015
    Messages:
    216
    Likes Received:
    77
    This thread is already at 11 - lets kick it up a notch to 12.

    ill be looking to water cooling the ICX 6610-48P to make it quiet and (hopefully) use less power.

    will also look into replacing the PSU with a high efficiency platinum one.

    @fohdeesha has kindly shared that it requires 12V and 52V (PoE).

    we also need to trick the fan rpm sense pins since no more stock fans.. hopefully its the same ATX standard and pump and rad fan output can be used.
     
    #2282
  3. juey

    juey Member

    Joined:
    Oct 1, 2018
    Messages:
    42
    Likes Received:
    11
    The original Delta oem psu is platinum certified.
     
    #2283
    fohdeesha likes this.
  4. blinkenlights

    blinkenlights New Member

    Joined:
    May 24, 2019
    Messages:
    12
    Likes Received:
    1
    I'm impressed, as some of these power supplies are nearly as old as the Platinum certification itself. For those who are skeptical, it's clearly listed on the label of a different OEM version: Delta 1000W PowerEdge AWF-2DC-1000W Switching Power Supply EDC14020023 A258 | eBay

    I could track down the 1000W PSU (AWF-2DC-1000W) - anyone know if the 250W PSU (DPS-250AB-96) is also Platinum certified?
     
    #2284
  5. levi danzer

    levi danzer New Member

    Joined:
    Sep 29, 2019
    Messages:
    3
    Likes Received:
    1
    First off, big thanks to fohdeesha for this amazing thread. It made me pull the trigger on a ICX6610-24i which is massive overkill for my home. I don't even have a lab setup yet!

    I've got to wire up a serial cable and find my usb-serial so I can get my switch setup. The ebay posting showed it as having the fcx08010.bin firmware so i'll be following the guide shortly to get everything updated and reset.

    The 40-gbe is working with a DAC. but the SFP+ ports don't seem to have 10gbe unlocked. Kinda annoying as that was the main use for this switch. But as a switch its been plug and play... grab the old 1gig copper, and internet on the 40gbe port. Kinda funny to me.
    This is one of my first steps into enterprise gear. It sounds much louder than the data sheet. Wondering if I have a fan bearing going out, was unbearable with only one PSU plugged in. Will dig into the system stats soon. Maybe i should swap out for a 6450?

    I'm debating forgoing the 40gbe and picking up a 7xxx series so i can be sure to have 10gbe.... or is there a better route to go? (the 40gbe is massive overkill for my uses, heck i'll be surprised if i can use half the 10gbe)

    Thanks!

    -Levi
     
    #2285
  6. BobTB

    BobTB New Member

    Joined:
    Jul 19, 2019
    Messages:
    16
    Likes Received:
    3
    In 6610-48, I inserted a single mode, duplex transceiver, show media shows it as
    Port 1/3/1: Type : 1G M-LX(SFP)

    When I connect the cable, nothing happens. I tried swapping the A an B, waiting a minute etc... but link never goes up.I have factory reset the switch, and touched nothing else but added the IP to the switch. The other end is connected to fiber provider. The transceiver is the one I got from them, to plug in some other switch, so it is the correct one.

    Is there something else I have to do to enable the port? Nothing is written in the log of the switch when I plug the cable in the transceiver or unplug it.
     
    #2286
    Last edited: Sep 29, 2019
  7. e97

    e97 Active Member

    Joined:
    Jun 3, 2015
    Messages:
    216
    Likes Received:
    77
    Semiconductor technology advances quickly but the power designs should be nearly the same. Now, highest efficiency rating is 80+ Titantium.

    Measured with a Kill-A-Watt a 3+ yr old 94% platinum certified enterprise PSU and a recent Seasonic 80+ Bronze PSU -- on the same system. Both at 120V, the 80+ bronze showed lower idle and load Watts by about 30W.

    Not sure if this holds at 240V
     
    #2287
  8. T_Minus

    T_Minus Moderator

    Joined:
    Feb 15, 2015
    Messages:
    6,837
    Likes Received:
    1,491
    What was the wattage of the Enterprise Plat and the Seasonic 80+ Bronze?
    Kind of apples\oranges unless their wattage is the same.
     
    #2288
  9. e97

    e97 Active Member

    Joined:
    Jun 3, 2015
    Messages:
    216
    Likes Received:
    77
    900W enterprise 94% efficient
    550W consumer ATX 80+ Bronze

    It used to be the case that load significantly impacted efficiency. Advances in modern switching semiconductors have drastically reduced those losses:

    Here's Efficiency vs Load for Seasonic Platinum 1000W Power Supply
    [​IMG]

    2% peak to trough = ~6W for the system under test

    80 PLUS Platinum efficiency; What does it mean, and what's the benefit to me? - Corsair has a nice breakdown.

    If you switch to even newer technologies like GaN or SiC you can achieve ~99% across a larger range due to faster switching speed and lower losses which also reduces heat and required cooling and size.

    Following links show how these advances are being applied to IT power system design at scale:

    * Google Contest Builds More Efficient Inverters

    * https://www.opencompute.org/files/External-2018-OCP-Summit-Google-48V-Update-Flatbed-and-STC-20180321.pdf

    At scale, a few watts x 1000s of systems drastically affects power, cooling and space.
     
    #2289
    Last edited: Sep 29, 2019
  10. T_Minus

    T_Minus Moderator

    Joined:
    Feb 15, 2015
    Messages:
    6,837
    Likes Received:
    1,491
    Great info!

    Your 3+ Year old Platinum likely isn't as efficient as these new modern ones you speak of though ??

    Going from 920 or 1200 I forget which to 550w I noticed 10-15w on Kill-a-watt between PSU, that's pretty significant (10-12%) on the system I was testing. *25 systems in a rack and that's like 1 or 2 additions systems you can run :D
     
    #2290
  11. e97

    e97 Active Member

    Joined:
    Jun 3, 2015
    Messages:
    216
    Likes Received:
    77
    I was pretty surprised myself which is why I did the Kill-A-Watt test. I have another energy measurement device for all the systems to double check.

    The enterprise is rated at 1200W at 240V and I suspect the 94% efficiency only applies at 240V.

    It was nice to watch the power consumption drop after running a dedicated 12/2 wire to the systems vs a double stack of surge protectors from an outlet across the room.

    Next project is upgrading the circuit breaker panel. Then I can switch to and test 240V and be able to monitor and log whole house energy consumption! Home | OpenEnergyMonitor
     
    #2291
  12. juey

    juey Member

    Joined:
    Oct 1, 2018
    Messages:
    42
    Likes Received:
    11
    You simply cannot compare a 550W psu with a 1000W psu, and it doesn't makes it better if you don't use calibrated measurement gear.
    Buy three different Kill-A-Watt meters, and its likely that they show you three different values for the same gear attached.

    Sure, at large scale, it does make sense to dig deeper.
    You can compare your Seasonic 550W Bronze (SSR-550GB3 ?) to the Brocade ICX6610's RPS16-E aka Delta ECD14020023 aka Dell AWF-2DC-1000W
    https://clearesult5.sharepoint.com/..._SSR-550GB3_550W_SOCE 5441_Report 12-6-18.pdf
    https://clearesult5.sharepoint.com/...CS INC._AWF-2DC-1000W_1000W_SO-366_Report.pdf


    All in all, i highly doubt that it makes sense to replace the OEM psu of the 6610 with a titanium one, even if its possible and you find one that actually fits in the tray. For the difference of money you can get that thing running for a long time.
     
    #2292
    fohdeesha likes this.
  13. e97

    e97 Active Member

    Joined:
    Jun 3, 2015
    Messages:
    216
    Likes Received:
    77
    Didn't know all the 80Plus testing results were public. Thanks for the links! (The links are one time use so its helpful to link the page or upload the PDFs.)

    plugloadsolutions.com has all the 80Plus results !

    I'm using the Delta DPS-1200 series (certified 2011-2013) and Seasonic S12II-620 Bronze (certified 2009, purchased 2017). The data on 80Plus shows the Delta at 230V and the Seasonic at 115V.

    The label on the DPS-1200 says only 94% efficient at 220V+ so I originally thought it was a combination of 120V and poor efficiency of the picoPSU.

    However, when testing another system with DPS-1200 + picoPSU, it shows lower idle and load power consumption vs the Seasonic. => Either the DPS-1200, breakout board, picoPSU or wiring is faulty.

    To your other comments:
    Using the same instrument, the Kill-a-watt, propagates the same error so its negligible. The relative delta W between the PSUs is more relevant in this case. Plus I have access to a NIST calibrated multimeter if accuracy is required.

    All my network gear runs on a single/redundant DPS-1200, so its more efficient power and space wise than running multiple low load AC/DC switching supplies.
    Switching to a titanium gives me even better efficiency and pays for itself in ~6 months with my electricity rates.

    After that, I'll switch to GaN PSUs http://www.ti.com/lit/wp/sszy033/sszy033.pdf which gets me 99% conversion efficiency. TI has nice reference designs if anyone is interested.
    The next step after that is converting to a 48VDC system.
    After that its renewable energy =)
     
    #2293
    Last edited: Sep 29, 2019
  14. infoMatt

    infoMatt Member

    Joined:
    Apr 16, 2019
    Messages:
    43
    Likes Received:
    19
    Mine has a high pitched noise too, maybe it was a batch of Foxconn fans that does this... I'd swap fans on mine too.
    Provided that the seller has reset the configuration, you can use it like a "dumb" unmanaged switch, and maybe connect to it via telnet on a DHCP address? Not sure about it, but you might try...

    Please, don't do it. There are many components beside the main ASIC that rely on airflow to cool themselves... power stages, port driver, PoE board if fitted... Those things are pretty well engineered, I wouldn't bother to """"optimize"""" it... ;)

    Oh, and for the power efficiency: in this case, provided that you cannot remove all the fans (otherwise the switch won't boot), and still there's a fan inside the power supply, you'll be adding power requirement for the pump ;)
     
    #2294
    fohdeesha likes this.
  15. juey

    juey Member

    Joined:
    Oct 1, 2018
    Messages:
    42
    Likes Received:
    11
    Oh, didnt know that, thanks ! Yeah the results are online available at plugloadsolutions.com (along with other sites).

    Switching from 230V input to 115V drops the efficiency only a couple percent, maybe its the picoPSU.

    You always have to look at your desired output wattage and choose the right psu for that specific load. Of course, your 550W 80Plus Bronze psu can draw less power at a low load as a 1000W 80Plus Platinum, 80Plus to 80Plus Platinum don't even have requirements for loads less than 20 % of their rated output, so its basically unknown or subject to further testing what their efficiency look like at low currents.

    Running any gear of only one or two big psu is indeed the most efficient way for powering a rack, but, for me, it only makes sense for a expensive 80 Plus Titanium psu, if the load will be less than 20 % the most time. The RPS16-E has a efficiency of 88.99% at 10 % load, which is only 1 % away from the requirements of 80Plus Titanium, efficiencies at other loads are, compared to Titanium psus, in the range of 1-2%. I doubt it would pay itself in 6 months if you switch from 80Plus Platinum to 80Plus Titanium.

    That looks interesting, but current products seems limited to industrial psus ?


    +1
     
    #2295
  16. noscoe30

    noscoe30 New Member

    Joined:
    Sep 30, 2019
    Messages:
    1
    Likes Received:
    0
    Is anybody using the 08.0.90 switch firmware on an ICX7250-24p and ICX7250-48p in a stack? I have a meraki mx upstream and want to stay with router on a stick for now.
     
    #2296
  17. am45931472

    am45931472 Member

    Joined:
    Feb 26, 2019
    Messages:
    45
    Likes Received:
    9
    If you can find a way to defeat all the safty/security of this switch that prevents it from being quieter I'd love to know.

    I love my 6610 48P and its features, I hate its noise. I have a home lab of mostly silent machines except for this 1 switch. Cant replace the PSU fans, or the chassis fans without triggering something that stopps it from booting.
     
    #2297
    e97 likes this.
  18. fohdeesha

    fohdeesha Kaini Industries

    Joined:
    Nov 20, 2016
    Messages:
    1,396
    Likes Received:
    1,124
    as I've said a bunch in this thread if you want 10gb licenses just PM me
     
    #2298
  19. fohdeesha

    fohdeesha Kaini Industries

    Joined:
    Nov 20, 2016
    Messages:
    1,396
    Likes Received:
    1,124

    if you want to be halfway serious about 3 and 4 percent changes in PSU efficiency, you need to throw away your killawatt. It is not true that it "propagates the same error", if that were true the factory could just program in a static offset and they would match up with a reference grade meter. It's error varies wildly depending on the actual percent of it's full scale measurement running through it (eg 500w versus 900w), the ambient temperature, how long it's been powered up and the resistor network inside it has been allowed to warm up, etc. I've been using a reference grade power meter ( https://ctlsys.com/support/wattnode-modbus/ ) and I have caught killawatts anywhere from 5 to 15 percent off depending on environmental conditions. I would actually consider 5% pretty good for those things based on how bad I've seen them, and obviously that's not going to be of much help when measuring much smaller variances. You simply can't use them for anything other than "yeah, this thing is drawing some rough amount of power". And while a reference grade multimeter is nice to have, and can be used to measure DC power accurately (and even then, you need two of them), they cannot be used to accurately measure AC. They'll show you apparent power, but not real power. You need a proper power meter that will actually measure and show you AC power factor

    Also, while I commend the effort (thinking outside the box is good and how we got a lot of the cool hacks on this forum), I think if you spend probably 20 man-hours replacing the OEM delta's with the best efficiency power supply you find, the 3 or 4 percent efficiency gain you may find, at the barely 100w of draw, is going to equal a lot of work to reduce your draw by literally 4 watts. Keep in mind the switch talks to the PSUs via PMbus (kind of a specialized version of i2c for power supplies), and reads 30 or 40 registers as well as identifying EEPROMs with serial codes etc, so if you want to replace the power supplies you'll need to engineer a microcontroller to fake all that, so perhaps 20 man-hours of work is a low estimation
     
    #2299
  20. fohdeesha

    fohdeesha Kaini Industries

    Joined:
    Nov 20, 2016
    Messages:
    1,396
    Likes Received:
    1,124
    apologies to the 9000 missed PMs, I think I just finished replying to all of them
     
    #2300
    Last edited: Sep 30, 2019
Similar Threads: Brocade Series
Forum Title Date
Networking [SOLVED] Help Needed - Brocade ICX 6450 + Ruckus R720 Nov 25, 2019
Networking Brocade ICX 6610 - what does dhcp-client enable do? Nov 19, 2019
Networking MikroTik CSS326-24G-2S+RM vs Brocade ICX6450-24 Nov 1, 2019
Networking Brocade ICX6450-24P vs Aruba S2500-24P for Homelab Sep 22, 2019
Networking Brocade VDX 6720 - what do I need to know? Sep 16, 2019

Share This Page