Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

Wolfstar

Active Member
Nov 28, 2015
156
81
28
45
I got my icx6450-24p today, but my console cable won't be here until next week.. I take there is not really anything I can do until I get that?

A fan mod is definitely in the pipe for this thing,, I think the fans may actually failing as they have more of a high pitch ring than I was expecting.
That's actually quite probable. Mine has no ring to it and there isn't even much of the 40mm fan whine really. Same switch - ICX6450-24P.
 

e97

Active Member
Jun 3, 2015
245
111
43
This thread is already at 11 - lets kick it up a notch to 12.

ill be looking to water cooling the ICX 6610-48P to make it quiet and (hopefully) use less power.

will also look into replacing the PSU with a high efficiency platinum one.

@fohdeesha has kindly shared that it requires 12V and 52V (PoE).

we also need to trick the fan rpm sense pins since no more stock fans.. hopefully its the same ATX standard and pump and rad fan output can be used.
 

juey

Member
Oct 1, 2018
56
15
8
Germany
This thread is already at 11 - lets kick it up a notch to 12.

ill be looking to water cooling the ICX 6610-48P to make it quiet and (hopefully) use less power.

will also look into replacing the PSU with a high efficiency platinum one.

@fohdeesha has kindly shared that it requires 12V and 52V (PoE).

we also need to trick the fan rpm sense pins since no more stock fans.. hopefully its the same ATX standard and pump and rad fan output can be used.
The original Delta oem psu is platinum certified.
 
  • Like
Reactions: fohdeesha

blinkenlights

Active Member
May 24, 2019
143
64
28

levi danzer

New Member
Sep 29, 2019
3
1
3
First off, big thanks to fohdeesha for this amazing thread. It made me pull the trigger on a ICX6610-24i which is massive overkill for my home. I don't even have a lab setup yet!

I've got to wire up a serial cable and find my usb-serial so I can get my switch setup. The ebay posting showed it as having the fcx08010.bin firmware so i'll be following the guide shortly to get everything updated and reset.

The 40-gbe is working with a DAC. but the SFP+ ports don't seem to have 10gbe unlocked. Kinda annoying as that was the main use for this switch. But as a switch its been plug and play... grab the old 1gig copper, and internet on the 40gbe port. Kinda funny to me.
This is one of my first steps into enterprise gear. It sounds much louder than the data sheet. Wondering if I have a fan bearing going out, was unbearable with only one PSU plugged in. Will dig into the system stats soon. Maybe i should swap out for a 6450?

I'm debating forgoing the 40gbe and picking up a 7xxx series so i can be sure to have 10gbe.... or is there a better route to go? (the 40gbe is massive overkill for my uses, heck i'll be surprised if i can use half the 10gbe)

Thanks!

-Levi
 

BobTB

Member
Jul 19, 2019
33
4
8
In 6610-48, I inserted a single mode, duplex transceiver, show media shows it as
Port 1/3/1: Type : 1G M-LX(SFP)

When I connect the cable, nothing happens. I tried swapping the A an B, waiting a minute etc... but link never goes up.I have factory reset the switch, and touched nothing else but added the IP to the switch. The other end is connected to fiber provider. The transceiver is the one I got from them, to plug in some other switch, so it is the correct one.

Is there something else I have to do to enable the port? Nothing is written in the log of the switch when I plug the cable in the transceiver or unplug it.
 
Last edited:

e97

Active Member
Jun 3, 2015
245
111
43
The original Delta oem psu is platinum certified.
I'm impressed, as some of these power supplies are nearly as old as the Platinum certification itself. For those who are skeptical, it's clearly listed on the label of a different OEM version: Delta 1000W PowerEdge AWF-2DC-1000W Switching Power Supply EDC14020023 A258 | eBay

I could track down the 1000W PSU (AWF-2DC-1000W) - anyone know if the 250W PSU (DPS-250AB-96) is also Platinum certified?
Semiconductor technology advances quickly but the power designs should be nearly the same. Now, highest efficiency rating is 80+ Titantium.

Measured with a Kill-A-Watt a 3+ yr old 94% platinum certified enterprise PSU and a recent Seasonic 80+ Bronze PSU -- on the same system. Both at 120V, the 80+ bronze showed lower idle and load Watts by about 30W.

Not sure if this holds at 240V
 

T_Minus

Build. Break. Fix. Repeat
Feb 15, 2015
7,046
1,583
113
CA
Semiconductor technology advances quickly but the power designs should be nearly the same. Now, highest efficiency rating is 80+ Titantium.

Measured with a Kill-A-Watt a 3+ yr old 94% platinum certified enterprise PSU and a recent Seasonic 80+ Bronze PSU -- on the same system. Both at 120V, the 80+ bronze showed lower idle and load Watts by about 30W.

Not sure if this holds at 240V
What was the wattage of the Enterprise Plat and the Seasonic 80+ Bronze?
Kind of apples\oranges unless their wattage is the same.
 

e97

Active Member
Jun 3, 2015
245
111
43
What was the wattage of the Enterprise Plat and the Seasonic 80+ Bronze?
Kind of apples\oranges unless their wattage is the same.
900W enterprise 94% efficient
550W consumer ATX 80+ Bronze

It used to be the case that load significantly impacted efficiency. Advances in modern switching semiconductors have drastically reduced those losses:

Here's Efficiency vs Load for Seasonic Platinum 1000W Power Supply


2% peak to trough = ~6W for the system under test

80 PLUS Platinum efficiency; What does it mean, and what's the benefit to me? - Corsair has a nice breakdown.

If you switch to even newer technologies like GaN or SiC you can achieve ~99% across a larger range due to faster switching speed and lower losses which also reduces heat and required cooling and size.

Following links show how these advances are being applied to IT power system design at scale:

* Google Contest Builds More Efficient Inverters

* https://www.opencompute.org/files/External-2018-OCP-Summit-Google-48V-Update-Flatbed-and-STC-20180321.pdf

At scale, a few watts x 1000s of systems drastically affects power, cooling and space.
 
Last edited:

T_Minus

Build. Break. Fix. Repeat
Feb 15, 2015
7,046
1,583
113
CA
Great info!

Your 3+ Year old Platinum likely isn't as efficient as these new modern ones you speak of though ??

Going from 920 or 1200 I forget which to 550w I noticed 10-15w on Kill-a-watt between PSU, that's pretty significant (10-12%) on the system I was testing. *25 systems in a rack and that's like 1 or 2 additions systems you can run :D
 

e97

Active Member
Jun 3, 2015
245
111
43
Great info!

Your 3+ Year old Platinum likely isn't as efficient as these new modern ones you speak of though ??

Going from 920 or 1200 I forget which to 550w I noticed 10-15w on Kill-a-watt between PSU, that's pretty significant (10-12%) on the system I was testing. *25 systems in a rack and that's like 1 or 2 additions systems you can run :D
I was pretty surprised myself which is why I did the Kill-A-Watt test. I have another energy measurement device for all the systems to double check.

The enterprise is rated at 1200W at 240V and I suspect the 94% efficiency only applies at 240V.

It was nice to watch the power consumption drop after running a dedicated 12/2 wire to the systems vs a double stack of surge protectors from an outlet across the room.

Next project is upgrading the circuit breaker panel. Then I can switch to and test 240V and be able to monitor and log whole house energy consumption! Home | OpenEnergyMonitor
 

juey

Member
Oct 1, 2018
56
15
8
Germany
Semiconductor technology advances quickly but the power designs should be nearly the same. Now, highest efficiency rating is 80+ Titantium.

Measured with a Kill-A-Watt a 3+ yr old 94% platinum certified enterprise PSU and a recent Seasonic 80+ Bronze PSU -- on the same system. Both at 120V, the 80+ bronze showed lower idle and load Watts by about 30W.

Not sure if this holds at 240V
You simply cannot compare a 550W psu with a 1000W psu, and it doesn't makes it better if you don't use calibrated measurement gear.
Buy three different Kill-A-Watt meters, and its likely that they show you three different values for the same gear attached.

900W enterprise 94% efficient
550W consumer ATX 80+ Bronze

It used to be the case that load significantly impacted efficiency. Advances in modern switching semiconductors have drastically reduced those losses:

Here's Efficiency vs Load for Seasonic Platinum 1000W Power Supply


2% peak to trough = ~6W for the system under test

80 PLUS Platinum efficiency; What does it mean, and what's the benefit to me? - Corsair has a nice breakdown.

If you switch to even newer technologies like GaN or SiC you can achieve ~99% across a larger range due to faster switching speed and lower losses which also reduces heat and required cooling and size.

Following links show how these advances are being applied to IT power system design at scale:

* Google Contest Builds More Efficient Inverters

* https://www.opencompute.org/files/External-2018-OCP-Summit-Google-48V-Update-Flatbed-and-STC-20180321.pdf

At scale, a few watts x 1000s of systems drastically affects power, cooling and space.
Sure, at large scale, it does make sense to dig deeper.
You can compare your Seasonic 550W Bronze (SSR-550GB3 ?) to the Brocade ICX6610's RPS16-E aka Delta ECD14020023 aka Dell AWF-2DC-1000W
https://clearesult5.sharepoint.com/..._SSR-550GB3_550W_SOCE 5441_Report 12-6-18.pdf
https://clearesult5.sharepoint.com/...CS INC._AWF-2DC-1000W_1000W_SO-366_Report.pdf


All in all, i highly doubt that it makes sense to replace the OEM psu of the 6610 with a titanium one, even if its possible and you find one that actually fits in the tray. For the difference of money you can get that thing running for a long time.
 
  • Like
Reactions: fohdeesha

e97

Active Member
Jun 3, 2015
245
111
43
You simply cannot compare a 550W psu with a 1000W psu, and it doesn't makes it better if you don't use calibrated measurement gear.
Buy three different Kill-A-Watt meters, and its likely that they show you three different values for the same gear attached.


Sure, at large scale, it does make sense to dig deeper.
You can compare your Seasonic 550W Bronze (SSR-550GB3 ?) to the Brocade ICX6610's RPS16-E aka Delta ECD14020023 aka Dell AWF-2DC-1000W
https://clearesult5.sharepoint.com/sites/PLS/Shared Documents/SEA SONIC ELECTRONICS CO., LTD._SSR-550GB3_550W_SOCE 5441_Report 12-6-18.pdf
https://clearesult5.sharepoint.com/sites/PLS/Shared Documents/DELTA ELECTRONICS INC._AWF-2DC-1000W_1000W_SO-366_Report.pdf


All in all, i highly doubt that it makes sense to replace the OEM psu of the 6610 with a titanium one, even if its possible and you find one that actually fits in the tray. For the difference of money you can get that thing running for a long time.
Didn't know all the 80Plus testing results were public. Thanks for the links! (The links are one time use so its helpful to link the page or upload the PDFs.)

plugloadsolutions.com has all the 80Plus results !

I'm using the Delta DPS-1200 series (certified 2011-2013) and Seasonic S12II-620 Bronze (certified 2009, purchased 2017). The data on 80Plus shows the Delta at 230V and the Seasonic at 115V.

The label on the DPS-1200 says only 94% efficient at 220V+ so I originally thought it was a combination of 120V and poor efficiency of the picoPSU.

However, when testing another system with DPS-1200 + picoPSU, it shows lower idle and load power consumption vs the Seasonic. => Either the DPS-1200, breakout board, picoPSU or wiring is faulty.

To your other comments:
Using the same instrument, the Kill-a-watt, propagates the same error so its negligible. The relative delta W between the PSUs is more relevant in this case. Plus I have access to a NIST calibrated multimeter if accuracy is required.

All my network gear runs on a single/redundant DPS-1200, so its more efficient power and space wise than running multiple low load AC/DC switching supplies.
Switching to a titanium gives me even better efficiency and pays for itself in ~6 months with my electricity rates.

After that, I'll switch to GaN PSUs http://www.ti.com/lit/wp/sszy033/sszy033.pdf which gets me 99% conversion efficiency. TI has nice reference designs if anyone is interested.
The next step after that is converting to a 48VDC system.
After that its renewable energy =)
 
Last edited:

infoMatt

Active Member
Apr 16, 2019
166
65
28
I got my icx6450-24p today, but my console cable won't be here until next week.. I take there is not really anything I can do until I get that?

A fan mod is definitely in the pipe for this thing,, I think the fans may actually failing as they have more of a high pitch ring than I was expecting.
Mine has a high pitched noise too, maybe it was a batch of Foxconn fans that does this... I'd swap fans on mine too.
Provided that the seller has reset the configuration, you can use it like a "dumb" unmanaged switch, and maybe connect to it via telnet on a DHCP address? Not sure about it, but you might try...

ill be looking to water cooling the ICX 6610-48P to make it quiet and (hopefully) use less power.
Please, don't do it. There are many components beside the main ASIC that rely on airflow to cool themselves... power stages, port driver, PoE board if fitted... Those things are pretty well engineered, I wouldn't bother to """"optimize"""" it... ;)

Oh, and for the power efficiency: in this case, provided that you cannot remove all the fans (otherwise the switch won't boot), and still there's a fan inside the power supply, you'll be adding power requirement for the pump ;)
 
  • Like
Reactions: fohdeesha

juey

Member
Oct 1, 2018
56
15
8
Germany
Didn't know all the 80Plus testing results were public. Thanks for the links! (The links are one time use so its helpful to link the page or upload the PDFs.)

plugloadsolutions.com has all the 80Plus results !
Oh, didnt know that, thanks ! Yeah the results are online available at plugloadsolutions.com (along with other sites).

The label on the DPS-1200 says only 94% efficient at 220V+ so I originally thought it was a combination of 120V and poor efficiency of the picoPSU.
Switching from 230V input to 115V drops the efficiency only a couple percent, maybe its the picoPSU.

To your other comments:
Using the same instrument, the Kill-a-watt, propagates the same error so its negligible. The relative delta W between the PSUs is more relevant in this case. Plus I have access to a NIST calibrated multimeter if accuracy is required.
You always have to look at your desired output wattage and choose the right psu for that specific load. Of course, your 550W 80Plus Bronze psu can draw less power at a low load as a 1000W 80Plus Platinum, 80Plus to 80Plus Platinum don't even have requirements for loads less than 20 % of their rated output, so its basically unknown or subject to further testing what their efficiency look like at low currents.

All my network gear runs on a single/redundant DPS-1200, so its more efficient power and space wise than running multiple low load AC/DC switching supplies.
Switching to a titanium gives me even better efficiency and pays for itself in ~6 months with my electricity rates.
Running any gear of only one or two big psu is indeed the most efficient way for powering a rack, but, for me, it only makes sense for a expensive 80 Plus Titanium psu, if the load will be less than 20 % the most time. The RPS16-E has a efficiency of 88.99% at 10 % load, which is only 1 % away from the requirements of 80Plus Titanium, efficiencies at other loads are, compared to Titanium psus, in the range of 1-2%. I doubt it would pay itself in 6 months if you switch from 80Plus Platinum to 80Plus Titanium.

After that, I'll switch to GaN PSUs http://www.ti.com/lit/wp/sszy033/sszy033.pdf which gets me 99% conversion efficiency. TI has nice reference designs if anyone is interested.
The next step after that is converting to a 48VDC system.
After that its renewable energy =)
That looks interesting, but current products seems limited to industrial psus ?


Please, don't do it. There are many components beside the main ASIC that rely on airflow to cool themselves... power stages, port driver, PoE board if fitted... Those things are pretty well engineered, I wouldn't bother to """"optimize"""" it... ;)
+1
 

noscoe30

New Member
Sep 30, 2019
2
0
1
Is anybody using the 08.0.90 switch firmware on an ICX7250-24p and ICX7250-48p in a stack? I have a meraki mx upstream and want to stay with router on a stick for now.
 

am45931472

Member
Feb 26, 2019
75
17
8
This thread is already at 11 - lets kick it up a notch to 12.

ill be looking to water cooling the ICX 6610-48P to make it quiet and (hopefully) use less power.

will also look into replacing the PSU with a high efficiency platinum one.

@fohdeesha has kindly shared that it requires 12V and 52V (PoE).

we also need to trick the fan rpm sense pins since no more stock fans.. hopefully its the same ATX standard and pump and rad fan output can be used.
If you can find a way to defeat all the safty/security of this switch that prevents it from being quieter I'd love to know.

I love my 6610 48P and its features, I hate its noise. I have a home lab of mostly silent machines except for this 1 switch. Cant replace the PSU fans, or the chassis fans without triggering something that stopps it from booting.
 
  • Like
Reactions: e97

fohdeesha

Kaini Industries
Nov 20, 2016
1,960
1,784
113
29
fohdeesha.com
First off, big thanks to fohdeesha for this amazing thread. It made me pull the trigger on a ICX6610-24i which is massive overkill for my home. I don't even have a lab setup yet!

I've got to wire up a serial cable and find my usb-serial so I can get my switch setup. The ebay posting showed it as having the fcx08010.bin firmware so i'll be following the guide shortly to get everything updated and reset.

The 40-gbe is working with a DAC. but the SFP+ ports don't seem to have 10gbe unlocked. Kinda annoying as that was the main use for this switch. But as a switch its been plug and play... grab the old 1gig copper, and internet on the 40gbe port. Kinda funny to me.
This is one of my first steps into enterprise gear. It sounds much louder than the data sheet. Wondering if I have a fan bearing going out, was unbearable with only one PSU plugged in. Will dig into the system stats soon. Maybe i should swap out for a 6450?

I'm debating forgoing the 40gbe and picking up a 7xxx series so i can be sure to have 10gbe.... or is there a better route to go? (the 40gbe is massive overkill for my uses, heck i'll be surprised if i can use half the 10gbe)

Thanks!

-Levi
as I've said a bunch in this thread if you want 10gb licenses just PM me
 

fohdeesha

Kaini Industries
Nov 20, 2016
1,960
1,784
113
29
fohdeesha.com
Didn't know all the 80Plus testing results were public. Thanks for the links! (The links are one time use so its helpful to link the page or upload the PDFs.)

plugloadsolutions.com has all the 80Plus results !

I'm using the Delta DPS-1200 series (certified 2011-2013) and Seasonic S12II-620 Bronze (certified 2009, purchased 2017). The data on 80Plus shows the Delta at 230V and the Seasonic at 115V.

The label on the DPS-1200 says only 94% efficient at 220V+ so I originally thought it was a combination of 120V and poor efficiency of the picoPSU.

However, when testing another system with DPS-1200 + picoPSU, it shows lower idle and load power consumption vs the Seasonic. => Either the DPS-1200, breakout board, picoPSU or wiring is faulty.

To your other comments:
Using the same instrument, the Kill-a-watt, propagates the same error so its negligible. The relative delta W between the PSUs is more relevant in this case. Plus I have access to a NIST calibrated multimeter if accuracy is required.

All my network gear runs on a single/redundant DPS-1200, so its more efficient power and space wise than running multiple low load AC/DC switching supplies.
Switching to a titanium gives me even better efficiency and pays for itself in ~6 months with my electricity rates.

After that, I'll switch to GaN PSUs http://www.ti.com/lit/wp/sszy033/sszy033.pdf which gets me 99% conversion efficiency. TI has nice reference designs if anyone is interested.
The next step after that is converting to a 48VDC system.
After that its renewable energy =)

if you want to be halfway serious about 3 and 4 percent changes in PSU efficiency, you need to throw away your killawatt. It is not true that it "propagates the same error", if that were true the factory could just program in a static offset and they would match up with a reference grade meter. It's error varies wildly depending on the actual percent of it's full scale measurement running through it (eg 500w versus 900w), the ambient temperature, how long it's been powered up and the resistor network inside it has been allowed to warm up, etc. I've been using a reference grade power meter ( https://ctlsys.com/support/wattnode-modbus/ ) and I have caught killawatts anywhere from 5 to 15 percent off depending on environmental conditions. I would actually consider 5% pretty good for those things based on how bad I've seen them, and obviously that's not going to be of much help when measuring much smaller variances. You simply can't use them for anything other than "yeah, this thing is drawing some rough amount of power". And while a reference grade multimeter is nice to have, and can be used to measure DC power accurately (and even then, you need two of them), they cannot be used to accurately measure AC. They'll show you apparent power, but not real power. You need a proper power meter that will actually measure and show you AC power factor

Also, while I commend the effort (thinking outside the box is good and how we got a lot of the cool hacks on this forum), I think if you spend probably 20 man-hours replacing the OEM delta's with the best efficiency power supply you find, the 3 or 4 percent efficiency gain you may find, at the barely 100w of draw, is going to equal a lot of work to reduce your draw by literally 4 watts. Keep in mind the switch talks to the PSUs via PMbus (kind of a specialized version of i2c for power supplies), and reads 30 or 40 registers as well as identifying EEPROMs with serial codes etc, so if you want to replace the power supplies you'll need to engineer a microcontroller to fake all that, so perhaps 20 man-hours of work is a low estimation