Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

pr09

New Member
Jun 21, 2020
6
0
1
How big are those fans? I have some doubled up fans out of Proliant servers, they are really loud at full power but pretty quiet when spun down a bit. There was a lot of design research into those fans to move air efficiently.
The originals are 28mm deep; both replacements I've tried are 20mm deep. Doubling up is an interesting idea, would need a bit more wiring though. For now I'm going to try faster/louder fans in the hope that they'll be powerful enough on low.
 

Jason Antes

Member
Feb 28, 2020
76
20
8
Twin Cities
So these are doubled up, with brackets are about 80mm deep (40mm fan). The front is a 4 blade fan, the rear is 5 blades, and in between there is an airflow modifier assembly. Hard to see in the pictures and if I can get it apart I'll get better ones. The set pictured is 4 fans total.

20200701_154546.jpg20200701_155000.jpg20200701_154939.jpg
 

infoMatt

Active Member
Apr 16, 2019
157
60
28
So these are doubled up, with brackets are about 80mm deep (40mm fan). The front is a 4 blade fan, the rear is 5 blades, and in between there is an airflow modifier assembly. Hard to see in the pictures and if I can get it apart I'll get better ones. The set pictured is 4 fans total.
Those are optimized for high static pressure... they draw a lot of power and make A LOT of noise at full tilt. Watch out for your ears ;)

EDIT: actually, it's written on the label of the rear one: 1.82A @12V, or if you prefer ~20ishW EACH!
 
  • Like
Reactions: Jason Antes

Spearfoot

Member
Apr 22, 2015
65
25
18
62
If, like me, you're one of those unfortunate ICX7150-C12P owners whose power supply has died, take comfort. There are replacement units available. Digikey has at least two: items LB240S56K and GCS350PS56 at the top of the list linked below. These are both 56VDC supplies which will deliver adequate power even after derating for convection cooling. They're the same form factor and use the same connectors as the original Ruckus PSU.

Warning: they both cost over $100!

The LB240S56K is a 'big brother' of the 130W LB130S56 @fohdeesha mentioned in a Reddit post on this subject (see below), and which I've purchased on eBay and intend to try out. I'm only running a single PoE device off my switch (an AP) so the lower power rating shouldn't be a problem -- I hope. If it doesn't work out, I'm either going to spring for a new C12P with warranty, or one of the PSUs above. Haven't made up my mind about that yet.
https://www.reddit.com/r/homelab/comments/dz989v
I received an XP Power GCS350PS56 power supply today and installed it this afternoon in my eBay-sourced ICX 7150-C12P -- the one in which the power supply died recently.

I am very pleased to report that it works.

DigiKey link to the GCS350PS56: GCS350PS56 XP Power | Power Supplies - External/Internal (Off-Board) | DigiKey

This is a 56VDC PSU; if you find yourself needing to replace the supply in your switch and are concerned about the slight over-voltage, you may want to use the GlobalTek 54VDC supply below. Not only is it the same voltage as the original Ruckus, it's $34 cheaper, too. On the other hand, my unit has a significant power margin, being capable of delivering 200W even with only convection cooling. The Ruckus only needs 124W, which is close to the maximum available from the GlobalTek supply when derated for convection cooling.

DigiKey link to the GlobalTek PSU: GTM91110P24054FSIM(R GlobTek, Inc. | Power Supplies - External/Internal (Off-Board) | DigiKey

I had to rewire the 6-lead DC power connector as the Ruckus is wired exactly bass-ackwards from the rest of the world of engineering design! This was easy to do: I just used a paper-clip to extract the wires from the power connector and re-insert them to match the new PSU. I was able to mount the replacement supply in the original Ruckus cage and made use of the OEM insulators as well.

All in all this was a pretty easy fix, other than having to shell out dough for the PSU.
 
Last edited:
  • Like
Reactions: Mitsubishi

Ryan Haver

New Member
Apr 7, 2020
6
1
3
I am seeing some really weird behavior when using two Mellanox MCX4131A-GCAT ConnectX-4 adapters connected to my Brocade ICX6610.

100% Reproducible behavior
  1. When using ConnectX-4 adapters plugged into both QSFP+ ports (1/2/1 and 1/2/6) the ConnectX-4 adapters exhibit a massive amount of packet loss.
  2. When one QSFP+ port is disabled then the adapter plugged into the other port no longer experiences packet loss.
  3. When using a Mellanox MCX313A-BCBT ConnectX-3 in one port and a Mellanox MCX4131A-GCAT ConnectX-4 in the other port there are no issues with packet loss.
Any help here is appreciated, as this doesn't make any sense to me. For the sake of troubleshooting, I'm going to test a turning off a few features within the ConnectX-4 adapter firmware like SR-IOV.


Edit: After swapping back to using both of the ConnectX-4 adapters this is no longer an issue....really weird.
 
Last edited:

Jason Antes

Member
Feb 28, 2020
76
20
8
Twin Cities
Those are optimized for high static pressure... they draw a lot of power and make A LOT of noise at full tilt. Watch out for your ears ;)

EDIT: actually, it's written on the label of the rear one: 1.82A @12V, or if you prefer ~20ishW EACH!
Yeah, these are for older Gen7ish Proliants. I'd look for newer Gen9/10 as they are much quieter. Of course, they only spin as fast as you allow them too by the power you feed them. Not hat it makes it any better, but for the power you are feeding more than 1 fan. ;)

I had gotten some of these as I was collecting them for a friend who wanted to use them to power his RC planes. I had more than he wanted at the time so I kept these.
 

tommybackeast

Active Member
Jun 10, 2018
247
80
28
Nope, nothing by default. It does support lldp and cdp for device discovery, and with those on your cable modem will see the mac address of your switch and get confused, which I'm sure is what happened with your nortel (cable modem should only be seeing 1 mac, the mac of your router). neither of these things are on by default however so after following my guide (and make a separate vlan for wan) it will work out of the box
ICX7250 firmware question :

I see your download page continues to list 8080 (for ICX7250). - this is the firmware my ICX 7250-48p is using.

I also have a Ruckus r510 AP: and from some reading, seems there's a single pane of glass for ICX7250 on 8090 firmware used in conjuction with the Ruckus r510AP.

Gently asking : are there problems using 8090 firmware with ICX 7250 ? thanks ...
 

tommybackeast

Active Member
Jun 10, 2018
247
80
28
just switch the fan direction around and monitor temps, I have a feeling it'll be fine. If I remember right somewhere here with a 7250 swapped the fann direction and actually saw slightly cooler temps but I could be misremembering
FYI: my ICX 7250-48p is rear mounted in an enclosed server rack; and I never reversed the fans and temps are just fine. I'm only using two 1G ports for POE; and am using 6 10GB ports + 20-25? 1GB non-POE ports
 

EngineerNate

Member
Jun 3, 2017
61
10
8
31
Seller sent me a non-PoE unit. :mad:

My ToR switch needs to be PoE but with a partial refund, the price was good enough with the rthat I'll hang onto it for now to either use elsewhere or sell, but I'm back on the market for myself.

I think I'll look for a 6610-24p... I didn't realize when I first purchased that the two non-breakout capable QSFP+ ports could be used to hook up to hypervisor servers at 40G. That's worth the ~30w to me.
 

hmw

Active Member
Apr 29, 2019
199
61
28
I think I'll look for a 6610-24p... I didn't realize when I first purchased that the two non-breakout capable QSFP+ ports could be used to hook up to hypervisor servers at 40G. That's worth the ~30w to me.
Yep - and it works beautifully. I have the ICX6610-24p connected at 40 GbE via a ConnectX-3 in my ESXi server. Again thanks to this forum, found a pair of 3m NetApp DAC cables for $9

Waiting for ConnectX-4 LX prices to go down so I can use SR-IOV. And maybe the next step is to build out vSAN ...
 

EngineerNate

Member
Jun 3, 2017
61
10
8
31
What exactly is the difference between PoE and non-PoE in terms of internals? Is it just the PSUs? Or there's an internal PoE board as well?
I'd guess there's a poe board. I know there is in some of the other ICX servers from reading this thread.

In better news, I found one of those sub $200 7150-C12Ps so my upstairs networking closet is sorted.

Can't decide now whether to homerun the upstairs access point or run it off the switch. The switch was originally going to be for other stuff, but if I have 10G up to the closet up there... Not a lot of downside to power it locally vs from the basement and it would be one less cat6 run to clog up the conduit.
 

safrax

New Member
Jun 21, 2020
5
0
1
Is this normal for an ICX7250-24 with stock fan sitting in a room at about ~24C?

Fan controlled temperature: 64.5 deg-C

Fan speed switching temperature thresholds:
Speed 1: NM<----->98 deg-C
Speed 2: 67<----->105 deg-C (shutdown)

Fan 1 Air Flow Direction: Front to Back
Slot 1 Current Temperature: 65.0 deg-C (Sensor 1)
Slot 2 Current Temperature: NA
Warning level.......: 100.0 deg-C
Shutdown level......: 105.0 deg-C


My ICX 6610-48P sitting in my un-airconditioned garage is idling at about ~71C.
 

fohdeesha

Kaini Industries
Nov 20, 2016
1,907
1,713
113
29
fohdeesha.com
Is this normal for an ICX7250-24 with stock fan sitting in a room at about ~24C?

Fan controlled temperature: 64.5 deg-C

Fan speed switching temperature thresholds:
Speed 1: NM<----->98 deg-C
Speed 2: 67<----->105 deg-C (shutdown)

Fan 1 Air Flow Direction: Front to Back
Slot 1 Current Temperature: 65.0 deg-C (Sensor 1)
Slot 2 Current Temperature: NA
Warning level.......: 100.0 deg-C
Shutdown level......: 105.0 deg-C


My ICX 6610-48P sitting in my un-airconditioned garage is idling at about ~71C.
very normal
 

infoMatt

Active Member
Apr 16, 2019
157
60
28
Are there any concerns I should be aware of if I move it out to my garage which at the top end is about ~29C,90F and somewhat humid?
If you put the unit in an environment 5 degrees hotter, its internal temp would also rise by 5°, reaching 70°C, giving you another 30°C or so of thermal headroom.
No worries, those are "older" process silicon, they are meant to run "hot". As you've seen, it wouldn't ramp up the fans until 98°C.

As for the humidity, those aren't exactly rugged units, but if it isn't condensing-on-the-walls damp, it should survive, but it could suffer of premature oxidation of the contacts inside the RJ45 or on the metal case. If it's human-bearable, it shouldn't be a problem.
 

EngineerNate

Member
Jun 3, 2017
61
10
8
31
Just scored a 6610-24P -I with two rev B power supplies, two fans, rack ears, and power cables included. Price was a bit above average but I figure for the full compliment of power supplies/fans, rack ears, and the PSUs being rev B it was worth it.

What QSFP+ pci-e cards is everyone using for the 40G connections? Any compatibility issues?
 

itronin

Active Member
Nov 24, 2018
399
245
43
Denver, Colorado
What QSFP+ pci-e cards is everyone using for the 40G connections? Any compatibility issues?
I don't know about "everyone" but the very first post in this thread towards the bottom has a recommendation for an OEM HPE ethernet/IB card that can be flashed to stock Mellanox ConnectX 3 40Gbe. I have two in use and they work great and were inexpensive. I can verify the card works in Windows, FreeNAS, and ESXI 6.5 and 6.7.

Netapp QSFP cables were fine for short run interconnects where super flexible cable bends are not required. They are also inexpensive.
 
Last edited:
  • Like
Reactions: EngineerNate

hmw

Active Member
Apr 29, 2019
199
61
28
Just scored a 6610-24P -I with two rev B power supplies, two fans, rack ears, and power cables included. Price was a bit above average but I figure for the full compliment of power supplies/fans, rack ears, and the PSUs being rev B it was worth it.

What QSFP+ pci-e cards is everyone using for the 40G connections? Any compatibility issues?
Preflashed ConnectX-3: HP 544QSFP MCX354A-FCBT 649281-B21 656089-001 VPI FDR 40GbE Mellanox OEMFirmware | eBay

DAC cable: Original NetApp 112-00178 X6559-R6 External SAS QSFP+ TO QSFP+ QSFP Cable 5m | eBay

The ConnectX-3 will work in ESXi 6.7 or 7 but will NOT do SR-IOV