Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

[Spearfoot]

I had the same experience this morning! Bought an ICX7150-C12P 32 days ago on eBay... this morning it suddenly died. When I tested it I found that the front panel LED's briefly flash when it's plugged in, same as you. So now I'm off to find a new power supply. Thanks for the link.

This 54VDC may be a better choice. It's hard to tell what the original PSU voltage is; mine has no markings of any kind, and -- being dead -- I can't just meter it. I did see a post on the Reddit thread stating that the OEM PSU on these units is 54VDC.

GTM91110P24054FSIM(R GlobTek, Inc. | Power Supplies - External/Internal (Off-Board) | DigiKey

Order today, ships today. GTM91110P24054FSIM(R – Open Frame AC DC Converters 1 Output 54V 90 ~ 264 VAC Input from GlobTek, Inc.. Pricing and Availability on millions of electronic components from Digi-Key Electronics.

www.digikey.com

 

[fohdeesha]

Do these (ICX7250) have anything like that I'll need to turn off? I don't have a cable modem right now but the config worked perfectly when I moved and switched to municipal fiber, so I'm inclined to set the new switch up identically before switching over (port mappings and all) and then work from there.

Nope, nothing by default. It does support lldp and cdp for device discovery, and with those on your cable modem will see the mac address of your switch and get confused, which I'm sure is what happened with your nortel (cable modem should only be seeing 1 mac, the mac of your router). neither of these things are on by default however so after following my guide (and make a separate vlan for wan) it will work out of the box

 

[fohdeesha]

Hey Everyone,

Does anyone have info on using the SDN capability in a 6610?

Apologies if I didn't read up to my answer yet (I'm 'only' on page 78).

-J

download the firmware and documentation zip on my update guide, linked in the very first post of this thread. In that zip, you'll find fastiron-08030-sdnguide.pdf

 

[EngineerNate]

Nope, nothing by default. It does support lldp and cdp for device discovery, and with those on your cable modem will see the mac address of your switch and get confused, which I'm sure is what happened with your nortel (cable modem should only be seeing 1 mac, the mac of your router). neither of these things are on by default however so after following my guide (and make a separate vlan for wan) it will work out of the box

Great! The Nortel gave me fits because I'd turned off all of discovery protocols I knew about (lldp etc) in the gui and it was still tripping up the modem. I had to mirror the modem's port and wireshark it to see what it was seeing to know that it was still sending out packets for it's own proprietary thing. It's been a great switch otherwise and at the time I purchased it it was a cheap into into 10G and PoE.

 

[jazzyy]

The Brocade ICX6450-24P does not support IPv6 Prefix Delegation, correct?

 

[3nodeproblem]

As my homelab is growing, it has become time for my first "serious" switch. After a couple of weeks of research, I am feeling ready to push the "buy" button on one of the Brocade models - the sweet spot for me seems to be the ICX6450-24P. My needs:

• 8 PoE 1G ports (these are currently on an unmanaged switch that I could reuse, but would be nice to fit them all on the main switch. SBCs for Consul/Nomad/Vault servers on the same VLAN, so I think it could actually be OK with a shared 1G for them all. PoE budget is pretty low and even 80W total should be more than plenty)
• 3 10G ports (glusterfs servers. I still haven't figured out if I really *need* 3 or if 2 would be sufficient as the third will be a glusterfs arbiter and I haven't figured out if full writes are sent from clients to arbiter if it's reduced to metadata on the client side)
• Min 8 non-PoE 1G ports for other nodes
• Low-power
• Quiet

Now, it seems that either 7150-24P, 7250-24P, or 6450-24P would be reasonable choices here. (6610 disregarded for power and noise). I'm no stranger to fan modding. If I reuse my current PoE switch, I could also fit my current requirements in a C12P, at the cost of easy future scalability. However, having followed the domestic (Japan) and eBay secondary market for the past week or two, it looks like I have to do some tradeoffs. While eBay is an option, domestic would be nicer due to COVID and customs. My options seem to be:

1. Domestic ICX6450-24 for ~100$. Reuse current switch for PoE.
2. Domestic ICX6610-48P for ~100$. Severe overkill but I'll never (knock on wood) grow out of this.
3. eBay International ICX6450-48P for ~180$.
4. eBay International ICX6450-24P for ~460$.
5. eBay International ICX7250-24P for ~540$.
6. eBay International XX50-C12-PD for 200~250$, coupled with a ~160$ MikroTik CRS305-1G-4S+IN for SFP+ ports and current GB switch for SBC cluster.
7. Hold my horses and closely watch the markets until something more suitable comes up.

So the main reason why I'm hesitant to go for either of the beefier but cheaper ones are power consumption and noise - the ratings are way higher for 48 port ones, and for 6610 compared to the others. If that is what I can expect under similar load, I'm thinking it *could* still be worth shell out 3x the price for the ICX6450-24P, or settle for the non-PoE version at the cost of needing two switches and not being able to add additional PoE devices (which I could make use of with new SBCs).

Now my main questions:

1. Under similar load, is should I really expect power consumption to be significantly higher (around double) for the 48-port versions over the 24 port ones? Can something be done either in hardware or software to disable the higher 24 ports?
2. Does the 6610 *have* to be such a noisy and hungry beast-hog compared to the 6450/7250? Can something be done to bring it down in line with the others considering I won't be utilizing near max capacity? The ICX6610-48P for 100$ looks really attractive but I'm afraid it'll end up a source of frustration.

 

[EngineerNate]

The 6610 fans are non-moddable from what I've read in this thread.

The 6610-24p would also meet your requirements and be significantly better on power (80w vs 110w) than the 48p, as a middle ground.

Rev B and later PSUs are quieter. Quiet enough? That's a question only you can answer.

All that said, I didn't think the power draw between the two port counts of the 6450 was all that different. I'd probably go with the 6450-48p if I were you.

 

[3nodeproblem]

All that said, I didn't think the power draw between the two port counts of the 6450 was all that different. I'd probably go with the 6450-48p if I were you.

Thanks for your input. The parent post states 25W for 24, 50W for 48 (double!), but my main question here is if that's considering all plugs utilized, or if power usage is more comparable with the same load on the two.

 

[infoMatt]

Thanks for your input. The parent post states 25W for 24, 50W for 48 (double!), but my main question here is if that's considering all plugs utilized, or if power usage is more comparable with the same load on the two.

My 6450-24p idles at 33-34W, so take those figures with a grain of salt. The only times it draws 25W is during boot, when the main switching ASIC isn't yet managed. Disabling the unused ports doesn't change the power draw.
The -48/-48P version has two switching ASIC, so that explains the difference in power consumption... if I'd have to guess stated the difference I saw during boot, it would add some 15ish W at the plug (please keep in mind I'm in EU so 230V) for a grand total of ~50W.

 

[bbqdt]

Does the 6610 *have* to be such a noisy and hungry beast-hog compared to the 6450/7250? Can something be done to bring it down in line with the others considering I won't be utilizing near max capacity? The ICX6610-48P for 100$ looks really attractive but I'm afraid it'll end up a source of frustration.

https://forums.servethehome.com/index.php?threads/anyone-done-a-brocade-6610-fan-mod.24039/post-264368

My ICX6610-24P idles around 75w.

48p will be around 110w. I think you can remove one of the poe boards though?

 

[infoMatt]

I think you can remove one of the poe boards though?

Maybe, but the PoE board by themselves doesn't draw much, maybe you can shave off a couple of watts, but is it worth the effort? I suspect that the major difference is due to the +54V supply rail idle draw than the boards. If there's no PD attached of course, in that case you have to add the power needed by the devices.

 

[3nodeproblem]

https://forums.servethehome.com/index.php?threads/anyone-done-a-brocade-6610-fan-mod.24039/post-264368

My ICX6610-24P idles around 75w.

48p will be around 110w. I think you can remove one of the poe boards though?

Oh wow, that fan mod is impressive and admittedly a bit above my level

From the replies I gather that the surplus draw from the 6610 and 48 port models is high enough that I should disregard them for now and keep sending out offers for 6450/7150/7250-24p until something sticks (energy is not just about cost but also about ecology IMO!)

 

[pr09]

Ended up mounting my ICX7250-24P with a CyberPower 4POSTRAIL kit. A bit overkill, but definitely no sag, and it's close enough to 1U to fit in right below a shelf. No width issues, which I was concerned about with some other shelves.

Now I'm onto fan woes. Tried the Noctua, but it couldn't keep it cool -- over 70C completely idle, even on full speed. Got the Sunon MF40201VX-1000U-G99, and it can keep it cool on full speed, but is very noticeably whiny. On low it's quiet, but goes over 90C. Time to keep trying more options, I suppose.

With all these fan changes, I realized that the unusual pinout is actually just the regular pinout backwards. So, rather than continuing to splice cables, I just bent the tab down and plugged it in backwards. Works perfectly, and doesn't make the fans unusable for other purposes.

 

[EngineerNate]

With a lot of those plugs you can just use a pin to release the actual wire+contact from the plastic housing and swap things around.

 

[pr09]

With a lot of those plugs you can just use a pin to release the actual wire+contact from the plastic housing and swap things around.

Well I feel silly now. Thank you, a couple minutes after reading your comment I now have two pigtails that do exactly the swap needed.

 

[Jason Antes]

How big are those fans? I have some doubled up fans out of Proliant servers, they are really loud at full power but pretty quiet when spun down a bit. There was a lot of design research into those fans to move air efficiently.

 

[pr09]

How big are those fans? I have some doubled up fans out of Proliant servers, they are really loud at full power but pretty quiet when spun down a bit. There was a lot of design research into those fans to move air efficiently.

The originals are 28mm deep; both replacements I've tried are 20mm deep. Doubling up is an interesting idea, would need a bit more wiring though. For now I'm going to try faster/louder fans in the hope that they'll be powerful enough on low.

 

[Jason Antes]

So these are doubled up, with brackets are about 80mm deep (40mm fan). The front is a 4 blade fan, the rear is 5 blades, and in between there is an airflow modifier assembly. Hard to see in the pictures and if I can get it apart I'll get better ones. The set pictured is 4 fans total.

 

[infoMatt]

So these are doubled up, with brackets are about 80mm deep (40mm fan). The front is a 4 blade fan, the rear is 5 blades, and in between there is an airflow modifier assembly. Hard to see in the pictures and if I can get it apart I'll get better ones. The set pictured is 4 fans total.

Those are optimized for high static pressure... they draw a lot of power and make A LOT of noise at full tilt. Watch out for your ears

EDIT: actually, it's written on the label of the rear one: 1.82A @12V, or if you prefer ~20ishW EACH!

 

[Spearfoot]

If, like me, you're one of those unfortunate ICX7150-C12P owners whose power supply has died, take comfort. There are replacement units available. Digikey has at least two: items LB240S56K and GCS350PS56 at the top of the list linked below. These are both 56VDC supplies which will deliver adequate power even after derating for convection cooling. They're the same form factor and use the same connectors as the original Ruckus PSU.

Warning: they both cost over $100!

Electronic Components and Parts Search | DigiKey Electronics

Digi-Key is your authorized distributor with over a million in stock products from the world’s top suppliers. Rated #1 in content and design support!

www.digikey.com

The LB240S56K is a 'big brother' of the 130W LB130S56 @fohdeesha mentioned in a Reddit post on this subject (see below), and which I've purchased on eBay and intend to try out. I'm only running a single PoE device off my switch (an AP) so the lower power rating shouldn't be a problem -- I hope. If it doesn't work out, I'm either going to spring for a new C12P with warranty, or one of the PSUs above. Haven't made up my mind about that yet.
https://www.reddit.com/r/homelab/comments/dz989v

I received an XP Power GCS350PS56 power supply today and installed it this afternoon in my eBay-sourced ICX 7150-C12P -- the one in which the power supply died recently.

I am very pleased to report that it works.

DigiKey link to the GCS350PS56: GCS350PS56 XP Power | Power Supplies - External/Internal (Off-Board) | DigiKey

This is a 56VDC PSU; if you find yourself needing to replace the supply in your switch and are concerned about the slight over-voltage, you may want to use the GlobalTek 54VDC supply below. Not only is it the same voltage as the original Ruckus, it's $34 cheaper, too. On the other hand, my unit has a significant power margin, being capable of delivering 200W even with only convection cooling. The Ruckus only needs 124W, which is close to the maximum available from the GlobalTek supply when derated for convection cooling.

DigiKey link to the GlobalTek PSU: GTM91110P24054FSIM(R GlobTek, Inc. | Power Supplies - External/Internal (Off-Board) | DigiKey

I had to rewire the 6-lead DC power connector as the Ruckus is wired exactly bass-ackwards from the rest of the world of engineering design! This was easy to do: I just used a paper-clip to extract the wires from the power connector and re-insert them to match the new PSU. I was able to mount the replacement supply in the original Ruckus cage and made use of the OEM insulators as well.

All in all this was a pretty easy fix, other than having to shell out dough for the PSU.