Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

[fohdeesha]

Hello!
I am the reddit person with said failed switch. I have just registered to reply here.

Original post:




I assume you are right on every account. They HAVE been tampered with.



I linked the power meter very closely resembling the one I have, as I couldn't find the my Poland/Germany only model on US Amazon. I also have DIN rail mounted current meter.






After reading the replies out there and looking into all the photos and other threads I could find of the switch and its insides, I came to conclusion, that I either "scored" engineering sample or qualification sample or got a result of some not-exactly-professional-but-still-somewhat-competent person frankensteining this device from scrapped or stolen lot.

It can be either, since it's impossible to modify fan curve, and knowing devices like that usually turn into "survival mode" (fan speed to max) when some sensors are off, it behaved surprisingly well, suggesting either someone found a way to alter it, to fool the sensors or it came with pre-release fw or client-targeted one.

I should have opened it right after receiving, but I only did coursory look to see of there are no loose components or screws, threw cables in and it worked, so I left it as it is. Only thing I did was swap the PSUs around. Switch and extra PSU were bought from same place (an eBay auction, and it was private person as far as I remember).

Using this photo:

https://people.ucsc.edu/~warner/Bufs/icx-6610.JPG

as reference, I found:

- switch has no serial number, neither on the device itself or on the motherboard, and all stickers were removed. Can't say anything about the firmware now, but you saw the zeros. Only markings I could find are laser etched "AR2054-01-011" between CPU and stack connector card and "Brocade MV1194V-0 / AB 1 026-3" under the card;
- memory stick was kaptoned to the slot from all sides and all over, and I have Smart sg57a648bro535y1sj EP2-5300c-555-13-zz. This is the only thing that has intact serial number in whole device, but I didn't find anything about this particular stick.
- all PSUs had a sticker with revision saying S5, where there would be A, B or C originally, underneath it the original rev has been scratched off. Stickers with QR and s/n are missing. They all bear marks of being opened multiple times (a lot of scratches around the screws and tabs were clearly abused). I missed this since for some time I used to handle device scrapping at my workplace so I got resistant to noticing scratches.
- I went over the motherboard with good light and found solder flux residue around almost all power components;
- almost all of electrolytic capacitors are random brands;
- Boot flash chip has "fused!" handwritten with a marker pen and flux residue around its legs; This probably explains the bricking and null s/n.
- Battery socket, after removing the battery shows signs of cleaned up corrosion (I don't ever recall seeing Lithium battery leak!); Current battery reads 2.9 V.
- headers J2, J10 and U6 had been clearly removed;
- place where POE headers go in PoE-equipped models were clearly soldered on and cleaned up;
- One of the fan tray connectors on the board had mangled pins and its mounting screw was held in place by copious amount of Locktite or similar glue. I had to wrestle it off. Fan modules look okay. There is a trace of s/n stickers being removed from them on the inside between the fans.

One thing that stands to me now, is that I never noticed the switch kicking up the fans to speeds anywhere close those heard during boot sequence, they did went up but not much - but at the same time it really did pull that much power off the wall. On idle the exhaust was moderately warm to very warm, but I wouldn't describe it as hot, like for example Dell R640 going full tilt can get, and with max traffic I could put on it it got really, really hot, enough to make keeping hand in the airstream very uncomfortable to painful.

I should also note that I misread the specifications! I assumed that both PSUs work in unison and share power, and to boot it only from one I need higher tier, 750 ow 1000 W ones. I am used to moving around 2 kW+ switches at work, so this is really why I paid no mind to inability of my device to turn on with one and the power consumption.

With all of that, and switch now bricked, even if I managed to find someone actually competent who would repair it I don't exactly feel safe putting it back into my homelab, little late perhaps, but oh well.

If anything I consider this to be a warning not to blindly trust enterprise gear. It never dawned on me someone would just go over a device like this to fix it. It's not that big blow financially, these switches go under $100 routinely, and I have other 10G gear, 40G was very cool addition, but I can live without it.

holy shit what the hell lol. Can you post pictures of this monstrosity? Also your assumption is correct, they only need one PSU to run. In fact, that's how 90% of them were configured/sold. You can use any wattage too, you can boot a PoE version with the non-PoE small 250w supply (PoE just won't be enabled)

 

[beren]

Sorry if this was answered in this monster thread or somewhere else, but I just got my 6610 with dual rev A power bricks. Would it be quieter to remove one and block it vs leaving it unplugged?

Also was thinking using the 40G breakout with DAC might be simpler than front ports, would I lose any diag info going with a non-brocade programmed cable? I know with optics what you lose but never found dac info.

 

[heromode]

So this was with 3x Arctic 4028-6K fans installed:

show chassis
The stack unit 1 chassis info:

Power supply 1 (NA - AC - PoE) present, status ok
Power supply 2 not present
Power supply 3 not present

Fan 1 ok, speed (auto): [[1]]<->2
Fan 2 ok, speed (auto): [[1]]<->2
Fan 3 ok, speed (auto): [[1]]<->2

Fan controlled temperature: 63.5 deg-C

Fan speed switching temperature thresholds:
                Speed 1: NM<----->65       deg-C
                Speed 2:       56<-----> 79 deg-C (shutdown)

Sensor B Temperature Readings:
        Current temperature : 56.5 deg-C
Sensor A Temperature Readings:
        Current temperature : 63.5 deg-C
        Warning level.......: 69.0 deg-C
        Shutdown level......: 79.0 deg-C

Obviously the 6K rpm Arctic's are not sufficient, at 4.5 volts calculating i presume they are spinning at 2250rpm. ((4.5/12)*6000). That's not much for a 40mm fan, and one can barely feel any airflow.

So this is the result of my next try:



3 cheap chinese wide voltage range (3V - 12V) fans connected to the fan headers using split cables. The fans are 40mm x 40mm x 10mm. (the distance from the top of the ASIC heatsinks to the top of the switch is exactly 20mm, so 10mm thick fans are the only option)

The fans are easy to attach without any screws using two small zip ties that fit between the heatsink fins, preventing the zip-tie to slide out over the edge, and locking them using only the lock part from two additional zip-ties. The fans at 4.5 volts are completely silent and can't be heard outside the case.

Clearly, now instead of the heat concentrating around the ASIC's, it's spread evenly across the whole case: note the below results are from a room that is 29 degrees celcius ambient (been the hottest august in 60 years here)

show chassis
The stack unit 1 chassis info:

Power supply 1 (NA - AC - PoE) present, status ok
Power supply 2 not present
Power supply 3 not present

Fan 1 ok, speed (auto): [[1]]<->2
Fan 2 ok, speed (auto): [[1]]<->2
Fan 3 ok, speed (auto): [[1]]<->2

Fan controlled temperature: 57.5 deg-C

Fan speed switching temperature thresholds:
                Speed 1: NM<----->65       deg-C
                Speed 2:       56<-----> 79 deg-C (shutdown)

Sensor B Temperature Readings:
        Current temperature : 57.5 deg-C
Sensor A Temperature Readings:
        Current temperature : 57.5 deg-C
        Warning level.......: 69.0 deg-C
        Shutdown level......: 79.0 deg-C

All the readings are exactly the same, 57.5 degrees. The upside of this setup is the switch is silent, and no ASIC is close to triggering a higher speed even in a 29 degree room. And should the temp go above 65 degrees, all 6 fans will spin up.

Basically, since the Arctic's move so little air out from behind, but the heat is spread evenly across the case, the top cover of the case now effectively works as a heatsink. It also works as a radiator, heating my small computer room. This is not an issue during winter, but during summer it is.

I'm sure it's been mentioned in the thread many times, and i plan to go through all 400 pages once i start configuring the switch, but please remind me:

Can i remove the POE board, and reduce power consumption/heat?
Can i reduce power consumption by disabling ports?

Initially i tried to order the 24 port model from that dutch dealer in the Great deals forum, but he had just sold the last one to a guy paying with paypal while my bank transfer was still pending, so i got a 48 port version for the same price. i'd still love to be able to shave about 10W from idle consumption on this.

 

[IceBrew]

Hi all,

I've been following this post for a few years at this point waiting for a good point to dive in, mainly concerned with power consumption but think I'm ready to bite the bullet. Because we're slowly creeping towards 2023 though I have to ask:

Are these switches still a good value buy in 2022 / 2023?

Of course you're getting up to 48 PoE ports with some 10G ports thrown in for $100 delivered in some cases, but from what I can see the 6450 in particular was EOL in 2018. So we're buying a 5+ year old switch. My main concern is what's the actual life expectancy of network switches? Searching online just gives the "replace your gear every 5 years" but doesn't give any indication of "They start failing after X years or time on".

Can I buy one now in the hopes it'll last another 5 years? My use case is dropping by the day to the point where for now I could get away with less than 10 ports and it'll be a fair few years before that amount would need to increase so I want to be happy with the investment rather than spending the extra $100 for a brand new switch with admittedly less ports, but with a much longer life expectancy, lower power draw and newer features (but admittedly no 10G). What's peoples thoughts? Thanks!

 

[i386]

Are these switches still a good value buy in 2022 / 2023?

I'm struggling with the same question. But after separating "want"s and "need"s the answer is yes.
- high quality
- 1gbe is (still) fast enough to stream multiple 4k streams (I mean remuxed UHD disc quality, not the netflix stuff )
- good documentation, the official and the in official by fohdeesha (I always mistype that name ._.) and other forum members
- free firmware updates (you need a ruckus account)

 

[IceBrew]

I'm struggling with the same question. But after separating "want"s and "need"s the answer is yes.
- high quality
- 1gbe is (still) fast enough to stream multiple 4k streams (I mean remuxed UHD disc quality, not the netflix stuff )
- good documentation, the official and the in official by fohdeesha (I always mistype that name ._.) and other forum members
- free firmware updates (you need a ruckus account)

Thanks for the thoughts, it's very hard to argue with $100 that's for sure and if it lasts 5 years I'd consider that amazing value so long as the running costs aren't drastically higher. Hopefully the one I order doesn't end up being the beat up one that's been abused.

And for sure documentation is king in this thread. Will be attempting to swap out for quieter fans as it'll go in a rack under a desk (attempting to rack mount my main rig and my old rig (as a server) too, so hopefully the noise and heat are palatable.

 

[safrax]

So FiOS finally turned on IPv6 to my house within the last few weeks. I've been trying to get it to propagate throughout my network but I'm a bit stumped. I have the following (simplified) setup:


I have an OPNSense firewall sitting between a FiOS ONT and my ICX-7250. The ICX-7250 and Firewall are configured with a 2-port LAG acting as a transit network. I have a few different VLANs on the 7250, VLAN1-Deprecated, VLAN99-Transit, VLAN101-Data, VLAN102-Servers, VLAN103-WiFi, VLAN104-IoT. The ICX-7250 is currently acting as the DHCP Server for all VLANs except 1 and 99. I have set the following IPv6 related configuration options in the ICX-7250:

ver 08.0.95T213
ipv6 dhcp-relay accept-broadcast
ipv6 unicast-routing
ipv6 router ospf
ipv6 neighbor inspection vlan 1
ipv6 neighbor inspection vlan 99
ipv6 neighbor inspection vlan 101 to 104
interface lag 1
dhcp6 snooping trust
ipv6-neighbor inspection trust

The OPNSense firewall has a rule to allow DHCP traffic to pass to the upstream DHCP server. I've also configured the appropriate IPv6 Tracking options for the LAG interface in OPNSense.

I'm seeing OPNSense make router advertisements on the LAG but a server downstream sitting on VLAN102 isn't seeing them. At this point I'm not sure what else I need to do. I read through the documentation on Brocade's website and to some extent it was over my head. I'm a sysadmin that does Linux, I'm not a network person.

As for why I went with a transit network over letting the OPNSense firewall do all the routing: performance. The OPNSense machine is more than capable of handling FiOS' ~1Gbps speeds, but I'm fairly certain it'd fall over if I threw a 10Gbps NIC in it and asked it to route as well. If there's a way to get rid of the transit network, I'm all ears, I feel like it causes me more problems than it's worth.

 

[heromode]

So FiOS finally turned on IPv6 to my house within the last few weeks.
If there's a way to get rid of the transit network, I'm all ears, I feel like it causes me more problems than it's worth.

IPv6 Is a Total Nightmare — This is Why

 

[safrax]

IPv6 Is a Total Nightmare — This is Why

Part of why I'm doing this is a learning exercise. The other part is I'm trying to figure out whether or not I can solve some issues I've been experiencing with NAT and some games. I don't actually have very high hopes that the games support IPv6, but on the off chance they do it could alleviate some spouse related annoyances.

That said, thank you for the article. It was informative and I do agree IPv6 is fairly insane. It's the best we currently have to help alleviate some of the IPv4 related issues and it would have been nice if the IETF had designed something else , with lessons learned, 10-15 years ago.

 

[heromode]

Part of why I'm doing this is a learning exercise. The other part is I'm trying to figure out whether or not I can solve some issues I've been experiencing with NAT and some games. I don't actually have very high hopes that the games support IPv6, but on the off chance they do it could alleviate some spouse related annoyances.

That said, thank you for the article. It was informative and I do agree IPv6 is fairly insane. It's the best we currently have to help alleviate some of the IPv4 related issues and it would have been nice if the IETF had designed something else , with lessons learned, 10-15 years ago.

we need ipv5

 

[DavidRa]

You literally cannot have something that's completely compatible with IPv4 as well as providing increased address spaces and changing all the code, all the tools, all the IPv4-aware hardware etc.

All the IPv4 code in the world has 32 little bit-sized boxes to put the IP address into. The moment you have more bits, no matter what you do, that code is insufficient.

IMO IPv6 isn't actually that hard, the big problems are:

• It seems like the designers wanted the network people to control everything - which is why DHCPv6 doesn't actually do much without the routers "permitting" it;
• The whole shebang is "design by committee" where no-one had the ability or control to stop adding stuff;
• Almost all the corporate network people threw their hands in the air because it was "change" and "different" and "we don't need it", and combined with the afore-mentioned "routers control everything" the result that IPv6 just can't get deployed;
• IT people seem to hate DNS and "Waaah I can't remember IPv6 addresses".

In short, people are stupid and shortsighted. Most of the problems in that blog rant are either unsolvable, irrelevant or "thinking is hard". I have had working IPv6 for nearly a decade - using a HE.net tunnel, because most ISPs won't provision IPv6 in Australia. And I'm by no means a network engineer.

 

[tfran1990]

Hi, is this the place for me to ask a question about my configuration for my ICX 6450?

 

[DavidRa]

Hi, is this the place for me to ask a question about my configuration for my ICX 6450?

Go for it, yes.

 

[tfran1990]

I used the guide in the very beginning of this thread and i was able to configure everything as far a ve and vlans with the attached ports. my problem is i cant ping a laptop connected to one of the vlans. is this the right place to post a config?

 

[kpfleming]

In short, people are stupid and shortsighted. Most of the problems in that blog rant are either unsolvable, irrelevant or "thinking is hard". I have had working IPv6 for nearly a decade - using a HE.net tunnel, because most ISPs won't provision IPv6 in Australia. And I'm by no means a network engineer.

Same here, I've had IPv6 in my home network (in multiple variations) for years and it works well and solves actual problems.

Back to the OP's question: with the configuration you have, you'll need to setup IPv6 address configuration and routing at each hop, you can't expect RAs to *pass through* layer 3 network elements (routers). RAs are *local* configuration, they don't pass through routers.

Assuming that FIOS allows you to use DHCPv6-PD to get a prefix delegation of reasonable size (/56 or larger), which they don't yet in my FIOS neighborhood, you'll need to do various things:

* The OPNsense firewall will need to obtain a delegation from FIOS using DHCPv6-PD. It may also get its own address (via SLAAC or DHCPv6) but that is not mandatory, as it will already have a link-local address that can be used for IPv6 transit between itself and the FIOS network. The delegation will need to be at least a /56 in size, although a /48 would be better given your configuration; if it's a /64, you're stuck because that can't be broken down further.

* The 7250 'core router' will need to be given a block of addresses it can use on the VLANs it manages. Just like the FIOS-OPNsense link there is no need for addresses to be assigned to the transit LAN, that LAN can use the automatic link-local addresses. The block can be given to the 7250 via DHCPv6-PD *from OPNsense* (not FIOS) if OPNsense is able to do that; if not, it can be manually configured on both ends.

* Each of the VLANs on the 7250 will need to be manually configured with a sub-block out of the 7250's block (but no smaller than /64). The 7250 would then be configured to emit RAs on the LANs so that hosts can get IPv6 addresses and have routing information. It can also provide DHCPv6 services if you desire, but that is not necessary.

* There is no need for any dynamic routing protocols for this to work; the subnet addresses form a 'tree' starting from the block delegated by FIOS, and each router in the tree knows about the downstream routers, but there's no need for them to know about 2nd or 3rd level routers/routes below them.

* "passing through" DHCPv6 traffic via the OPNsense firewall isn't likely to work out well. That sort of thing can work OK for host addresses (not prefix delegation) if the device doing the passthrough supports NDP proxying (similar to ARP proxying in IPv4), but that isn't going to work with DHCPv6-PD.

 

[Oodaloop]

I am having connectivity problems hooking up my ICX-7250 to a Dell Poweredge R740 server . The Dell has an Intel X520/I350 daughter card. My connection uses a DAC cable from one of the 10G ports on the Intel card to one 10G port on the ICX-7250.
My connection is very instable , with frequent loss of connectivity. Not sure how to troubleshoot this - change the DAC cable? I previously had stp setup on the brocade, so I turned it off because I thought the Dell Poweredge was bringing down the network with a broadcast storm. The network was rock solid until I included the Dell server. What should I be looking for?

Did you ever get this figured out? I'm having a similar issue with a R710 server. The 10G ports on the ICX work when initially booted then just shutdown eventually and I can't get them to turn back on until after a reboot.

Anyone else have this experience or could walk me through some troubleshooting steps to figure out the problem or atleast a way to get the ports back on without having to power cycle the entire switch?

 

[i386]

Does anyone know of the replacement for the ICX6650?

ICX7750-48F (10GBE)
ICX7850-48F (25GBE)

Edit: Source Ethernet Switches

 

[Vesalius]

i386, do you know if the 25G model is backwards compatible to 10G?

With a free account, you can verify your options on the ruckus site. A few are available among the 7750/7850 models.

Ruckus ICX 7750 Campus Switch Technical Specifications | Technical Documents | Ruckus Wireless Support

support.ruckuswireless.com

RUCKUS ICX 7850 Switch Technical Specifications | Technical Documents | Ruckus Wireless Support

support.ruckuswireless.com

 

[LodeRunner]

I'm trying to hunt down packet loss in a stack of 7450's. Work has a stack of 6, full ring using QSFP modules in the back, and up until a few days ago everything was fine.

Now, despite a full stack reload and reflash to 8095g we're getting measurable packet loss to many hosts across all the units in the stack. Ruckus has a document about using the MAC filter to identify packet loss, but it's locked behind paid support. Any advice?

Edit: sh int and sh stats on the stack ports show all zeroes in the error counters. Ports used for testing area also showing no errors.

Edit 2: symptoms may have actually begun a week ago and presented as a reporting service generating blank reports (reporting service finalizes the file after receiving data or timing out). In addition to a full stack reload yesterday and a ISSU upgrade today, we reloaded all downstream switches in the building.

We traced it all to a workstation; once that port was disabled, we stopped losing traffic. Plugging that workstation into a different port immediately created the issue again. So diagnosing the home run and the workstation NIC will be fun.

 

[heromode]

I'm not happy with the chinese fans on the ASIC's, they're actually creating noise that can be heard outside the case. Onto the next option, 3x Noctua nf-a4x10-flx fans with low noise adapter on the asic's, spinning at 3700 RPM, and fed from the 12V rail of the PSU. Then maybe 1x Noctua NF-A4x20 FLX at the rear also fed with 12V and a low noise adapter, to slightly increase the outbound airflow, because the Arctic S4028-6K's barely move any air at 4.5V.

I'm bummed that i wasn't able to get the 24 port ICX6450, thermal management would have been so much easier with 25W. I ordered the last one but had to pay with bank transfer, and while the seller was waiting for that transfer, some yahoo came along and bought the last 24 version with paypal, and the seller then offered me the 48 at the same price.

Now looking at the markets i see the 48 port models are plentiful and selling for cheap, but the 24 port models are few and far between, and much more expensive. Man i wish i could lose that extra 25W idle

ATM at idle and the summer heatwave gone, i'm running at 55 degrees in a small room with lots of old gear producing heat, which will all soon be gone. But i'm determined to make my 48p model absolutely silent, it's just the 25 Watts running 24/7 is gonna cost me about 25 EUR annually until december 2024, and after that, who knows..

edit anyone have a spare good condition 6450-24, poe or no poe, and wanna swap with a -48p in excellent condition in europe, lemme know.