Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

[ewer0012]

yeah I would definitely open it up and look inside, especially if there's an identical switch above it running much cooler. the PSUs are responsible for most of the airflow, so I would visually inspect those and feel the air coming out of them and see if it's roughly the same airflow as the switch above it as well

I checked the fans/PSUs last night for proper airflow by taking a piece of cardboard and putting it between the two switches to isolate the exhausts. From what I can tell from this super accurate and by no means silly test, the fans are operating at roughly the same speed/CFM on the low fan setting.

I haven't gotten a chance to take it out of the rack, yet (why couldn't this have shown up while I had it on my desk instead of installed in my rack, buried under a berjillion ethernet/fiber cables?), but I'll do that tonight and crack it open to see what the deal is. Hopefully it's something fixable... Otherwise I might see about picking up a decent priced Arista DCS-7050S-64-R and tinkering with that.

Thanks for your help, foh.

 

[klui]

I checked the fans/PSUs last night for proper airflow by taking a piece of cardboard and putting it between the two switches to isolate the exhausts. From what I can tell from this super accurate and by no means silly test, the fans are operating at roughly the same speed/CFM on the low fan setting.

I haven't gotten a chance to take it out of the rack, yet (why couldn't this have shown up while I had it on my desk instead of installed in my rack, buried under a berjillion ethernet/fiber cables?), but I'll do that tonight and crack it open to see what the deal is. Hopefully it's something fixable... Otherwise I might see about picking up a decent priced Arista DCS-7050S-64-R and tinkering with that.

Thanks for your help, foh.

Clogged fans or heatsink issues? It sounds like what will happen when a notebook has its inlet vents clogged with dust/dirt. Or it could be that the temperature sensor is failing. Get compressed air and go to town.

 

[Vesalius]

Finally pulled the trigger on 2 Ruckus R720 APs. Will install the unleashed controller firmware on both APs. I'm still confused on the 803.3af vs 802.3at POE option on the AP and which port on the R720 to connect to my ICX6610 switch.

In addition, on the ICX6610, it looks like LLDP needs to be turned on and the switch needs to advertise power via LLDP.

Replying to an old post, but just found the same to be true for my Ruckus AP's. below is how to manage LLDP on the ICX switches per instructions from Ruckus.

PoE Switches negotiate PoE power after identifying other end devices such as AP, Cameras using LLDP-MED.

By default, LLDP is disabled in ICX 7000 and 6ooo series switches. To enable it, the following global command has to be issued:
ICX6450-48P Router(config)#lldp run

The switch will have to advertise power via LLDP and the following commands have to be issued either for the specific port e.g. port 1/1/5,
ICX6450-48P Router(config-if-e1000-1/1/5)#lldp advertise power-via-mdi ports ethernet 1/1/5

or if all ports have to be configured identically
ICX6450-48P Router(config-if-e1000-1/1/5)#lldp advertise power-via-mdi ports all

 

[juey]

Replying to an old post, but just found the same to be true for my Ruckus AP's. below is how to manage LLDP on the ICX switches per instructions from Ruckus.

PoE Switches negotiate PoE power after identifying other end devices such as AP, Cameras using LLDP-MED.

By default, LLDP is disabled in ICX 7000 and 6ooo series switches. To enable it, the following global command has to be issued:
ICX6450-48P Router(config)#lldp run

The switch will have to advertise power via LLDP and the following commands have to be issued either for the specific port e.g. port 1/1/5,
ICX6450-48P Router(config-if-e1000-1/1/5)#lldp advertise power-via-mdi ports ethernet 1/1/5

or if all ports have to be configured identically
ICX6450-48P Router(config-if-e1000-1/1/5)#lldp advertise power-via-mdi ports all

Hm, that steps weren‘t necessery for my UAP AC-HD. 802.3at was auto detected just by firing „inline power“ on the port.

 

[Vesalius]

My 3 Ruckus AP also worked with just inline power to the port, but the AP complained via logs and through the newest unleashed App, afte the latest firmware update this week, that they were not getting their full allocation of power. Looking at the ICX power stats before the change it seems while the APs were correctly listed as 802.3at devices by the ICX they were each being allocated a MAX 15.4 W power. After enabling LLDP on each of the APs ports the ICX now reports that each AP has now negotiated a Max draw of around 26.5 W, if memory serves.

Not sure if that will affect the AP's performance or not, but I have seen reports of only 2-3 out 4 antennas working when they are under-powered.

Copied from wikipedia but 802.3at Type 2 "PoE+" has Four power class levels (1-4) negotiated by signature (I think this is the best you can get with just inline power) or 0.1 W steps negotiated by LLDP.

I am unsure if the Unifi AP's support LLDP or not as the Ruckus AP's do, but unless hear or read otherwise enabling LLDP on a port does not seem to have any negative impact if you plug a POE device in that doesn't support it.

 

[ewer0012]

Clogged fans or heatsink issues? It sounds like what will happen when a notebook has its inlet vents clogged with dust/dirt. Or it could be that the temperature sensor is failing. Get compressed air and go to town.

Just took this thing apart. It looks and smells brand new. Not a spec of dust in it... Weird.

 

[exvarkin]

Wanted to post an update about the 6610-48P I bought from Techretire on eBay:

Received the switch, it's in great physical condition. 2 PSU, 2 fans (Was surprised for $125). I was able to update all firmware and configure everything for my lab network, and so far, so good... ish.

The switch keeps surging its fans to the high setting every 6ish minutes for about 30-45 seconds. I was able to confirm this with the logs (can post a sample later if needed). When I look at the temp readings from "sh chassis" right after a fan surge the switch cpu is sitting at 70C, and it kicks up to high at 76C.

I find this temp to be strange, considering I have an identical switch right above it running no more than 60C on any sensor, and it's running 40Gbe to my storage array, 2 breakout ports to my Nutanix cluster, and all 8 SFP+ ports are populated... So all of my heavy network traffic, VLANs, replication, transfers... All going through my first 6610-48P and it's rock solid at 60C.

I haven't gotten a chance to crack open the new switch, yet, but was going to do that to check for dust and such when I get home. Can anyone think of anything else I should take a look at when I have it open?

I may be experiencing something similar to you.

I bought a second 6610 48p. It was listed as PoE failed, and no PSU/fan, but I got it for 25 + shipping and had a spare PSU and fan form my first switch. Figured it be worth the risk for a possible spare.

I just got it last night so I haven't got too much time to play with it yet. I got it configured, removed the non working PoE board and everything seems to be "working". Just it does seem to be getting much hotter the my first 6610 (my original is a 24p). Part of me expects that, as it double the ports, but I don't know, looks similar to what you are seeing. Haven't let it run log enough to see the temps stabilize, but it on its way to hit the fan ramp up temps.

 

[sash]

Found this PDF document about officially supported Brocade optics. Thought it would be interesting for those who need monitoring for modules:

https://www.proficomms.cz/files/datasheets/Brocade/Brocade-Ethernet-Optics-Family.pdf

 

[jd.developer]

I was able to snag a Rev C PSU & a PSU blank from ebay. Unfortunately, when I boot the switch with only that PSU, it says:

Fatal Error: PSU fan is not present or FAN has failed

I tried swapping the PSU into a different slot, and mixing the Rev C & Rev A PSUs, but every time it shows the above message when the Rev C is inserted. I noticed the Rev C is made by Delta, while the Rev A is made by Emerson. Both PSUs have a PNO of RPS15-I & EID (17V) 1TNPO. They both have the same output ratings too - so I believe this is the correct part.

The power supply fan also appears to turn on when the Rev C PSU is inserted. I noticed this problem was reported in an earlier post, but the problem was dismissed as likely a defective part. Hoping it's not defective .

Does anyone know of settings that must be changed after switching to a different Rev PSU?

 

[u238]

Is there any way to see the stack port utilization of the icx6450s?

 

[juey]

Is there any way to see the stack port utilization of the icx6450s?

SNMP

 

[fohdeesha]

Is there any way to see the stack port utilization of the icx6450s?

sh stat e 1/2/1
sh e 1/2/1

 

[fohdeesha]

I was able to snag a Rev C PSU & a PSU blank from ebay. Unfortunately, when I boot the switch with only that PSU, it says:

Fatal Error: PSU fan is not present or FAN has failed

I tried swapping the PSU into a different slot, and mixing the Rev C & Rev A PSUs, but every time it shows the above message when the Rev C is inserted. I noticed the Rev C is made by Delta, while the Rev A is made by Emerson. Both PSUs have a PNO of RPS15-I & EID (17V) 1TNPO. They both have the same output ratings too - so I believe this is the correct part.

The power supply fan also appears to turn on when the Rev C PSU is inserted. I noticed this problem was reported in an earlier post, but the problem was dismissed as likely a defective part. Hoping it's not defective .

Does anyone know of settings that must be changed after switching to a different Rev PSU?

that message happens when the RPM reported by the PSU fan is not within the normal operating threshold, so it's either failing and spinning too slow, or it's RPM reporting/wire is busted altogether

 

[exvarkin]

Regarding the 6610, is the non PoE PSU (DPS-250AB-96 I believe?) compatible with PoE modle? Obviously no PoE at that point, but would the normal switch still operate?
I'm currently powering my second 6610 (removed faulty PoE daughter board) with the 1000W Rev A. Was think I might try to pick up a Rev C or B PSU but not sure if I still need the 1000W version (AWF-2DC-1000W-E) if the switch cant do PoE anyways.

 

[fohdeesha]

Regarding the 6610, is the non PoE PSU (DPS-250AB-96 I believe?) compatible with PoE modle? Obviously no PoE at that point, but would the normal switch still operate?
I'm currently powering my second 6610 (removed faulty PoE daughter board) with the 1000W Rev A. Was think I might try to pick up a Rev C or B PSU but not sure if I still need the 1000W version (AWF-2DC-1000W-E) if the switch cant do PoE anyways.

Yep works fine

 

[fohdeesha]

Also the PSU part numbers for maximum eBay search results are


brocade rps-15 (non Poe)
brocade rps-16 (Poe)

 

[exvarkin]

Would you expect any benefit going for the rps-15 in this case?? e.g. heat/fan noise/efficiency, or just get whatever has the best price.

 

[camper8080]

Just purchased the ICX 7250-48P. Installed latest firmware. The fans are hovering around ~7000 RPM. Is this normal for this switch?

ICX7250-48P Router#show flash
Stack unit 1:
Compressed Pri Code size = 29826604, Version:08.0.80eT213 (SPR08080e.bin)
Compressed Sec Code size = 31678660, Version:08.0.30hT213 (SPR08030h.bin)
Compressed Boot-Monitor Image size = 786944, Version:10.1.14T215
Code Flash Free Space = 1816481792

ICX7250-48P Router#show chassis
The stack unit 1 chassis info:
Power supply 1 (AC - PoE) present, status ok
Power supply 2 not present
Power supply 3 not present
Fan 1 ok, speed (auto): [[1]]<->2
Fan 2 ok, speed (auto): [[1]]<->2
Fan 3 ok, speed (auto): [[1]]<->2
Fan controlled temperature:
Rule 1/2 (MGMT THERMAL PLANE): 59.0 deg-C
Rule 2/2 (AIR OUTLET NEAR PSU): 21.0 deg-C
Fan speed switching temperature thresholds:
Rule 1/2 (MGMT THERMAL PLANE):
Speed 1: NM<-----> 95 deg-C
Speed 2: 85<----->105 deg-C (shutdown)
Rule 2/2 (AIR OUTLET NEAR PSU):
Speed 1: NM<-----> 41 deg-C
Speed 2: 34<----->105 deg-C (shutdown)
Fan 1 Air Flow Direction: Front to Back
Fan 2 Air Flow Direction: Front to Back
Fan 3 Air Flow Direction: Front to Back
Slot 1 Current Temperature: 59.0 deg-C (Sensor 1), 21.0 deg-C (Sensor 2)
Slot 2 Current Temperature: NA
Warning level.......: 100.0 deg-C
Shutdown level......: 105.0 deg-C


ICX7250-48P Router#dm fan-speed
Fan 1 Speed at 7180 RPM.
Fan 2 Speed at 7458 RPM.
Fan 3 Speed at 7258 RPM.


Thanks for everyone's time.

 

[vangoose]

Just purchased the ICX 7250-48P. Installed latest firmware. The fans are hovering around ~7000 RPM. Is this normal for this switch?

ICX7250-48P Router#show flash
Stack unit 1:
Compressed Pri Code size = 29826604, Version:08.0.80eT213 (SPR08080e.bin)
Compressed Sec Code size = 31678660, Version:08.0.30hT213 (SPR08030h.bin)
Compressed Boot-Monitor Image size = 786944, Version:10.1.14T215
Code Flash Free Space = 1816481792

ICX7250-48P Router#show chassis
The stack unit 1 chassis info:
Power supply 1 (AC - PoE) present, status ok
Power supply 2 not present
Power supply 3 not present
Fan 1 ok, speed (auto): [[1]]<->2
Fan 2 ok, speed (auto): [[1]]<->2
Fan 3 ok, speed (auto): [[1]]<->2
Fan controlled temperature:
Rule 1/2 (MGMT THERMAL PLANE): 59.0 deg-C
Rule 2/2 (AIR OUTLET NEAR PSU): 21.0 deg-C
Fan speed switching temperature thresholds:
Rule 1/2 (MGMT THERMAL PLANE):
Speed 1: NM<-----> 95 deg-C
Speed 2: 85<----->105 deg-C (shutdown)
Rule 2/2 (AIR OUTLET NEAR PSU):
Speed 1: NM<-----> 41 deg-C
Speed 2: 34<----->105 deg-C (shutdown)
Fan 1 Air Flow Direction: Front to Back
Fan 2 Air Flow Direction: Front to Back
Fan 3 Air Flow Direction: Front to Back
Slot 1 Current Temperature: 59.0 deg-C (Sensor 1), 21.0 deg-C (Sensor 2)
Slot 2 Current Temperature: NA
Warning level.......: 100.0 deg-C
Shutdown level......: 105.0 deg-C


ICX7250-48P Router#dm fan-speed
Fan 1 Speed at 7180 RPM.
Fan 2 Speed at 7458 RPM.
Fan 3 Speed at 7258 RPM.


Thanks for everyone's time.

Yes.
I have 7250-24/P. The fans run avg 6500 rpm but it's very quiet.
Your is 48 port so fan runs a bit faster.

 

[Wolfstar]

These "stacking" models only take the 2-port 10gbE XFP card. Search ebay for "ES4625M" to find these cards around $40. This is the easiest option if you want a good lab switch with 2x 10gbE ports for around $70 total.

Correction on this. The Stacking models take either the FCX-2XG or the FCX-2SFPP, which is a 2-port SFP+ 10G uplink model. However, there is precisely one that I can find on eBay, and they're gouging people for $995. So if you actually WANT available ports, just buy XFPs and some fiber. Tomorrow I'll see if any of my guys have modules left over that can take a solid pic for me and see if I can find an alternate search key. (Source: some thousand or so FCX-648 switches scattered about our "enterprise" using the term loosely. We've been upgrading them for the last 3+ years with FCX-2SFPP modules for 10G uplinks.)

Also, please for the love of all that is sacred, DO NOT UNDER ANY CIRCUMSTANCES buy FCX switches capable of PoE. There are two reasons for this.

First, Fohdeesha isn't overstating the noise - we had one in a closed cabinet two rooms away from the nap room of one site's daycare center, and we had to remove it because the littles couldn't sleep because it was too loud.

Second, I personally have seen two, and I've seen photos and talked to the people who had to deal with another EIGHT, PoE FCX switches that caught fire. Specifically, their PoE boards fried - literally - and were blackened and charred. (Switches were still running, too, surprisingly. No PoE loads on them either.)

Trust me, PoE FCX switches are bad juju. Don't do it. I love the FastIron stuff for the most part, but stay away from FCX PoE.