Drag to reposition cover

Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

anomaly

Active Member
Jan 8, 2018
228
45
28
If any of you can tell me which UK seller you got the faulty units from, I would like to share my experience too and confirm if it is the same.
Got a 48P from the UK today, it runs an ancient rev from 2012 (hooray for sysadmins who don't care about running old *** **** that is bugged and can expose things to compromise). The PWM whine is insane after a fan swap using a Nidec from a 24 unit (no PoE) and a faster model fo Sunon that can go down to dead silent. I observed some whine with the 24 model when using lower speed fans, but this one is something else.

This is my chassis information:

Code:
1#show chassis
The stack unit 1 chassis info:

Power supply 1 (NA - AC - Regular) present, status ok
Power supply 2 not present
Power supply 3 not present

Fan 1 failed
Fan 2 failed
Fan 3 failed

Fan controlled temperature: 39.0 deg-C

Fan speed switching temperature thresholds:
                Speed 1: NM<----->61       deg-C
                Speed 2:       56<-----> 79 deg-C (shutdown)

Sensor B Temperature Readings:
        Current temperature : 34.0 deg-C
Sensor A Temperature Readings:
        Current temperature : 39.0 deg-C
        Warning level.......: 69.0 deg-C
        Shutdown level......: 79.0 deg-C
Regarding custom u-boot: I'm really interested on altering the fan speeds. I can probably reduce whine if I get the Sunons to operate at full speed. The Nidec is OK but it does have the characteristic PWM whine at lower speeds, still nothing like the PWM controller itself in either the PSU or main board.

I would bet money it is either EEPROM stored or I2C control.

Another option is fooling the PWM driver to output full throttle and then disregard it (using something between the fan and the controller).
 

mkstrom

New Member
Sep 6, 2018
2
1
3
Hi

I recently got my ICX6450-48P and followed @fohdeesha excellent guide to update it. I also bought my switch from a UK seller, and suspect that it might be the same seller that has been mentioned in regards to misbehaving fans. I do however think that my fans behave normally. They start out really loud for a short time during boot, but calm down quite a bit after boot is complete. I wouldn't say that they are quiet, but they aren't loud either.

This is my chassis information after all updates.

Code:
ICX6450-1>show chassis
The stack unit 1 chassis info:

Power supply 1 (NA - AC - PoE) present, status ok
Power supply 2 not present
Power supply 3 not present

Fan 1 ok, speed (auto): [[1]]<->2
Fan 2 ok, speed (auto): [[1]]<->2
Fan 3 ok, speed (auto): [[1]]<->2

Fan controlled temperature: 35.0 deg-C

Fan speed switching temperature thresholds:
                Speed 1: NM<----->65       deg-C
                Speed 2:       56<-----> 79 deg-C (shutdown)

Sensor B Temperature Readings:
        Current temperature : 32.0 deg-C
Sensor A Temperature Readings:
        Current temperature : 35.0 deg-C
        Warning level.......: 69.0 deg-C
        Shutdown level......: 79.0 deg-C
I also have an additional question regarding licensing. Below is my output from the "show license" command, am I reading it right that the last number under "Lic capacity", a 2, means that only 2 of the SFP+ ports are in 10G mode?

Code:
ICX6450-1>show license
Index    Lic Mode        Lic Name               Lid/Serial No  Lic Type    Status     Lic Period    Lic Capacity
Stack unit 1:
1        Node Lock       ICX6450-10G-LIC-POD    xxxxxxxxxxx    Normal      Active     Unlimited         2
ICX6450-1>
 
  • Like
Reactions: tony&

kapone

Well-Known Member
May 23, 2015
1,059
635
113
Each Brocade 10G license is a "4 pack" and is applicable to 4 ports. So the quantity of 2 is exactly what it should be.
 

fohdeesha

Kaini Industries
Nov 20, 2016
2,583
2,775
113
31
fohdeesha.com
he's on the 6450 - 2 out of the 4 ports come unlocked by default, the quantity of 2 unlocks the last 2 ports, so you have all 4 unlocked total. You're missing the premium routing license that unlocks stuff like ospf and ipv6 routing etc, if you want that shoot me a pm
 

kapone

Well-Known Member
May 23, 2015
1,059
635
113
he's on the 6450 - 2 out of the 4 ports come unlocked by default, the quantity of 2 unlocks the last 2 ports, so you have all 4 unlocked total. You're missing the premium routing license that unlocks stuff like ospf and ipv6 routing etc, if you want that shoot me a pm
I need more coffee...
 

ljvb

Member
Nov 8, 2015
97
32
18
46
So.. pointed here from my request helping to decide between cisco hp (and a few other) switches.. and apparently after noting that there is still active development on the Brocades, I'm sold. Will be picking up 2 6610 48p switches.. way overkill on the port numbers, but they are cheaper than the 6450-24p for some reason.. in fact in general the 48 port ones seem cheaper than the 24p ones...

Anyways. Just throwing my 2 cents in and subscribing to the thread.
 
  • Like
Reactions: Emanuele

fohdeesha

Kaini Industries
Nov 20, 2016
2,583
2,775
113
31
fohdeesha.com
So.. pointed here from my request helping to decide between cisco hp (and a few other) switches.. and apparently after noting that there is still active development on the Brocades, I'm sold. Will be picking up 2 6610 48p switches.. way overkill on the port numbers, but they are cheaper than the 6450-24p for some reason.. in fact in general the 48 port ones seem cheaper than the 24p ones...

Anyways. Just throwing my 2 cents in and subscribing to the thread.
You won't be disappointed, if you've read the 30 page history, I don't think anyone has been disappointed now that I think about it

I remember you saying in your other thread you were weary of learning brocade's OS and are familiar with cisco - I wouldn't worry too much, brocade fastiron is something like 80% identical to cisco IOS.

And yup, they're all certainly still under active dev, the 8030sa firmware release was just earlier this month
 

fohdeesha

Kaini Industries
Nov 20, 2016
2,583
2,775
113
31
fohdeesha.com
How do I do "this" ? ...... Brocade 6450 ........ I would like to set up Brocade 6450's Port 20 to be a port that only has LAN access but no WAN access. So that way, whatever I plug in there will be on the LAN, but cannot have any internet access.

Is this possible?
I haven't forgotten about this, just been crazy week with all the IDRAC stuff. Will hopefully have time to do this tomorrow, but for now I'm going to sleep for approximately the rest of the weekend
 

anomaly

Active Member
Jan 8, 2018
228
45
28
BTW, to get clean 12V DC, what solder points on the top of the board and connectors are exposed that wouldn't involve damaging anything in the existent cabling? Splicing works, but it's a little ghetto.

I'm going to make a small board that leaves GND and PWM pins connected to the original slots in the fans, but takes voltage directly from the PSU, so they spin at full speed (these are swapped fans with lower RPMs, not ultra silent as you still need good thermal, but much quieter than the Foxconn fans in my 48P) and keep the PWM components happy. It is my suspicion that brocade actually "value-designed" those, hence why the ceramic caps and coils whine when using lower RPM fans. Only one kind of capacitor is immune to acting as a piezo electric speaker... and they are certainly not used here.
 

kapone

Well-Known Member
May 23, 2015
1,059
635
113
Won't work. The RPMs of the fans have to be within range of "what's expected", otherwise the switch throws up.

Edit: On the 6610 that is.
 

sean

Member
Sep 26, 2013
66
33
18
CT
Ever since swapping out the fans, I've been unable to use the management interface. I have only the 6450 so I plugged the management port into the switch and that was fine until the fan swap. The interface will get an IP via DHCP but I still can't access it. I moved it to a vlan with no change. I can access another device on that vlan though. Is it possible to have the management port plugged into the switch itself for remote administration? I've been doing everything via the console and it's a pain.
 

fohdeesha

Kaini Industries
Nov 20, 2016
2,583
2,775
113
31
fohdeesha.com
Is it possible to have the management port plugged into the switch itself for remote administration? I've been doing everything via the console and it's a pain.
Don't do this, it's designed to be used on a totally seperate domain. it shares a MAC address with the first VE, so if you plug it into itself you'll get collisions. If you followed the update guide, you should have no need for the management port as you gave the VE an IP address, that you can now access from any of the normal ports via ssh/telnet/etc

If the management port was physically linking up but you couldn't access it, it was user/config error. If you were plugging it into the same network as the rest of your stuff, that would be why. You also can't tag or otherwise do anything with VLANs on the management port, so that tells me you were definitely doing something else
 

anomaly

Active Member
Jan 8, 2018
228
45
28
Won't work. The RPMs of the fans have to be within range of "what's expected", otherwise the switch throws up.

Edit: On the 6610 that is.
Yeah, I gave no indication about using a 6610, I only use 6450s here. Different animal. I also implicitly assume someone swapping fans will be doing his homework and verifying that the thermal conditions of the switch are similar before and after the swap. Or that he adequately calculates or verifies that different workloads won't vastly affect thermal (ex. 48P with full PoE I could see a lot of heat being generated from the wasted power wherever conversion isn't 100% efficient... which is nowhere, so a watt or two times 48 will add up very, very quickly). Your average 10-13 watt 10G NIC (SFP+, 10GBASE-T is worse) gets up to 70C or more without good airflow. Imagine that in 1U, several times over.

Anyhow, once we establish we aren't doing something retarded, we can do the following:

  • Spoof PWM signal with a small MCU that simply fools the RPM reading into thinking everything is peachy. Because the switches only have two modes afaik, and I don't see much in the way of fine tuning of the speed, we could have two or three speeds, that translate to slower ones from the "OK" assumed speeds of the switch.
  • Simply wire PWM and GND to the headers, and supply voltage straight out of a 12V source. The readings will report a constant speed but this won't cause the PWM value-designed drivers to whine. A small board for this can be prototyped easily.
There are some PWM controllers off Aliexpress and eBay and other internet dollar stores that take 12V in. You would just need to solder some fan headers that expose the PWM/GND pins and connect those. I have one so I might give it a go. They also have PWM control via temperature, but it seems without altering the fw or caps/resistors the temp values are fixed.

@fohdeesha, since this is a mildly annoying trait of our beloved switches, do you want to work with me on whipping out something people can put together on their own? (possibly under a non commercial license to avoid earthworm tactics). This could also help with other switches.

Oh, did I say don't f*ck up the thermal by putting fans unable to exhaust enough air volume for your particular environment? Good news is, these switches have thermal cutoff safeguards. You can't bake them. They cut off way before reaching any temperatures dangerous to the ICs.

Is it possible to have the management port plugged into the switch itself for remote administration? I've been doing everything via the console and it's a pain.
Unless you have a dedicated management network not connected to anything else, with no conflicts with the rest, I suggest you use the mgmt port as a fallback/emergency port when you have lost access to the switch somehow, and can combine that with serial access for fixing things, and also for firmware updates. I use the mgmt for initial config and bootstrapping, and leave it assigned with an IP in the range of 10.90.250.x, so I can just configure a laptop with static IP and get TFTP up and running.

You can create a virtual interface in a management VLAN if you are so inclined and have unlimited/near unlimited IPs bound to the switch per VLAN for management if you need to. Honestly, if someone has a protocol level vulnerability in your network gear, you are SOL no matter what containment measures you have taken. I had a conversation in that juvenile labbing community we all have heard of, with someone claiming he could protect Cisco IOS via policy and blah blah. Before he ran out of arguments (that were compsci and OS design 101: if someone executes code at ring 0 you are screwed, end of the game, nothing further beyond, even with a security coprocessor it would still need to be able to authenticate whatever code you are running, on realtime) and claimed that his company "controlled CPU execution in IOS" (LOL hmmkay!), I commented on how IOS mitigations have been bypassed throughout the years (as late as past year there is a public talk in CCC showcasing a type of ROP against IOS_.

FastIron has no mitigations of any kind. It's a barebones ARM based Linux under the hood. Leave your management facilities out of band if paranoid. Be conservative with SNMP and that's that. Console access also yields password override: bootloader can disable password checks in the image.

Hopefully this clears out any questions with "mom and pops" "hardening" ideas.
 
Last edited:

sean

Member
Sep 26, 2013
66
33
18
CT
More switches won't increase reading comprehension.
It was a joke based on homelabbers propensity to acquire more stuff. I've dealt with datacenter-class switches before but someone else handled the wiring and assigning of IPs and I took it from there. I'm new to wiring up switches, so I'm going to do dumb stuff at some point. Pointing out where in the documentation it says what to do is a lot more helpful than insulting people.
 

anomaly

Active Member
Jan 8, 2018
228
45
28
It was a joke based on homelabbers propensity to acquire more stuff. I've dealt with datacenter-class switches before but someone else handled the wiring and assigning of IPs and I took it from there. I'm new to wiring up switches, so I'm going to do dumb stuff at some point. Pointing out where in the documentation it says what to do is a lot more helpful than insulting people.
My response was a joke, not an insult or anything intended to offend. I am familiar with the propensity to waste money and time on sometimes absolutely dead-from-the-get-go ideas (the other day I heard someone talking about buying multiple RPIs to "simulate networking"... as soon as I told him to use GNS or something similar and forego the dumb idea for his wallet and sanity's sake (after all, after the boards he will need the stack of Cisco gear and what not...... and the cabling, and, and....), a bunch of people egged him on to do it anyway.

Anyhow, my point was simple: forget the mgmt port unless you need emergency access to TFTP and the like. Create a VE interface for an isolated VLAN only used for the purpose of managing the network gear, and you are good to go.

I'm also not a network guy by trade, and I have dumb questions every so often. I will probably ask here about L3 routing between vlans soon, as I want to avoid hitting the firewall or router for simple inter-vlan routing that can be done in FastIron with the L3 capabilities. Since I need to use ACLs there, I might end up needing assistance with PVLANs too.
 

mixmansc

Member
Feb 15, 2016
46
26
18
Quick one on the 6610. Can the fan speed thresholds be changed? I did not see the fan-threshold command in the cli but I might be looking in the wrong place. On mine, I put it in the rack today and about every 10 minutes the fans kick up to high for about 5 seconds. Does not take them long to cool it back down. My closet where I have the switch sits at about 80 degrees F.

It appears to be the MAC 2 temp that is triggering it. I'm hoping I can raise the thresholds by about 5 to 10 degrees. Worst case I'll deal with the noise this week. No time to mess with pulling it back out until next weekend. :( I might have to take it down then and re-do the heatsinks with some decent tim like AS5. The stock paste is likely rubbish and I can probably shave a few degrees there and maybe even figure a way to put a larger heatsink on. The CPU temp is also getting up there too. I'm really not worried about the shutdown temp (the fans almost instantly cool it back below the high fan threshold) but I'd like to raise that to probably 90c if possible. When they kick on the temps quickly drop to about 72c.

By today's standards 80 degrees is actually not that bad for modern datacenters. Many have started letting them go a bit warmer than in the past as it saves a ton on energy and the hardware, realistically can handle it just fine. This 6610 is a hot running bugger. I can hear those fans kick up to high from several rooms away. lol

Another thought.... Maybe I can find a second fan module. That might help it too. It has a slot for it.

Code:
The stack unit 1 chassis info:

Power supply 1 (AC - Regular) present, status ok
        Model Number:   23-0000144-01
        Serial Number:  019
        Firmware Ver:    A
Power supply 1 Fan Air Flow Direction:  Front to Back
Power supply 2 (AC - Regular) present, status ok
        Model Number:   23-0000144-01
        Serial Number:  0GR
        Firmware Ver:    A
Power supply 2 Fan Air Flow Direction:  Front to Back

Fan 1 ok, speed (auto): [[1]]<->2
Fan 2 not present

Fan controlled temperature: 80.0 deg-C

Fan speed switching temperature thresholds:
                Speed 1: NM<----->80       deg-C
                Speed 2:       75<-----> 85 deg-C (shutdown)

Fan 1 Air Flow Direction:  Front to Back
MAC 1 Temperature Readings:
        Current temperature : 60.5 deg-C
MAC 2 Temperature Readings:
        Current temperature : 80.5 deg-C
CPU Temperature Readings:
        Current temperature : 76.5 deg-C
sensor A Temperature Readings:
        Current temperature : 60.5 deg-C
sensor B Temperature Readings:
        Current temperature : 66.0 deg-C
sensor C Temperature Readings:
        Current temperature : 57.0 deg-C
stacking card Temperature Readings:
        Current temperature : 58.5 deg-C
        Warning level.......: 82.0 deg-C
        Shutdown level......: 85.0 deg-C
 
Last edited: