Drag to reposition cover

Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

arglebargle

H̸̖̅ȩ̸̐l̷̦͋l̴̰̈ỏ̶̱ ̸̢͋W̵͖̌ò̴͚r̴͇̀l̵̼͗d̷͕̈
Jul 15, 2018
657
241
43
That looks the same as mine (minus the ACLs and the SSH access group) - eyeballing it I would think it would work as long as you are coming from a 192.168.17.0/24 address (I do all my ACLs on my untangle firewall)
Yeah, access was a bit weird. My desktop (Win10) on vlan1/192.168.17 couldn't ssh or access https, nor could my chromebook on vlan20/10.0.20. But my little arm box that I host docker containers on could (vlan1/192.168.17.) so I ended up bouncing through that to add the ACLs. Meanwhile I've had no problems accessing my second switch (6450-24, non PoE) from any of my non-firewalled vlans or subnets.

I factory reset both switches and upgraded to the most recent firmware before configuring them, so they should be completely clean.
 

Zervun

Member
Feb 2, 2019
44
9
8
Oregon
Yeah, access was a bit weird. My desktop (Win10) on vlan1/192.168.17 couldn't ssh or access https, nor could my chromebook on vlan20/10.0.20. But my little arm box that I host docker containers on could (vlan1/192.168.17.) so I ended up bouncing through that to add the ACLs. Meanwhile I've had no problems accessing my second switch (6450-24, non PoE) from any of my non-firewalled vlans or subnets.

I factory reset both switches and upgraded to the most recent firmware before configuring them, so they should be completely clean.
Have you tried to run nmap against the switch IPs to see what ports are open (or telnet to the ports)? verifying that 22/443 are open (without the ACLs unless you have the deny all there, which would then require the ACLs, as you have the deny, you can remove the access list)
 

exvarkin

New Member
Apr 13, 2017
7
0
1
29
My 6610 arrived and I was successful in getting it up and running thanks to the info from this thread!

Has anyone tried to replace the fans in the two fan modules? I know I read somewhere in the thread that replacing the psu fan wouldn't really be possible. I'm hoping they can be replaced seeing as the switch runs with just one module, but I guess its possible it'll reject slower/out of expected parameter fans and not use the modified fan module at all?

I'm currently running with one PSU and one fan module to see how the temps react. Think it may be feasible for light home use.

my temps after about a day
Code:
Fan 1 not present
Fan 2 ok, speed (auto): [[1]]<->2

Fan controlled temperature: 51.5 deg-C

Fan speed switching temperature thresholds:
                Speed 1: NM<----->78       deg-C
                Speed 2:       73<-----> 87 deg-C (shutdown)

Fan 2 Air Flow Direction:  Front to Back
MAC 1 Temperature Readings:
        Current temperature : 51.5 deg-C
CPU Temperature Readings:
        Current temperature : 42.5 deg-C
sensor A Temperature Readings:
        Current temperature : 24.5 deg-C
sensor B Temperature Readings:
        Current temperature : 42.5 deg-C
sensor C Temperature Readings:
        Current temperature : 22.0 deg-C
sensor D Temperature Readings:
        Current temperature : 17.5 deg-C
stacking card Temperature Readings:
        Current temperature : 47.5 deg-C
        Warning level.......: 84.0 deg-C
        Shutdown level......: 87.0 deg-C
Side note, does anyone know if you can find a psu block off plate? My unit only had one PSU and the other side is just an open hole. I'm currently using a makeshift block (cardboard and tape :/) to not have it wide open and change the airflow too much, but it would be nice to get a proper cover if its not absurdly expensive.
 

Zervun

Member
Feb 2, 2019
44
9
8
Oregon
My 6610 arrived and I was successful in getting it up and running thanks to the info from this thread!

Has anyone tried to replace the fans in the two fan modules? I know I read somewhere in the thread that replacing the psu fan wouldn't really be possible. I'm hoping they can be replaced seeing as the switch runs with just one module, but I guess its possible it'll reject slower/out of expected parameter fans and not use the modified fan module at all?
I haven't looked inside at them but they might be the same as the PSU fans. Since they went through the logic of really monitoring the PSU fans amps/rpms or whatever it is exactly they probably did that with the chassis fans as well even if they are different fan models. The 6610's and I'm guessing more so with the POE/48 port ones put out a decent amount of heat (potentially quite a bit with POE enabled on all the ports). It is a lot of ports slammed into a fairly small space on them considering it also has quite a few 10g's and a few 40g's ports (again I'm assuming those asics run a bit hotter than 1g ports).

All speculation on my part.

Below are my readings on one of my switches with 2x PS, 2x Fans, with only one POE port running current (migrating other things over to it), ~20 other ports, both 40g's used. In a crawlspace that is at 57 deg with lots of ventilation. Not hot, but isn't exactly what I would consider cool on some of the sensors (Mac 2/Cpu). Have not tested heat under high load with transferring my Milli Vanilli Mp3 collection and live concert videos over 40G yet so this is at idle/general usage for the entire home with a bunch of IoT devices, servers and wife is watching netflix right now.

Fan speed switching temperature thresholds:
Speed 1: NM<----->76 deg-C
Speed 2: 71<-----> 80 deg-C (shutdown)

Fan 1 Air Flow Direction: Front to Back
Fan 2 Air Flow Direction: Front to Back
MAC 1 Temperature Readings:
Current temperature : 40.0 deg-C
MAC 2 Temperature Readings:
Current temperature : 47.0 deg-C
CPU Temperature Readings:
Current temperature : 53.5 deg-C
sensor A Temperature Readings:
Current temperature : 40.5 deg-C
sensor B Temperature Readings:
Current temperature : 42.0 deg-C
sensor C Temperature Readings:
Current temperature : 31.5 deg-C
 

kapone

Well-Known Member
May 23, 2015
1,056
627
113
It's even worse. The fans in the power supplies are controlled by the switch over i2c as well. If you take apart one of the power supplies (and I did), the fan/power led of the PSU connects to a header inside the PSU. If you remove this wire, the PSU won't turn on... :)

If you replace the fan IN the PSU with a slower one, the PSU will turn on, but the switch won't boot...'cause..well Brocade said FU when it came to silencing this switch.

That said, in all fairness, the 6610 is 1U switch with almost o.5tbps of switching bandwidth. That's a lot. The 6610 does get seriously hot, and probably needs these screamers, unless they change the chassis design. After letting the switch run for a few hours, open it up, and you can barely touch the heatsinks for more than a few seconds. They are that hot and that's WITH the screamers. If you even manage to replace the fans with something slower, there's a better than even chance, you'll cook the switch.

I went a slightly different route to silence mine. After realizing I can't change the fans, can't remove them, can't slow them down...well..it's all about airflow and temps, so give the switch more airflow and lower the temps WITH all stock fans. That will make the switch happy, and it'll slow down the fans internally.

So...I cut apart the top cover to mount THREE 120mm fans pretty much covering the front (or back, depending on how you look at it) 3rd of the width. These are nothing special (Yate Loon fans I had lying around) and standard 3 pin 12v fans. Now came the question of how to power them from the switch.

The PSU I had hacked earlier...hacked it even more and soldered wires internally on the big connector (that mates to the switch) to get 12v and GND out. Then routed these two wires carefully out the back of the PSU (next to the power LED), so that I can connect them to the fan header for the 3 fans.

It's pretty "hacky", but I had to get the noise down, my home office is about 10ft from where the rack is and even with a wall in between, in stock configuration, I could hear it...a lot...even on the first floor. :)
Quoting this once again...

TLDR - Can't change/mod the fans in a 6610, in the PSU or the fan trays.
 

exvarkin

New Member
Apr 13, 2017
7
0
1
29
Quoting this once again...

TLDR - Can't change/mod the fans in a 6610, in the PSU or the fan trays.
I knew I read that post, but failed to read/remember the one a few above it:

i2c...already been down this road. And even then, the switch is "hard coded" for the RPMs of the stock fans. Ask me how I know... :) Hacked apart one of the fan assemblies and put in slower fans...switch won't boot.
Thanks for pointing it out, apologies for my ignorance.

I did remove the one fan modal after the switch was booted, but I never tried booting it up in that state.
 

ViciousXUSMC

Active Member
Nov 27, 2016
227
123
43
39
So the thought of paying $55 for a RMK + $10 more in shipping on eBay didn't sit well with me and today I made these (not 100% finished here).



I made it from 1/8" steel flat stock, which was way more material than I actually needed for just 1 switch.



Material:
  • 1/8" x 2" x 4' flat steel piece ($12)
  • Pack of 12 M4x.70 x 12 flat philips machine screws ($2)
  • Flat Black paint
The tools used were
  • 5" Vise
  • 4lb sledge hammer
  • 3/8" drill bit
  • 3/8" counter-sink
  • Hacksaw
  • Angle Grinder
  • Torch
  • Center Punch
I made a template with card stock and used that to transfer over the proper length and hole locations. I cut the pieces to length with the hack saw (could have also used the angle grinder I guess) then clamped them in the vise to cut off the excess width. After that I marked where the bend needed to be and placed the piece in the vise, heated it with the torch and used the sledge hammer to bend it over. I then threw it in a bucket of water to cool off and center punched the mounting holes and drilled them out. After test fitting to make sure the holes were properly placed, I counter sunk the screw holes. I cleaned the pieces off with some degreaser and painted them with some flat black paint so they won't rust.

Since I already had all the tools and the paint, my total investment was only $14. It took me about 90 minutes to get it all done, plus a few hours for the paint to dry completely.
Very Nice! I tried to do the same with another switch and just figured I could not do it without something to bend the metal (no vice) and wanted a drill press. Guess punching it first and then drilling would work with the proper metal bits.
 

arglebargle

H̸̖̅ȩ̸̐l̷̦͋l̴̰̈ỏ̶̱ ̸̢͋W̵͖̌ò̴͚r̴͇̀l̵̼͗d̷͕̈
Jul 15, 2018
657
241
43
Does anyone know how to trap chassis notifications? The Fan Up/Down notifications are making it hard to concentrate on configuring my switches.

This is what I'm talking about:

Code:
Rear Side Fan 1  is down
Rear Side Fan 1  is up
Rear Side Fan 1  is down
Rear Side Fan 1  is up
Rear Side Fan 1  is down
Rear Side Fan 1  is up
Rear Side Fan 1  is down
Rear Side Fan 1  is up
Rear Side Fan 1  is down
Rear Side Fan 1  is up
Rear Side Fan 1  is down
Rear Side Fan 1  is up
Rear Side Fan 1  is down
Rear Side Fan 1  is up
Rear Side Fan 1  is down
Rear Side Fan 1  is up
Rear Side Fan 1  is down
Rear Side Fan 1  is up
Rear Side Fan 1  is down
Rear Side Fan 1  is up

core02(config)#
core02(config)#Rear Side Fan 1  is down
Rear Side Fan 1  is up
Rear Side Fan 1  is down
Rear Side Fan 1  is up
Rear Side Fan 1  is down
Rear Side Fan 1  is up
 

fohdeesha

Kaini Industries
Nov 20, 2016
2,491
2,668
113
31
fohdeesha.com
what switch model is that? I've never seen fan up/down messages on any brocade model unless I physically pulled out a fan tray during operation, does one of your fan trays have a loose connection or something?
 

arglebargle

H̸̖̅ȩ̸̐l̷̦͋l̴̰̈ỏ̶̱ ̸̢͋W̵͖̌ò̴͚r̴͇̀l̵̼͗d̷͕̈
Jul 15, 2018
657
241
43
what switch model is that? I've never seen fan up/down messages on any brocade model unless I physically pulled out a fan tray during operation, does one of your fan trays have a loose connection or something?
It's a 6450-24. I looked at the log more closely today and realized the same thing, the switch thinks it's losing connection to the fan. I'm not sure if it's actually loose or if the fan is drawing so little current that the switch thinks it's off once in a while. I'll pull the cover off and keep an eye on it for a bit. Thanks, this was driving me nuts.
 

Blue)(Fusion

Active Member
Mar 1, 2017
130
46
28
Chicago
I got my second 6450-24P from an eBay seller that obliged me in getting me a replacement unit after having a PoE module issue. Apparently, it is self-inflicted from the firmware update. But not sure how to proceed....

The repeating console errors are:
Code:
PoE Error: Device 0 failed to start on PoE module.
PoE Error: Device 1 failed to start on PoE module.
Resetting module in slot 1 again to recover from dev fault
PoE Info: Hard Resetting in slot 1....

The old firmware is version 07.4.00bT313. When running the old firmware, I get no errors from the PoE module as it initializes correctly. When booting the new firmware, it fails with the above errors. When I try to update the PoE module firmware from the new OS firmware, the update fails. When I try to update the PoE firmware from the old OS firmware, it gives no errors but fails to update. I've waited 15 minutes. No progress or errors show on the console or in the log.

Please advise on ideas. This same problem is occurring on two ICX6450-24P switches.

Edit: The PoE firmware version is 02.1.0 Build 002.
 
Last edited:

fohdeesha

Kaini Industries
Nov 20, 2016
2,491
2,668
113
31
fohdeesha.com
it's a faulty PoE board, older firmware just wasn't as verbose in letting you know about it (you'd have to wait until you tried all the PoE ports to find out)
 

Blue)(Fusion

Active Member
Mar 1, 2017
130
46
28
Chicago
Two switches with the PoE boards bad but not showing up until the new OS firmware and then becomes unusable when trying to flash th new PoE firmware from the old OS firmware? Seems unlikely.....
 

fohdeesha

Kaini Industries
Nov 20, 2016
2,491
2,668
113
31
fohdeesha.com
very likely, PoE boards are the most common failure mode in PoE units next to power supplies, they're directly connected to external devices outdoors half the time incredibly prone to static discharge and other similar issues, I have a box of dead PoE daughterboards. As mentioned, they were still faulty on the old OS, v7 just doesn't let you know about I2C bus errors. Load up v7 and try all the PoE ports for functionality, guarantee you find dead ones (if not all of them). The fact you couldn't dump a firmware image to it double guarantees it, that's as simple as writing to the PD controller EEPROM - if it can't even make that happen, the PD chip is #1 dead
 

Blue)(Fusion

Active Member
Mar 1, 2017
130
46
28
Chicago
It seems very weird that there were zero errors from the old PoE and OS firmware when testing at first (although no PoE devices plugged in) and went downhill with the new OS firmware. It became a brick when trying to flash the new PoE firmware from the old OS firmware.

With that said, whether the damage was there prior to the attempt, I can't be sure, but oh boy there's damage:

Burnt out PoE modules ICX6450
 

fohdeesha

Kaini Industries
Nov 20, 2016
2,491
2,668
113
31
fohdeesha.com
It's because as I said v7 does not report low level errors like this to the user. They go into a debug log that you can only view with a `dm debug` something or other command I can't remember off the top of my head. I can 100% promise a firmware upgrade did not blow a f**king hole in your PoE IC :p

looks exactly like the chips I got on units that survived indirect strikes on cameras outdoors, clean hole
 

Ouraing

Member
Dec 31, 2018
25
27
13
On the 6450 I had with a bad daughterboard you could configure a port to be enabled and it wouldn't work, or you could disable one of the ports that did work but it would still be enabled and power up a device. It seemed to be frozen at whatever previously working configuration it had, I couldn't control the state of any of the ports at all.
 

arglebargle

H̸̖̅ȩ̸̐l̷̦͋l̴̰̈ỏ̶̱ ̸̢͋W̵͖̌ò̴͚r̴͇̀l̵̼͗d̷͕̈
Jul 15, 2018
657
241
43
Two switches with the PoE boards bad but not showing up until the new OS firmware and then becomes unusable when trying to flash th new PoE firmware from the old OS firmware? Seems unlikely.....
Something to think about - these two switches likely came from the same environment, so it's totally possible that they arrive with similar wear/damage.
 

tjk

Active Member
Mar 3, 2013
380
122
43
www.servercentral.com
Hey Folks,

Can someone link me to the console cable needed for the 7750 switches? I know it is in this thread somewhere, but I can't search all 75 pages and don't feel like scrolling through them.

Thanks!