Drag to reposition cover

Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

vangoose

Active Member
May 21, 2019
268
70
28
Canada
Has anyone setup snmp to monitor temps for their 7250? I'm having a tough time trying to figure out which OID to use.

snChasActualTemperature .1.3.6.1.4.1.1991.1.1.1.1.18 returns an integer of 163 and the description states each unit is 0.5 degrees Celcius but when doing the math doesn't add up to what sh chassis has for either the mgmt plane or the psu temps.

What am I missing?
I use Zabbix to monitor with Brocade Foundry template.
 

3nodeproblem

Member
Jun 28, 2020
42
11
8
This is probably a stupid question but don't find it mentioned in the thread: In absence of a serial cable (still waiting for delayed delivery out in the sticks), is there any management that can be done by default on 6450?

Tried connecting the mgmt port to a computer running a dhcp server - I see it getting an IP address assigned and can ping it but it doesn't seem any TCP ports are open, and only UDP 67/68.
 

PnoT

Active Member
Mar 1, 2015
609
141
43
Texas
I use Zabbix to monitor with Brocade Foundry template.
Thanks for the template I found a few items in there. I figured out why the numbers weren't adding up... I have 2 brocades and was pulling info from the wrong one so the temps didn't match oops.
 

Dave Corder

Active Member
Dec 21, 2015
119
49
28
38
I recently acquired another ICX6610-48P from eBay for dirt cheap. It was sold as being bricked ("It stopped booting fully after a firmware upgrade"), but that's fine with me - my primary reason for purchasing it was to swap the 40 GbE daughtercard into my existing, functional 6610 that has one defective 40 GbE port.

Just for grins, though (and before I go through all the effort of taking both of them apart), I pulled a PSU and fan tray from my working unit and stuck it in this one to see what state it's in.

It's stuck in a boot loop of some kind. The PSU fans rev up upon having AC applied, run for a couple seconds, rev down for a few seconds more, and then rev back up again. Repeat x inifinity.

On the console, this is what I get (over and over again). Hitting 'a' or 'b' as prompted looks like they work, but they do nothing to interrupt the boot looping - it just reboots again a few seconds later.

Code:
ICX Boot Code Version 10.1.00 (grz10100)
Enter 'a' to stop at memory test
Enter 'b' to stop at boot monitor
Boot(Mem-Test)>aaaaaaaaaaaaaaa$
ICX Boot Code Version 10.1.00 (grz10100)
Enter 'a' to stop at memory test
Enter 'b' to stop at boot monitor
Boot(Mem-Test)>bbbbbbb$
ICX Boot Code Version 10.1.00 (grz10100)
Enter 'a' to stop at memory test
Enter 'b' to stop at boot monitor
***** Interrupted by entering 'b' *****
.$
ICX Boot Code Version 10.1.00 (grz10100)
Enter 'a' to stop at memory test
Enter 'b' to stop at boot monitor
***** Interrupted by entering 'b' *****
.$
ICX Boot Code Version 10.1.00 (grz10100)
Enter 'a' to stop at memory test
Enter 'b' to stop at boot monitor
Any hope of recovering this short of JTAG?
 

fohdeesha

Kaini Industries
Nov 20, 2016
2,003
1,824
113
29
fohdeesha.com
I recently acquired another ICX6610-48P from eBay for dirt cheap. It was sold as being bricked ("It stopped booting fully after a firmware upgrade"), but that's fine with me - my primary reason for purchasing it was to swap the 40 GbE daughtercard into my existing, functional 6610 that has one defective 40 GbE port.

Just for grins, though (and before I go through all the effort of taking both of them apart), I pulled a PSU and fan tray from my working unit and stuck it in this one to see what state it's in.

It's stuck in a boot loop of some kind. The PSU fans rev up upon having AC applied, run for a couple seconds, rev down for a few seconds more, and then rev back up again. Repeat x inifinity.

On the console, this is what I get (over and over again). Hitting 'a' or 'b' as prompted looks like they work, but they do nothing to interrupt the boot looping - it just reboots again a few seconds later.

Code:
ICX Boot Code Version 10.1.00 (grz10100)
Enter 'a' to stop at memory test
Enter 'b' to stop at boot monitor
Boot(Mem-Test)>aaaaaaaaaaaaaaa$
ICX Boot Code Version 10.1.00 (grz10100)
Enter 'a' to stop at memory test
Enter 'b' to stop at boot monitor
Boot(Mem-Test)>bbbbbbb$
ICX Boot Code Version 10.1.00 (grz10100)
Enter 'a' to stop at memory test
Enter 'b' to stop at boot monitor
***** Interrupted by entering 'b' *****
.$
ICX Boot Code Version 10.1.00 (grz10100)
Enter 'a' to stop at memory test
Enter 'b' to stop at boot monitor
***** Interrupted by entering 'b' *****
.$
ICX Boot Code Version 10.1.00 (grz10100)
Enter 'a' to stop at memory test
Enter 'b' to stop at boot monitor
Any hope of recovering this short of JTAG?
I don't even think jtag would fix that, it seems the bootloader is intact but something has gone wrong enough with the CPU, RAM, or something else that's it's going into reset as soon as it tries to run anything. you could try reseating the RAM as a last hope
 

Dave Corder

Active Member
Dec 21, 2015
119
49
28
38
I don't even think jtag would fix that, it seems the bootloader is intact but something has gone wrong enough with the CPU, RAM, or something else that's it's going into reset as soon as it tries to run anything. you could try reseating the RAM as a last hope
Alrighty, I guess I'm popping off the lid and having a look inside...
 

Dave Corder

Active Member
Dec 21, 2015
119
49
28
38
I don't even think jtag would fix that, it seems the bootloader is intact but something has gone wrong enough with the CPU, RAM, or something else that's it's going into reset as soon as it tries to run anything. you could try reseating the RAM as a last hope
Looks like it was actually a bad PSU I was using. I had two Rev. A PSUs sitting in my shelf and I knew one of them was a dud. I assumed it was the one with the bent handle (you know, the physically damaged one...) and tried the other one. On a whim today during my lunch break I tried the other one, and it started booting!

Although it does seem to be getting stuck a little ways into the boot process. This is what I get:

Code:
$
ICX Boot Code Version 10.1.00 (grz10100)
Enter 'a' to stop at memory test
Enter 'b' to stop at boot monitor
BOOT INFO: load monitor from boot flash, cksum = 71f1
BOOT INFO: verify flash files..........
BOOT INFO: load image from primary copy...

platform type = 12
PCIE-1 LTSSM status: 22
PCIE Switch status: 0
.............................
Firmware integrity checksum passed
.......
Starting Main Task ...CPSS DxCh Version: cpss3.4p1 release
Pre Parsing Config Data ...
Gonna tinker a little bit here before I get back to my day job...

Edit: apparently I just needed to be more patient. It continued startup pretty much as soon as I posted my reply.
 
  • Like
Reactions: fohdeesha

fohdeesha

Kaini Industries
Nov 20, 2016
2,003
1,824
113
29
fohdeesha.com
yeah I dunno what it is but the 6610's boot sloooowwww. The FCX has the same CPU and has to unpack the exact same firmware to memory and it boots like 5x faster
 

i386

Well-Known Member
Mar 18, 2016
2,120
562
113
31
Germany
The FCX has the same CPU and has to unpack the exact same firmware to memory and it boots like 5x faster
Is it unpacking with the same settings (all cores/threads vs single thread)?
Is the boot device connected in the same way?
Is the boot device on pair?
 

fohdeesha

Kaini Industries
Nov 20, 2016
2,003
1,824
113
29
fohdeesha.com
Is it unpacking with the same settings (all cores/threads vs single thread)?
Is the boot device connected in the same way?
Is the boot device on pair?
the management plane between the two is identical (that's why they run the same firmware and even the same bootloader image). Both are single core single thread PPC procs booting from CFI flash, the flash part number is even the same between them (S29GL512P). the main difference is the ASIC of course, perhaps the newer marvell in the 6610 takes longer to initialize
 

karpuzvepeynir

New Member
Aug 19, 2020
4
0
1
I did my brocade switches stack like I wish with fohdeesha's description below.
https://forums.servethehome.com/ind...p-powerful-10gbe-40gbe-switching.21107/page-7

But I faced a strange issue. Some rear ports 1/2/2 to 1/2/5, 1/2/7 to 1/2/10 and same ports on the second switch don’t work when they connected to servers. There is no problem with network cards of servers.

I would be very happy if you suggest anything about the issue.
 

karpuzvepeynir

New Member
Aug 19, 2020
4
0
1
This log shows what happens about not working ports:

Code:
Aug 19 18:45:51:I:System: Interface ethernet 2/2/4, state down
Aug 19 18:45:51:I:STP: VLAN 1 Port 2/2/4 STP State -> DISABLED (PortDown)
Aug 19 18:45:51:I:STP: VLAN 1 Port 2/2/4 STP State -> LISTENING (PortDown)
Aug 19 18:45:50:I:System: Interface ethernet 2/2/4, state up
Aug 19 18:45:50:I:STP: VLAN 1 Port 2/2/4 STP State -> LISTENING (MakeFwding)
Aug 19 18:44:31:I:System: Interface ethernet 2/2/4, state down
Aug 19 18:44:31:I:STP: VLAN 1 Port 2/2/4 STP State -> DISABLED (PortDown)
Aug 19 18:44:31:I:STP: VLAN 1 Port 2/2/4 STP State -> LISTENING (PortDown)
Aug 19 18:44:30:I:System: Interface ethernet 2/2/4, state up
Aug 19 18:44:30:I:STP: VLAN 1 Port 2/2/4 STP State -> LISTENING (MakeFwding)
This time STP is disabled on the port:

Code:
Aug 19 18:51:38:I:System: Interface ethernet 2/2/4, state down
Aug 19 18:51:37:I:System: Interface ethernet 2/2/4, state up
Aug 19 18:51:22:I:System: Interface ethernet 2/2/4, state down
Aug 19 18:51:22:I:System: Interface ethernet 2/2/4, state up
 

karpuzvepeynir

New Member
Aug 19, 2020
4
0
1
Posting your 6610 configuration would be helpful.
Thanks for your reply itronin.

Code:
SSH@ICX6610-24 Switch#sh run
Current configuration:
!
ver 08.0.30tT7f1
!
stack unit 1
  module 1 icx6610-24-port-management-module
  module 2 icx6610-qsfp-10-port-160g-module
  module 3 icx6610-8-port-10g-dual-mode-module
  priority 128
  stack-port 1/2/1 1/2/6
stack unit 2
  module 1 icx6610-24-port-management-module
  module 2 icx6610-qsfp-10-port-160g-module
  module 3 icx6610-8-port-10g-dual-mode-module
  stack-port 2/2/1 2/2/6
stack enable
stack mac xxxx.xxxx.xx74
!
!
!
lag xxxx dynamic id 1
 ports ethernet 1/3/8 ethernet 2/3/8
 primary-port 1/3/8
  deploy
 port-name xxxx ethernet 1/3/8
 port-name xxxx ethernet 2/3/8
!
lag xxxx dynamic id 2
 ports ethernet 1/3/3 ethernet 2/3/3
 primary-port 1/3/3
 deploy
!
lag xxxx dynamic id 3
 ports ethernet 1/3/4 ethernet 2/3/4
 primary-port 1/3/4
 deploy
!
!
vlan 1 name DEFAULT-VLAN by port
!
vlan xxxx name xxxx by port
 tagged ethe 1/3/1 ethe 1/3/8 ethe 2/3/1 ethe 2/3/8
 spanning-tree 802-1w
!
vlan xxxx name xxxx by port
 tagged ethe 1/3/1 ethe 1/3/8 ethe 2/3/1 ethe 2/3/8
  spanning-tree 802-1w
!
vlan xxxx name xxxx by port
 tagged ethe 1/1/1 to 1/1/24 ethe 1/2/2 to 1/2/5 ethe 1/2/7 to 1/2/10 ethe 1/3/1 to 1/3/7 ethe 2/1/1 to 2/1/24 ethe 2/2/2 to 2/2/3 ethe 2/2/5 ethe 2/2/7 to 2/2/10 ethe 2/3/1 to 2/3/7
 spanning-tree 802-1w
!
vlan xxxx name xxxx by port
 tagged ethe 1/1/1 to 1/1/24 ethe 1/2/2 to 1/2/5 ethe 1/2/7 to 1/2/10 ethe 1/3/1 to 1/3/7 ethe 2/1/1 to 2/1/24 ethe 2/2/2 to 2/2/3 ethe 2/2/5 ethe 2/2/7 to 2/2/10 ethe 2/3/1 to 2/3/7
 spanning-tree 802-1w
!
vlan xxxx name xxxx by port
 tagged ethe 1/1/1 to 1/1/24 ethe 1/2/2 to 1/2/5 ethe 1/2/7 to 1/2/10 ethe 1/3/1 to 1/3/7 ethe 2/1/1 to 2/1/24 ethe 2/2/2 to 2/2/3 ethe 2/2/5 ethe 2/2/7 to 2/2/10 ethe 2/3/1 to 2/3/7
 spanning-tree 802-1w
!
vlan xxxx name xxxx by port
 tagged ethe 1/1/1 to 1/1/24 ethe 1/2/2 to 1/2/5 ethe 1/2/7 to 1/2/10 ethe 1/3/1 to 1/3/7 ethe 2/1/1 to 2/1/24 ethe 2/2/2 to 2/2/3 ethe 2/2/5 ethe 2/2/7 to 2/2/10 ethe 2/3/1 to 2/3/7
 spanning-tree 802-1w
!
vlan xxxx name xxxx by port
 tagged ethe 1/1/1 to 1/1/24 ethe 1/2/2 to 1/2/5 ethe 1/2/7 to 1/2/10 ethe 1/3/1 to 1/3/7 ethe 2/1/1 to 2/1/24 ethe 2/2/2 to 2/2/3 ethe 2/2/5 ethe 2/2/7 to 2/2/10 ethe 2/3/1 to 2/3/7
 spanning-tree 802-1w
!
!
!
!
!
aaa authentication enable default local
aaa authentication login default local
boot sys fl sec
jumbo
enable telnet authentication
ip address x.x.x.x x.x.x.x
ip dns server-address 8.8.8.8
no ip dhcp-client enable
ip default-gateway x.x.x.x
!
username xxxx password .....
!
!
clock timezone gmt GMT+03
!
!
ntp
 disable serve
 server 192.36.143.130
 server 213.251.52.185
!
!
hitless-failover enable
interface ethernet 1/1/17
 dual-mode  xxxx
!
interface ethernet 1/2/1
 port-name xxxx
!
interface ethernet 1/3/1
 port-name xxxx
 speed-duplex 10G-full
!
interface ethernet 1/3/2
 speed-duplex 10G-full
!
interface ethernet 1/3/3
 speed-duplex 10G-full
!
interface ethernet 1/3/4
 speed-duplex 10G-full
!
interface ethernet 1/3/5
 dual-mode  xxxx
 speed-duplex 10G-full
!
interface ethernet 1/3/6
 speed-duplex 10G-full
!
interface ethernet 1/3/7
 speed-duplex 10G-full
!
interface ethernet 1/3/8
 port-name xxxx
 speed-duplex 10G-full
!
interface ethernet 2/2/4
 no spanning-tree
!
interface ethernet 2/3/1
 speed-duplex 10G-full
!
interface ethernet 2/3/2
 speed-duplex 10G-full
!
interface ethernet 2/3/5
 speed-duplex 10G-full
!
interface ethernet 2/3/6
 speed-duplex 10G-full
!
interface ethernet 2/3/7
 speed-duplex 10G-full
!
!
!
!
!
!
ip ssh  timeout 60
!
!
end
 

liberty

New Member
Nov 4, 2019
4
2
3
I have implemented the @RoachedCoach Mechatronics and Sunon fan mods discussed here on a ICX7250-48P. The new fans are definitely quieter, but I am not getting the results that others have. From a cold boot, the ASIC temps rise to mid-60's which is no problem, however the air outlet temps rise to the speed 2 threshold, the fans go to high and the temp drops to exactly 34c, the threshold for dropping back to speed 1. So once speed 2 becomes active, it never drops back. I tried just running with 1 fan, air outlet temps go into the 60's, but still have the one fan running at high speed. With the fan(s) running at high speed, the 7250-48P is louder than a 6610-24F with 2 Rev. A power supplies.

Code:
Fan 1 ok, speed (auto): 1<->[[2]]
Fan 2 ok, speed (auto): 1<->[[2]]
Fan 3 ok, speed (auto): 1<->[[2]]

Fan controlled temperature:
        Rule 1/2 (MGMT THERMAL PLANE): 51.7 deg-C
        Rule 2/2 (AIR OUTLET NEAR PSU): 34.0 deg-C

Fan speed switching temperature thresholds:
        Rule 1/2 (MGMT THERMAL PLANE):
                Speed 1: NM<-----> 95       deg-C
                Speed 2:        85<----->105 deg-C (shutdown)
        Rule 2/2 (AIR OUTLET NEAR PSU)***active***:
                Speed 1: NM<-----> 41       deg-C
                Speed 2:        34<----->105 deg-C (shutdown)

Fan 1 Air Flow Direction:  Front to Back
Fan 2 Air Flow Direction:  Front to Back                         
Fan 3 Air Flow Direction:  Front to Back
Slot 1 Current Temperature: 51.7 deg-C (Sensor 1), 34.0 deg-C (Sensor 2)
Slot 2 Current Temperature: NA
        Warning level.......: 100.0 deg-C
        Shutdown level......: 105.0 deg-C

Due to a comedy of seller errors, I also have a 7250-24 and 7250-24P with dead PoE. The 7250-24 with its single OEM fan runs happily in the same rack with the fan at low speed. But I am not fond of the idea of going back to PoE power bricks.

Any ideas on getting a 7250-48P cool enough so the fans run at low speed?
 

PnoT

Active Member
Mar 1, 2015
609
141
43
Texas
I have implemented the @RoachedCoach Mechatronics and Sunon fan mods discussed here on a ICX7250-48P. The new fans are definitely quieter, but I am not getting the results that others have. From a cold boot, the ASIC temps rise to mid-60's which is no problem, however the air outlet temps rise to the speed 2 threshold, the fans go to high and the temp drops to exactly 34c, the threshold for dropping back to speed 1. So once speed 2 becomes active, it never drops back. I tried just running with 1 fan, air outlet temps go into the 60's, but still have the one fan running at high speed. With the fan(s) running at high speed, the 7250-48P is louder than a 6610-24F with 2 Rev. A power supplies.

Due to a comedy of seller errors, I also have a 7250-24 and 7250-24P with dead PoE. The 7250-24 with its single OEM fan runs happily in the same rack with the fan at low speed. But I am not fond of the idea of going back to PoE power bricks.

Any ideas on getting a 7250-48P cool enough so the fans run at low speed?
I just finished doing the exact same modification this afternoon on my 7250-48P but instead of the Sunon fan I replaced it with an EFB0612LA-R00 because Digi-Key was out of stock on the Sunon.

Mods and temps:

24P - 2 fans
48P - 3 fans + ASIC

1597960298038.png

I'd love to be able to pull the fan speed RPMs via SNMP but can't find an OID :(
 
Last edited:
  • Like
Reactions: liberty

3nodeproblem

Member
Jun 28, 2020
42
11
8
Having replaces 2/3 fans with Noctua NF-A4x20 FLX, reqiring the pins as others suggested (black -> black, red -> yellow, yellow -> green), the fans show as "failed" - is this all good? The fans are spinning at a very low RPM but temps look OK (haven't gone further than idle);


Code:
ICX6450-48P Switch#show chassis

The stack unit 1 chassis info:


Power supply 1 (NA - AC - Regular) present, status ok

Power supply 2 not present

Power supply 3 not present


Fan 1 failed

Fan 2 failed

Fan 3 failed


Fan controlled temperature: 41.5 deg-C


Fan speed switching temperature thresholds:

                Speed 1: NM<----->61       deg-C

                Speed 2:       56<-----> 79 deg-C (shutdown)


Sensor B Temperature Readings:

        Current temperature : 37.0 deg-C

Sensor A Temperature Readings:

        Current temperature : 41.5 deg-C

        Warning level.......: 69.0 deg-C

        Shutdown level......: 79.0 deg-C
 

fohdeesha

Kaini Industries
Nov 20, 2016
2,003
1,824
113
29
fohdeesha.com
it's probably showing failed because the fans are spinning slower than the "failed" threshold RPM. to check if it's still getting RPM data just run "dm fan-speed". Either way on the 6450 it doesn't matter, as long as your temps are good. if it were a 7 series it'd be refusing to boot because of it but the 6 series doesn't give a shiza about no dang ole fans
 
  • Like
Reactions: 3nodeproblem

3nodeproblem

Member
Jun 28, 2020
42
11
8
it's probably showing failed because the fans are spinning slower than the "failed" threshold RPM. to check if it's still getting RPM data just run "dm fan-speed". Either way on the 6450 it doesn't matter, as long as your temps are good. if it were a 7 series it'd be refusing to boot because of it but the 6 series doesn't give a shiza about no dang ole fans
Code:
All fans in Low speed (0x83)
This is with only 2/3 fans connected to the 6450-48p btw. Will see how temps look once it actually has work to do.

BTW, my power meter between the switch and outlet reports only 97-99V, any idea how sensitive it is for undervoltage? Old house with likely shitty wiring.