Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)
- Thread starter fohdeesha
- Start date
This is probably a stupid question but don't find it mentioned in the thread: In absence of a serial cable (still waiting for delayed delivery out in the sticks), is there any management that can be done by default on 6450?
Tried connecting the mgmt port to a computer running a dhcp server - I see it getting an IP address assigned and can ping it but it doesn't seem any TCP ports are open, and only UDP 67/68.
Tried connecting the mgmt port to a computer running a dhcp server - I see it getting an IP address assigned and can ping it but it doesn't seem any TCP ports are open, and only UDP 67/68.
Thanks for the template I found a few items in there. I figured out why the numbers weren't adding up... I have 2 brocades and was pulling info from the wrong one so the temps didn't match oops.
I recently acquired another ICX6610-48P from eBay for dirt cheap. It was sold as being bricked ("It stopped booting fully after a firmware upgrade"), but that's fine with me - my primary reason for purchasing it was to swap the 40 GbE daughtercard into my existing, functional 6610 that has one defective 40 GbE port.
Just for grins, though (and before I go through all the effort of taking both of them apart), I pulled a PSU and fan tray from my working unit and stuck it in this one to see what state it's in.
It's stuck in a boot loop of some kind. The PSU fans rev up upon having AC applied, run for a couple seconds, rev down for a few seconds more, and then rev back up again. Repeat x inifinity.
On the console, this is what I get (over and over again). Hitting 'a' or 'b' as prompted looks like they work, but they do nothing to interrupt the boot looping - it just reboots again a few seconds later.
Any hope of recovering this short of JTAG?
Just for grins, though (and before I go through all the effort of taking both of them apart), I pulled a PSU and fan tray from my working unit and stuck it in this one to see what state it's in.
It's stuck in a boot loop of some kind. The PSU fans rev up upon having AC applied, run for a couple seconds, rev down for a few seconds more, and then rev back up again. Repeat x inifinity.
On the console, this is what I get (over and over again). Hitting 'a' or 'b' as prompted looks like they work, but they do nothing to interrupt the boot looping - it just reboots again a few seconds later.
Code:
ICX Boot Code Version 10.1.00 (grz10100)
Enter 'a' to stop at memory test
Enter 'b' to stop at boot monitor
Boot(Mem-Test)>aaaaaaaaaaaaaaa$
ICX Boot Code Version 10.1.00 (grz10100)
Enter 'a' to stop at memory test
Enter 'b' to stop at boot monitor
Boot(Mem-Test)>bbbbbbb$
ICX Boot Code Version 10.1.00 (grz10100)
Enter 'a' to stop at memory test
Enter 'b' to stop at boot monitor
***** Interrupted by entering 'b' *****
.$
ICX Boot Code Version 10.1.00 (grz10100)
Enter 'a' to stop at memory test
Enter 'b' to stop at boot monitor
***** Interrupted by entering 'b' *****
.$
ICX Boot Code Version 10.1.00 (grz10100)
Enter 'a' to stop at memory test
Enter 'b' to stop at boot monitor
I don't even think jtag would fix that, it seems the bootloader is intact but something has gone wrong enough with the CPU, RAM, or something else that's it's going into reset as soon as it tries to run anything. you could try reseating the RAM as a last hope
Looks like it was actually a bad PSU I was using. I had two Rev. A PSUs sitting in my shelf and I knew one of them was a dud. I assumed it was the one with the bent handle (you know, the physically damaged one...) and tried the other one. On a whim today during my lunch break I tried the other one, and it started booting!
Although it does seem to be getting stuck a little ways into the boot process. This is what I get:
Code:
$
ICX Boot Code Version 10.1.00 (grz10100)
Enter 'a' to stop at memory test
Enter 'b' to stop at boot monitor
BOOT INFO: load monitor from boot flash, cksum = 71f1
BOOT INFO: verify flash files..........
BOOT INFO: load image from primary copy...
platform type = 12
PCIE-1 LTSSM status: 22
PCIE Switch status: 0
.............................
Firmware integrity checksum passed
.......
Starting Main Task ...CPSS DxCh Version: cpss3.4p1 release
Pre Parsing Config Data ...Edit: apparently I just needed to be more patient. It continued startup pretty much as soon as I posted my reply.
yeah I dunno what it is but the 6610's boot sloooowwww. The FCX has the same CPU and has to unpack the exact same firmware to memory and it boots like 5x faster
Is it unpacking with the same settings (all cores/threads vs single thread)?
Is the boot device connected in the same way?
Is the boot device on pair?
the management plane between the two is identical (that's why they run the same firmware and even the same bootloader image). Both are single core single thread PPC procs booting from CFI flash, the flash part number is even the same between them (S29GL512P). the main difference is the ASIC of course, perhaps the newer marvell in the 6610 takes longer to initialize
I did my brocade switches stack like I wish with fohdeesha's description below.
https://forums.servethehome.com/ind...p-powerful-10gbe-40gbe-switching.21107/page-7
But I faced a strange issue. Some rear ports 1/2/2 to 1/2/5, 1/2/7 to 1/2/10 and same ports on the second switch don’t work when they connected to servers. There is no problem with network cards of servers.
I would be very happy if you suggest anything about the issue.
https://forums.servethehome.com/ind...p-powerful-10gbe-40gbe-switching.21107/page-7
But I faced a strange issue. Some rear ports 1/2/2 to 1/2/5, 1/2/7 to 1/2/10 and same ports on the second switch don’t work when they connected to servers. There is no problem with network cards of servers.
I would be very happy if you suggest anything about the issue.
This log shows what happens about not working ports:
This time STP is disabled on the port:
Code:
Aug 19 18:45:51:I:System: Interface ethernet 2/2/4, state down
Aug 19 18:45:51:I:STP: VLAN 1 Port 2/2/4 STP State -> DISABLED (PortDown)
Aug 19 18:45:51:I:STP: VLAN 1 Port 2/2/4 STP State -> LISTENING (PortDown)
Aug 19 18:45:50:I:System: Interface ethernet 2/2/4, state up
Aug 19 18:45:50:I:STP: VLAN 1 Port 2/2/4 STP State -> LISTENING (MakeFwding)
Aug 19 18:44:31:I:System: Interface ethernet 2/2/4, state down
Aug 19 18:44:31:I:STP: VLAN 1 Port 2/2/4 STP State -> DISABLED (PortDown)
Aug 19 18:44:31:I:STP: VLAN 1 Port 2/2/4 STP State -> LISTENING (PortDown)
Aug 19 18:44:30:I:System: Interface ethernet 2/2/4, state up
Aug 19 18:44:30:I:STP: VLAN 1 Port 2/2/4 STP State -> LISTENING (MakeFwding)
Code:
Aug 19 18:51:38:I:System: Interface ethernet 2/2/4, state down
Aug 19 18:51:37:I:System: Interface ethernet 2/2/4, state up
Aug 19 18:51:22:I:System: Interface ethernet 2/2/4, state down
Aug 19 18:51:22:I:System: Interface ethernet 2/2/4, state upThanks for your reply itronin.
Code:
SSH@ICX6610-24 Switch#sh run
Current configuration:
!
ver 08.0.30tT7f1
!
stack unit 1
module 1 icx6610-24-port-management-module
module 2 icx6610-qsfp-10-port-160g-module
module 3 icx6610-8-port-10g-dual-mode-module
priority 128
stack-port 1/2/1 1/2/6
stack unit 2
module 1 icx6610-24-port-management-module
module 2 icx6610-qsfp-10-port-160g-module
module 3 icx6610-8-port-10g-dual-mode-module
stack-port 2/2/1 2/2/6
stack enable
stack mac xxxx.xxxx.xx74
!
!
!
lag xxxx dynamic id 1
ports ethernet 1/3/8 ethernet 2/3/8
primary-port 1/3/8
deploy
port-name xxxx ethernet 1/3/8
port-name xxxx ethernet 2/3/8
!
lag xxxx dynamic id 2
ports ethernet 1/3/3 ethernet 2/3/3
primary-port 1/3/3
deploy
!
lag xxxx dynamic id 3
ports ethernet 1/3/4 ethernet 2/3/4
primary-port 1/3/4
deploy
!
!
vlan 1 name DEFAULT-VLAN by port
!
vlan xxxx name xxxx by port
tagged ethe 1/3/1 ethe 1/3/8 ethe 2/3/1 ethe 2/3/8
spanning-tree 802-1w
!
vlan xxxx name xxxx by port
tagged ethe 1/3/1 ethe 1/3/8 ethe 2/3/1 ethe 2/3/8
spanning-tree 802-1w
!
vlan xxxx name xxxx by port
tagged ethe 1/1/1 to 1/1/24 ethe 1/2/2 to 1/2/5 ethe 1/2/7 to 1/2/10 ethe 1/3/1 to 1/3/7 ethe 2/1/1 to 2/1/24 ethe 2/2/2 to 2/2/3 ethe 2/2/5 ethe 2/2/7 to 2/2/10 ethe 2/3/1 to 2/3/7
spanning-tree 802-1w
!
vlan xxxx name xxxx by port
tagged ethe 1/1/1 to 1/1/24 ethe 1/2/2 to 1/2/5 ethe 1/2/7 to 1/2/10 ethe 1/3/1 to 1/3/7 ethe 2/1/1 to 2/1/24 ethe 2/2/2 to 2/2/3 ethe 2/2/5 ethe 2/2/7 to 2/2/10 ethe 2/3/1 to 2/3/7
spanning-tree 802-1w
!
vlan xxxx name xxxx by port
tagged ethe 1/1/1 to 1/1/24 ethe 1/2/2 to 1/2/5 ethe 1/2/7 to 1/2/10 ethe 1/3/1 to 1/3/7 ethe 2/1/1 to 2/1/24 ethe 2/2/2 to 2/2/3 ethe 2/2/5 ethe 2/2/7 to 2/2/10 ethe 2/3/1 to 2/3/7
spanning-tree 802-1w
!
vlan xxxx name xxxx by port
tagged ethe 1/1/1 to 1/1/24 ethe 1/2/2 to 1/2/5 ethe 1/2/7 to 1/2/10 ethe 1/3/1 to 1/3/7 ethe 2/1/1 to 2/1/24 ethe 2/2/2 to 2/2/3 ethe 2/2/5 ethe 2/2/7 to 2/2/10 ethe 2/3/1 to 2/3/7
spanning-tree 802-1w
!
vlan xxxx name xxxx by port
tagged ethe 1/1/1 to 1/1/24 ethe 1/2/2 to 1/2/5 ethe 1/2/7 to 1/2/10 ethe 1/3/1 to 1/3/7 ethe 2/1/1 to 2/1/24 ethe 2/2/2 to 2/2/3 ethe 2/2/5 ethe 2/2/7 to 2/2/10 ethe 2/3/1 to 2/3/7
spanning-tree 802-1w
!
!
!
!
!
aaa authentication enable default local
aaa authentication login default local
boot sys fl sec
jumbo
enable telnet authentication
ip address x.x.x.x x.x.x.x
ip dns server-address 8.8.8.8
no ip dhcp-client enable
ip default-gateway x.x.x.x
!
username xxxx password .....
!
!
clock timezone gmt GMT+03
!
!
ntp
disable serve
server 192.36.143.130
server 213.251.52.185
!
!
hitless-failover enable
interface ethernet 1/1/17
dual-mode xxxx
!
interface ethernet 1/2/1
port-name xxxx
!
interface ethernet 1/3/1
port-name xxxx
speed-duplex 10G-full
!
interface ethernet 1/3/2
speed-duplex 10G-full
!
interface ethernet 1/3/3
speed-duplex 10G-full
!
interface ethernet 1/3/4
speed-duplex 10G-full
!
interface ethernet 1/3/5
dual-mode xxxx
speed-duplex 10G-full
!
interface ethernet 1/3/6
speed-duplex 10G-full
!
interface ethernet 1/3/7
speed-duplex 10G-full
!
interface ethernet 1/3/8
port-name xxxx
speed-duplex 10G-full
!
interface ethernet 2/2/4
no spanning-tree
!
interface ethernet 2/3/1
speed-duplex 10G-full
!
interface ethernet 2/3/2
speed-duplex 10G-full
!
interface ethernet 2/3/5
speed-duplex 10G-full
!
interface ethernet 2/3/6
speed-duplex 10G-full
!
interface ethernet 2/3/7
speed-duplex 10G-full
!
!
!
!
!
!
ip ssh timeout 60
!
!
endI have implemented the @RoachedCoach Mechatronics and Sunon fan mods discussed here on a ICX7250-48P. The new fans are definitely quieter, but I am not getting the results that others have. From a cold boot, the ASIC temps rise to mid-60's which is no problem, however the air outlet temps rise to the speed 2 threshold, the fans go to high and the temp drops to exactly 34c, the threshold for dropping back to speed 1. So once speed 2 becomes active, it never drops back. I tried just running with 1 fan, air outlet temps go into the 60's, but still have the one fan running at high speed. With the fan(s) running at high speed, the 7250-48P is louder than a 6610-24F with 2 Rev. A power supplies.
Due to a comedy of seller errors, I also have a 7250-24 and 7250-24P with dead PoE. The 7250-24 with its single OEM fan runs happily in the same rack with the fan at low speed. But I am not fond of the idea of going back to PoE power bricks.
Any ideas on getting a 7250-48P cool enough so the fans run at low speed?
Code:
Fan 1 ok, speed (auto): 1<->[[2]]
Fan 2 ok, speed (auto): 1<->[[2]]
Fan 3 ok, speed (auto): 1<->[[2]]
Fan controlled temperature:
Rule 1/2 (MGMT THERMAL PLANE): 51.7 deg-C
Rule 2/2 (AIR OUTLET NEAR PSU): 34.0 deg-C
Fan speed switching temperature thresholds:
Rule 1/2 (MGMT THERMAL PLANE):
Speed 1: NM<-----> 95 deg-C
Speed 2: 85<----->105 deg-C (shutdown)
Rule 2/2 (AIR OUTLET NEAR PSU)***active***:
Speed 1: NM<-----> 41 deg-C
Speed 2: 34<----->105 deg-C (shutdown)
Fan 1 Air Flow Direction: Front to Back
Fan 2 Air Flow Direction: Front to Back
Fan 3 Air Flow Direction: Front to Back
Slot 1 Current Temperature: 51.7 deg-C (Sensor 1), 34.0 deg-C (Sensor 2)
Slot 2 Current Temperature: NA
Warning level.......: 100.0 deg-C
Shutdown level......: 105.0 deg-CDue to a comedy of seller errors, I also have a 7250-24 and 7250-24P with dead PoE. The 7250-24 with its single OEM fan runs happily in the same rack with the fan at low speed. But I am not fond of the idea of going back to PoE power bricks.
Any ideas on getting a 7250-48P cool enough so the fans run at low speed?
I just finished doing the exact same modification this afternoon on my 7250-48P but instead of the Sunon fan I replaced it with an EFB0612LA-R00 because Digi-Key was out of stock on the Sunon.
Mods and temps:
24P - 2 fans
48P - 3 fans + ASIC
I'd love to be able to pull the fan speed RPMs via SNMP but can't find an OID
Last edited:
Having replaces 2/3 fans with Noctua NF-A4x20 FLX, reqiring the pins as others suggested (black -> black, red -> yellow, yellow -> green), the fans show as "failed" - is this all good? The fans are spinning at a very low RPM but temps look OK (haven't gone further than idle);
Code:
ICX6450-48P Switch#show chassis
The stack unit 1 chassis info:
Power supply 1 (NA - AC - Regular) present, status ok
Power supply 2 not present
Power supply 3 not present
Fan 1 failed
Fan 2 failed
Fan 3 failed
Fan controlled temperature: 41.5 deg-C
Fan speed switching temperature thresholds:
Speed 1: NM<----->61 deg-C
Speed 2: 56<-----> 79 deg-C (shutdown)
Sensor B Temperature Readings:
Current temperature : 37.0 deg-C
Sensor A Temperature Readings:
Current temperature : 41.5 deg-C
Warning level.......: 69.0 deg-C
Shutdown level......: 79.0 deg-Cit's probably showing failed because the fans are spinning slower than the "failed" threshold RPM. to check if it's still getting RPM data just run "dm fan-speed". Either way on the 6450 it doesn't matter, as long as your temps are good. if it were a 7 series it'd be refusing to boot because of it but the 6 series doesn't give a shiza about no dang ole fans
Code:
All fans in Low speed (0x83)BTW, my power meter between the switch and outlet reports only 97-99V, any idea how sensitive it is for undervoltage? Old house with likely shitty wiring.