Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

[Zer0Day]

I have seen some switches whose management ports don't work. It's an old HW rev, or engineering sample, if you will. I thought I read that some switches could have their management ports disabled/damaged? Mellanox (I2C pins)?

@fohdeesha would know for sure about Brocades.

Don't think that's the case here, as I am able to successfully TFTP an image via the management port into RAM after setting all my environment variables to my local subnet:

u-boot> md 64000000
64000000: 4c494e58 02130000 01c727f8 3bd58087    LINX......'.;...
64000010: 36d212a7 00000000 00000000 6c000000    6...........l...
64000020: 00000000 00000000 00000000 00000000    ................
64000030: 9e138c5e 08005006 53505230 38303830    ...^..P.SPR08080
64000040: 66000000 00000000 08005000 00000000    f.........P.....
64000050: 00000000 00000000 00000000 00000000    ................
64000060: 00000000 00000000 00000000 00000000    ................
64000070: 00000000 00000000 00000000 00000000    ................
64000080: 00000000 00000000 00000000 00000000    ................
64000090: 00000000 00000000 00000000 00000000    ................
640000a0: 00000000 00000000 00000000 00000000    ................
640000b0: 00000000 00000000 00000000 00000000    ................
640000c0: 00000000 00000000 00000000 00000000    ................
640000d0: 00000000 00000000 00000000 00000000    ................
640000e0: 00000000 00000000 00000000 00000000    ................
640000f0: 00000000 00000000 00000000 00000000    ................
u-boot>

I was hoping to follow roughly the process Fodeesha described HERE, but with my hardware I'm sure the offset addresses will be different. It may be a moot point however, as I tried getting more info from uboot and found this:

u-boot> bdinfo
arch_number = 0x0000127F
boot_params = 0x60200000
DRAM bank   = 0x00000000
-> start    = 0x60000000
-> size     = 0x3FE00000
eth0name    = bcm_xgs_gmac-0
ethaddr     = 78:A6:E1:42:04:F4
current eth = bcm_xgs_gmac-0
ip_addr     = 10.225.100.217
baudrate    = 9600 bps
TLB addr    = 0x9FDF0000
relocaddr   = 0x9FD0C000
reloc off   = 0xAFD0C000
irq_sp      = 0x9F4FBEF0
sp start    = 0x9F4FBEE0

u-boot> coninfo
List of available devices:
serial   00000003 IO stdin stdout stderr
eserial0 00000003 IO

u-boot> version

U-Boot 2016.01-Broadcom XLDK-3.8.1-svn20827 (Dec 12 2016 - 09:36:45 +0800)
armeb-linux-gcc.br_real (Buildroot 2015.11.1-svn18338) 4.9.3
GNU ld (GNU Binutils) 2.24

u-boot> flinfo

Bank # 1: missing or unknown FLASH type
u-boot>

..which seems to indicate that the flash is missing or corrupt, so I'm not sure if I'd be able to copy to it in the first place. Input welcome but if I cant see flash this one's probably going back from whence it came. Unless it could be configured to load an image from USB/TFTP on every bootup?

 

[LodeRunner]

Do any of the nand commands work for getting info?

As far as USB boot, I see the USB boot command in the uboot list, but I've never set that up, so no idea what formatting requirements are.

 

[Zer0Day]

Yes, from the init output it sees nand but not flash:

Spoiler

Flash: 0 Bytes
NAND: PNOR flash is not present - switch mux back for NAND
Micron MT29F16G08CBACA, blocks per lun: 800 lun count: 1
1024 KiB blocks, 4 KiB pages, 27B OOB, 8-bit
NAND: chipsize 2048 MiB
...
NAND read: device 0 offset 0x0, size 0x5000000
83886080 bytes read: OK
## Loading kernel from FIT Image at 65000000 ...
Using 'conf@1' configuration
Trying 'kernel@1' kernel subimage
Description: Broadcom iProc Linux
Type: Kernel Image
Compression: uncompressed
Data Start: 0x650000d4
Data Size: 32951392 Bytes = 31.4 MiB
Architecture: ARM
OS: Linux
Load Address: 0x61008000
Entry Point: 0x61008000
Hash algo: crc32
Hash value: af2d6a92
Verifying Hash Integrity ... crc32+ OK

I was able to do an sflash test from foxdiag and it passed:

Spoiler

DIAG>sflashtest 1 0 0 0
FLASH test OK - offset 0x00000000 , sector 0
FLASH test OK - offset 0x00010000 , sector 1
FLASH test OK - offset 0x00020000 , sector 2
FLASH test OK - offset 0x00030000 , sector 3
FLASH test OK - offset 0x00040000 , sector 4
FLASH test OK - offset 0x00050000 , sector 5
FLASH test OK - offset 0x00060000 , sector 6
FLASH test OK - offset 0x00070000 , sector 7
FLASH test OK - offset 0x00080000 , sector 8
FLASH test OK - offset 0x00090000 , sector 9
FLASH test OK - offset 0x000A0000 , sector 10
FLASH test OK - offset 0x000B0000 , sector 11
SPI flash test (1) - PASS

If I break into uboot I can see nand and SPI flash:

Spoiler

u-boot> nand info

Device 0: nand0, sector size 1024 KiB
Page size 4096 b
OOB size 224 b
Erase size 1048576 b
subpagesize 4096 b
options 0x 10200
bbt options 0x 0
u-boot>

u-boot> sf probe
Access set to SECONDARY FL..
SF: Detected MX25L6405D with page size 256 Bytes, erase size 64 KiB, total 8 MiB
u-boot>

I verified that mtd partitions exist on nand:

Spoiler

u-boot> mtdparts

device nand0 <brcmnand.0>, # parts = 4
#: name size offset mask_flags
0: brcd_primary_image 0x04000000 0x00000000 0
1: brcd_secondary_image0x04000000 0x04000000 0
2: configs_logs 0x58000000 0x08000000 0
3: resources 0x20000000 0x60000000 0

active partition: nand0,0 - (brcd_primary_image) 0x04000000 @ 0x00000000

defaults:
mtdids : nand0=nand_iproc.0
mtdparts: mtdparts=nand_iproc.0:2M(nboot),4M(nenv),10M(nsystem),48M(nrootfs),-(ncustfs)
u-boot>

The USB subsystem also works, and I can mount FAT drives (and potentially use them as a source for a file copy using fatload, rather than TFTP):

Spoiler

u-boot> usb start
starting USB...
USB0: Bring usb2h_out of reset.......
USB EHCI 1.00
scanning bus 0 for devices... 2 USB Device(s) found
scanning usb for storage devices... 1 Storage Device(s) found
u-boot> usb storage
Device 0: Vendor: Rev: PMAP Prod: USB DISK 3.0
Type: Removable Hard Disk
Capacity: 14814.0 MB = 14.4 GB (30339072 x 512)
u-boot> fatls usb0
** No device specified **
u-boot> fatls usb 0
system volume information/
786944 mnz10114.bin
29829112 spr08080f.bin
icx7150/

2 file(s), 2 dir(s)

u-boot>
u-boot> ? fatload
fatload - load binary file from a dos filesystem

Usage:
fatload <interface> [<dev[art]> [<addr> [<filename> [bytes [pos]]]]]
- Load binary file 'filename' from 'dev' on 'interface'
to address 'addr' from dos filesystem.
'pos' gives the file position to start loading from.
If 'pos' is omitted, 0 is used. 'pos' requires 'bytes'.
'bytes' gives the size to load. If 'bytes' is 0 or omitted,
the load stops on end of file.
If either 'pos' or 'bytes' are not aligned to
ARCH_DMA_MINALIGN then a misaligned buffer warning will
be printed and performance will suffer for the load.
u-boot>

So I *think* I just need to copy the new bootloader (mnz10114.bin) somewhere I can boot from, then from there copy the new spr08080 image to flash, optionally erasing/formatting/partitioning the target device first. Just want to make sure my target filesystems and offsets are good before writing onto them.

 

[fohdeesha]

Yes, from the init output it sees nand but not flash:

Spoiler

Flash: 0 Bytes
NAND: PNOR flash is not present - switch mux back for NAND
Micron MT29F16G08CBACA, blocks per lun: 800 lun count: 1
1024 KiB blocks, 4 KiB pages, 27B OOB, 8-bit
NAND: chipsize 2048 MiB
...
NAND read: device 0 offset 0x0, size 0x5000000
83886080 bytes read: OK
## Loading kernel from FIT Image at 65000000 ...
Using 'conf@1' configuration
Trying 'kernel@1' kernel subimage
Description: Broadcom iProc Linux
Type: Kernel Image
Compression: uncompressed
Data Start: 0x650000d4
Data Size: 32951392 Bytes = 31.4 MiB
Architecture: ARM
OS: Linux
Load Address: 0x61008000
Entry Point: 0x61008000
Hash algo: crc32
Hash value: af2d6a92
Verifying Hash Integrity ... crc32+ OK

I was able to do an sflash test from foxdiag and it passed:

Spoiler

DIAG>sflashtest 1 0 0 0
FLASH test OK - offset 0x00000000 , sector 0
FLASH test OK - offset 0x00010000 , sector 1
FLASH test OK - offset 0x00020000 , sector 2
FLASH test OK - offset 0x00030000 , sector 3
FLASH test OK - offset 0x00040000 , sector 4
FLASH test OK - offset 0x00050000 , sector 5
FLASH test OK - offset 0x00060000 , sector 6
FLASH test OK - offset 0x00070000 , sector 7
FLASH test OK - offset 0x00080000 , sector 8
FLASH test OK - offset 0x00090000 , sector 9
FLASH test OK - offset 0x000A0000 , sector 10
FLASH test OK - offset 0x000B0000 , sector 11
SPI flash test (1) - PASS

If I break into uboot I can see nand and SPI flash:

Spoiler

u-boot> nand info

Device 0: nand0, sector size 1024 KiB
Page size 4096 b
OOB size 224 b
Erase size 1048576 b
subpagesize 4096 b
options 0x 10200
bbt options 0x 0
u-boot>

u-boot> sf probe
Access set to SECONDARY FL..
SF: Detected MX25L6405D with page size 256 Bytes, erase size 64 KiB, total 8 MiB
u-boot>

I verified that mtd partitions exist on nand:

Spoiler

u-boot> mtdparts

device nand0 <brcmnand.0>, # parts = 4
#: name size offset mask_flags
0: brcd_primary_image 0x04000000 0x00000000 0
1: brcd_secondary_image0x04000000 0x04000000 0
2: configs_logs 0x58000000 0x08000000 0
3: resources 0x20000000 0x60000000 0

active partition: nand0,0 - (brcd_primary_image) 0x04000000 @ 0x00000000

defaults:
mtdids : nand0=nand_iproc.0
mtdparts: mtdparts=nand_iproc.0:2M(nboot),4M(nenv),10M(nsystem),48M(nrootfs),-(ncustfs)
u-boot>

The USB subsystem also works, and I can mount FAT drives (and potentially use them as a source for a file copy using fatload, rather than TFTP):

Spoiler

u-boot> usb start
starting USB...
USB0: Bring usb2h_out of reset.......
USB EHCI 1.00
scanning bus 0 for devices... 2 USB Device(s) found
scanning usb for storage devices... 1 Storage Device(s) found
u-boot> usb storage
Device 0: Vendor: Rev: PMAP Prod: USB DISK 3.0
Type: Removable Hard Disk
Capacity: 14814.0 MB = 14.4 GB (30339072 x 512)
u-boot> fatls usb0
** No device specified **
u-boot> fatls usb 0
system volume information/
786944 mnz10114.bin
29829112 spr08080f.bin
icx7150/

2 file(s), 2 dir(s)

u-boot>
u-boot> ? fatload
fatload - load binary file from a dos filesystem

Usage:
fatload <interface> [<dev[art]> [<addr> [<filename> [bytes [pos]]]]]
- Load binary file 'filename' from 'dev' on 'interface'
to address 'addr' from dos filesystem.
'pos' gives the file position to start loading from.
If 'pos' is omitted, 0 is used. 'pos' requires 'bytes'.
'bytes' gives the size to load. If 'bytes' is 0 or omitted,
the load stops on end of file.
If either 'pos' or 'bytes' are not aligned to
ARCH_DMA_MINALIGN then a misaligned buffer warning will
be printed and performance will suffer for the load.
u-boot>

So I *think* I just need to copy the new bootloader (mnz10114.bin) somewhere I can boot from, then from there copy the new spr08080 image to flash, optionally erasing/formatting/partitioning the target device first. Just want to make sure my target filesystems and offsets are good before writing onto them.

you need to write brocade's bootloader to SPI flash, it will have the correct flash driver and be able to see everything. If you want to be sure/careful I'll have to do it myself over teamviewer (need to check and verify the offsets, never looked on a 7450)

 

[Zer0Day]

Appreciate the offer and the help you've provided to the community, but wanted to try to do this myself as a learning experience. Fortunately, I did not learn how to brick a switch, but rather how to manually load the bootloader code, which will be a valuable skill as I manage a fleet of these.

I first stumbled across the 6430/6450 steps, but instead followed the 7150-C12P recover that was documented in the end of Nov, in which you used:

##RECOVER
##this example uses the mnz10114 bootloader from the 8080 codetrain/zip
tftp 0x80000000 mnz10114.bin
sf probe 0:0
sf erase 0 +11F3D8
sf write 0x80000200 0 0x11F3D8

##VERIFY
sf probe 0:0
sf read 0x80000000 0 100
md 0x80000000

The version of uboot on this switch is exactly the same as that 7150-C12P, the only difference being that this is it's bigger brother (7150-48zp). So I'm leaning toward that being the more correct approach in my case. I had gotten as far as loading the mnz10114.bin file from TFTP/USB into memory, and validating that it wrote successfully. I also saw both sflash chips:

u-boot> sf probe 0:0
Access set to SECONDARY FL..
SF: Detected MX25L6405D with page size 256 Bytes, erase size 64 KiB, total 8 MiB

u-boot> sf probe 0:1
Flash set to PRIMARY FL
SF: Detected MX25L6405D with page size 256 Bytes, erase size 64 KiB, total 8 MiB
u-boot>

Following your example of the C12P, you erased the first MB (+11F3D8) on secondary flash (0:0) starting at the beginning (0) and just copied over the ram contents that were populated from the bootloader file loaded from TFTP. That all seems fairly straightforward. I took a leap of faith and performed the exact same steps to write the new bootloader onto the start of secondary flash, crossed my fingers, and it booted into the Brocade 08080 bootloader ca. 2018. From there, re-applying the official boot image and finally full Brocade OS was trivial.

I am still getting the device key or cert file not available errors, even after issuing sz disable from conf t and reloading. Otherwise, life is good, and I will continue to upgrade this to the most recent UFI codebase and maybe even open up telnet to the Linux environment as you've described.

Thanks again for all you've done for the community, and add 7150-48zp to the list of hardware that these steps are known to work for.

 

[Rand__]

Also, can I connect the management interface to a port on the switch? Having some issues with that one... its not working on the 7450 itself and not on one other switch, but on a third. Very weird

Edit - just verified that the ports are all good. One of the runs is rather long (30m), but it works fine with a pi attached. Both Cisco (one working [short run], one not [long run]) have identical config (VLAN1 untagged/access)

Edit 2 - its not the length of the run, added another small switch in between, switch is reachable but 7450 not. This one is really giving me a hard time...

This one is still quite vexing ...
I directly connected a box to the mgmt port (via the long run) and that's working fine. So why in hell is it not working with a switch?. It links up just fine, just not passing traffic

 

[nerdalertdk]

Picked up a 7150-48zp recently (the multigig poe+ model with redundant internal PSU/fan support), and it seems to have the early dev uboot firmware that's been found in the wild recently (boot output attached). I can setenv and TFTP successfully, but update_primary and update_uboot aren't options. uboot help menu shows the following commands:

u-boot> help
? - alias for 'help'
base - print or set address offset
bdinfo - print Board Info structure
boot - boot default, i.e., run 'bootcmd'
bootd - boot default, i.e., run 'bootcmd'
bootelf - Boot from an ELF image in memory
bootm - boot application image from memory
bootp - boot image via network using BOOTP/TFTP protocol
bootvx - Boot vxWorks from an ELF image
bootz - boot Linux zImage image from memory
chpart - change active partition
cmp - memory compare
coninfo - print console devices and information
cp - memory copy
cplddl - cplddl - To perform cpld download
crc32 - checksum calculation
dhcp - boot image via network using DHCP/TFTP protocol
echo - echo args to console
editenv - edit environment variable
env - environment handling commands
erase - erase FLASH memory
exit - exit script
ext2load- load binary file from a Ext2 filesystem
ext2ls - list files in a directory (default /)
false - do nothing, unsuccessfully
fatinfo - print information about filesystem
fatload - load binary file from a dos filesystem
fatls - list files in a directory (default /)
fatsize - determine a file's size
fatwrite- write file into a dos filesystem
fdt - flattened device tree utility commands
flinfo - print FLASH memory information
go - start application at address 'addr'
gpio_read- Read from GPIO
gpio_write- write to GPIO
hash - compute hash message digest
help - print command description/usage
i2c - I2C sub-system
iminfo - print header information for application image
imxtract- extract a part of a multi-image
itest - return true/false on integer compare
loadb - load binary file over serial line (kermit mode)
loads - load S-Record file over serial line
loadx - load binary file over serial line (xmodem mode)
loady - load binary file over serial line (ymodem mode)
loop - infinite loop on address range
md - memory display
mdc - memory display cyclic
mdio - MDIO utility commands
mii - MII utility commands
mm - memory modify (auto-incrementing address)
mtdparts- define flash/nand partitions
mtest - simple RAM read/write test
mw - memory write (fill)
mwc - memory write cyclic
nand - NAND sub-system
nboot - boot from NAND device
nfs - boot image via network using NFS protocol
nm - memory modify (constant address)
ping - send ICMP ECHO_REQUEST to network host
printenv- print environment variables
protect - enable or disable FLASH write protection
reset - Perform RESET of the CPU
run - run commands in an environment variable
saveenv - save environment variables to persistent storage
setenv - set environment variables
sf - SPI flash sub-system
showvar - print local hushshell variables
sleep - delay execution for some time
source - run script from memory
test - minimal test like /bin/sh
tftp -
tftpboot- boot image via network using TFTP protocol
tftpput - TFTP put command, for uploading files to a server
time - run commands and summarize execution time
true - do nothing, successfully
ubi - ubi commands
ubifsload- load file from an UBIFS filesystem
ubifsls - list files in a directory
ubifsmount- mount UBIFS volume
ubifsumount- unmount UBIFS volume
usb - USB sub-system
usbboot - boot from USB device
version - print monitor, compiler and linker version
u-boot>

I can tftp to memory, but from there how can I tell what flash address to copy the bin files to for this model?

Hi

i have the same switch and I upgraded by following ruckus upgrade guide, to get it on the newest you have to upgrade to the first version that supports ufi boot, ruckus has an document where it’s all described how to do

 

[rocketpanda40]

This one is still quite vexing ...
I directly connected a box to the mgmt port (via the long run) and that's working fine. So why in hell is it not working with a switch?. It links up just fine, just not passing traffic

I can't pretend to be an expert, but I have the same experience with my 7150-c12. I can connect to the mgmt port directly with my laptop, and from there ping the mgmt ip and all the switch's SVIs, but traffic on the mgmt port won't seem to egress the switch – that is, I can ping the mgmt ip and say the ve 16 interface, but cant ping any other hosts on vlan 16 connected to the switch, while the switch itself can ping thosw hosts just fine.

This isn't an issue for me, but the experience (problem?) seems to be the same.

It's possible this is due to the mgmt port being a separate VRF and the switch unsure how to route it. I honestly don't remember the exact config for my 7150, but I know on my ciscos I configured the management ports as independent VRFs from the main routing table.

 

[Rand__]

Well its documented that the mgmt interface won't connect with the other ports, so thats to be expected.
My issue is slightly different, the mgmt interface works under some conditions (one switch, local port) but not under other (different switches) and I am not able to find why there is a difference (since all settings I checked are identical)

 

[Surfarn_]

I have the ICX 6610 and have been trying to setup vlans so I can run two opnsense boxes (one virtual running on proxmox (2 nics 1G and 10G) and one physical (router on a stick 1G))

Have put the vlan 99 as incomming Internet and it works fine.

I now try to add vlan 98 for pfsync. The vlan seems to work fine for the physical master Opnsense server (port 1/1/3 Tagged mode), but not for the virtual backup server (port 1/3/2 tagged mode).

I managed to get it to work when I put port 1/1/3 in dual mode and did not have a specific vlan selected in proxmox (or in Opnsense).






How should I setup the vlan for port 1/3/2? What could I have done wrong? I have been using the web interface. I would also like that proxmox would be able to to access vlan1 on this port, but not Opnsense.

VLAN config from running show run

vlan 1 name DEFAULT-VLAN by port
router-interface ve 1
spanning-tree 802-1w
!
vlan 98 name pfsync by port
tagged ethe 1/1/3 ethe 1/3/2
spanning-tree 802-1w
!
vlan 99 name WAN by port
tagged ethe 1/1/3 ethe 1/3/2
untagged ethe 1/1/1
spanning-tree 802-1w

 

[Surfarn_]

Hi again,

I have done some more testing. I can get vlan 98 to work in proxmox if I send it to the 1gig network card (1/1/2) instead of 1/3/2



vlan 1 name DEFAULT-VLAN by port
router-interface ve 1
spanning-tree 802-1w
!
vlan 98 name pfsync by port
tagged ethe 1/1/2 to 1/1/3 ethe 1/3/2
spanning-tree 802-1w
!
vlan 99 name WAN by port
tagged ethe 1/1/3 ethe 1/3/2
untagged ethe 1/1/1
spanning-tree 802-1w

This works

But not this

 

[csementuh]

EDIT It looks like my log output is being truncated here and there are current entries. I can't figure out the correct syntax to get current date info.
show sz logs | begin xxx may work if I can use it to search for today.

fohdeesha, do you know of a way to get the most current part of the log file for "show sz logs"? It looks like I am only getting the beginning of the file and the current data is being truncated. I'm still seeing lots and lots of DNS hits from the switch. The rest of the info a few posts up. Thank you!

 

[tubs-ffm]

Introducing: ICX7250​

This medium-rare beefer has 24 or 48 1gbE ports (PoE models available) with 8x 10gbE SFP+ ports. You can think of it as a big ICX6450, or a little ICX6610. It has fans, has the same sound level as the ICX6450, and draws about 50 watts. If you wished the ICX6610 cut it's power draw and sound level in half, and were willing to trade the rear QSFP+ ports and redundant PSU's for that, this is your switch.

Again, this 7 series runs v8080, so no licenses physically required.

• L3 - all the same L3 features as the ICX6610 (ipv4/ipv6 routing, OSPF, VRRP, PIM, VRF's) except BGP
• Same ACL features as 6450/6610
• non-PoE models easily converted to 12v DC power (like the ICX6450)
• 256gbps switching capacity
• 190mpps forwarding capacity
• ASIC: BCM56344
• Management CPU: 1GHZ dual-core ARM
• Datasheet

Price: the price on these vary more wildly. If you knew what I picked some up for, you'd stab me. They seemed to have levelled out at around $300 to $400.

Thank you very much for this article.

As home user I always was a big fan in buying used enterprise IT stuff. You get more value for your money in comparison to consumer stuff. OK, not always simple and useful for everybody.

But a device like this with a couple of SPF+ ports (4+), a couple of 1 gbE ports (8+) and L3 routing capability is what I could fine nowhere else at low costs. A cheap solution build-up with two MicroTik devices already is more expensive.

I will give it a trial.

 

[fohdeesha]

fohdeesha, do you know of a way to get the most current part of the log file for "show sz logs"? It looks like I am only getting the beginning of the file and the current data is being truncated. I'm still seeing lots and lots of DNS hits from the switch. The rest of the info a few posts up. Thank you!

Not offhand, sorry. I tried to reproduce your issue here locally with a wireshark dump, but I see no DNS requests (or anything, really) after running those sz disable commands. that was on an icx7250 running 8092d

have you rebooted the switch since running all the SZ disable stuff? Only other thing I can think to try. you SURE you're seeing requests from the switch itself?

 

[fohdeesha]

Well its documented that the mgmt interface won't connect with the other ports, so thats to be expected.
My issue is slightly different, the mgmt interface works under some conditions (one switch, local port) but not under other (different switches) and I am not able to find why there is a difference (since all settings I checked are identical)

are you plugging the management port into something/a switch that eventually ends up connected to the ICX's regular ports? The switches typically use the same MAC for the management port, and the first interface/VE on the regular ports, so if they end up on the same overall L2 network somehow, you'll get conflicts and it won't work right. the mgmt port is intended for totally seperate isolated management networks - if you're plugging it somewhere that ends up back on the same network as your normal ports, just configure an in-band management IP on a VE like the config guide details

 

[Rand__]

Hm that is actually possible, since in the end my Workstation VM has access to all the VLANs... so there is some path at least
Would have expected to see stp shutdowns on the other switches then , but maybe this is different then a regular loop.

Thanks, will have a look at that, it sounds like a reasonable explanation.

 

[fohdeesha]

Appreciate the offer and the help you've provided to the community, but wanted to try to do this myself as a learning experience. Fortunately, I did not learn how to brick a switch, but rather how to manually load the bootloader code, which will be a valuable skill as I manage a fleet of these.

I first stumbled across the 6430/6450 steps, but instead followed the 7150-C12P recover that was documented in the end of Nov, in which you used:

##RECOVER
##this example uses the mnz10114 bootloader from the 8080 codetrain/zip
tftp 0x80000000 mnz10114.bin
sf probe 0:0
sf erase 0 +11F3D8
sf write 0x80000200 0 0x11F3D8

##VERIFY
sf probe 0:0
sf read 0x80000000 0 100
md 0x80000000

The version of uboot on this switch is exactly the same as that 7150-C12P, the only difference being that this is it's bigger brother (7150-48zp). So I'm leaning toward that being the more correct approach in my case. I had gotten as far as loading the mnz10114.bin file from TFTP/USB into memory, and validating that it wrote successfully. I also saw both sflash chips:

u-boot> sf probe 0:0
Access set to SECONDARY FL..
SF: Detected MX25L6405D with page size 256 Bytes, erase size 64 KiB, total 8 MiB

u-boot> sf probe 0:1
Flash set to PRIMARY FL
SF: Detected MX25L6405D with page size 256 Bytes, erase size 64 KiB, total 8 MiB
u-boot>

Following your example of the C12P, you erased the first MB (+11F3D8) on secondary flash (0:0) starting at the beginning (0) and just copied over the ram contents that were populated from the bootloader file loaded from TFTP. That all seems fairly straightforward. I took a leap of faith and performed the exact same steps to write the new bootloader onto the start of secondary flash, crossed my fingers, and it booted into the Brocade 08080 bootloader ca. 2018. From there, re-applying the official boot image and finally full Brocade OS was trivial.

I am still getting the device key or cert file not available errors, even after issuing sz disable from conf t and reloading. Otherwise, life is good, and I will continue to upgrade this to the most recent UFI codebase and maybe even open up telnet to the Linux environment as you've described.

Thanks again for all you've done for the community, and add 7150-48zp to the list of hardware that these steps are known to work for.

d'oh - thought you were on a 7450 for some reason. Indeed if you have an icx7150 variant, they all have the same boot process/loader/flash locations. glad you got it figured out!

the key thing to note is the brocade-shipped bootloader file has a bunch of metadata at the beginning, and if you write that to flash, it's not gunna boot. You'll note in my instructions, you copy in the whole bootloader file to 0x80000000 in RAM, but then when writing to SPI flash, the RAM address you tell it to pull from is 0x80000200 - skipping the first 0x200 bytes of the copied-in brocade bootloader, so it starts writing from the actual bootloader start. One detail is this offset/metadata size could change at anytime (totally up to ruckus) which is one of many reasons I'm weary to give out instructions instead of doing it myself, I don't want people to brick their stuff if the offset changes and they don't know how to verify it. It's pretty obvious when opening the bootloader in a hex editor like HxD, it's the metadata, empty space and then the actual bootloader start, which for these ARM devices typically always has "EA" at the end of the first row of bytes:

so indeed in this example the offset is still 0x200 bytes

the other big thing is verifying where in SPI the actual bootloader needs to be written to. this is almost always at the very beginning, but as usual it's subject to change on different platforms so it needs to be verified first. typically to verify you would read the very first couple hundred bytes of SPI flash, and see if it looks like the very beginning of a bootloader. If it doesn't, that's not where you should write it. it should look similar to the starting bytes of our example bootloader above:

##to read boot flash for sanity check
(0:0 is primary boot flash, 0:1 is secondary)
sf probe 0:0
sf read 0x80000000 0 100
md 0x80000000

you should see something very similar to the following, the DEADBEEF on the fourth line is a dead giveaway you're in the right place:

if you get something close to that, you know the bootloader is supposed to be written right at the start of said SPI and you're good to go

 

[fohdeesha]

Hm that is actually possible, since in the end my Workstation VM has access to all the VLANs... so there is some path at least
Would have expected to see stp shutdowns on the other switches then , but maybe this is different then a regular loop.

Thanks, will have a look at that, it sounds like a reasonable explanation.

it's not exactly a loop as the management port has no access or connection to the regular ports, so it won't "complete the loop". It does have the same MAC though, and of course two "devices" having the same MAC is going to confuse the shit out of clients when they try to access it

 

[Zer0Day]

d'oh - thought you were on a 7450 for some reason. Indeed if you have an icx7150 variant, they all have the same boot process/loader/flash locations. glad you got it figured out!

the key thing to note is the brocade-shipped bootloader file has a bunch of metadata at the beginning, and if you write that to flash, it's not gunna boot. You'll note in my instructions, you copy in the whole bootloader file to 0x80000000 in RAM, but then when writing to SPI flash, the RAM address you tell it to pull from is 0x80000200 - skipping the first 0x200 bytes of the copied-in brocade bootloader, so it starts writing from the actual bootloader start. One detail is this offset/metadata size could change at anytime (totally up to ruckus) which is one of many reasons I'm weary to give out instructions instead of doing it myself, I don't want people to brick their stuff if the offset changes and they don't know how to verify it. It's pretty obvious when opening the bootloader in a hex editor like HxD, it's the metadata, empty space and then the actual bootloader start, which for these ARM devices typically always has "EA" at the end of the first row of bytes:

so indeed in this example the offset is still 0x200 bytes

the other big thing is verifying where in SPI the actual bootloader needs to be written to. this is almost always at the very beginning, but as usual it's subject to change on different platforms so it needs to be verified first. typically to verify you would read the very first couple hundred bytes of SPI flash, and see if it looks like the very beginning of a bootloader. If it doesn't, that's not where you should write it. it should look similar to the starting bytes of our example bootloader above:

##to read boot flash for sanity check
(0:0 is primary boot flash, 0:1 is secondary)
sf probe 0:0
sf read 0x80000000 0 100
md 0x80000000

you should see something very similar to the following, the DEADBEEF on the fourth line is a dead giveaway you're in the right place:

if you get something close to that, you know the bootloader is supposed to be written right at the start of said SPI and you're good to go

Excellent insight and yes as always your mileage will vary, so I'd certainly recommend people be careful when performing this type of surgery. I did use the 200 byte offset and also verified the hex values were what I expected and where I expected them before copying from ram onto SPI, especially the consecutive string of 14F09FE5.

 

[Zer0Day]

it's not exactly a loop as the management port has no access or connection to the regular ports, so it won't "complete the loop". It does have the same MAC though, and of course two "devices" having the same MAC is going to confuse the shit out of clients when they try to access it

Right, this isn't a spanning tree issue as the mgmt port isn't participating in STP or switching traffic, it's an issue with the CAM and ARP tables on other switches binding the MGMT IP/MAC to a specific port.