Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

[Haim Gelfenbeyn]

I just bought ICX6450-48P off eBay! Seems like the firmware is relatively easy to "open": I used BinWalk to extract everything from the image file:

binwalk -e -M ./Images/ICX64R08030r.bin

And then you have everything extracted as separate files, including etc/init.sh which actually starts everything when the switch boots. Too bad the actual switch app is just one, statically-linked monolith (and compressed with xz for some reason).

I wonder how hard it would be to add additional services to the switch (e.g. add/remove files and repack the firmware, then update), and how much space is still available on that flash... AFAIK the firmware is not signed in any way, am I correct? I wonder if anyone already tried doing something like that...

 

[fohdeesha]

wonder if anyone already tried doing something like that...

oh yeah, I have done some insane stupid stuff to these switches firmware. Some of it is covered back midway in the thread, can't remember where. You can also get access to the live linux FS as the switch runs and load in whatever you want there via tftp etc, methods for that are posted somewhere in the first page

fyi brocade images (at least for linux based switches) are u-boot FIT images with some brocade specific headers in front. Instead of using binwalk which mostly has to guess, you can install u-boot-tools, which comes with the dumpimage binary, which can walk through and take apart the FIT image, you just have to trim out the first 512 bytes (brocade's header): hastebin

you can then use dumpimage to selectively extract any part of it:

root@testing:~# dumpimage -T flat_dt -p 5 -i trimmed.bin ICX7250-ramdisk.zip
Extracted:
 Image 5 (ramdisk@1)
  Description:  Ramdisk -rootfs
  Created:      Tue Apr  9 06:25:31 2019
  Type:         RAMDisk Image
  Compression:  lzma compressed
  Data Size:    25430519 Bytes = 24834.49 KiB = 24.25 MiB
  Architecture: ARM
  OS:           Linux
  Load Address: 0x00000000
  Entry Point:  0x00000000
  Hash algo:    crc32
  Hash value:   960fb2cd

then you have the nicely packaged nix filesystem in an lzma compressed ZIP archive. You can do the same to extract the kernel and device trees, make any modifications you want, then use another u-boot-tools utility, mkimage, to repackage it back into a FIT compliant file ready to flash back. some docu here: Confluence

fun fact: the ICX7150, ICX7250, and ICX7450 all run the same firmware image. They have codenames:

ICX7150 = MN (Minion)
ICX7250 = SI (Sica)
ICX7450 = SP (Spatha)

As you can see in the FIT dump above, the unified firmware image detects which platform it's booting on, then loads up a unique combination of a device tree, ramdisk, and kernel for that specific platform:

 Default Configuration: 'conf@1'
 Configuration 0 (conf@1)
  Description:  Boot MN Linux kernel with FDT blob
  Kernel:       kernel@2
  Init Ramdisk: ramdisk@1
  FDT:          fdt@3
 Configuration 1 (conf@2)
  Description:  Boot SP  Linux kernel with FDT blob
  Kernel:       kernel@1
  Init Ramdisk: ramdisk@1
  FDT:          fdt@1
 Configuration 2 (conf@3)
  Description:  Boot SI Linux kernel with FDT blob
  Kernel:       kernel@1
  Init Ramdisk: ramdisk@1
  FDT:          fdt@2

if you're wondering how I got the codenames, they're buried in code comments in a preinit.sh script in the above image ramdisk: hastebin

 

[fohdeesha]

fun fact #2: they started protecting the linux install underneath with a root password, however they didn't bother hashing it, so you can get the password by just looking at /etc/passwdshadow from the above extracted ramdisk

 

[maes]

Great, now I'm curious to see if there are any unused IOs anywhere on there. Some of the empty eeprom pads might be promising for I2C or SPI, to allow extra storage (say, SD cards) or more... unusual... options. GPS module, maybe, to give it a full-blown internal stratum-1 ntp server?

@fohdeesha , in your teardowns, did you ever write down the exact model of the CPU in the 6450? Or is it integrated in one of the switching ASICs?

 

[fohdeesha]

in your teardowns, did you ever write down the exact model of the CPU in the 6450? Or is it integrated in one of the switching ASICs?

https://fohdeesha.com/data/other/brocade/Marvell-88F6281.pdf

https://fohdeesha.com/data/other/brocade/Kirkwood_88F6281_2_Hardware_Spec.pdf

 

[maes]

Thanks! Shame the kernel available on the Brocade/Arris/Ruckus sourceforge is absolutely prehistoric (looks like 2.6.12-r3!) so that limits a few things, but it might be fun to experiment.

There's definitely a lot of connectivity to those CPUs, it's a matter of seeing what's traced out and available.

 

[fohdeesha]

Thanks! Shame the kernel available on the Brocade/Arris/Ruckus sourceforge is absolutely prehistoric (looks like 2.6.12-r3!) so that limits a few things, but it might be fun to experiment.

There's definitely a lot of connectivity to those CPUs, it's a matter of seeing what's traced out and available.

that's just the super old icx6xxx line, the newer stuff (like v8091) is using at least 4.4.0 looks like:

Linux kernel version "4.4.0 (swrel@l42-ub-ecbld-03) (gcc version 4.9.3 (Buildroot 201gcc version 4.9.3 (Buildroot 2015.11.1) )

 

[fohdeesha]

quick guide on extracting brocade images:

-------Extracting Brocade Firmware (v8060 and above)
##Remove the brocade-specific header info from the file
##For regular images, this is 512 bytes:
dd if="SPR08080e.bin" of="trimmed.bin" bs=512 skip=1

##For UFI images, (v8091 and up only have UFI images available), it's 1360 bytes:
dd if=SPR08091ufi.bin of="trimmed.bin" bs=1360 skip=1


#Install u-boot-tools
apt install u-boot-tools
##Debian 9 (and maybe others) have a very old uboot-tools version in the default repo, and does not come with the required dumpimage
##In that case, just manually install recent tools:
wget http://ftp.us.debian.org/debian/pool/main/u/u-boot/u-boot-tools_2019.01+dfsg-7_amd64.deb
dpkg -i u-boot-tools_2019.01+dfsg-7_amd64.deb

##Now use dumpimage to view a list of all the components in the firmware package:
dumpimage -l trimmed.bin

##You'll get a list of everything in the package:

FIT description: Linux kernel and FDT blob
Created:         Fri Jun  7 00:12:13 2019
 Image 0 (kernel@1)
  Description:  Ruckus Linux SP/SI
  Created:      Fri Jun  7 00:12:13 2019
  Type:         Kernel Image
  Compression:  lzma compressed
  Data Size:    2386204 Bytes = 2330.28 KiB = 2.28 MiB
  Architecture: ARM
  OS:           Linux
  Load Address: 0x61008000
  Entry Point:  0x61008000
  Hash algo:    crc32
  Hash value:   19ecdaa8
 Image 1 (kernel@2)
  Description:  Ruckus Linux MN VER=08.0.91
  Created:      Fri Jun  7 00:12:13 2019
  Type:         Kernel Image
  Compression:  lzma compressed
  Data Size:    2401701 Bytes = 2345.41 KiB = 2.29 MiB
  Architecture: ARM
  OS:           Linux
  Load Address: 0x61008000
  Entry Point:  0x61008000
  Hash algo:    crc32
  Hash value:   8c4ccc81
----trimmed for brevity----

##Choose a part to extract to a separate file, putting the image number after the -p argument:
##Note: -T must be kept set to "-T flat_dt", even if you are extracting a different image type
dumpimage -T flat_dt -p 0 -i trimmed.bin kernel.zip

#If you extract firmware device trees and want to see them in human-readable form:
apt install device-tree-compiler
fdtdump FDT.bin

 

[maes]

that's just the super old icx6xxx line, the newer stuff (like v8091) is using at least 4.4.0 looks like:

I'm only running a icx6540 so 'super old' is appropriate. Fortunately, the 88f6281 is so common even on other hardware, if Brocade hasn't done anything too weird to the old kernel it might be possible to run a far more recent one. I know the sheevaplug (and successors) and a gaggle of NAS boxes use the exact same CPU.

Still, not quite something I'm too tempted to attempt until I at least have a spare to tinker with.

 

[fohdeesha]

I know the sheevaplug (and successors) and a gaggle of NAS boxes use the exact same CPU

heh yup, when I had to write a JTAG config from scratch for openOCD for the 6450 I borrowed heavily from some of the existing sheevaplug configs

I wouldn't worry about messing around with the OS firmware, it's not like you're re-writing the bootloader. You can flash the most screwed up image possible and if it doesn't work you can just drop into u-boot and flash the stock firmware back. Even if you did flash a new u-boot for some reason and it went wrong, the boot memory on the 6450 is SPI (mx25l25635emi-12g) so you could most likely just use a 10 dollar SPI reader/writer and a chip clip to dump u-boot back on it

 

[Smbaker]

Thanks again to fohdeesha for this thread. I was all prepared to spend around $350 on a new Mikrotik and instead I have this wonderful ICX6450-24P for just under two hundred bucks. Following the guide and licensing went without issue.

I installed a couple digikey 259-1633-ND in place of the factory fans and it's now silent at least to the ability of my hearing. It's not that the stock fans were loud, not compared to some other equipment I've owned, but they were loud enough. I don't even hear the PWM noise that people have complained of in this thread; it's possible it's either above my auditory range or a perfect match for my tinnitus.

This is going to allow me to get rid of a pair of Cisco SG300 switches that I had, going down to just one switch, with more PoE than I had before, and the 10G ports.

 

[Roelf Zomerman]

Given I'm putting up camera's outside the house (and therefore extending the LAN to outside). I was wondering what the best way is to Mac-Lock the ports on the 6450? So, if someone just adds something on the wire, the switch would deny it?

(I know not the best method, but it's a start... - the cams dont support 802.1x)

 

[fohdeesha]

you can use the generic MAC ACLs but it's simpler to use the MAC security feature, which was made for your particular application. basically you specify "secure" MACs and these are allowed to work, nothing else is (either on one port, or globally). If that port sees another mac, it can disable the port, creates a snmp trap, generates a log entry in the switch (as well as syslog server if you have one configured)

something like:
int e 1/1/5
port security
secure-mac-address 3adc.9f45.7a76
#if the port has tagged vlans on it, which I doubt yours does with the camera, you put the vlan ID on the end (not necessary for untagged ports)
secure-mac-address 3adc.9f45.7a76 10
#tell it to shutdown the port if it sees another mac, and for how long in minutes. 0 means forever until you manually re-enable it
violation shutdown 5

#show status
show port security e 1/1/5

more details starting on page 189 of fastiron-08030n-securityguide.pdf in the firmware zip from the update guide. If your camera has a sticker on it that includes the MAC address, I advise removing it if relying on this type of security

 

[Roelf Zomerman]

Thanks!

 

[tojoski]

So I did some testing on my 6450 last night with the replacement sunon fans and something is amiss.

The more I look at it, I think these might be knock-offs.. they make a sound almost like old hard drive would.

Comparing the one I am holding to a used one I found on ebay, the ones that i received seem to be of notably lesser quality, both in construction and the label.



I bought them off of a Amazon, but they were a third-party seller. Anyone else run into this?

 

[fohdeesha]

Anyone need licenses I only have 1.9660494e+39 left

 

[Smbaker]

My ICX6450-24P with Sunon fan mod is sitting at 56.5 degrees on Sensor B and 60.5 degrees on Sensor A. Currently running at fan speed 1. This is with four PoE cams, a PoE Access point, and a PoE Raspberry pi. About 30 watts of PoE total. Plus about a dozen ports of regular Ethernet.

This is only about 3.5 degrees away from engaging fan speed 2, which happens at 64C.

Is there a way to see the fan RPMs on this switch?

Do these temperatures seem reasonable for a modded unit? Nothing to worry about?

 

[fohdeesha]

those temps are fine, as you said it hasn't even decided it's warm enough to kick up to speed 2. If it's trying to run speed 2 all the time and approaching warning temp, that's when you should start to worry

there's a hidden debug command to get raw fan RPM, but I can't remember if it works on the 6450:

dm fan-speed

 

[kzoo]

Anyone need licenses I only have 1.9660494e+39 left

Thanks to all of your hard work, I have an ICX7450 on the way, and I may be getting an ICX6450. So yes, I may be asking you for a license soon!

 

[tojoski]

Anyone need licenses I only have 1.9660494e+39 left

I might, I haven't gotten that far with mine yet.

I did find and order a suitable set of fans.. These are 7.7CFM @ 21DB, have the RD output and best of all available from a reputable seller.

MF40201V2-1000U-A99 Sunon Fans | Fans, Thermal Management | DigiKey