Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

TheCodeLife

New Member
Mar 29, 2019
20
3
3
@LodeRunner and @fohdeesha Thank you both for the responses! I will see what I can do about copying the data from flash and copying the keys out. I purchased the switch for less than $50, so it's not a big loss if I can't get it working. I'll let you know if I have any success. If I'm successful with that, do you expect the keys will also work for the switch @LodeRunner has? I'm certainly happy to share the keys with him if I can successfully extract them from this switch.
 

LodeRunner

Member
Apr 27, 2019
42
21
8
I was able to mount a USB stick while running the underlying OS, if you can at least get Linux to boot. I haven't tried anything since working on it with you @fohdeesha. The last thing I tried was to boot my working 7150 to Linux, copied the pem files I could find there and load the onto the broken one, but it never worked.

There are TPM related commands in the firmware, but they fail:
Code:
ICX7150-C12 Router#dm create_device_profile_and_trustpoint
PKI: Error in opening certificate file - Manufacturing certificate file.

Error: File not found
Info: Device certificate import is failed ..!!, ret: 16
Error: read_private_key_from_tpm, Private key file ../opt/tpm/mfg-wrapped-key.pem does not exists...!!
pki_import_device_key_file, load tpm private key is failed..!!
Error: key do not exist
Info: Device lable creation is failed ..!!, ret :24
These are the files I found on the working 7150 and copied to the broken one:
Code:
/opt/tpm/system.data
/opt/tpm/bkp-mfg-system.data
/opt/tpm/mfg-wrapped-key.pem
/opt/tpm/mfg-md5sum.txt
/opt/tpm/mfg-cert.pem
/opt/tpm/mfg-key.pem
@TheCodeLife If you can back them up on yours and restore them and it works after the NAND erase, I would love to see if they can be used to resurrect my unit. Per work with Fohdeesha, the issue on my switch is that the tpm-tools package is ripped out of the firmware, so I can't just drop to Linux and use those to reinitialize. He thinks, and I agree, that Ruckus must have some sort of way to reinit these (and other TPM based switches) without binning them when they get RMA's for bad flash chips.


I can easily setup a system on a Debian Live image and hook it up by serial to the problem child for anyone who wants to take a crack at it.

Is there JTAG access on the 7150's? I don't have JTAG tools, but maybe one could use JTAG to read out memory of a working switch and write it to one that's not?

Per Ruckus, if a switch with a TPM goes screwy on you, you must open a RMA: Troubleshooting ICX-to-SmartZone Connectivity
On non-TPM switches, there are OS commands to regenerate/replace the certificates.

Post history from April on this:

So the solution appears to be either: hope that files can be copied to the proper locations and work, or compile tpm-tools for whatever Linux the 7150 is running.
 
Last edited:

infoMatt

Active Member
Apr 16, 2019
146
53
28
@TheCodeLife If you can back them up on yours and restore them and it works after the NAND erase, I would love to see if they can be used to resurrect my unit.
I won't place any bet on it; the private keys are embeded inside the TPM chips and they can't be extracted in any way. The only feasible operation is to regenerate another keypair, but, as you said, you'll need the correct tool to communicate with the chip to do so.
 

fohdeesha

Kaini Industries
Nov 20, 2016
1,850
1,644
113
29
fohdeesha.com
yeah that's the main problem, the pub keys on NAND that you'll erase match with what's stored in the TPM. If you just copy someone else's in, they're not going to match the priv keys in your TPM and it's going to fail in the same manner. It's really obnoxious, the 7150, the lowest end model, is the only 7 series with a TPM keystore as far as I know. have not seen it on 7250s, 7450s, or 7750s, where you can just regenerate a keypair because it's stored in a dumb folder
 

noise850

New Member
Feb 28, 2020
10
0
1
I just replaced the stock fan in a 6450-48 non PoE with a Sunon MB40201V3-000U-G99 and I am a bit concerned about heat. The ambient temperature of the room is around 75 F right now and with the fan running on high from the dd set_pwrfan_high command I am getting the following temps.

Code:
Power supply 1 (NA - AC - Regular) present, status ok
Power supply 2 not present

Fan ok, speed (auto): [[1]]<->2

Fan speed switching temperature thresholds:
        1 -> 2 @ 69 deg-C
        1 <- 2 @ 64 deg-C

Sensor B Temperature Readings:
        Current temperature : 48.0 deg-C
Sensor A Temperature Readings:
        Current temperature : 60.5 deg-C
        Warning level.......: 66.0 deg-C
        Shutdown level......: 76.0 deg-C
The noise from the fan is noticeable in high mode and there is also a low pitch whirring noise only present with the lid on (can't hear it with the lid off on the bench). I've only got one Gb trunk line going in and one Gb access port in use as a test, with very low utilization and I am a bit worried about cooking this thing under a full load or if I turn the fan back down with dd set_pwrfan_low.

What do you all think?

Edit:

set_pwrfan_low as a test and ran it for 20 minutes and these are the current readings. Am I misunderstanding something as it looks like Sensor A is about to hit Warning level?

Code:
Fan ok, speed (auto): [[1]]<->2

Fan speed switching temperature thresholds:
        1 -> 2 @ 69 deg-C
        1 <- 2 @ 64 deg-C

Sensor B Temperature Readings:
        Current temperature : 53.0 deg-C
Sensor A Temperature Readings:
        Current temperature : 65.0 deg-C
        Warning level.......: 66.0 deg-C
        Shutdown level......: 76.0 deg-C
Edit 2:

It's now the hottest part of the day for this room. Sensor B is at 55. Sensor A at 67. Nothing in the log file except my ssh connections to check temps and logging level is set at ACDMEINW. I am now really confused.
 
Last edited:

TheCodeLife

New Member
Mar 29, 2019
20
3
3
@fohdeesha I was able to unlock the hidden bootloader and I do see a tftpput command. However, I'm struggling to understand how to dump the nand into a file. I was hoping to do something like piping the output from the nand read.raw command into the tftpput utility, but I've been unsuccesful with that so far. I can create empty files using the tftpput command, so I know it's connecting to my tftp server.

Here are the nand menu options if it's helpful at all:

Code:
nand info - show available NAND devices
nand device [dev] - show or set current device
nand read - addr off|partition size
nand write - addr off|partition size
    read/write 'size' bytes starting at offset 'off'
    to/from memory address 'addr', skipping bad blocks.
nand read.raw - addr off|partition [count]
nand write.raw - addr off|partition [count]
    Use read.raw/write.raw to avoid ECC and access the flash as-is.
nand erase[.spread] [clean] off size - erase 'size' bytes from offset 'off'
    With '.spread', erase enough for given file size, otherwise,
    'size' includes skipped bad blocks.
nand erase.part [clean] partition - erase entire mtd partition'
nand erase.chip [clean] - erase entire chip'
nand bad - show bad blocks
nand dump[.oob] off - dump page
nand scrub [-y] off size | scrub.part partition | scrub.chip
    really clean NAND erasing bad blocks (UNSAFE)
nand markbad off [...] - mark bad block(s) at offset (UNSAFE)
nand biterr off - make a bit error at offset (UNSAFE)

EDIT: I have discovered that nand read will put the data directly in RAM. I can then use tftpput to dump a user specified portion of the RAM into a file on the tftp server. The biggest problem with this process that I'm encountering is that nand read seems to hang instead of reading past the bad sectors. I was hoping to use nand read.raw to get around this issue, but nand read.raw doesn't seem to place the read data into RAM, so I don't know how to get it. nand read.raw also reads a different amount of data than nand.read.

nand read.raw example:
Code:
ICX7150-Boot>nand read.raw 0 0 1

NAND read:  4320 bytes read: OK
nand read example:
Code:
ICX7150-Boot>nand read 0 0 1

NAND read: device 0 offset 0x0, size 0x1
1 bytes read: OK
If anyone has any ideas I would appreciate the input.

EDIT 2: I'm also having problems with tftpput only working when uploading small amounts of data (~1KB). Larger amounts of data transfer nothing and just timeout. It might just be my TFTP program on Windows, so I'll try setting up a TFTP server on Linux later and try again.
 
Last edited:

aidenpryde

New Member
Apr 30, 2020
21
1
3
So, I'm looking to quiet down the 1 fan that this ICX 6450-24 has in it. I did a little searching in this thread and I've seen the NF-A4x20 FLX and the Sunon MB40201VX-000U-G99 has been used by a couple folks, but it looks like these fans will cause a boot loop until the stock fan is put back in.

Can anyone tell me if they've managed to get a quieter fan inside this that doesn't have that issue?
 

infoMatt

Active Member
Apr 16, 2019
146
53
28
So, I'm looking to quiet down the 1 fan that this ICX 6450-24 has in it. I did a little searching in this thread and I've seen the NF-A4x20 FLX and the Sunon MB40201VX-000U-G99 has been used by a couple folks, but it looks like these fans will cause a boot loop until the stock fan is put back in.

Can anyone tell me if they've managed to get a quieter fan inside this that doesn't have that issue?
6450 will boot even without any fans plugged in, no problem.
 

OptimusPrime

New Member
Apr 21, 2020
25
4
3
This is fun. I got all 4 of my RJ45 SFP transceivers installed into my 6450. All three PCs have their 10Gbe cards. Looks like I can transfer between a PC’s M2 card and another PC’s SSD at 3.5 Gbps max…which is close to maxing out the SATA interface on SSD.

However, mechanical drives are disappointing. Even though they are SATA III, my transfer speeds when reading from one of the mechanical drives averages 111 MB/s. It’s a little less when writing to them.
 

infoMatt

Active Member
Apr 16, 2019
146
53
28
This is fun. I got all 4 of my RJ45 SFP transceivers installed into my 6450. All three PCs have their 10Gbe cards. Looks like I can transfer between a PC’s M2 card and another PC’s SSD at 3.5 Gbps max…which is close to maxing out the SATA interface on SSD.

However, mechanical drives are disappointing. Even though they are SATA III, my transfer speeds when reading from one of the mechanical drives averages 111 MB/s. It’s a little less when writing to them.
Yes, those are the expected performance out of a spinning hard drive. It will vary with the size of the disk, number of plates and speed, but it should start around 120-180MB/s for the outer tracks, and end at say 80-120MB/s for the innermost tracks.

To get faster speeds, you'll need a RAID 0/5/10 configuration. ;)
 

infoMatt

Active Member
Apr 16, 2019
146
53
28
I figured that out later. Will the 6450 take whatever 3-pin fan I give it?
It should... They are standard +12V GND and PWM signal, but pay attention that the pinout is not the standard one used for computer fans; a user made a video, reposted in the last few pages, in which it shows the correct pins to swap.
 

aidenpryde

New Member
Apr 30, 2020
21
1
3
Thanks. Does anyone have a guide on how to get these working with pfSense. I don't think I'm capable of translating some of the guides you see around for Netgear and Cisco switches into Brocade. If I can't get this working I'm going to have to return this or resell it.
 

bubsterboo

Member
Dec 15, 2019
33
11
8
Thanks. Does anyone have a guide on how to get these working with pfSense. I don't think I'm capable of translating some of the guides you see around for Netgear and Cisco switches into Brocade. If I can't get this working I'm going to have to return this or resell it.
Unless you need to get into vlans or any kind of more advanced configuration then there's really not much special to it.
Follow the guide from the OP to get your switch in a good default state. Plug the LAN side of your pfsense into any port on the switch and that's really it!
Ofcourse there's a ton of features and goodies if you want to get more into it. But you don't need to!
 

OptimusPrime

New Member
Apr 21, 2020
25
4
3
Is there a good primer thread on pfSense for home hobbyists? It seems to be a recurring topic here, I'd like to learn more about it.
 

aidenpryde

New Member
Apr 30, 2020
21
1
3
Unless you need to get into vlans or any kind of more advanced configuration then there's really not much special to it.
Follow the guide from the OP to get your switch in a good default state. Plug the LAN side of your pfsense into any port on the switch and that's really it!
Ofcourse there's a ton of features and goodies if you want to get more into it. But you don't need to!
That's the thing, I don't want anything complex, but no matter what I do here it doesn't seem as though I'm understanding.

All I want is 5 VLANS similar to the guide here: pfSense baseline guide with VPN, Guest and VLAN support

But this is to too much for me to do via CLI without some kind of guide. If I could get the web interface up when connected to pfSense, maybe I could do it, but I can't do it with my knowledge.
 

bubsterboo

Member
Dec 15, 2019
33
11
8
That's the thing, I don't want anything complex, but no matter what I do here it doesn't seem as though I'm understanding.

All I want is 5 VLANS similar to the guide here: pfSense baseline guide with VPN, Guest and VLAN support

But this is to too much for me to do via CLI without some kind of guide. If I could get the web interface up when connected to pfSense, maybe I could do it, but I can't do it with my knowledge.
I don't mean to be a debby downer or anything. But.. That guide isn't exactly for a simple network setup. It's not the most complicated. It's not impossible. But it isn't simple either. It will require a lot of reading and some patience to get working properly. Likewise you will need a bit of patience to learn how to manage VLANs on the brocade switches. Its not complicated and you don't need to fear the CLI. It took me about half a day to learn about VLANs and read through the brocade fastiron documentation to get something similar going.
 

klui

Active Member
Feb 3, 2019
131
50
28
Not a good experience with going to 08.0.92b on an ICX 7150-C12P from 08.0.70c.

I couldn't go directly from 70 to 92 because 70 is a non-UFI while 92 is. I had to go to 08.0.80 non-UFI first. I decided to have primary on 80f and secondary 92b. The upgrade to 80f was straightforward. copy from tftp to flash the 80f's bootrom, image to primary, then reload. The nice thing was after bootup the PoE FW is automatically updated. And the system prints progress to the console. The ICX6610 running 08.0.30u does not print progress but prevents you from rebooting until the PoE FW is done after a manual upgrade. show logging will display progress.

The problem was upgrading the secondary to 92b. Because it's a unified image, the bootrom is embedded in the ufi.bin file. When I boot back to 80f, I'm met with a boot-monitor version mismatch at startup. Back in 92b, it shows there are primary and secondary boot code partitions. They both are the version recommended for 92 but the copy to flash bootrom command has been deprecated and there are no commands to replace the primary boot code. Under 80f, there is only one flash bootrom command and there is no way to specify which partition to use. What's more 92b displays "Moving app to flash...." upon every boot and takes an annoyingly long (~15 sec) time. Going back from 92b to 80f loses some stacking/trunk port definitions if there is no pre-8090-startup-backup file. What was a convenience of PoE FW auto flashing is now a hassle. Switching between versions will cause during first reboot into the partition upgrade/downgrade of PoE FW which takes around 2 minutes.

The above served me right because reading the Software Upgrade guide shows Ruckus recommending upgrades of pre-08.0.80 by

flash 80 bootrom
flash non-UFI 80 image primary
reboot
flash 92 image primary
flash 92 image secondary
reboot

There is no support for different versions between partitions if their recommended bootroms are different.

Keeping 80f on primary for now.
 

aidenpryde

New Member
Apr 30, 2020
21
1
3
I don't mean to be a debby downer or anything. But.. That guide isn't exactly for a simple network setup. It's not the most complicated. It's not impossible. But it isn't simple either. It will require a lot of reading and some patience to get working properly. Likewise you will need a bit of patience to learn how to manage VLANs on the brocade switches. Its not complicated and you don't need to fear the CLI. It took me about half a day to learn about VLANs and read through the brocade fastiron documentation to get something similar going.
Yeah, okay, that's annoying. Figure I should have gone with Netgear or Cisco as this is essentially a dumb switch for me now.
 

gb00s

Active Member
Jul 25, 2018
169
45
28
Malta
That's the thing, I don't want anything complex, but no matter what I do here it doesn't seem as though I'm understanding.

All I want is 5 VLANS similar to the guide here: pfSense baseline guide with VPN, Guest and VLAN support

But this is to too much for me to do via CLI without some kind of guide. If I could get the web interface up when connected to pfSense, maybe I could do it, but I can't do it with my knowledge.
Go to and check out Terry Henry's YT channel. he has some nice and easy to understand vid's about setting up & configuring a Brocade switch via CLI in FastIron OS. It's super cool and I'm glad I didn't give up. It's going to be fun to install a switch via CLI and you will be able to do it as well and learn something. Yes, you can buy a Cisco SG200-10 and just 'copy & paste' and learn nothing. But what if you want to change something in the setup later or you have to troubleshoot a problem and no blog in the WWW can guide you with your specific issue?

In my personal opinion, if you have the Layer3 setup on your future switches and you want to combine it with pfSense, it's not fun either CLI or not.

It's not hard to learn. I'm, just trying to push you and encourage you in a positive way. If you got your VLAN in FastIron working, you will be able to do the next in almost every Cisco switch via CLI as well. Both are so close to each other that you can use what you learned. Give yourself a push. Brocades are awesome.

Regards

Mike
 
Last edited: