Drag to reposition cover

Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

infoMatt

Active Member
Apr 16, 2019
222
100
43
Why don't you move your DAC that's in 1/3/1 throughout all the ports to get an idea what ports are licensed?
No need to mess around, just issue a "show pod" command ad it will tell you which port are licensed.

@Ryan Haver Have you tried looping back with a known-working DAC ethernet 1/3/2 and 1/3/3 toghether? Does the link come up in this case?
Try also a "show pod" just for sake of completeness.
The ports are working, because at 1G they come up, it's strange that they won't work at 10G...
 

klui

Well-Known Member
Feb 3, 2019
824
453
63
I don't remember the logical port numbers but the rear QSFP+ ports are labeled indicating the 2 left ports (XL1 and XL6) only permit 40G QSFP+ while the right 2 ports (XL2-5 and XL7-10) only permit breakouts. See https://forums.servethehome.com/ind...gbe-40gbe-switching.21107/page-79#post-223244

My 6610 came licensed the same as yours and I don't plan on applying the "lab" license until I read off my current license through I2C. All my licensed ports work at 10G. I don't recall what happened placing a breakout in XL1 and XL6. The breakouts should be designated for 1/2/2--1/2/5, and 1/2/7--1/2/10.
 

fohdeesha

Kaini Industries
Nov 20, 2016
2,728
3,076
113
33
fohdeesha.com
Ports 1/3/1 to 1/3/4 are the licensed ports because 1/3/5 to 1/3/8 are stuck in 1Gig mode and throw the following error when I try to change them to 10Gig "No license present for port". The licensed ports are still acting problematic though.

Ports 1/1/1 to 1/1/48
1Gig copper ports and all seem to work fine

Ports 1/2/1 to 1/2/10
Ports 1/2/1 and 1/2/6 are currently working (running at 40Gbps with QSFP+ DAC cables)
Ports 1/2/2 and 1/2/7 are currently working (running at 10Gig with QSFP+ to 4 x SFP+ Breakout DAC)
Ports 1/2/3 to 1/2/5 (1st QSFP+ breakout cable) and 1/2/8 to 1/2/10 (2nd QSFP+ breakout cable) none seem to work, need to test further

Ports 1/3/1 to 1/3/8
Ports 1/3/1 and 1/3/4 working at both 1Gig and 10Gig (tested with an assortment of transceivers and DAC Cables)
Ports 1/3/2 and 1/3/3 work only at 1Gig, 10Gig not working (tested with 1Gig copper transceivers and 10Gig DAC and Copper transceivers)
Ports 1/3/5 to 1/3/8 are not licensed and thus only work at 1Gig, but may have the same issues other ports have after being licensed

So ... part of the issue could be licensing, but then once licensed ports 1/3/5 to 1/3/8 may still not work at 10Gig, just like the other misbehaving. I'm going to wait on a response back from fohdeesha on the licensing and once all ports are licensed then I'll test some more.

There is a new firmware 08.0.30u that was released at the end of last month which fixes defects....but it isn't clear what those defects are. It'd be pretty funny if this is one of those defects that is fixed, but honestly, I'm still trying to make sure it isn't some weird configuration mistake on my part.
I can almost promise it's license related, I'll send you some when I get up in a few hours
 
  • Like
Reactions: tommybackeast

fohdeesha

Kaini Industries
Nov 20, 2016
2,728
3,076
113
33
fohdeesha.com
I don't remember the logical port numbers but the rear QSFP+ ports are labeled indicating the 2 left ports (XL1 and XL6) only permit 40G QSFP+ while the right 2 ports (XL2-5 and XL7-10) only permit breakouts. See https://forums.servethehome.com/ind...gbe-40gbe-switching.21107/page-79#post-223244

My 6610 came licensed the same as yours and I don't plan on applying the "lab" license until I read off my current license through I2C. All my licensed ports work at 10G. I don't recall what happened placing a breakout in XL1 and XL6. The breakouts should be designated for 1/2/2--1/2/5, and 1/2/7--1/2/10.
You can apply the licenses I sent while leaving your current ones intact so they don't need to be deleted. If you still want to read them off to back them up, you'll need to do a flash dump from the bootloader as described here: Extracting Licenses - Fohdeesha Docs

I can also generate you some new lics unique to your original license ID, but I try not to make a habit of that
 

OptimusPrime

Member
Apr 21, 2020
44
6
8
Well, that doesn't sound very healty indeed... Probably a bad card; let's hope for the best for your second one ;).
After three cards, we have success...installed card, plugged in the cable, and --- voila! --- it just worked. I don't have to do anything to either the card settings or the switch. It auto negotiates regardless if plugged into a Gbe or 10GBe port on the switch and it just works!
 

aidenpryde

New Member
Apr 30, 2020
27
1
3
So, I followed the instructions to update the firmware, but no matter what I do I can't get to the webUI, and none of the Brocade documentation seems to apply as it's now a Layer 3 switch and the documentation assumes that you're using the Layer 2 firmware. Getting frustrated at this point.
 

fohdeesha

Kaini Industries
Nov 20, 2016
2,728
3,076
113
33
fohdeesha.com
So, I followed the instructions to update the firmware, but no matter what I do I can't get to the webUI, and none of the Brocade documentation seems to apply as it's now a Layer 3 switch and the documentation assumes that you're using the Layer 2 firmware. Getting frustrated at this point.
did you follow my documentation ALL the way through? I assume not, otherwise the web UI would be coming up. Paste the output of "show run" - also all the documentation is included in the firmware zip in my guide, they are split up by category. the only document that assumes layer 2 is the layer 2 pdf
 
  • Like
Reactions: Jason Antes

aidenpryde

New Member
Apr 30, 2020
27
1
3
So, I think a lot of my frustration was learning CLI. I also did a dumb thing and didn't consider that since the switch's DHCP server was off that I had to use a static IP on the client to get to the webUI. Then, once I registered the switch with the router's DHCP I was able to give the switch itself a static IP and other clients can access it now too. Thanks.
 
  • Like
Reactions: tommybackeast

tommybackeast

Active Member
Jun 10, 2018
286
105
43
did you follow my documentation ALL the way through? I assume not, otherwise the web UI would be coming up. Paste the output of "show run" - also all the documentation is included in the firmware zip in my guide, they are split up by category. the only document that assumes layer 2 is the layer 2 pdf
as a non IT Professional - I would just like to compliment you on your clarity of instructions. You write an excellent How-To
 

kousuke

New Member
Jul 18, 2017
24
0
1
40
out of interest is it possible to flash newer firmware on ICX6610? I'm so into ICX now that i'm bought a number of R710 APs running in unleashed mode. ICX 08.0.90 or later firmware is required to manage the switch though the Ruckus unleashed interface.
 

fohdeesha

Kaini Industries
Nov 20, 2016
2,728
3,076
113
33
fohdeesha.com
out of interest is it possible to flash newer firmware on ICX6610? I'm so into ICX now that i'm bought a number of R710 APs running in unleashed mode. ICX 08.0.90 or later firmware is required to manage the switch though the Ruckus unleashed interface.
nope, 6 series is 8030 only
 

TheCodeLife

New Member
Mar 29, 2019
25
3
3
I purchased a broken ICX7150-C12P in hopes it would have a failed PSU issue and have an easy fix. I had success with 3 of these in the past, but unfortunately this switch has a different problem that I'm struggling to fix. The PSU seems to be functional, albeit fairly noisy. Here is the interesting output I received during boot:

Code:
Starting kernel ...

booting with 4.4 kernel
NAND Type: Micron NAND 2GiB (x 1)
PLATFORM MN
dd cmd: UBI
ubi0 error: ubi_io_read: error -74 (ECC error) while reading 1040384 bytes from PEB 1132:8192, read 1040384 bytes
UBIFS error (ubi0:0 pid 565): ubifs_recover_leb: corruption -3
UBIFS error (ubi0:0 pid 565): ubifs_check_node: bad node length 536871026
UBIFS error (ubi0:0 pid 565): ubifs_check_node: bad node at LEB 730:524288
        magic          0x6101831
        crc            0x217bf7f2
        node_type      1 (data node)
        group_type     0 (no node group)
        sqnum          288230444905571979
        len            536871026
        key            (1073742972, data, 7640)
        size           2181039616
        compr_typ      1
        data size      536870978
Immediately following this is a data dump which I don't think would be useful. I followed the guide here using the SPR08080e.bin file in place of the one listed in the link: Recovering Software in an attempt to correct the issue, but it fails again when I use the boot_primary command.

I'm wondering if the flash chip in the switch has failed. Does anyone know what, if anything, I can do to fix this issue?
 

LodeRunner

Active Member
Apr 27, 2019
540
227
43
I purchased a broken ICX7150-C12P in hopes it would have a failed PSU issue and have an easy fix. I had success with 3 of these in the past, but unfortunately this switch has a different problem that I'm struggling to fix. The PSU seems to be functional, albeit fairly noisy. Here is the interesting output I received during boot:

Code:
Starting kernel ...

booting with 4.4 kernel
NAND Type: Micron NAND 2GiB (x 1)
PLATFORM MN
dd cmd: UBI
ubi0 error: ubi_io_read: error -74 (ECC error) while reading 1040384 bytes from PEB 1132:8192, read 1040384 bytes
UBIFS error (ubi0:0 pid 565): ubifs_recover_leb: corruption -3
UBIFS error (ubi0:0 pid 565): ubifs_check_node: bad node length 536871026
UBIFS error (ubi0:0 pid 565): ubifs_check_node: bad node at LEB 730:524288
        magic          0x6101831
        crc            0x217bf7f2
        node_type      1 (data node)
        group_type     0 (no node group)
        sqnum          288230444905571979
        len            536871026
        key            (1073742972, data, 7640)
        size           2181039616
        compr_typ      1
        data size      536870978
Immediately following this is a data dump which I don't think would be useful. I followed the guide here using the SPR08080e.bin file in place of the one listed in the link: Recovering Software in an attempt to correct the issue, but it fails again when I use the boot_primary command.

I'm wondering if the flash chip in the switch has failed. Does anyone know what, if anything, I can do to fix this issue?
You can try erasing the NAND. I ran into this with one of the two I bought with dead PSU's, but then ended up with a TPM issue that I should repost here and see if there're any new ideas (switch boots fine, complains about TPM errors, and will idle just fine, as long as none of the switch ports are connected; once that happens, it kernel panics in short order).

"nand erase.chip" is the command, then you would follow that recovery document to reflash it.

Here's what mine was doing on boot prior to erasing the NAND:

Code:
Starting kernel ...

booting with 4.4 kernel
NAND Type: Micron NAND 2GiB (x 1)
PLATFORM MN
dd cmd: UBI
ubi0 error: ubi_io_read: error -74 (ECC error) while reading 1040384 bytes from PEB 841:8192, read 1040384 bytes
UBIFS error (ubi0:0 pid 565): ubifs_recover_leb: corruption -3
UBIFS error (ubi0:0 pid 565): ubifs_check_node: bad node length 263218
UBIFS error (ubi0:0 pid 565): ubifs_check_node: bad node at LEB 1134:884736
    magic          0x6101831
    crc            0x87f8e43e
    node_type      1 (data node)
    group_type     128 (unknown)
    sqnum          1378685144455
    len            263218
    key            (4194457, xentry, 0x00205e)
    size           335548416
    compr_typ      1344
    data size      263170
    data:
    00000000: d6 c1 c7 35 d4 a1 e1 f2 c1 d0 ef 02 7e 99 b9 63 4f 17 43 76 22 8a 8d f7 1e d1 49 19 d3 0b a3 60
    [trimmed]
    00025fa0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
{ bunch of stack trace stuff }
Segmentation fault

Mounting Config partition failed, non-recoverable file system corruption
Reformatting the flash, please download config and keys again ...

ubidetach: error!: cannot detach mtd2
           error 16 (Device or resource busy)
ubiformat: error!: please, first detach mtd2 (/dev/mtd2) from ubi0
Formatting Doneubi0 error: ubi_attach_mtd_dev: mtd2 is already attached to ubi0

ubiattach: error!: cannot attach mtd2
           error 17 (File exists)
ubimkvol: error!: UBI device does not have free logical erasebloubi1: attaching mtd3
cks
!!!ERROR:ubimkvol failed for config partition
ubi1: scanning is finished
ubi1: attached mtd3 (name "resources", size 512 MiB)
ubi1: PEB size: 1048576 bytes (1024 KiB), LEB size: 1040384 bytes
ubi1: min./max. I/O unit sizes: 4096/4096, sub-page size 4096
ubi1: VID header offset: 4096 (aligned 4096), data offset: 8192
ubi1: good PEBs: 504, bad PEBs: 8, corrupted PEBs: 0
ubi1: user volume: 1, internal volumes: 1, max. volumes count: 128
ubi1: max/mean erase counter: 1/0, WL threshold: 4096, image sequence number: 677597375
ubi1: available PEBs: 0, total reserved PEBs: 504, PEBs reserved for bad PEB handling: 32
ubi1: background thread "ubi_bgt1d" started, PID 582
UBIFS (ubi1:0): background thread "ubifs_bgt1_0" started, PID 586
UBIFS (ubi1:0): recovery needed
UBIFS (ubi1:0): recovery completed
UBIFS (ubi1:0): UBIFS: mounted UBI device 1, volume 0, name "resources"
UBIFS (ubi1:0): LEB size: 1040384 bytes (1016 KiB), min./max. I/O unit sizes: 4096 bytes/4096 bytes
UBIFS (ubi1:0): FS size: 476495872 bytes (454 MiB, 458 LEBs), journal size 23928832 bytes (22 MiB, 23 LEBs)
UBIFS (ubi1:0): reserved for root: 4952683 bytes (4836 KiB)
UBIFS (ubi1:0): media format: w4/r0 (latest is w4/r0), UUID 8068C335-3096-4AA2-983E-4659B278DEBA, small LPT model
UBIFS (ubi1:0): full atime support is enabled.
Resource partition is mounted!!
Restore is Done
Kernel crashdump is disabled.
Backup is fine
dma_mem_base: 0x96000000, dma_mem_len: 0x8000000, warm_mem_base: 0x9f500000, warm_mem_len: 0xb00000
Creating TUN device
Starting TPM Infra
Group tss and User tss created
kernel.hostname = localhost
hostname set to localhost
Starting TCSD Daemon
TCSD Up and Running
Enabling time stamp.
Starting the FastIron.
USER=root
HOME=/
GCOV_PREFIX=/fast_iron
ethaddr=78:a6:e1:2e:48:29?
TERM=vt102
PATH=/sbin:/usr/sbin:/bin:/usr/bin
crashkernel=64M@0x70000000
ip=192.168.0.210:192.168.0.14::255.255.255.0:MN:eth0:off
SHELL=/bin/sh
PWD=/
i2c-0    i2c          Broadcom iProc I2C adapter          I2C adapter    2
i2c-1    i2c          Broadcom iProc I2C adapter          I2C adapter    2
!!!WARNING: Unable to read Filesystem information!!!
Total no. of blocks in FS = 0
Total no. of free blocks = 0
FIPS Disabled:PORT NOT DISABLED
platform type 90
OS>
Loaded Image SPR08080e from Pri Built on (UTC): Tue Apr  9 10:25:33 2019
Enabling Console Logging
flash_file=0.
Upgrade from pre 8.0.80 image. Dual mode to VLAN config upgrade needed

Hotplugger Daemon Initialized ..
External USB disk is not mounted or plugged in.
set_board_level: gi_board_type = 116[  628.362222] linux-kernel-bde: map phys range 0x3200000-0x3240000 to 0xb0d09000-0xb0d49000
[  628.544467] linux-kernel-bde: map phys range 0x88100000-0x90100000 to 0xa8c00000-0xb0c00000
DMA pool size: 134217728
PHY Reset De-asserted

sw_pp_sdk_init:137 ToR init

SOC unit 0 attached to PCI device BCM56160_B0
Firmware version from File: 2.1.1
Pre Parsing Config Data ...


Done with Pre-allocating nexthop id's for Unicast & Openflow
 
  • Like
Reactions: TheCodeLife

fohdeesha

Kaini Industries
Nov 20, 2016
2,728
3,076
113
33
fohdeesha.com
I purchased a broken ICX7150-C12P in hopes it would have a failed PSU issue and have an easy fix. I had success with 3 of these in the past, but unfortunately this switch has a different problem that I'm struggling to fix. The PSU seems to be functional, albeit fairly noisy. Here is the interesting output I received during boot:

Code:
Starting kernel ...

booting with 4.4 kernel
NAND Type: Micron NAND 2GiB (x 1)
PLATFORM MN
dd cmd: UBI
ubi0 error: ubi_io_read: error -74 (ECC error) while reading 1040384 bytes from PEB 1132:8192, read 1040384 bytes
UBIFS error (ubi0:0 pid 565): ubifs_recover_leb: corruption -3
UBIFS error (ubi0:0 pid 565): ubifs_check_node: bad node length 536871026
UBIFS error (ubi0:0 pid 565): ubifs_check_node: bad node at LEB 730:524288
        magic          0x6101831
        crc            0x217bf7f2
        node_type      1 (data node)
        group_type     0 (no node group)
        sqnum          288230444905571979
        len            536871026
        key            (1073742972, data, 7640)
        size           2181039616
        compr_typ      1
        data size      536870978
Immediately following this is a data dump which I don't think would be useful. I followed the guide here using the SPR08080e.bin file in place of the one listed in the link: Recovering Software in an attempt to correct the issue, but it fails again when I use the boot_primary command.

I'm wondering if the flash chip in the switch has failed. Does anyone know what, if anything, I can do to fix this issue?
EDIT: Loderunner beat me to it, but don't erase the chip before reading my second paragraph

the NAND flash has at least one bad sector - the issue is you can erase and reformat the NAND, and it will map that out and work normally, but it erases encryption keys that only the ICX7150 uses, and the OS will constantly bitch about the missing key files and crash when interfaces are connected. @LodeRunner and I went through exactly this and never really found a solution, you may want to contact him (or file a return with the seller)

I suppose since you haven't erased it yet, in the bootloader you could copy out the flash contents (specifically the area where the linux filesystem is), open it with 7zip or whatever and pull the keys out of the CPIO file system. Can't remember what commands u-boot has on the 7xxx series for copying FROM flash, I'd imagine after unlocking the dev bootloader mode there might be a copy to tftp option
 
  • Like
Reactions: TheCodeLife

TheCodeLife

New Member
Mar 29, 2019
25
3
3
@LodeRunner and @fohdeesha Thank you both for the responses! I will see what I can do about copying the data from flash and copying the keys out. I purchased the switch for less than $50, so it's not a big loss if I can't get it working. I'll let you know if I have any success. If I'm successful with that, do you expect the keys will also work for the switch @LodeRunner has? I'm certainly happy to share the keys with him if I can successfully extract them from this switch.
 

LodeRunner

Active Member
Apr 27, 2019
540
227
43
I was able to mount a USB stick while running the underlying OS, if you can at least get Linux to boot. I haven't tried anything since working on it with you @fohdeesha. The last thing I tried was to boot my working 7150 to Linux, copied the pem files I could find there and load the onto the broken one, but it never worked.

There are TPM related commands in the firmware, but they fail:
Code:
ICX7150-C12 Router#dm create_device_profile_and_trustpoint
PKI: Error in opening certificate file - Manufacturing certificate file.

Error: File not found
Info: Device certificate import is failed ..!!, ret: 16
Error: read_private_key_from_tpm, Private key file ../opt/tpm/mfg-wrapped-key.pem does not exists...!!
pki_import_device_key_file, load tpm private key is failed..!!
Error: key do not exist
Info: Device lable creation is failed ..!!, ret :24
These are the files I found on the working 7150 and copied to the broken one:
Code:
/opt/tpm/system.data
/opt/tpm/bkp-mfg-system.data
/opt/tpm/mfg-wrapped-key.pem
/opt/tpm/mfg-md5sum.txt
/opt/tpm/mfg-cert.pem
/opt/tpm/mfg-key.pem
@TheCodeLife If you can back them up on yours and restore them and it works after the NAND erase, I would love to see if they can be used to resurrect my unit. Per work with Fohdeesha, the issue on my switch is that the tpm-tools package is ripped out of the firmware, so I can't just drop to Linux and use those to reinitialize. He thinks, and I agree, that Ruckus must have some sort of way to reinit these (and other TPM based switches) without binning them when they get RMA's for bad flash chips.


I can easily setup a system on a Debian Live image and hook it up by serial to the problem child for anyone who wants to take a crack at it.

Is there JTAG access on the 7150's? I don't have JTAG tools, but maybe one could use JTAG to read out memory of a working switch and write it to one that's not?

Per Ruckus, if a switch with a TPM goes screwy on you, you must open a RMA: Troubleshooting ICX-to-SmartZone Connectivity
On non-TPM switches, there are OS commands to regenerate/replace the certificates.

Post history from April on this:

So the solution appears to be either: hope that files can be copied to the proper locations and work, or compile tpm-tools for whatever Linux the 7150 is running.
 
Last edited:

infoMatt

Active Member
Apr 16, 2019
222
100
43
@TheCodeLife If you can back them up on yours and restore them and it works after the NAND erase, I would love to see if they can be used to resurrect my unit.
I won't place any bet on it; the private keys are embeded inside the TPM chips and they can't be extracted in any way. The only feasible operation is to regenerate another keypair, but, as you said, you'll need the correct tool to communicate with the chip to do so.
 

fohdeesha

Kaini Industries
Nov 20, 2016
2,728
3,076
113
33
fohdeesha.com
yeah that's the main problem, the pub keys on NAND that you'll erase match with what's stored in the TPM. If you just copy someone else's in, they're not going to match the priv keys in your TPM and it's going to fail in the same manner. It's really obnoxious, the 7150, the lowest end model, is the only 7 series with a TPM keystore as far as I know. have not seen it on 7250s, 7450s, or 7750s, where you can just regenerate a keypair because it's stored in a dumb folder