Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

[LodeRunner]

Yeah, I copied those files from my working 7150 to the one that I had to erase the entire NAND chip on (it wouldn't just let me erase a partition). I get the following on boot:

ICX7150-C12 Router>2820269152:error:8006F06D:tpm engine:TPM_ENGINE_LOAD_KEY:request failed:e_tpm.c:672:
                                                                                                       2820269152:error:26096080:lib(38):func(150):reason(128):NA:0:
                     /vobs/fdry/build/../../../../..///vobs/mucho/mp/cmds/web_cmds.c:1605 Couldn't load TPM key "../opt/tpm/mfg-wrapped-key.pem" from file.
            update_tls_client_db_for_trustpoint, TPM key file load failed..!!

From the OS console (CTRL+Y, M over serial after booting):

OS>tpm key
TPM Engine:Couldn't load TPM Key File
Library error is: error:8006F06D:tpm engine:TPM_ENGINE_LOAD_KEY:request failed

dm create_device_profile_and_trustpoint generates a similar (identical, it seems) error as is displayed on boot:

ICX7150-C12 Router#dm create_device_profile_and_trustpoint 2788738144:error:8006F06D:tpm engine:TPM_ENGINE_LOAD_KEY:request failed:e_tpm.c:672:
2788738144:error:26096080:lib(38):func(150):reason(128):NA:0:
                                                             /vobs/fdry/build/../../../../..///vobs/mucho/mp/cmds/web_cmds.c:1605 Couldn't load TPM key "../opt/tpm/mfg-wrapped-key.pem" from file.

update_tls_client_db_for_trustpoint, TPM key file load failed..!!

fohdeesha noted that all the tpm related binaries had been removed from the BusyBox image; I tried to run armhf versions of them downloaded from the Debian Jessie repo as the libraries on the system appeared to meet the dependency requirements, but I encountered an error that's either due to compiler/library differences, or the fact that the switch runs sh and not bash.

Edit to add:
This switch really doesn't like having interfaces connected when the TPM doesn't work:

stack: 0147f0f0 0145a420 b6bc37c1 
Application received signal -> SIGNUM#11
Tuning CFS scheduler parameters...
Copying fitrace errorlog file to flash
[  123.059976] [BrcdSoftlockup]: sim_softwatchdog thread is detached on core=0 
CORE_PATTERN:PID=1055 UID=0 GID=0 sig=11
Sat Apr 18 21:20:00 UTC 2020: Dumping core file to /tmp.gz, this will take couple of minutes ...
Segmentation fault (core dumped)
Sat Apr 18 21:24:37 UTC 2020: Core file collected as /tmp.gz, processing the core file ...
Sat Apr 18 21:24:37 UTC 2020: Removing the oldest core file:core_1055-12_2020-04-18_21-07-50.gz of size 8484 kbytes
seq 13
Sat Apr 18 21:24:38 UTC 2020: Calculating checksum: core_1055-14_2020-04-18_21-24-38.gz
Sat Apr 18 21:24:38 UTC 2020: Checking integrity of core_1055-14_2020-04-18_21-24-38.gz

 

[fohdeesha]

That was a DES encrypted password with salt.

openssl passwd --crypt --salt=wY --stdin <pswd

where pswd contains the default barcode password, see attached file.View attachment 13647

Really? that's strange, I've logged into my 7250 using that plaintext string numerous times. Wonder if they either forgot to hash it on older versions (can't remember the last time I tried), or if it's something specific to the TPM switches

 

[fohdeesha]

well indeed, specific to the 7150, so that would explain it (from preinit.sh)

if [[ "$platform" == "MN" ]] ; then
    cp /etc/passwdshadow /etc/passwd
fi

ICX7150 = MN Minion

ICX7250 = SI Sica

ICX7450 = SP Spatha

it did also respond nicely to just being told to boot in single user mode where no password was needed at all (but it's good to know fibranne now): setenv extra_bootargs noautostart single

it seems his problem is the TPM needs cleared and re-initialized, it's managed by the tcsd daemon which is present, but there's no packages like tpm-tools present to actually pass manual commands to the daemon. as far as I can tell he's gunna have to compile tpm-tools for the switch specifically. there's a bootarg "notpm" which skips initialization of the TPM completely (stops the execution of tpm-infra.sh entirely) but the switch still complained, even more if I remember right. It really wants to see /opt/tpm with keys matching what's held in the TPM

I'm going to have another look through the fastiron binary tonight if I get a chance, they clearly have at least a couple TPM related commands hidden buried in it, hopefully one of them will re-initialize the thing with new keys. I can't imagine switches RMAd with this issue need a custom toolchain loaded to fix it, that would be a huge waste of time on everyone's part. Whatever we do load obviously won't be Ruckus's keys so it won't link up with their smartzone controllers any longer, but at least the switch will function normally again

 

[muhfugen]

How loud are the icx 7450 switches? The data sheet says 46db for the non-PoE and 49db for the PoE model. A Catalyst 3750E is 45db and my current FlexFabric 5800 is 42db. is it really going to be a negligible difference compared to a Catalyst 3750E, or are the data sheets not reflective of reality? Also can the fans be modded on the icx 7450?

 

[fohdeesha]

How loud are the icx 7450 switches? The data sheet says 46db for the non-PoE and 49db for the PoE model. A Catalyst 3750E is 45db and my current FlexFabric 5800 is 42db. is it really going to be a negligible difference compared to a Catalyst 3750E, or are the data sheets not reflective of reality? Also can the fans be modded on the icx 7450?

why do you want a 7450? same sound level as the 6610 with half the highspeed ports and twice as expensive

 

[sth]

I think the 7450 has a place still, albeit at a slightly higher price point than the 6 series.
The 7450 I use provides
- 8 * 2.5gbps uplinks for wifi access points
- 12 * 10gbps ports (8 copper, 4 optical through 3 easily swappable modules)
The native 10gig copper ports are more reliable than using SFP+ > RJ45 converters.

 

[fohdeesha]

I think the 7450 has a place still, albeit at a slightly higher price point than the 6 series.
The 7450 I use provides
- 8 * 2.5gbps uplinks for wifi access points
- 12 * 10gbps ports (8 copper, 4 optical through 3 easily swappable modules)
The native 10gig copper ports are more reliable than using SFP+ > RJ45 converters.

you got quite lucky, the only 7450 model that has 2.5gbE ports is the ICX7450-32ZP, and I think I've seen a single one of these for sale on ebay in the ~year or two I've been looking

 

[muhfugen]

why do you want a 7450? same sound level as the 6610 with half the highspeed ports and twice as expensive

The 6610 uses more power at 120 watts for the 6610-24P compared to 75 watts for the 7450-24P. Thats why i was looking at it.

 

[fohdeesha]

The 6610 uses more power at 120 watts for the 6610-24P compared to 75 watts for the 7450-24P. Thats why i was looking at it.

The 24 port 6610 draws ~80w, it's the 48 port (two ASICs) that draws 120

 

[muhfugen]

The 24 port 6610 draws ~80w, it's the 48 port (two ASICs) that draws 120

Not according to the data sheet:
https://www.dataswitchworks.com/datasheets/switches/brocade-icx-6610-ds.pdf

 

[fohdeesha]

the datasheets are worst case, both sound levels and watts. a 24 port 6610 with a few 1gbe and 10gbe ports active registers ~82w on a wattnode. I've never seen them pull anywhere near that, especially the 165 watts for the 48 port lmao

 

[fohdeesha]

icx6610-48p (twice the ASIC) with a rev.b PSU, 9 ports active and 1 PoE cam active:

 

[fohdeesha]

anyone have an icx6610 PSU slot blank they want to sell me? (just the blank slot cover)

 

[hemmy]

I think I have one lying around. Let me check and I’ll pm you.

 

[Wolfstar]

A note for those who are concerned about sound: Do NOT UNDER ANY CIRCUMSTANCES get yourself intake-model fans/power supplies. If it's an ICX6610-24-I, keep shopping for a -24-E. The fans (both the fan modules and the PSU fans) run significantly faster, and therefore louder, for the intake models.

 

[Mithril]

No picture but I can confirm my ICX6610-24 runs about 80watts according to my killawatt meter once it is fully booted and the fans spin down from "takeoff speed".

 

[cthompson]

At 171 pages I've had to skim some of this thread, so I apologize if this is covered.

Going way back to the first page it talks about flashing MCX354A-QCBT to -FCBT. That was like two years ago.

Right now I can get pre-flashed -FCBT on ebay for under $30.

HP 544QSFP MCX354A-FCBT 649281-B21 656089-001 VPI FDR 40GbE Mellanox OEMFirmware | eBay

My plan is to buy a 6610-48P and hook it to the pfsense/opnsense router I'm building using one of these cards.

My google skills are weak and what I can find is contradictory. Can anyone verify whether or not these work under freebsd/pfsense/opnsense?

And more relevant to this thread, assuming I have drivers, just one of those cards, a short QSFP+ DAC and the back of the 6610 and I'm good?

It seems too good to be true

 

[camper8080]

I run the Sunon KDE1204PKV3-MS ones, I have all 3 in there, and I live in Dubai. It survived last summer where the ambient 100+ F room temp was no problem at all for the switch..

I purchased (3) Sunon KDE1204PKV3-MS fans but the 3-pin connector is smaller than the original Foxconn PIA040H12P fans and does not fit. Do I need to modify the new fans or buy a different model?

 

[fohdeesha]

At 171 pages I've had to skim some of this thread, so I apologize if this is covered.

Going way back to the first page it talks about flashing MCX354A-QCBT to -FCBT. That was like two years ago.

Right now I can get pre-flashed -FCBT on ebay for under $30.

HP 544QSFP MCX354A-FCBT 649281-B21 656089-001 VPI FDR 40GbE Mellanox OEMFirmware | eBay

My plan is to buy a 6610-48P and hook it to the pfsense/opnsense router I'm building using one of these cards.

My google skills are weak and what I can find is contradictory. Can anyone verify whether or not these work under freebsd/pfsense/opnsense?

And more relevant to this thread, assuming I have drivers, just one of those cards, a short QSFP+ DAC and the back of the 6610 and I'm good?

It seems too good to be true

yeah those cards work fine in all of those and yes you just need to install them and connect them with a DAC. you'll still want to follow the crossflashing post even if they come pre-crossflashed, as the eeprom configuring stuff (deleting bootrom, forcing link type to ethernet only) prevents a lot of annoying issues in the future. second half of this in an ubuntu livecd or similar https://forums.servethehome.com/ind...net-dual-port-qsfp-adapter.20525/#post-198015

 

[EK701]

ICX6450-C12-PD POE question:

If the switch is powered by POE+, can it also provide POE power to a device or two? It’s not clear from any of the documentation I could find.

Thanks!