Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

[fiftyclick]

I heard great things about Wiitek 100m SFP+/RJ45 adapters which are supposed to run a lot cooler than other SFP+ options. While they certainly run much cooler, I experienced transient issues under load which would cause latency to suddenly spike, and couldn't be resolved without disconnecting / reconnecting the Ethernet cable. Maybe this works better with other switches, but it didn't work with my ICX 7250-24p on latest 8.0.x firmware. It is able to negotiate multi-gigabit speeds just fine, but fails when under load, and doesn't recover until Ethernet link is re-connected.

P/N: UF-RJ45-10G-100 / 10GBase-T SFP+ RJF45 100m UBQ (there is no Brocade variant, yet at least...)
Link: Amazon.com: Wiitek 100 Meters, 10Gb SFP+ to RJ45 Module, 2.5G/5G/10GBase-T Ethernet Copper Transceiver for Ubiquiti UF-RJ45-10G, Plugin 10Gb SFP+ Port, Low Power Consumption : Electronics

10:34:10.561246 64 bytes from xx.xx.xx.xx: icmp_seq=5872 ttl=63 time=1.327 ms
10:34:11.562444 64 bytes from xx.xx.xx.xx: icmp_seq=5873 ttl=63 time=1.366 ms
10:34:12.571736 64 bytes from xx.xx.xx.xx: icmp_seq=5874 ttl=63 time=1.394 ms
10:34:13.576855 64 bytes from xx.xx.xx.xx: icmp_seq=5875 ttl=63 time=1.420 ms
10:34:14.586627 64 bytes from xx.xx.xx.xx: icmp_seq=5876 ttl=63 time=1.551 ms
10:34:15.591651 64 bytes from xx.xx.xx.xx: icmp_seq=5877 ttl=63 time=1.295 ms
10:34:16.592268 64 bytes from xx.xx.xx.xx: icmp_seq=5878 ttl=63 time=1.272 ms
10:34:17.600854 64 bytes from xx.xx.xx.xx: icmp_seq=5879 ttl=63 time=1.350 ms
10:34:18.609619 64 bytes from xx.xx.xx.xx: icmp_seq=5880 ttl=63 time=1.460 ms
--> iperf load test starts here, a few more packets continue without issue
10:34:19.619624 64 bytes from xx.xx.xx.xx: icmp_seq=5881 ttl=63 time=1.261 ms
10:34:20.622211 64 bytes from xx.xx.xx.xx: icmp_seq=5882 ttl=63 time=1.258 ms
10:34:21.632491 64 bytes from xx.xx.xx.xx: icmp_seq=5883 ttl=63 time=1.332 ms
--> iperf fails, continual timeouts
10:34:23.648688 Request timeout for icmp_seq 5884
10:34:24.001331 64 bytes from xx.xx.xx.xx: icmp_seq=5884 ttl=63 time=1359.971 ms
10:34:25.661465 Request timeout for icmp_seq 5886
10:34:26.666641 Request timeout for icmp_seq 5887
10:34:27.667586 Request timeout for icmp_seq 5888
10:34:28.672650 Request timeout for icmp_seq 5889
10:34:28.681126 64 bytes from xx.xx.xx.xx: icmp_seq=5886 ttl=63 time=4024.911 ms
10:34:30.683637 Request timeout for icmp_seq 5891

After spending almost all day troubleshooting my R670 access point, I thought I traced the issue back to the PoE++ injector, but ultimately it was the SFP+ adapter. Every time I disconnected / reconnected the PoE adapter, latency would drop back <1ms, but with any load (e.g. YouTube, Disney+, etc..) latency would suddenly spike >1600ms with constant timeouts. Having exhausted all options, I finally tried changing SFP+ ports and switched to another adapter I had laying around (10Gtek mulit-gigabit 30m adapter) when things started working. Swapped out the SFP transceivers in the same port it originally was, and it still worked.

Curious if anyone else had the same issue; maybe it is just incompatibility between the UBQ skew and Brocade, but this would be the first SPF+ I've used that didn't just work out of the box.

TLDR; If you are troubleshooting sudden latency spikes under load, double check your SFP+ transceiver isn't the issue; you will save yourself a lot of time!

 

[nereith]

I heard great things about Wiitek 100m SFP+/RJ45 adapters which are supposed to run a lot cooler than other SFP+ options. While they certainly run much cooler, I experienced transient issues under load which would cause latency to suddenly spike, and couldn't be resolved without disconnecting / reconnecting the Ethernet cable. Maybe this works better with other switches, but it didn't work with my ICX 7250-24p on latest 8.0.x firmware. It is able to negotiate multi-gigabit speeds just fine, but fails when under load, and doesn't recover until Ethernet link is re-connected.

Someone reported high retransmission rates when that Wiitek module transmits from 10G to NBaseT.

I've seen latency spikes and no Pause frames with the same module in a similar scenario.

 

[dbvader]

Does anyone have any experience with FIPS?

No.

Did or can you run 'factory set-default' from the bootloader?

 

[vehicledune]

No.

Did or can you run 'factory set-default' from the bootloader?

Running factory set-default in the bootloader did not disable FIPS. setenv fipsreset also wasn't of any aid. It think it may be related to the FIPS Firmware Integrity Test which I believe permanently requires a kind of signature to be copied into flash along with the firmware after FIPS has been enabled (and even after disabling it).

But somehow, maybe through running factory set-default or no password in the bootloader, or maybe it arrived this way... I was able to login with the default username and password for FastIron v09.0.10h. From there, I did the following:

! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! 
! FLASHING A FIPS-CURSED RUCKUS ICX 7150-C12P WITH FASTIRON v08.0.95s UFI IMAGE...
! I used the following commands to downgrade from v09.0.10h to v08.0.95s...
! v08.0.95s is the current recommended software and stability release for the ICX 7150-C12P as of 1 September 2025...
! See [ https://support.ruckuswireless.com/products?view_type=recommended_releases_table ] for more info...
! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! 

! ! ! Enter global config mode...
enable
conf t

! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! !
! Some people may now need to setup an IP address and netmask,
! but I don't think I had to do this because
! I had already entered some setenv parameters in the bootloader from following 
! fohdeesha's guide...
! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! !

! ! ! Make sure that FIPS is disabled and enable tftp
crypto key zeroize
no fips enable
no tftp disable

! ! ! Begin transfer/flashing of signature and image...
copy tftp flash <your-server's-ip> SPR08095sufi.sig fips-ufi-primary-sig
copy tftp flash <your-server's-ip> SPR08095sufi.sig fips-ufi-secondary-sig
copy tftp flash <your-server's-ip> SPR08095sufi.bin primary
copy tftp flash <your-server's-ip> SPR08095sufi.bin secondary

! ! ! Verify that the new version of firmware is now in flash memory before writing and restarting...
show flash
write memory
reload

After this, I reset all of the bootloader parameters to their original state and rebooted into the new firmware. I was able to successfully downgrade from v09.0.10h to v08.0.95s. Though, it would be nice to be able to copy signatures and images via the bootloader if that were required.

 

[dbvader]

Good going.

Thanks for showing the commands. While it was probably frustrating for you I'm sure your feedback is going to become a useful resource for others.

 

[fohdeesha]

Strange, factory set default in the bootloader wipes the fips bit in the chassis eeprom on icx6xxx/8030 series. They must have changed it for the 7 series

 

[dbvader]

Could v9 firmware perhaps have different semantics? Seems like that's where the issue was occurring.

 

[fohdeesha]

Could v9 firmware perhaps have different semantics? Seems like that's where the issue was occurring.

looked at the u-boot source code for icx7xxx stuff and it's certainly supposed to reset FIPS mode when you run the factory set default command from bootloader like my guide instructs -


"factory set-default" command routine -

+int do_factory_default(cmd_tbl_t *cmdtp, int flag, int argc, char *argv[])
+{
+        int ret = 1;
+        char ch;
+
+        switch (argc) {
+                case 2:
+                        if(strcmp(argv[1],"set-default") == 0) {
+                                printf("Execution of \"factory set-default\" will remove all user data like config, keys etc.\n\n");
+                                printf("Do you want to continue? (Y/N) ");
+
+                                ch = getc();
+                                printf("%c\n", ch);
+
+                                if (ch == 'Y') {
+                                        ret = do_set_default();
+                                } else {
+                                        printf("\nAborting operation\n");
+                                }
+                        } else {
+                                printf("Unknown command '%s' - try 'help'\n",argv[0]);
+                        }
+                        break;
+                default:
+                        printf("Unknown command '%s' - try 'help'\n",argv[0]);
+                        break;
+        }
+        return ret;

You can see if you confirm by hitting Y (capital Y, not lowercase, like my guide says). it runs do_set_default, which resets FIPS:

+static int do_set_default (void) {
+    fips_status = 0;
+    fips_reset = 1;
+    fips_reset_flag = 0;
+    setenv("fips_reset", "fipsreset");
+    setenv("fips_status", NULL);
+    saveenv();
+    reserve_cmd = normal_mode_cmd;
+    fips_reset_flag = 1;
+
+    printf("\nCommand executed successfully\n");
+
+    return 0;

 

[klui]

Commscope advises the signature file should be loaded after disabling FIPS before loading the software image file.

Changes to the running configuration are not saved to the startup configuration; therefore, when the device reloads, it returns to FIPS mode.

Use the write memory command to save the running configuration.

docs.commscope.com/bundle/fastiron-10010-fipscc-config/page/GUID-807FB990-C80B-48F0-A39C-35B8FB974520.html

 

[fohdeesha]

Commscope advises the signature file should be loaded after disabling FIPS before loading the software image file.



docs.commscope.com/bundle/fastiron-10010-fipscc-config/page/GUID-807FB990-C80B-48F0-A39C-35B8FB974520.html

the sig files are only needed if you want to re-enable fips. factory set default in the bootloader should disable fips completely, and it will be saved, I'm not sure how @dbvader managed to have it re-enable

 

[dbvader]

I'm not sure how @dbvader managed to have it re-enable

@vehicledune had the issue. I arrived on scene after the accident.

 

[fohdeesha]

@vehicledune had the issue. I arrived on scene after the accident.

I am still learning to read

 

[vehicledune]

Here is some more information I came across that helped me.

FIRMWARE INTEGRITY TEST

I couldn't find a RUCKUS FastIron FIPS and Common Criteria Configuration Guide for v08.0.95 or v09.X.X, but I was able find the ones for v.08.0.70 and 10.0.0.

RUCKUS FastIron FIPS and Common Criteria Configuration Guide, 08.0.70,
"Upgrading and Downgrading Software on FIPS-Enabled Devices, Downgrading from FIPS to Non-FIPS Mode."
https://docs.commscope.com/bundle/fastiron-08070-fipscc-config/resource/fastiron-08070-fipscc-config.pdf:

NOTE
Once FIPS mode is enabled on the system, even if the mode is disabled later, a firmware integrity test will always be carried out on the device when the image is copied.

RUCKUS FastIron FIPS and Common Criteria Configuration Guide, 10.0.10,
"Upgrading and Downgrading Software on FIPS-Enabled Devices, Downgrading from FIPS to Non-FIPS Mode."
https://docs.commscope.com/bundle/fastiron-10010-fipscc-config/page/GUID-F8E0EFB8-75BF-49A7-AB01-ED74233AF413.html#:~:text=Once%20the%20ICX%20device%20is,configuration%20with%20the%20reload%20command:

Once the ICX device is enabled for FIPS, it remains enabled internally, so that the signature file must be copied first whenever the image is copied.

Once you have downgraded to non-FIPS mode, you must still load the relevant signature (.sig) file before the image (.bin) file every time you upgrade or downgrade an image in the future. This prevents the ICX device from looping.

...

Note: Once FIPS mode is enabled on the system, even if the mode is disabled later, a firmware integrity test will always be carried out on the device when the image is copied.

SIGNATURE FILES

The following seems to suggest that the FIPS Firmware Integrity Test is dependent on the correct signature file being in place:

RUCKUS FastIron FIPS and Common Criteria Configuration Guide, 08.0.70,
"FIPS Configuration, Placing the Device in FIPS Mode, Performing a FIPS Self-test."
https://docs.commscope.com/bundle/fastiron-08070-fipscc-config/resource/fastiron-08070-fipscc-config.pdf:

If the test fails, make sure that the correct signature file was copied for the correct image file and version, and recopy as needed.

RUCKUS FastIron FIPS and Common Criteria Configuration Guide, 10.0.10,
"FIPS Configuration, Placing the Device in FIPS Mode, Performing a FIPS Self-test."
https://docs.commscope.com/bundle/fastiron-10010-fipscc-config/page/GUID-0274905A-BCD2-44B7-9BE6-98BEB8172F84.html:

If the software integrity test fails, make sure that the correct signature file was copied for the correct image file and version, and recopy as needed. If the problem persists, contact RUCKUS technical support.

 

[OKGolombRuler]

Gurus, I have Found A New Error on my ICX7150-24 after an unexpected cable-yank power loss:

PTP Feature Enabled
mem PTP_LABEL_RANGE_PROFILE_TABLE is invalid
ERROR: Assertion failed: (SOC_MEM_IS_VALID(unit, mem)) at /home/jenkins/workspace/fi_release_08095x_patch_build_SPS/ven8
stack: 000368a4 0003aefc b6d287d0 b6bad7c1 02799794

...and then she dumps core and bootloops.

Anything to try before I factory-reset, reflash, etc?


ETA:
swver: 08.0.95s
Bootloader: 10.1.27T225

 

[OKGolombRuler]

Just to save folks time: I found an old (but new enough to have my interface details) backup config so I hit it with the factory-reset stick and am taking this... uh.... opportunity... to review and refine my config.

 

[tongboy]

Any possibility of using 10g for one link of a 7450 3 switch stack?

I have 3 switches currently in a daisy chain, and I have one very unreliable link. While waiting for additional QSFP's so I can finish the ring topo: can I temporarily use a set of spare 10g ports from the 'end' switches to close the ring?

generally seems to be a no, based on my config attempts always replacing each other. Was hoping someone might have a big brain option.

I'd like to use 1/2/4 and 3/2/4 to finish out the ring.

#show stack
T=5d2h32m30.8: alone: standalone, D: dynamic cfg, S: static
ID   Type          Role    Mac Address    Pri State   Comment
1  S ICX7450-24P   member  0000.0000.0000   0 reserve
2  S ICX7450-24P   active  609c.9f3a.5c58   0 local   Ready
3  S ICX7450-24P   standby cc4e.248e.5af8   0 remote  Ready

    active       standby
     +---+        +---+
  3/1| 2 |4/1--3/1| 3 |
     +---+        +---+

attempts:

#stack-port 3/2/1
Error! It may break the link 3/3/1 -- 2/4/1.
Error! It may break the link 3/3/1 -- 2/4/1.
stack-port 3/2/1 would remove stack-port 3/3/1 to satisfy (3/2/1, 3/2/3, 2).
The replacement is to meet the following rule(s):

ICX7450 stack-port must be icx7400-xgf-4port-40g-module at module 2, or icx7400-qsfp-1port-40g-module at module 3 or 4.

The following stack-ports/trunks are allowed.
Linear-topo trunk (exact ports): allowed only in one-direction
stack-trunk 3/2/1 to 3/2/4
General stack-port/trunk that can be in up to two directions: Both directions must belong to the same valid-stack-port-set (dir_0_1st_port, dir_1_1st_port, max_#_ports):
(3/3/1, 3/4/64, 1), (3/2/1, 3/2/3, 2)
E.g., (1/2/1, 1/2/3, 2): each direction can have 1-2 ports.
     dir 0: stack-port 1/2/1, or stack-trunk 1/2/1 to 1/2/2
     dir 1: stack-port 1/2/3, or stack-trunk 1/2/3 to 1/2/4

 

[fohdeesha]

(finally) updated the guide to 8095s, including refreshing/updating all manual pdfs with the latest revisions. will get the guide up to 9x when I have some time to robustly test the non-ufi > ufi 9x upgrade process

 

[fohdeesha]

finally updated OP prices to reflect 2025 reality - my favorite is the 7250 going from 300 to 70, sheeesh

also if anyone near Indianapolis wants to buy a bunch of ICXs I will cut you a comical deal, I just don't want to deal with shipping

 

[ttabbal]

Working with a ICX-6610. I can't get a 40gb link to work well. I think the config is good, I'll paste it below.

SSH@ICX6610-48P Router#show run
Current configuration:
!
ver 08.0.30uT7f3
!
stack unit 1
  module 1 icx6610-48p-poe-port-management-module
  module 2 icx6610-qsfp-10-port-160g-module
  module 3 icx6610-8-port-10g-dual-mode-module
  no legacy-inline-power
stack disable
!

Everything else is just basic VLAN configs.

1/2/1 will link, but performance on 3 NICs and 2 DACS, one a Brocade that came with the switch, get a lot of retries in iperf3 to a 10G machine. Those can iperf 10g-1g without issues. It's just the 40gb that doesn't want to work. The 1/2/6 port doesn't link with any config I have tried. NICs are 2x ConnectX-3, and 1 Intel XL710. I even tried the breakout ports, not expecting them to work without breakout cables. The Mellanox cards could do ok, but the Intel was in the 2 Mbps range.

I'm thinking the only constant is the switch, so the "stack" module appears to be bad. Anything else to try?

SSH@ICX6610-48P Router#show license
Index    Lic Mode        Lic Name               Lid/Serial No  Lic Type    Status     Lic Period    Lic Capacity
Stack unit 1:
1        Node Lock       ICX6610-PREM-LIC-SW    dnrHKKHiFGt    Normal      Invalid    Unlimited         65535
2        Node Lock       ICX6610-10G-LIC-POD    H4CKTH3PLN8    Normal      Active     Unlimited         8
3        Node Lock       ICX6610-ADV-LIC-SW     H4CKTH3PLN8    Normal      Active     Unlimited         1
4        Node Lock       ICX-MACSEC-LIC         H4CKTH3PLN8    Normal      Active     Unlimited         1

 

[BoGs]

Any of you know what model of fan ICX7150-48p/48PF uses? and if there is a quieter one available that would work with the 4 pin fan header?