Drag to reposition cover

Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

AgentXXL

New Member
Apr 23, 2020
23
5
3
When I spoke about using your storage under a different VLAN, I should have stated that is if you had other devices that would be dedicated to talking to that storage. For your normal access to the storage server, you would use a different VLAN to communicate to it.

I tend to forget that most people do not run as a complicated setup as I do, where my servers that do a bunch of tasks like virtualization on top of the storage use the higher MTU, whereas the clients on the virtualization do not see the underlying storage because it has been virtualized. I then expose the storage virtually for desktops and servers to make use of it.

So for most using a higher MTU would not make sense to do if you are exposing your unRaid storage server directly to the client machines. If you were creating a cephs storage cluster or something like that you would want to use higher MTUs. A virtualization storage system that communicated between machines would only be via one VLAN for storage and another VLAN for external access like the Internet or management.

If you are on a flat network and will be accessing that server directly, you would never set that system up for a large MTU. Keep it at 1500.

One example for the use of higher MTU would be, something like, you have another system set up to do rsync of your unRaid box for backup to another, you could set a VLAN up for just that alone to increase throughput.

Never have used unRaid before, but if it supported a redundant system to backup to, then that would be a case as well. Your VLAN for backups or syncing would make use of it.

It is common in the industry to use a separate VLAN for backups, syncing, and other high throughput services that would never be connected via a client-side network.

Based on what you have said so far, you can set the MTU on the switch as high as you want, but set all of your interfaces at the correct MTU for use if the unRaid box needs Internet and client access.
Thanks for those comments. Yes, my home network is very much a flat topology - I'm retired so I don't need to build out a full homelab that resembles what is done at the corporate/enterprise level. As it turns out, as @Dave Corder mentioned earlier in this thread, it appears that it's the Wiitek SFP module that's causing my issues. As the 10G ports on the 6610 can only negotiate a 1 or 10 Gbps connection, it's up to the SFP module to rate limit to 5Gbps. My Wiitek modules apparently aren't as compatible with the 6610 as they are with other switches. They're listed as capable of 1, 2.5, 5 and 10Gbps connections, but there's some incompatibility with it based on this thread:


I had tried to research which SFP modules to buy and based my decision on this STH review. Alas that reviewer didn't test the Wiitek on the 6610. It wouldn't surprise me if it worked fine on more modern switches.


I'm now looking for a supplier that has the working SFP modules that use the AQS-107 chip and have more buffer capacity. I'm seeing so many retransmits with the Wiitek that it's affecting the speed of the transfers. For the time being I've got the 5Gbps plugged into a 1Gbps port on the 6610 and at least with that config I'm seeing no retransmit errors.
 

RobstarUSA

Active Member
Sep 15, 2016
235
104
43
Thanks for those comments. Yes, my home network is very much a flat topology - I'm retired so I don't need to build out a full homelab that resembles what is done at the corporate/enterprise level. As it turns out, as @Dave Corder mentioned earlier in this thread, it appears that it's the Wiitek SFP module that's causing my issues. As the 10G ports on the 6610 can only negotiate a 1 or 10 Gbps connection, it's up to the SFP module to rate limit to 5Gbps. My Wiitek modules apparently aren't as compatible with the 6610 as they are with other switches. They're listed as capable of 1, 2.5, 5 and 10Gbps connections, but there's some incompatibility with it based on this thread:


I had tried to research which SFP modules to buy and based my decision on this STH review. Alas that reviewer didn't test the Wiitek on the 6610. It wouldn't surprise me if it worked fine on more modern switches.


I'm now looking for a supplier that has the working SFP modules that use the AQS-107 chip and have more buffer capacity. I'm seeing so many retransmits with the Wiitek that it's affecting the speed of the transfers. For the time being I've got the 5Gbps plugged into a 1Gbps port on the 6610 and at least with that config I'm seeing no retransmit errors.
If on the 6610 port with the mgig sfp, can you do a "rate-limit output shaping 4000000" and see if you get 4Gbit/s throughput ?
 
  • Like
Reactions: AgentXXL

jcstill

New Member
Aug 10, 2021
4
0
1
Southern California
Just received a new (to me) 7250-24p. Walking through the setup guide, the uboot image updates just fine, but the primary image fails.

All I get is:
Code:
ICX7250-Boot> update_uboot
Loading image to Uboot Partition 1
Using bcmiproc_eth-0 device
TFTP from server 10.22.102.17; our IP address is 10.22.102.200
Filename 'ICX7xxx/spz10118.bin'.
Load address: 0x61007dc0
Loading: #################################################################
         #################################################################
         ########################
done
Bytes transferred = 786944 (c0200 hex)
sf erase 0x0 0x100000
copying uboot image to flash, it will take sometime...
sf write 0x61007fc0 0x0 0xc0000 
TFTP to Flash Done.
ICX7250-Boot>
//
ICX7250-Boot> update_primary
Updating PRIMARY image over TFTP
Using bcmiproc_eth-0 device
TFTP from server 10.22.102.17; our IP address is 10.22.102.200
Filename 'ICX7xxx/SPR08090mc.bin'.
Load address: 0x1000000
Loading: #################################################################
         #################################################################
                              (100 lines cut)
         #################################################################
         #################################################################
         #######################################################Primary image download failed
ICX7250-Boot>
The switch came to me runing v08.0.30fT211. I thought I remember reading in this thread that updating from pre 08.0.80 to 8.0.90 needed to be incremental. ie 08.0.30 -> 08.0.80 -> 08.0.90. Is this the case?

Thanks
 

fohdeesha

Kaini Industries
Nov 20, 2016
2,893
3,404
113
34
fohdeesha.com
Just received a new (to me) 7250-24p. Walking through the setup guide, the uboot image updates just fine, but the primary image fails.

All I get is:
Code:
ICX7250-Boot> update_uboot
Loading image to Uboot Partition 1
Using bcmiproc_eth-0 device
TFTP from server 10.22.102.17; our IP address is 10.22.102.200
Filename 'ICX7xxx/spz10118.bin'.
Load address: 0x61007dc0
Loading: #################################################################
         #################################################################
         ########################
done
Bytes transferred = 786944 (c0200 hex)
sf erase 0x0 0x100000
copying uboot image to flash, it will take sometime...
sf write 0x61007fc0 0x0 0xc0000
TFTP to Flash Done.
ICX7250-Boot>
//
ICX7250-Boot> update_primary
Updating PRIMARY image over TFTP
Using bcmiproc_eth-0 device
TFTP from server 10.22.102.17; our IP address is 10.22.102.200
Filename 'ICX7xxx/SPR08090mc.bin'.
Load address: 0x1000000
Loading: #################################################################
         #################################################################
                              (100 lines cut)
         #################################################################
         #################################################################
         #######################################################Primary image download failed
ICX7250-Boot>
The switch came to me runing v08.0.30fT211. I thought I remember reading in this thread that updating from pre 08.0.80 to 8.0.90 needed to be incremental. ie 08.0.30 -> 08.0.80 -> 08.0.90. Is this the case?

Thanks
can you try to run that again (mainly the update u boot command), then reset the switch, (just send the command "reset" but be sure to stop it back into the bootloader again (if it boots all the way into the OS, it may downgrade uboot again). once you stop it back in uboot after the reboot, it should hopefully be in the new version of uboot, then try the update primary command again
 

acurax04

New Member
Feb 18, 2022
2
1
3
I am very inexperienced and looking for help setting up an ICX 7150.

Following the setup guide (ICX7150 - Fohdeesha Docs) and I cannot get the ICX7150 to fetch the firmware images from the TFTP server. I am using the USB-C port on the 7150 to connect directly to my PC. I am able to access the bootloader via Putty and assign temp IP/mask to the 7150 and direct to the TFTP server files, but the 'update_uboot' command results in this:

Timeout error.JPG

I'm using TFTPD64 from the zip file in the guide:

tftp64.JPG

I have disabled Windows Firewall. I only have the one PC connected to a MikroTik router at this time and the 7150 to the PC via USB-C port. I can't ping between the devices--not sure what I have set up incorrectly...
 

NablaSquaredG

Bringing 100G switches to homelabs
Aug 17, 2020
1,743
1,161
113
I currently do not have access to ICX6610 as I'm on the road

Is it possible to use the 4*10G QSFP+ breakout ports on the ICX6610 for stacking without using the regular QSFP+ ports for stacking too? i.e. stacking via the 4*10G QSFP+ breakout ports and using the other two QSFP+ port for regular connection?

I remember that I was able to use the regular QSFP+ ports for stacking and use the 4*10G ports for a link to the core switches - But that was not an optimal solution because a 4*10G with 802.1ad LACP is not as good as a regular QSFP+ port and I lost a port on the SX6036 (breakout disabled other ports on the SX6036)
 

Rain

Active Member
May 13, 2013
279
125
43
I am very inexperienced and looking for help setting up an ICX 7150.

Following the setup guide (ICX7150 - Fohdeesha Docs) and I cannot get the ICX7150 to fetch the firmware images from the TFTP server. I am using the USB-C port on the 7150 to connect directly to my PC. I am able to access the bootloader via Putty and assign temp IP/mask to the 7150 and direct to the TFTP server files, but the 'update_uboot' command results in this: ...

I have disabled Windows Firewall. I only have the one PC connected to a MikroTik router at this time and the 7150 to the PC via USB-C port. I can't ping between the devices--not sure what I have set up incorrectly...
If you can't ping the switch's IP address (192.168.88.100 in your example) from the computer running the TFTP server, something isn't connected properly.

Don't use the MikroTik router; it shouldn't be needed. As the guide mentions, don't use any of the regular switch ports yet; use the management ethernet port to update the bootloader. Connect an ethernet cable directly from the 7150's management port (regardless of what model you have, this should be the ethernet port just above the USB-A port) to your computer's ethernet port.
 

kpfleming

Active Member
Dec 28, 2021
440
226
43
Pelham NY USA
Is it possible to use the 4*10G QSFP+ breakout ports on the ICX6610 for stacking without using the regular QSFP+ ports for stacking too? i.e. stacking via the 4*10G QSFP+ breakout ports and using the other two QSFP+ port for regular connection?
Probably not; only the designated stacking ports can be used for stacking.
 

AgentXXL

New Member
Apr 23, 2020
23
5
3
If on the 6610 port with the mgig sfp, can you do a "rate-limit output shaping 4000000" and see if you get 4Gbit/s throughput ?
I'm a novice when it comes to configuring a managed switch like the 6610. I did a bit of reading and tried this command:

Device(config)#interface ethernet 1/3/2
Device(config-if-e1000-1/3/2)#rate input fixed 4000000

Traffic in one direction was OK with no retransmits, but the other still showed many as seen in the attached pic:ICX6610-5Gbe-RateLimit.jpg

The rate limit applied was to the machine which has the onboard 5Gbe NIC. Out of curiosity I applied the same rate limit to the other machine which has the onboard 10Gbe NIC. Alas I still see poor speeds and many retransmit errors.

As I'm new to a lot of the concepts in a managed switch, I'm doing a lot of reading to try and see if there's a solution. If you have other things you want me to try, let me know. Thanks!
 

NablaSquaredG

Bringing 100G switches to homelabs
Aug 17, 2020
1,743
1,161
113
Probably not; only the designated stacking ports can be used for stacking.
You know that *all* of them are stacking ports?

I can use the 40G QSFP ports for stacking or 40G QSFP AND 4*10G QSFP breakout together for stacking, but I haven't managed to only use the 4*10G QSFP breakouts for stacking
 
  • Like
Reactions: fohdeesha

Vesalius

Active Member
Nov 25, 2019
261
202
43
PSA for anyone that jumped ahead to the 9.0.* firmware.
FastIron software releases 09.0.00, 09.0.00a, and 09.0.10 are no longer available for download due to the recent discovery of a critical defect. RUCKUS recommends upgrading to FastIron software release 09.0.10a for all switches currently running the impacted FastIron releases 09.0.00, 09.0.00a, and 09.0.10. Refer to TSB-2022-001 FastIron 09.0.00 and 09.0.10 - Risk of Filesystem Corruption for more details.

 

fohdeesha

Kaini Industries
Nov 20, 2016
2,893
3,404
113
34
fohdeesha.com
I am very inexperienced and looking for help setting up an ICX 7150.

Following the setup guide (ICX7150 - Fohdeesha Docs) and I cannot get the ICX7150 to fetch the firmware images from the TFTP server. I am using the USB-C port on the 7150 to connect directly to my PC. I am able to access the bootloader via Putty and assign temp IP/mask to the 7150 and direct to the TFTP server files, but the 'update_uboot' command results in this:

View attachment 21733

I'm using TFTPD64 from the zip file in the guide:

View attachment 21734

I have disabled Windows Firewall. I only have the one PC connected to a MikroTik router at this time and the 7150 to the PC via USB-C port. I can't ping between the devices--not sure what I have set up incorrectly...
it sounds like you haven't actually plugged the switch into your network? The usb-c serial connection is not a network connection, as the guide states you need to connect the switches dedicated management ethernet port (NOT any of the regular ports) to your network
 
  • Like
Reactions: acurax04

crackelf

Member
Apr 11, 2021
74
6
8
Quick sanity check:

ICX 7450 with 2x 40G modules and stack disabled. Can I use Mellanox ConnectX-4 cards in 40G ethernet mode with these? Was planning on either some DACs or MTP-MTP with optics on either end.

Wanted to check before pulling the trigger on all these. Thanks!
 

fohdeesha

Kaini Industries
Nov 20, 2016
2,893
3,404
113
34
fohdeesha.com
Quick sanity check:

ICX 7450 with 2x 40G modules and stack disabled. Can I use Mellanox ConnectX-4 cards in 40G ethernet mode with these? Was planning on either some DACs or MTP-MTP with optics on either end.

Wanted to check before pulling the trigger on all these. Thanks!
yes they're regular 40gbe ports. instead of expensive annoying mtp you can grab these BiDi optics and run 40gbE over cheap regular singlemode duplex LC fiber XQX2502 KAIAM QSFP+40G-LR4 Lite OPTICAL MODULE NEW PULLS | eBay
 

LodeRunner

Active Member
Apr 27, 2019
553
235
43
Quick sanity check:

ICX 7450 with 2x 40G modules and stack disabled. Can I use Mellanox ConnectX-4 cards in 40G ethernet mode with these? Was planning on either some DACs or MTP-MTP with optics on either end.

Wanted to check before pulling the trigger on all these. Thanks!
On the ICX, the ports are just ports. Optics are probably safest; I don't know if they QSFP on a 7450 can understand a QSFP28 (is that what the CX4 is?).

On my 7450 I'm using both of them as a LAG to my Arista 7050 core.
 
  • Like
Reactions: crackelf

crackelf

Member
Apr 11, 2021
74
6
8
yes they're regular 40gbe ports. instead of expensive annoying mtp you can grab these BiDi optics and run 40gbE over cheap regular singlemode duplex LC fiber XQX2502 KAIAM QSFP+40G-LR4 Lite OPTICAL MODULE NEW PULLS | eBay
Thanks for the response and the link! I had been looking at these Brocade optics on eBay and was going to try out these Arista MTP-MTP 12 fibers OM4 (also eBay), but I much prefer the idea of cheap LC. I really appreciate your guidance on this! Will post back when everything arrives.

Last potential gotcha: is it possible to fall back to 10G with a 40G <---> 10G breakout cable between card <---> switch if the QSFP ports on the switch fail for whatever reason, or would a QSFP to SFP+ Transceiver Adapter make more sense?

On the ICX, the ports are just ports. Optics are probably safest; I don't know if they QSFP on a 7450 can understand a QSFP28 (is that what the CX4 is?).

On my 7450 I'm using both of them as a LAG to my Arista 7050 core.
I thought that made the most sense, but after living through various licensing hells I was spooked somehow they wouldn't allow them to be used as non-stacking ports. Thanks for the reassurance. I actually have no idea myself if the 7450 can handle the QSFP28, but I'll report back with my findings even if this all blows up in my face spectacularly.
 

LodeRunner

Active Member
Apr 27, 2019
553
235
43
Last potential gotcha: is it possible to fall back to 10G with a 40G <---> 10G breakout cable between card <---> switch if the QSFP ports on the switch fail for whatever reason, or would a QSFP to SFP+ Transceiver Adapter make more sense?
The ICX7450 does not support breakout. If you want 10G, you have to use the 4x10g module.

The BiDi posted by fohdeesha is known to work and is an excellent cheap choice. I don't think those 16Gb FC optics would work the way you want, assuming the switch can recognize them at all.

Edit: maybe that optic would work, it's 4x 16 Gb lanes. The BiDi is still cheaper at $12 vs $40.
 

crackelf

Member
Apr 11, 2021
74
6
8
The ICX7450 does not support breakout. If you want 10G, you have to use the 4x10g module.

The BiDi posted by fohdeesha is known to work and is an excellent cheap choice. I don't think those 16Gb FC optics would work the way you want, assuming the switch can recognize them at all.

Edit: maybe that optic would work, it's 4x 16 Gb lanes. The BiDi is still cheaper at $12 vs $40.
Sorry I probably didn't explain this the right way - I'm saying the breakout goes from card to switch so the QSFP side plugged into the Mellanox and 1 of the 10G plugged into the 4x10g module, but the more I type this out the more I realize the adapter is the way to go here. I completely believe that the breakout doesn't work from the switch side since it doesn't say anywhere in the software or on the hardware that breakout is doable.

I posted said Brocade optics because another thread on here (which I now for the life of me can't find) reporting them as functional with the 6610, but we have been blessed by now both fohdeesha himself & LodeRunner with better recommendations, so thanks for straightening me out! I barely skated by setting up my last 10g topology with a 7150, so my hilarious lack of fiber knowledge is really shining here.
 

LodeRunner

Active Member
Apr 27, 2019
553
235
43
Ah. The ICX would be expecting a LAG at that point, so if the card supports being broken out and then you LAG it in the OS, then I expect it would work.