Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

[AgentXXL]

1. Do you have a router that is routing already?
2. If no external router, then using the switch as a router, does have some limitations in what routing it does.
3. Setting jumbo still allows you on a per-port or VLAN basis the MTU
4. I would segment storage from other VLANs i.e. internal network on one VLAN storage another
5. I would then run a per VLAN MTU, Set the VLAN for normal communications under a 1500 MTU, storage under the max MTU and configure your ports on the storage to the max MTU your nics will support.
6. You would need to understand Brocade's network and routing to get all of that to work without some other routing device to do it for you.
7. Deciding whether or not to do jumbo is really dependent on what type of storage you are doing. Small files, large files etc. Smaller files may be best with standard mtu, I mean like 1k files versus larger megabyte files.

1. Yes, I have a physical pfSense system operating as my router, connected to my ISP via DOCSIS 3.1 modem in bridge mode.
2. N/A
3. When I tried to assign a new MTU value to a specific port, it always gave an error that MTU is an unrecognized command. I was able to assign a MTU to a VLAN, but right now I only have the one VLAN containing all ports. The larger MTU dropped my overall network speed so I changed it back to the default of 1500 for now.
4. I've seen that recommendation before but I'm not quite sure what it means... my unRAID servers act as more than just storage. Docker containers and VMs run on my unRAID systems. Right now I've left the z690 system with its 10Gbe Aquantia NIC attached to one of the front 10G ports but the x299 system with its 5Gbe NIC is currently plugged into a 1Gbps port. That has eliminated all of the retransmits that iperf3 reported when it was connected to a 10G port via the Wiitek SFP+ module, but of course that limits my max speed to 1Gbps.
5. My research has led me to the same conclusion so I'm still planning to create a new VLAN for all of the 10G ports. My main concern is how to let other devices access that VLAN and vice versa. I assume that's why you asked about a router - I suspected I would have to implement some routing on it to allow the VLAN to be seen by other devices. For example, my Nvidia Shield needs to be able to contact Plex on the main media unRAID system.
6. Any sites or tips you can provide to illustrate what kind of routing rules I need to configure, presumably on pfSense?
7. Most of the files on my unRAID systems are large - I only rip full remuxes from my physical discs. I do also use the second server for backups of other systems in my LAN. The backups contain lots of smaller files like documents and pictures - I suspect a large MTU will result in a lot of wasted overhead in packet transmission. The reason I want more LAN speed between the two servers is this: I rip and edit my full remuxes on the 2nd system. When I'm satisfied with the file, it then gets moved into one of the media folders (Movies, TV, Music, etc). Its this transfer between the two systems that I'm trying to improve. I've read good and bad about enabling jumbo frames but right now my feeling is to not enable it.

Note that my plan to put an Intel x520 10Gbe NIC into the x299 system wasn't implemented. Unfortunately I'm using all of the x8 slots that my motherboard supports. The x299 system does have Thunderbolt 3 ports on it so I was contemplating buying a TB3 10Gbe NIC. Alas that's an expensive option, even more than trying to find a known good SFP+ to RJ45 module with the Aquantia AQS-107 chip. There appear to be a LOT of 3rd party knock-offs of the Aquantia module - most of the ads I've seen state that they are 'compatible' but so was my Wiitek.

Thanks for the response and suggestions!

 

[kpfleming]

You might be misunderstanding something: the MTU is the *maximum* packet size, it's not the required size of all packets. Increasing the MTU can't cause transfers of small files to be slowed down. The network stack will only send large packets when it has enough data to fill them.

 

[AgentXXL]

You might be misunderstanding something: the MTU is the *maximum* packet size, it's not the required size of all packets. Increasing the MTU can't cause transfers of small files to be slowed down. The network stack will only send large packets when it has enough data to fill them.

Very likely... I'm retired now for health reasons but my career had me often working with our IT teams. I had never really wanted 10Gbe (or better) until recently but I had heard about jumbo frames and MTU. Whenever we needed faster transfers it was up to our network gurus to do the changes. Now I understand why they griped so much when I asked them to increase our throughput.

 

[fohdeesha]

how do deduce if you should increase MTU over the default:

year = > 2005? then > no

hyperscalars and HPC need not apply

 

[fohdeesha]

I was able to revert to the original licenses that came with the 6610 and do the factory reset. Alas my speeds between the 5Gbe and 10Gbe systems are now even slower, and retransmits are still quite large. It's entirely possible I did something wrong. I followed your guide this 2nd time other than reflashing the firmware and installing the licenses you kindly provide. I didn't see any need to reflash with the same versions of firmware as my initial flash worked - the R, S and POE firmwares are all updated to the version provided in the zip file.

The licenses mine came with are:

ICX6610-PREM-LIC
ICX6610-10G-LIC-POD (enabling all 8 front SFP+ ports)

I also had issues with the 2 SFP+ ports in that they wouldn't auto-negotiate. I had to go in and configure them as 10G full duplex before they came online - I didn't have to do this the first time I setup the switch, or I don't remember doing it. Perhaps that means the ICX-MACSEC-LIC isn't included in the ICX6610-PREM-LIC license? Even so, my understanding of MACSec is that it's only to enable point to point security at layer 2. Even if the PREM license doesn't include MACSec, I can't see why my speeds got even slower or why I had to manually set them to 10G full duplex.

In any case I'm going to try it again but I'm going to change things around a bit. On my new z690 system I'll pull the Intel x520 out and re-enable the onboard Marvell/Aquantia 10Gbe NIC. I'll use a CAT7 cable to connect the RJ45 from that system to one of my Wiitek modules and insert into one of the 10G enabled front SFP+ ports. I'll then install the Intel x520 NIC in my other system and disable the onboard 5Gbe NIC. The Intel x520 will be connected to the switch using a DAC cable. Then both systems will have 10G ports and hopefully the Wiitek module will perform better.

I did look into the Aquantia AQS-107 based modules and found some on eBay for $50 US each, but not sure if they are the ones that have the Aquantia chips or the Marvell chips with the smaller buffer. I see them at retailers for much more - $180 - $210 US a piece. It's also possible that the eBay ones are knock-offs as a few of the ads add the word 'compatible' after the AOM-AQS-107-B0C2-CX part number.

Anyhow, just taking a dinner break and then I'll go retry it again. Wish me luck!

there is no autonegotiation for SFP+ / 10gbase-sr. the ports by default are locked and configured to 1gbps. to run them at 10gbe, you must load a valid license, then go in and set them to 10gbps instead, which is covered in the guide: ICX6610 Licensing - Fohdeesha Docs

 

[AgentXXL]

there is no autonegotiation for SFP+ / 10gbase-sr. the ports by default are locked and configured to 1gbps. to run them at 10gbe, you must load a valid license, then go in and set them to 10gbps instead, which is covered in the guide: ICX6610 Licensing - Fohdeesha Docs

My 6610 came with the license to enable them and I guess the seller likely set them to that before shipping to me. Regardless, my revert attempt was a wild grasp at air, i.e. a waste of my time. I have since reloaded the licenses you provide and followed the 3 guides. Alas I still get far too many retransmits with the Wiitek handling the rate drop from 10Gbps to 5Gbps.

My unRAID server reports the connection as 5Gbps so the Wiitek is at least partially working. I'm looking for one of the Supermicro units with the AQS-107 chip and will likely buy it when I come across a supplier who has it in stock and will ship to Canada. It will be expensive... too bad the unRAID kernel doesn't support Thunderbolt 3 10Gbe NICs as they aren't as high priced.

Thanks again for all the information you've gathered and shared via your excellent site.

 

[linuxsrc]

My 6610 came with the license to enable them and I guess the seller likely set them to that before shipping to me. Regardless, my revert attempt was a wild grasp at air, i.e. a waste of my time. I have since reloaded the licenses you provide and followed the 3 guides. Alas I still get far too many retransmits with the Wiitek handling the rate drop from 10Gbps to 5Gbps.

My unRAID server reports the connection as 5Gbps so the Wiitek is at least partially working. I'm looking for one of the Supermicro units with the AQS-107 chip and will likely buy it when I come across a supplier who has it in stock and will ship to Canada. It will be expensive... too bad the unRAID kernel doesn't support Thunderbolt 3 10Gbe NICs as they aren't as high priced.

Thanks again for all the information you've gathered and shared via your excellent site.

When I spoke about using your storage under a different VLAN, I should have stated that is if you had other devices that would be dedicated to talking to that storage. For your normal access to the storage server, you would use a different VLAN to communicate to it.

I tend to forget that most people do not run as a complicated setup as I do, where my servers that do a bunch of tasks like virtualization on top of the storage use the higher MTU, whereas the clients on the virtualization do not see the underlying storage because it has been virtualized. I then expose the storage virtually for desktops and servers to make use of it.

So for most using a higher MTU would not make sense to do if you are exposing your unRaid storage server directly to the client machines. If you were creating a cephs storage cluster or something like that you would want to use higher MTUs. A virtualization storage system that communicated between machines would only be via one VLAN for storage and another VLAN for external access like the Internet or management.

If you are on a flat network and will be accessing that server directly, you would never set that system up for a large MTU. Keep it at 1500.

One example for the use of higher MTU would be, something like, you have another system set up to do rsync of your unRaid box for backup to another, you could set a VLAN up for just that alone to increase throughput.

Never have used unRaid before, but if it supported a redundant system to backup to, then that would be a case as well. Your VLAN for backups or syncing would make use of it.

It is common in the industry to use a separate VLAN for backups, syncing, and other high throughput services that would never be connected via a client-side network.

Based on what you have said so far, you can set the MTU on the switch as high as you want, but set all of your interfaces at the correct MTU for use if the unRaid box needs Internet and client access.

 

[i386]

yall mfs crazy with them fans gahdamnbah

Sth becomes a place where people frankenstein more and more enterprise stuff. Even stuff that is relatively quiet (eg for a 24 bay chassis or aristas running with fans at 30%)...

Not sure if it's a corona thing and people spending (too much) time at home

 

[AgentXXL]

When I spoke about using your storage under a different VLAN, I should have stated that is if you had other devices that would be dedicated to talking to that storage. For your normal access to the storage server, you would use a different VLAN to communicate to it.

I tend to forget that most people do not run as a complicated setup as I do, where my servers that do a bunch of tasks like virtualization on top of the storage use the higher MTU, whereas the clients on the virtualization do not see the underlying storage because it has been virtualized. I then expose the storage virtually for desktops and servers to make use of it.

So for most using a higher MTU would not make sense to do if you are exposing your unRaid storage server directly to the client machines. If you were creating a cephs storage cluster or something like that you would want to use higher MTUs. A virtualization storage system that communicated between machines would only be via one VLAN for storage and another VLAN for external access like the Internet or management.

If you are on a flat network and will be accessing that server directly, you would never set that system up for a large MTU. Keep it at 1500.

One example for the use of higher MTU would be, something like, you have another system set up to do rsync of your unRaid box for backup to another, you could set a VLAN up for just that alone to increase throughput.

Never have used unRaid before, but if it supported a redundant system to backup to, then that would be a case as well. Your VLAN for backups or syncing would make use of it.

It is common in the industry to use a separate VLAN for backups, syncing, and other high throughput services that would never be connected via a client-side network.

Based on what you have said so far, you can set the MTU on the switch as high as you want, but set all of your interfaces at the correct MTU for use if the unRaid box needs Internet and client access.

Thanks for those comments. Yes, my home network is very much a flat topology - I'm retired so I don't need to build out a full homelab that resembles what is done at the corporate/enterprise level. As it turns out, as @Dave Corder mentioned earlier in this thread, it appears that it's the Wiitek SFP module that's causing my issues. As the 10G ports on the 6610 can only negotiate a 1 or 10 Gbps connection, it's up to the SFP module to rate limit to 5Gbps. My Wiitek modules apparently aren't as compatible with the 6610 as they are with other switches. They're listed as capable of 1, 2.5, 5 and 10Gbps connections, but there's some incompatibility with it based on this thread:

https://forums.servethehome.com/index.php?threads/gbase-t-transceivers-performance-with-nbase-t-marvell-88x3300-v-s-aquantia-aqs-107.30004

I had tried to research which SFP modules to buy and based my decision on this STH review. Alas that reviewer didn't test the Wiitek on the 6610. It wouldn't surprise me if it worked fine on more modern switches.

https://www.servethehome.com/wiitek-sfp-10g-t-review-10gbase-t-adapter/

I'm now looking for a supplier that has the working SFP modules that use the AQS-107 chip and have more buffer capacity. I'm seeing so many retransmits with the Wiitek that it's affecting the speed of the transfers. For the time being I've got the 5Gbps plugged into a 1Gbps port on the 6610 and at least with that config I'm seeing no retransmit errors.

 

[RobstarUSA]

Thanks for those comments. Yes, my home network is very much a flat topology - I'm retired so I don't need to build out a full homelab that resembles what is done at the corporate/enterprise level. As it turns out, as @Dave Corder mentioned earlier in this thread, it appears that it's the Wiitek SFP module that's causing my issues. As the 10G ports on the 6610 can only negotiate a 1 or 10 Gbps connection, it's up to the SFP module to rate limit to 5Gbps. My Wiitek modules apparently aren't as compatible with the 6610 as they are with other switches. They're listed as capable of 1, 2.5, 5 and 10Gbps connections, but there's some incompatibility with it based on this thread:

https://forums.servethehome.com/index.php?threads/gbase-t-transceivers-performance-with-nbase-t-marvell-88x3300-v-s-aquantia-aqs-107.30004

I had tried to research which SFP modules to buy and based my decision on this STH review. Alas that reviewer didn't test the Wiitek on the 6610. It wouldn't surprise me if it worked fine on more modern switches.

https://www.servethehome.com/wiitek-sfp-10g-t-review-10gbase-t-adapter/

I'm now looking for a supplier that has the working SFP modules that use the AQS-107 chip and have more buffer capacity. I'm seeing so many retransmits with the Wiitek that it's affecting the speed of the transfers. For the time being I've got the 5Gbps plugged into a 1Gbps port on the 6610 and at least with that config I'm seeing no retransmit errors.

If on the 6610 port with the mgig sfp, can you do a "rate-limit output shaping 4000000" and see if you get 4Gbit/s throughput ?

 

[jcstill]

Just received a new (to me) 7250-24p. Walking through the setup guide, the uboot image updates just fine, but the primary image fails.

All I get is:

ICX7250-Boot> update_uboot
Loading image to Uboot Partition 1
Using bcmiproc_eth-0 device
TFTP from server 10.22.102.17; our IP address is 10.22.102.200
Filename 'ICX7xxx/spz10118.bin'.
Load address: 0x61007dc0
Loading: #################################################################
         #################################################################
         ########################
done
Bytes transferred = 786944 (c0200 hex)
sf erase 0x0 0x100000
copying uboot image to flash, it will take sometime...
sf write 0x61007fc0 0x0 0xc0000 
TFTP to Flash Done.
ICX7250-Boot>
//
ICX7250-Boot> update_primary
Updating PRIMARY image over TFTP
Using bcmiproc_eth-0 device
TFTP from server 10.22.102.17; our IP address is 10.22.102.200
Filename 'ICX7xxx/SPR08090mc.bin'.
Load address: 0x1000000
Loading: #################################################################
         #################################################################
                              (100 lines cut)
         #################################################################
         #################################################################
         #######################################################Primary image download failed
ICX7250-Boot>

The switch came to me runing v08.0.30fT211. I thought I remember reading in this thread that updating from pre 08.0.80 to 8.0.90 needed to be incremental. ie 08.0.30 -> 08.0.80 -> 08.0.90. Is this the case?

Thanks

 

[fohdeesha]

Just received a new (to me) 7250-24p. Walking through the setup guide, the uboot image updates just fine, but the primary image fails.

All I get is:

ICX7250-Boot> update_uboot
Loading image to Uboot Partition 1
Using bcmiproc_eth-0 device
TFTP from server 10.22.102.17; our IP address is 10.22.102.200
Filename 'ICX7xxx/spz10118.bin'.
Load address: 0x61007dc0
Loading: #################################################################
         #################################################################
         ########################
done
Bytes transferred = 786944 (c0200 hex)
sf erase 0x0 0x100000
copying uboot image to flash, it will take sometime...
sf write 0x61007fc0 0x0 0xc0000
TFTP to Flash Done.
ICX7250-Boot>
//
ICX7250-Boot> update_primary
Updating PRIMARY image over TFTP
Using bcmiproc_eth-0 device
TFTP from server 10.22.102.17; our IP address is 10.22.102.200
Filename 'ICX7xxx/SPR08090mc.bin'.
Load address: 0x1000000
Loading: #################################################################
         #################################################################
                              (100 lines cut)
         #################################################################
         #################################################################
         #######################################################Primary image download failed
ICX7250-Boot>

The switch came to me runing v08.0.30fT211. I thought I remember reading in this thread that updating from pre 08.0.80 to 8.0.90 needed to be incremental. ie 08.0.30 -> 08.0.80 -> 08.0.90. Is this the case?

Thanks

can you try to run that again (mainly the update u boot command), then reset the switch, (just send the command "reset" but be sure to stop it back into the bootloader again (if it boots all the way into the OS, it may downgrade uboot again). once you stop it back in uboot after the reboot, it should hopefully be in the new version of uboot, then try the update primary command again

 

[acurax04]

I am very inexperienced and looking for help setting up an ICX 7150.

Following the setup guide (ICX7150 - Fohdeesha Docs) and I cannot get the ICX7150 to fetch the firmware images from the TFTP server. I am using the USB-C port on the 7150 to connect directly to my PC. I am able to access the bootloader via Putty and assign temp IP/mask to the 7150 and direct to the TFTP server files, but the 'update_uboot' command results in this:



I'm using TFTPD64 from the zip file in the guide:



I have disabled Windows Firewall. I only have the one PC connected to a MikroTik router at this time and the 7150 to the PC via USB-C port. I can't ping between the devices--not sure what I have set up incorrectly...

 

[NablaSquaredG]

I currently do not have access to ICX6610 as I'm on the road

Is it possible to use the 4*10G QSFP+ breakout ports on the ICX6610 for stacking without using the regular QSFP+ ports for stacking too? i.e. stacking via the 4*10G QSFP+ breakout ports and using the other two QSFP+ port for regular connection?

I remember that I was able to use the regular QSFP+ ports for stacking and use the 4*10G ports for a link to the core switches - But that was not an optimal solution because a 4*10G with 802.1ad LACP is not as good as a regular QSFP+ port and I lost a port on the SX6036 (breakout disabled other ports on the SX6036)

 

[Rain]

I am very inexperienced and looking for help setting up an ICX 7150.

Following the setup guide (ICX7150 - Fohdeesha Docs) and I cannot get the ICX7150 to fetch the firmware images from the TFTP server. I am using the USB-C port on the 7150 to connect directly to my PC. I am able to access the bootloader via Putty and assign temp IP/mask to the 7150 and direct to the TFTP server files, but the 'update_uboot' command results in this: ...

I have disabled Windows Firewall. I only have the one PC connected to a MikroTik router at this time and the 7150 to the PC via USB-C port. I can't ping between the devices--not sure what I have set up incorrectly...

If you can't ping the switch's IP address (192.168.88.100 in your example) from the computer running the TFTP server, something isn't connected properly.

Don't use the MikroTik router; it shouldn't be needed. As the guide mentions, don't use any of the regular switch ports yet; use the management ethernet port to update the bootloader. Connect an ethernet cable directly from the 7150's management port (regardless of what model you have, this should be the ethernet port just above the USB-A port) to your computer's ethernet port.

 

[kpfleming]

Is it possible to use the 4*10G QSFP+ breakout ports on the ICX6610 for stacking without using the regular QSFP+ ports for stacking too? i.e. stacking via the 4*10G QSFP+ breakout ports and using the other two QSFP+ port for regular connection?

Probably not; only the designated stacking ports can be used for stacking.

 

[AgentXXL]

If on the 6610 port with the mgig sfp, can you do a "rate-limit output shaping 4000000" and see if you get 4Gbit/s throughput ?

I'm a novice when it comes to configuring a managed switch like the 6610. I did a bit of reading and tried this command:

Device(config)#interface ethernet 1/3/2
Device(config-if-e1000-1/3/2)#rate input fixed 4000000

Traffic in one direction was OK with no retransmits, but the other still showed many as seen in the attached pic:

The rate limit applied was to the machine which has the onboard 5Gbe NIC. Out of curiosity I applied the same rate limit to the other machine which has the onboard 10Gbe NIC. Alas I still see poor speeds and many retransmit errors.

As I'm new to a lot of the concepts in a managed switch, I'm doing a lot of reading to try and see if there's a solution. If you have other things you want me to try, let me know. Thanks!

 

[NablaSquaredG]

Probably not; only the designated stacking ports can be used for stacking.

You know that *all* of them are stacking ports?

I can use the 40G QSFP ports for stacking or 40G QSFP AND 4*10G QSFP breakout together for stacking, but I haven't managed to only use the 4*10G QSFP breakouts for stacking

 

[Vesalius]

PSA for anyone that jumped ahead to the 9.0.* firmware.

FastIron software releases 09.0.00, 09.0.00a, and 09.0.10 are no longer available for download due to the recent discovery of a critical defect. RUCKUS recommends upgrading to FastIron software release 09.0.10a for all switches currently running the impacted FastIron releases 09.0.00, 09.0.00a, and 09.0.10. Refer to TSB-2022-001 FastIron 09.0.00 and 09.0.10 - Risk of Filesystem Corruption for more details.

RUCKUS ICX FastIron 09.0.10a (GA) Software Release (.zip) | Software Downloads | Ruckus Wireless Support

support.ruckuswireless.com

 

[fohdeesha]

I am very inexperienced and looking for help setting up an ICX 7150.

Following the setup guide (ICX7150 - Fohdeesha Docs) and I cannot get the ICX7150 to fetch the firmware images from the TFTP server. I am using the USB-C port on the 7150 to connect directly to my PC. I am able to access the bootloader via Putty and assign temp IP/mask to the 7150 and direct to the TFTP server files, but the 'update_uboot' command results in this:

View attachment 21733

I'm using TFTPD64 from the zip file in the guide:

View attachment 21734

I have disabled Windows Firewall. I only have the one PC connected to a MikroTik router at this time and the 7150 to the PC via USB-C port. I can't ping between the devices--not sure what I have set up incorrectly...

it sounds like you haven't actually plugged the switch into your network? The usb-c serial connection is not a network connection, as the guide states you need to connect the switches dedicated management ethernet port (NOT any of the regular ports) to your network