Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

[LodeRunner]

Does the 6610 even support 5GbE? It's not listed in the datasheet.

Not natively, no. 1/10G only. There is a multi-gig SFP that some people have used, but I can’t recall if anyone tried it in a 6610.

 

[LodeRunner]

I suspect my ICX6610 recently acquired from ebay has a bad console management port. When I connect a known-good usb-to-rj45 and view the serial output over minicom it either stops at

(if no connection to LAN)

ICX6610-48 Switch>

Power supply 2  detected.


Power supply 2  detected.

Power supply 2  is up.

or just repeats tftp session timed out after that (if connected to LAN)

I'm able to view serial output, but none of my keystrokes register. So no show version or show license commands seem posible through serial-to-rj45 over minicom

I can telnet in and receive the below

telnet@ICX6610-48 Switch>show license
License record empty
telnet@ICX6610-48 Switch>show version
  Copyright (c) 1996-2016 Brocade Communications Systems, Inc. All rights reserved.
    UNIT 1: compiled on May 06 2017 at 08:15:28 labeled as FCXS08030n
        (7762230 bytes) from Primary FCXS08030n.bin
        SW: Version 08.0.30nT7f1
  Boot-Monitor Image size = 370555, Version:07.3.02T7f5 (grz07302)
  HW: Stackable ICX6610-48
==========================================================================
UNIT 1: SL 1: ICX6610-48 48-port Management Module
      Serial  #: BXxxxxxxxx
      License: BASE_SOFT_PACKAGE   (LID: xxxxxxxxx)
      P-ENGINE  0: type E02B, rev 01
      P-ENGINE  1: type E02B, rev 01
==========================================================================
UNIT 1: SL 2: ICX6610-QSFP 10-port 160G Module
==========================================================================
UNIT 1: SL 3: ICX6610-8-port Dual Mode(SFP/SFP+) Module
==========================================================================
  800 MHz Power PC processor 8544E (version 0021/0023) 400 MHz bus
65536 KB flash memory
  512 MB DRAM
STACKID 1  system uptime is 23 minute(s) 1 second(s)
The system : started=cold start

Is the console port bad? is there a way to check or some suggested troubleshooting?

serial output below:

ICX Boot Code Version 7.3.02 (grz07302)                                  

Enter 'a' to stop at memory test                                          

Enter 'b' to stop at boot monitor                                        

BOOT INFO: load monitor from boot flash, cksum = 1fef                    

BOOT INFO: verify flash files....                                        

BOOT INFO: load image from primary copy...                                

                                                                         

platform type = 9                                                        

PCIE-1 LTSSM status: 22                                                  

PCIE Switch status: 0                                                    

......................                                                    

Firmware integrity checksum passed

.....

Starting Main Task .Applying factory defaults..

INFO: startup config data is not available, try to read from backup

INFO: startup config data in the backup area is not available

..CPSS DxCh Version: cpss3.4p1 release

Pre Parsing Config Data ...

INFO: empty config data in the primary area, try to read from backup

INFO: empty config data in the backup area also


Parsing Config Data ...

INFO: empty config data in the primary area, try to read from backup

INFO: empty config data in the backup area also


System initialization completed...console going online.

  Copyright (c) 1996-2016 Brocade Communications Systems, Inc. All rights reserved.

    UNIT 1: compiled on May 06 2017 at 08:15:28 labeled as FCXS08030n

                (7762230 bytes) from Primary FCXS08030n.bin

        SW: Version 08.0.30nT7f1

  Boot-Monitor Image size = 370555, Version:07.3.02T7f5 (grz07302)

  HW: Stackable ICX6610-48

==========================================================================

UNIT 1: SL 1: ICX6610-48 48-port Management Module

         Serial  #: BXxxxxxxxxx

         License: BASE_SOFT_PACKAGE   (LID: xxxxxxxxxxx)

         P-ENGINE  0: type E02B, rev 01

         P-ENGINE  1: type E02B, rev 01

==========================================================================

UNIT 1: SL 2: ICX6610-QSFP 10-port 160G Module

==========================================================================

UNIT 1: SL 3: ICX6610-8-port Dual Mode(SFP/SFP+) Module

==========================================================================

  800 MHz Power PC processor 8544E (version 0021/0023) 400 MHz bus

65536 KB flash memory

  512 MB DRAM

STACKID 1  system uptime is 2 minute(s) 30 second(s)

The system : started=cold start


ICX6610-48 Switch>

Power supply 2  detected.


Power supply 2  detected.

Power supply 2  is up.

TFTP session timed out

What serial settings have you tried? I’ve had similar behavior with wrong flow control settings or a flaky ground (homemade keystone to mini-USB for a 7250).

 

[brb78]

What serial settings have you tried? I’ve had similar behavior with wrong flow control settings or a flaky ground (homemade keystone to mini-USB for a 7250).

I only changed the target to /dev/USB0 minicom defaulted to 9600 8N1 so I didn't make any other changes. You think there might be a flow control setting to tweak?

 

[AgentXXL]

Does the 6610 even support 5GbE? It's not listed in the datasheet.

The unRAID server reports the connection as 5Gbps, which as mentioned by @LodeRunner is supported by the Wiitek SFP+ to RJ45 module. When I was still using the old 6th gen Intel setup on my 2nd unRAID server, it only had a 1Gbps port and transfers between it and the main unRAID (using the 5Gbps NIC) were saturating at the 1Gbps source. To me that indicates that the switch has no problem with the SFP+ module connecting at 5Gbps.

It's only now that I've upgraded to a 10Gbps connection on the 2nd unRAID that I'm seeing these speed issues. As mentioned it's also affecting download speed from the internet as I was previously able to saturate my ISP connection when using the 1Gbps NIC. Of course having a 10Gbps NIC won't help with increasing my ISP speed, but instead it's made it worse.

Could it be that kernel 5.14 (used on the latest release candidate of unRAID - 6.10.0-rc2) doesn't have proper driver support for the Marvell/Aquantia 10Gbps NIC? I'm going to try booting a Linux distro that's using kernel 5.15 or if I can find one, 5.16.5 which was released very recently, within the last couple of weeks. If the speed improves then I likely have my answer and will have to add a 1Gbps NIC to the 2nd system until unRAID has a release with the new kernel.

As others on Discord have mentioned the 5Gbps might be causing issues, I could install an Intel x520 10Gbps NIC in the main unRAID server, but I suspect that it's the 10Gbps NIC on the other server that's causing my speed issues.

 

[LodeRunner]

I only changed the target to /dev/USB0 minicom defaulted to 9600 8N1 so I didn't make any other changes. You think there might be a flow control setting to tweak?

Minicom has a text based config menu with hardware and software flow control options. I don’t know what its defaults are; in Windows, PuTTY for example defaults to 9600, 8N1, XON/XOFF.

 

[brb78]

Minicom has a text based config menu with hardware and software flow control options. I don’t know what its defaults are; in Windows, PuTTY for example defaults to 9600, 8N1, XON/XOFF.

Thanks. In addition to 9600 8N1 those of us using minicom instead of putty need to set hardware flow control to no. Maybe this is common knowledge, but its not my first time using minicom and I didn't know this.

 

[brb78]

Any idea what TFTP: received error request -- code 2 might mean when runningcopy tftp flash ...? Seems like it might be a permission error on the tftp server, but not sure

 

[LodeRunner]

Any idea what TFTP: received error request -- code 2 might mean when runningcopy tftp flash ...? Seems like it might be a permission error on the tftp server, but not sure

If that’s a linux based TFTP server, check syslog. It may be attempting to read a no -existent path.

What was the full command you tried to execute on the switch?

 

[brb78]

If that’s a linux based TFTP server, check syslog. It may be attempting to read a no -existent path.

What was the full command you tried to execute on the switch?

Full command was copy tftp flash 192.168.78.10 grz10100.bin boot. I don't use tftp much, I'll give the logs a check. I appreciate the input, not sure if I'm missing something setting up the tftp server.

 

[bpye]

ah okay... yeah.. hmmm.. any idea on how to achieve this? perhaps pfsense or something?

From my reading of the command/security reference, I think you could do this selectively too. Have an ACL for all DNS traffic (by port), and set the next-hop to your firewall instance. Then your firewall can do whatever NATing you need. This is probably something I'm going to do as well as I like to run my own DNS server so that I can have a hostname block list.

 

[brb78]

Full command was copy tftp flash 192.168.78.10 grz10100.bin boot. I don't use tftp much, I'll give the logs a check. I appreciate the input, not sure if I'm missing something setting up the tftp server.

For those interested, not all tftp-hpa config files are formatted as described in the Brocade overview of the Fodeesha docs. Depending on implementation the following may be necessary:

# /etc/conf.d/tftpd

TFTPD_ARGS="--user=nobody --secure /srv/tftp/"

TFTP-ArchWiki

 

[adman_c]

Based on sensor from my wearable:
7250 at boot: 75 dB
7250 running: 55 dB
Old Cisco Catalyst 3750 I've had running for years in my wiring closet: 42 dB front, 45 dB back

Anyone tried these low speed/noise fans from this seller? 1x Quiet Replacement Fan (18dBA) for Brocade ICX 6430 ICX 6450 ICX 7250

I did not care for the noise that the Sunon maglev fans made in my 6450. I went with a couple of Delta FFB0412HN-5T20 fans that were quieter than stock but still moved plenty of air. If those aren't readily available, you can check out some of the fans linked in this post.

 

[AgentXXL]

I'm still troubleshooting my poor speed between 5Gbe and 10Gbe NICs on my unRAID servers. iperf3 is showing a LOT of retransmits, regardless of which system is chosen as the server or client. I've made so many settings adjustments on my ICX6610 that I really don't know the current state of it. Which method is the one that will NOT erase my licenses:

factory set-default
reset

or

erase startup-config
reload

TIA!

 

[Dave Corder]

Per this thread (https://forums.servethehome.com/ind...-t-marvell-88x3300-v-s-aquantia-aqs-107.30004), the Marvell-based NBase-T SFP+ adapters like the Wiitek aren't all that great in systems that don't natively support 2.5Gbps/5Gbps. Only the (expensive) Aquantia seems to handle those speeds well.

 

[AgentXXL]

Per this thread (https://forums.servethehome.com/ind...-t-marvell-88x3300-v-s-aquantia-aqs-107.30004), the Marvell-based NBase-T SFP+ adapters like the Wiitek aren't all that great in systems that don't natively support 2.5Gbps/5Gbps. Only the (expensive) Aquantia seems to handle those speeds well.

Interesting... some mildly conflicting info here on STH. Easy to happen with forums/communities like this. I based my purchase of the Wiitek on a review of it here on STH: https://www.servethehome.com/wiitek-sfp-10g-t-review-10gbase-t-adapter/

Of course their testing wasn't on an ICX6610, so it's possible that the Wiitek just doesn't work well with this switch. I've disabled the onboard 10Gbe Marvell/Aquantia NIC on the z690 system and added a Intel x520 PCIe x8 NIC connected via DAC cable directly to the switch. Alas I'm still seeing far too many re-transmits whether it acts as the iperf server or client.

I may try the same on the x299 system but I only have an older HPE/Qlogix dual SFP NIC in my spare parts. I had trouble with them overheating and dropping off the network so I won't trust it for long term use.

I'd still like to factory reset the ICX6610 just to make sure it's not one or more settings that I tried that's causing some of my issues. I just don't know which method to use to ensure I don't erase the licenses. I know I can just re-install the licenses from Fohdeesha but I was also planning to try and revert back to the licenses it came with.

It came with these but they're now invalid as I used the ones provided by Fohdeesha:

1 Node Lock ICX6610-PREM-LIC-SW
2 Node Lock ICX6610-10G-LIC-POD

From what I've read, the switch will have the same features whether I use the ones it came with or the ones provided by Fohdeesha, even though they are 3 licenses instead of the 2 it came with. I'd just prefer not to lose the licenses it came with and I don't want to spend the 24 hours downloading the full 64MB flash.

 

[AgentXXL]

Per this thread (https://forums.servethehome.com/ind...-t-marvell-88x3300-v-s-aquantia-aqs-107.30004), the Marvell-based NBase-T SFP+ adapters like the Wiitek aren't all that great in systems that don't natively support 2.5Gbps/5Gbps. Only the (expensive) Aquantia seems to handle those speeds well.

Oh, and it's even stranger as Marvell bought Aquantia, so the onboard NICs actually get drivers from the Marvell site. Alas running unRAID I'm not too sure I want to try installing the new Linux drivers that were released yesterday.

 

[fohdeesha]

I'm still troubleshooting my poor speed between 5Gbe and 10Gbe NICs on my unRAID servers. iperf3 is showing a LOT of retransmits, regardless of which system is chosen as the server or client. I've made so many settings adjustments on my ICX6610 that I really don't know the current state of it. Which method is the one that will NOT erase my licenses:

factory set-default
reset

or

erase startup-config
reload

TIA!

neither should erase licenses

 

[AgentXXL]

neither should erase licenses

I was able to revert to the original licenses that came with the 6610 and do the factory reset. Alas my speeds between the 5Gbe and 10Gbe systems are now even slower, and retransmits are still quite large. It's entirely possible I did something wrong. I followed your guide this 2nd time other than reflashing the firmware and installing the licenses you kindly provide. I didn't see any need to reflash with the same versions of firmware as my initial flash worked - the R, S and POE firmwares are all updated to the version provided in the zip file.

The licenses mine came with are:

ICX6610-PREM-LIC
ICX6610-10G-LIC-POD (enabling all 8 front SFP+ ports)

I also had issues with the 2 SFP+ ports in that they wouldn't auto-negotiate. I had to go in and configure them as 10G full duplex before they came online - I didn't have to do this the first time I setup the switch, or I don't remember doing it. Perhaps that means the ICX-MACSEC-LIC isn't included in the ICX6610-PREM-LIC license? Even so, my understanding of MACSec is that it's only to enable point to point security at layer 2. Even if the PREM license doesn't include MACSec, I can't see why my speeds got even slower or why I had to manually set them to 10G full duplex.

In any case I'm going to try it again but I'm going to change things around a bit. On my new z690 system I'll pull the Intel x520 out and re-enable the onboard Marvell/Aquantia 10Gbe NIC. I'll use a CAT7 cable to connect the RJ45 from that system to one of my Wiitek modules and insert into one of the 10G enabled front SFP+ ports. I'll then install the Intel x520 NIC in my other system and disable the onboard 5Gbe NIC. The Intel x520 will be connected to the switch using a DAC cable. Then both systems will have 10G ports and hopefully the Wiitek module will perform better.

I did look into the Aquantia AQS-107 based modules and found some on eBay for $50 US each, but not sure if they are the ones that have the Aquantia chips or the Marvell chips with the smaller buffer. I see them at retailers for much more - $180 - $210 US a piece. It's also possible that the eBay ones are knock-offs as a few of the ads add the word 'compatible' after the AOM-AQS-107-B0C2-CX part number.

Anyhow, just taking a dinner break and then I'll go retry it again. Wish me luck!

 

[AgentXXL]

A couple more questions that have probably been asked and answered numerous times, but I'm finding it quite difficult to search this long thread for some definitive answers. Is it still recommended to put the 10G capable ports into their own VLAN? I'm a rookie at VLAN configuration so if I did that, I suspect I'll need to setup some bridging or routing rules so that the 10G devices can see the 1G and 100M devices and vice versa. What do I need to configure so that they can use jumbo frames and/or a larger MTU? It looks like enabling jumbo frames affects all ports on the switch so I'm thinking that's going to create issues for slower connections.

I can set the MTU on my unRAID systems, but when I try to set the MTU of a specific port on the switch, it errors out. Is setting MTU a per-port option? Is it and/or enabling jumbo frames even recommended anymore? I've read lots of conflicting suggestions, some stating that you have to use a large MTU to gain the best speeds, but others stating not to enable jumbo frames and just leave the MTU at the default of 1500. I know that certain devices won't like large MTU packets that will likely lead to fragmentation. So what's the best way to configure things?

Again, TIA!

 

[linuxsrc]

A couple more questions that have probably been asked and answered numerous times, but I'm finding it quite difficult to search this long thread for some definitive answers. Is it still recommended to put the 10G capable ports into their own VLAN? I'm a rookie at VLAN configuration so if I did that, I suspect I'll need to setup some bridging or routing rules so that the 10G devices can see the 1G and 100M devices and vice versa. What do I need to configure so that they can use jumbo frames and/or a larger MTU? It looks like enabling jumbo frames affects all ports on the switch so I'm thinking that's going to create issues for slower connections.

I can set the MTU on my unRAID systems, but when I try to set the MTU of a specific port on the switch, it errors out. Is setting MTU a per-port option? Is it and/or enabling jumbo frames even recommended anymore? I've read lots of conflicting suggestions, some stating that you have to use a large MTU to gain the best speeds, but others stating not to enable jumbo frames and just leave the MTU at the default of 1500. I know that certain devices won't like large MTU packets that will likely lead to fragmentation. So what's the best way to configure things?

Again, TIA!

1. Do you have a router that is routing already?
2. If no external router, then using the switch as a router, does have some limitations in what routing it does.
3. Setting jumbo still allows you on a per-port or VLAN basis the MTU
4. I would segment storage from other VLANs i.e. internal network on one VLAN storage another
5. I would then run a per VLAN MTU, Set the VLAN for normal communications under a 1500 MTU, storage under the max MTU and configure your ports on the storage to the max MTU your nics will support.
6. You would need to understand Brocade's network and routing to get all of that to work without some other routing device to do it for you.
7. Deciding whether or not to do jumbo is really dependent on what type of storage you are doing. Small files, large files etc. Smaller files may be best with standard mtu, I mean like 1k files versus larger megabyte files.