Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

[BlackCat]

Can you check the firmware version on it? On mine when I run 'show version' it shows SW: Version 08.0.30uT313. Are you running 7.x? When I got mine it was running 7.4 and the fans didn't spin down much. After upgrading to 8.0.3 they spun down after booting and became a lot quieter. Way too loud for a living room, but much quieter than before. My replacement fans should arrive today.

The switch is running on sw version: 08.0.30uT313.

ICX6450-48>sh ver
Copyright (c) 1996-2016 Brocade Communications Systems, Inc. All rights reserved.
UNIT 1: compiled on Apr 23 2020 at 10:57:06 labeled as ICX64R08030u
(9871112 bytes) from Primary ICX64R08030u.bin
SW: Version
Boot-Monitor Image size = 786944, Version:10.1.05T310 (kxz10105)
HW: Stackable ICX6450-48

 

[fohdeesha]

All done! It went on rails once able to get the "factory set-default" to run.

Thanks again to all, particularly to Jon/Fohdeesha for the facility and a beer/coffee token is inbound!

Paul

Thanks for the donation! I'll put it to good use (altho I think wolf and ronin did a lot more helping here than anyone else lmao)

 

[clcorbin]

I've been having a problem with one port (work laptop 1Gb LAN port) since I switched to the ICX 6610 stack. One 6610 is in the hobby room in the computer stack, the other is out in the garage (one switch would have been MORE THAN ENOUGH ports, but about half my network drops led to the hobby room (stuff I've added) and half to the garage (original drops)) linked by dual 40GB fiber on x/2/1 and x/2/6 stacking ports.

One port from the garage switch on vlan 50 comes back to my hobby room (again, existing cabling in the right place) and goes to a basic, unmanaged Netgear GS108 switch. My work laptop and my personal laptop are connected to it.

The personal laptop (Windows 10 Pro on my AD domain) has worked flawlessly. But the work laptop is being a bit of a pain. It will disconnect the LAN from the network at (what appears to be) random times and then reconnect like nothing happened a few HOURS later. I've seen it happen two or three times in a day. There IS a physical connection during this time, but it gets the default ip address and has no connectivity to anything. But it DOES know there is a cable connected to SOMETHING out there.

Every single time it has happened, I've checked my personal laptop and it is connected just fine. I've checked (and then swapped) the cable with no change. Once, I saw a message on the network adapter show it had failed authentication. Which is a bit strange, as I don't have 802.1x setup on my network.

Using the same network (other than the switches), I never saw this problem in the 1 1/2 years I've had this laptop at my house on my network.

Also, the WiFi (from an Aruba IAP-305 connected to the server rack switch) has remained flawless as well. Same computer, but it IS brought into a different VLAN while connected to WiFi.

Any suggestions on what to look at would be greatly appreciated!

Clint

I did find the root cause of these problem: The patch cable going from the wall plate to the 6610 in the garage and the GS108 switch in my hobby room was bad. I know I tested it after I originally made it (I do NOT trust myself to make a perfect patch cable without testing to prove it!). I have to assume my personal laptop was just more tolerant of dropped frames than my work laptop.

Either way, when I retested that cable and it failed, I replaced it with one that passed the tester and the disconnect issue on my work laptop disappeared and has not come back in over a month.

 

[Lone Wolf]

The switch is running on sw version: 08.0.30uT313.

Bummer, I had hoped you had the same issue I did, which was running the 7.x version of the firmware. My fans ran at high speed until I updated to 8.0.3, and then they spun down to a slower speed and were a lot quieter. I don't know if there is a way to adjust those, other than replacing the fans with quieter ones.

 

[BlackCat]

Bummer, I had hoped you had the same issue I did, which was running the 7.x version of the firmware. My fans ran at high speed until I updated to 8.0.3, and then they spun down to a slower speed and were a lot quieter. I don't know if there is a way to adjust those, other than replacing the fans with quieter ones.

Totally agreed with you.
Already placed order on replacement Sunon fans - first 1.4w and second 3.8w. Will test them for optimal results.

 

[Lone Wolf]

Totally agreed with you.
Already placed order on replacement Sunon fans - first 1.4w and second 3.8w. Will test them for optimal results.

Let us know how you make out with those!

 

[BlackCat]

Let us know how you make out with those!

Shipping from China can take about month or two. I have to take deep breath...

 

[custom90gt]

Two potentially silly questions about my new to me Brocade 6610 (non-POE).

1. Can the 250W rev A power supply be replaced by a 1000w rev b/c power supply for any less noise?

2. Down the road would it ever be possible to add the POE module? Not sure what all is different between the two. Just in case I started running POE cameras. If not, my 6540 will do just fine there.

Thanks

 

[NablaSquaredG]

2. Down the road would it ever be possible to add the POE module? Not sure what all is different between the two. Just in case I started running POE cameras. If not, my 6540 will do just fine there

I believe this was discussed earlier in this thread and it turned out that the PoE and non-PoE versions use different components and the non-PoE boards miss some headers

 

[custom90gt]

I believe this was discussed earlier in this thread and it turned out that the PoE and non-PoE versions use different components and the non-PoE boards miss some headers

Appreciate it, I tried searching this thread but didn't come up with an answer to that or the other question. Either way I figure it's a good switch for a great deal, especially since I plan on doing more 10GB.

 

[klui]

https://forums.servethehome.com/index.php?search/901107/&q=non-poe&t=post&c[thread]=21107&o=relevance

The previews within the top 10 hits make it easy to determine what @NablaSquaredG was referring to.

 

[Kartoff]

Hello, I found some ICX7750 and thinking about to buy some... They look pretty good with 48x SFP+ and 6x QSFP instead of usually 4x
I have some VDX switches but no ICX so I have no experience with them... Will be nice to hear some impressions. Are they SFP capricious ? As I know ICX does not have locked ports right ? What is the normal power draw ?
BTW I already managed to download firmware from Ruckus site
Also I hope they can manage to handle like 7 Gbit multicast without problems... I am not so familiar with multicast and what I know from network admins who deal with it, not every switch is suitable for this purpose
Last but not least, what price to chase ?

Thank you

 

[fohdeesha]

Version 5.2 of tftpd-hpa (appears to be the current version for Ubuntu LTS 20.04, and potentially others) does not like having directory names with spaces in them, even when they are properly enclosed in quotes. It treats a directory with spaces in it as being 2 (or more) directories, and then throws an error. This is because it's just piping the Directory variable through a "wc -w" command, which is dumb, but whatever. Easily fixed. I removed the space in the "TFTP Content" directory and presto-magico everything works.

Thanks again to everyone who has contributed to this massive thread! Getting my 7250-24p up and running now.

thanks for the heads up, fixed: remove spaces from tftp dir name · Fohdeesha/lab-docu@2d24adc

 

[bpye]

I know the ICX6450 has pretty limited IPv6 support, but is the DHCPv6 relay expected to work fully? I was wondering if I could use it to have a routed WAN -> ICX6450 -> Gateway by relaying the DHCP requests from the gateway to the WAN - but it looks like these packets are getting swallowed?

1/1/37 is the connection to the gateway, and 1/2/1 is the WAN. Would I need to add some additional route to make this work, or is the issue that PD requests are getting filtered? I see that 'Prefix Delegation Information' is included in 'show ipv6 dhcp-relay' - and I see a number of PD related DHCP relay commands in the command reference, however these commands don't actually appear to exist, for example 'ipv6 dhcp-relay maximum-delegated-prefixes'.

SSH@ICX6450-48P Router(config-if-e1000-1/1/37)#show ipv6 dhcp-relay                                                   
Current DHCPv6 Relay Agent State: Enabled
DHCPv6 enabled interface(s): e 1/1/37 
DHCPv6 Relay Agent Statistics:
 Total DHCPv6 Packets, Received:12, Transmitted:0 
 Received DHCPv6 Packets: RELEASE:0,RELAY_FORWARD:0,RELAY_REPLY:0 
                          OtherServertoClient:0,OtherClinettoServer:12
DHCPv6 Relay Prefix Delegation Notification:
 Maximum Delegated Prefixes Allowed:0 Current Delegated Prefixes:0
SSH@ICX6450-48P Router(config-if-e1000-1/1/37)#show ipv6 dhcp-relay interface ethernet 1/1/37
DHCPv6 Relay Information for interface e 1/1/37: 
Destinations:
 Destination                    OutgoingInterface 
 ff05::1:3                      e 1/2/1
 ff02::1:2                      e 1/2/1
Options:
 Interface-Id: No     Remote-Id:No
Prefix Delegation Information:
 Current:0 Maximum:0 AdminDistance:10

 

[fohdeesha]

I know the ICX6450 has pretty limited IPv6 support, but is the DHCPv6 relay expected to work fully? I was wondering if I could use it to have a routed WAN -> ICX6450 -> Gateway by relaying the DHCP requests from the gateway to the WAN - but it looks like these packets are getting swallowed?

1/1/37 is the connection to the gateway, and 1/2/1 is the WAN. Would I need to add some additional route to make this work, or is the issue that PD requests are getting filtered? I see that 'Prefix Delegation Information' is included in 'show ipv6 dhcp-relay' - and I see a number of PD related DHCP relay commands in the command reference, however these commands don't actually appear to exist, for example 'ipv6 dhcp-relay maximum-delegated-prefixes'.

SSH@ICX6450-48P Router(config-if-e1000-1/1/37)#show ipv6 dhcp-relay                                                 
Current DHCPv6 Relay Agent State: Enabled
DHCPv6 enabled interface(s): e 1/1/37
DHCPv6 Relay Agent Statistics:
Total DHCPv6 Packets, Received:12, Transmitted:0
Received DHCPv6 Packets: RELEASE:0,RELAY_FORWARD:0,RELAY_REPLY:0
                          OtherServertoClient:0,OtherClinettoServer:12
DHCPv6 Relay Prefix Delegation Notification:
Maximum Delegated Prefixes Allowed:0 Current Delegated Prefixes:0
SSH@ICX6450-48P Router(config-if-e1000-1/1/37)#show ipv6 dhcp-relay interface ethernet 1/1/37
DHCPv6 Relay Information for interface e 1/1/37:
Destinations:
Destination                    OutgoingInterface
ff05::1:3                      e 1/2/1
ff02::1:2                      e 1/2/1
Options:
Interface-Id: No     Remote-Id:No
Prefix Delegation Information:
Current:0 Maximum:0 AdminDistance:10

Is there a reason you're doing it on raw l2 ports and not on an addressed VE? How have you specified the relay destination? can't help much unless you share the config. I've been using dhcpv6 relay on a 6 series for years (to redundant dhcpd servers as well) without issue, but I'm not doing prefix delegation:

interface ve 10
ip address 192.168.1.1 255.255.255.0
ip helper-address 1 172.16.110.2
ip helper-address 2 172.16.110.3
ipv6 address xxx:xxx/64
ipv6 enable
ipv6 dhcp-relay destination zzz:xxx::2
ipv6 dhcp-relay destination zzz:xxx::3
ipv6 dhcp-relay include-options interface-id remote-id
ipv6 nd managed-config-flag

you're also showing 0 transmitted dhcpv6 packets, I dunno if this is part of the cause, or just a symptom of your ICX getting PD requests and intentionally not forwarding them for some config issue reason

telnet@ICX1(config-vif-10)#show ipv6 dhcp-rela
Current DHCPv6 Relay Agent State: Enabled
DHCPv6 enabled interface(s): ve 10
DHCPv6 Relay Agent Statistics:
Total DHCPv6 Packets, Received:23352, Transmitted:32847
Received DHCPv6 Packets: RELEASE:3,RELAY_FORWARD:0,RELAY_REPLY:11607
                          OtherServertoClient:0,OtherClinettoServer:11742
DHCPv6 Relay Prefix Delegation Notification:
Maximum Delegated Prefixes Allowed:512 Current Delegated Prefixes:0

on my 6610 the maximum delegated seems to default to 512, but yeah I can't change it at first glance, from any config level:

telnet@ICX1(config)#ipv6 dhcp-relay maximum-delegated-prefixes 50
Invalid input -> dhcp-relay maximum-delegated-prefixes 50
Type ? for a list

my guess is the command was only introduced to 7 series switches so the command won't work on the 6 series fw builds, the limit seems hardcoded (but indeed strange it would be limited to 0 on the 6450, not sure why they would include it as a configurable feature in that case)

 

[fohdeesha]

OK yeah, it's fastiron 6 series bug ****ery. The command exists and works on my lab FCX, which runs **the exact same bootloader and fw image as the 6610**:

telnet@ORAN-FCX(config)#ipv6 dhcp-relay maximum-delegated-prefixes
  DECIMAL   Maximum delegated prefixes, <0-512>
telnet@ORAN-FCX(config)#ipv6 dhcp-relay maximum-delegated-prefixes 50
telnet@ORAN-FCX(config)#

On a 6610 next to it with the bit-identical fw image booted:

telnet@ICX1(config)#ipv6 dhcp-relay maximum-delegated-prefixes 50
Invalid input -> dhcp-relay maximum-delegated-prefixes 50
Type ? for a list

probably worth reporting to ruckus on their user forum at least so it's documented, but who knows if it'll ever be addressed (I think they have one more bug & security release in them before these are officially End-Of-Support in Nov 2023). At the very least their documentation should be updated to state the command only works on the FCX (I've ran into a couple other commands like this, that exist in the base firmware but only become active when booted on an FCX, namely "ip ssh key-exchange-method dh-group14-sha1", which is completely nonsensical to me as the FCX runs an identical management plane vs the 6610, exact PPC part number, although less RAM I suppose )

 

[bpye]

My bad - relevant config:

vlan 1 name DEFAULT-VLAN by port
 router-interface ve 1
!
vlan 100 name wan by port
 untagged ethe 1/2/1 
!
vlan 101 by port
 untagged ethe 1/1/37 
!
ipv6 unicast-routing
interface ethernet 1/1/1
 inline power
!
interface ethernet 1/1/37
 ipv6 enable
 ipv6 dhcp-relay destination ff02::1:2 outgoing-interface ethernet 1/2/1
 ipv6 dhcp-relay destination ff05::1:3 outgoing-interface ethernet 1/2/1
 ipv6 dhcp-relay include-options interface-id remote-id           
 ipv6 nd managed-config-flag
!
interface ethernet 1/2/1
 ipv6 enable
 ipv6 nd suppress-ra
 speed-duplex 1000-full-master
!
interface ve 1
 ip address 192.168.1.10 255.255.255.0

No real reason I'm using the L2 interfaces...

 

[fohdeesha]

My bad - relevant config:

vlan 1 name DEFAULT-VLAN by port
router-interface ve 1
!
vlan 100 name wan by port
untagged ethe 1/2/1
!
vlan 101 by port
untagged ethe 1/1/37
!
ipv6 unicast-routing
interface ethernet 1/1/1
inline power
!
interface ethernet 1/1/37
ipv6 enable
ipv6 dhcp-relay destination ff02::1:2 outgoing-interface ethernet 1/2/1
ipv6 dhcp-relay destination ff05::1:3 outgoing-interface ethernet 1/2/1
ipv6 dhcp-relay include-options interface-id remote-id          
ipv6 nd managed-config-flag
!
interface ethernet 1/2/1
ipv6 enable
ipv6 nd suppress-ra
speed-duplex 1000-full-master
!
interface ve 1
ip address 192.168.1.10 255.255.255.0

No real reason I'm using the L2 interfaces...

Does your icx6450 have an ipv6 address it can source traffic from to these ff02::1:2/3 destinations? eg can you run a v6 ping from the 6450 to those addresses and get a reply? guess in the dark here, but maybe because you're doing dhcpv6 relay on an interface with no IPv6 address? (well, other than the local link). maybe try assigning an ipv6 address to 1/1/37 just to test, then se if the delegation limit increases, if you're relying on link-local addresses for relay source and destination perhaps it doesn't like that (although technically in a perfect world it shouldn't matter)

 

[bpye]

Does your icx6450 have an ipv6 address it can source traffic from to these ff02::1:2/3 destinations? eg can you run a v6 ping from the 6450 to those addresses and get a reply? guess in the dark here, but maybe because you're doing dhcpv6 relay on an interface with no IPv6 address? (well, other than the local link). maybe try assigning an ipv6 address to 1/1/37 just to test, then se if the delegation limit increases, if you're relying on link-local addresses for relay source and destination perhaps it doesn't like that (although technically in a perfect world it shouldn't matter)

I'll give that a try later today after work, I am thinking I should maybe try and find a 7250, or ideally a 7150 for the fanless mode but I don't see many. The 6450 was cheap and does do the basic stuff I need, and whilst I knew it didn't have full IPv6 support I missed that it didn't support PBR, which is more annoying...

 

[tochnia]

Not sure if anyone have tested noise levels before, but I got calibrated Audio Analyser and measured noise from my 6610 with B revision Power supplies.
With only one Power Supply connected to power noise measurement are:

56dB at 1 meter distance
50dB at 2 meter distance

This is too much noise for me to stay permanently in the same room with switch..

66dB at 1 meter distance - Boot noise
60dB at 2 meter distance - Boot noise

Also Power Consumption measured on my 6610-48P is between 100-116W with no PoE enabled ports.