Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

[daleq]

glad to help - is there any reason you didn't continue following the guide and instead started trying to boot into the included L2 only firmware and use the web UI? <snip>. I would go back and follow the guide from the beginning, I've also added a sub-note right in the guide on how to fix the old bootloader/missing factory command issue: ICX6450 - Fohdeesha Docs

<snip> I highly recommend following the actual guide, it's been adjusted to account for these older loaders: ICX6450 - Fohdeesha Docs

Thanks @fohdeesha.

Eventually I caught on and followed the fine documentation. Of course it worked perfectly. Thanks!
I've finished the config/upgrade guide and will proceed with the advanced configuration tomorrow.

One small suggestion, my error text for the factory command was

ICX64XX-boot>> factory set-default
Unknown command 'factory' - try 'help'

So, perhaps change
- From: If you get "factory set-default is not a valid command"
- - - To: If you get "Unknown command 'factory'"

Again, thank you so much for your contribution here. I have appreciated it greatly.

 

[kevindd992002]

Based on the icx6450 documentation, without the external psu it only has 24 poe+ capable ports and 48 poe capable ports. Does that mean what it says it means? Only 24 ports can be used as poe+ even if the power budget is not exceeded?

 

[fohdeesha]

I have two 6450s connected via Brocade branded SFP+ modules. Can I use optical monitoring to check signal strength?

see example command: ICX6xxx Advanced - Fohdeesha Docs

 

[fohdeesha]

Thanks @fohdeesha.

Eventually I caught on and followed the fine documentation. Of course it worked perfectly. Thanks!
I've finished the config/upgrade guide and will proceed with the advanced configuration tomorrow.

One small suggestion, my error text for the factory command was

ICX64XX-boot>> factory set-default
Unknown command 'factory' - try 'help'

So, perhaps change
- From: If you get "factory set-default is not a valid command"
- - - To: If you get "Unknown command 'factory'"

Again, thank you so much for your contribution here. I have appreciated it greatly.

Thanks @fohdeesha.

Eventually I caught on and followed the fine documentation. Of course it worked perfectly. Thanks!
I've finished the config/upgrade guide and will proceed with the advanced configuration tomorrow.

One small suggestion, my error text for the factory command was

ICX64XX-boot>> factory set-default
Unknown command 'factory' - try 'help'

So, perhaps change
- From: If you get "factory set-default is not a valid command"
- - - To: If you get "Unknown command 'factory'"

Again, thank you so much for your contribution here. I have appreciated it greatly.

Thanks for the error text! I was wondering what it was exactly, didn't have a switch here to test with that old of bootloader. fixed on the guide: fix factory set default error · Fohdeesha/lab-docu@47a5f4a

 

[fohdeesha]

also pushed a fix for the cases where some users with icx7xxx series experienced it rebooting into an old bootloader (which then couldn't boot the new OS) even though they just flashed a new one. The issue was caused by the fact that the bootloader flashing command only flashes the currently active bootloader slot, for these users that would be "slot secondary". The guide then specified using "boot_primary" to boot into the OS - however this bootloader command turned out to also be specifying which bootloader slot to boot from as well (not just OS slot), so for these users who had just updated the secondary boot slot, booting from the primary boot slot would be an old bootloader. using reset instead of boot_primary always reboots using the currently active boot slot (the one we just updated), then once fastiron fully boots, the UEFI image will update all boot slots

fix icx7xxx booting into wrong bootloader · Fohdeesha/lab-docu@b96f01c

fix for users whose switches were booting into old bootloaders, "reset" will always reboot using the current bootloader (the new one we just updated)

github.com

 

[kimbo]

see example command: ICX6xxx Advanced - Fohdeesha Docs

Thank you for the link!

Everything is reporting as normal (see below), but the Tx Bias seems high to me. Is there anyway to see what the normal range is? I'm using LC UPC OS2 fibre if that makes a difference.

 Port  Temperature   Tx Power     Rx Power       Tx Bias Current
+----+-----------+--------------+--------------+---------------+
1/2/3   30.0703 C  -001.5187 dBm -005.5067 dBm   42.718 mA
        Normal      Normal        Normal         Normal

 

[pr09]

ICX 7250 / 08.0.90k / IPv6 neighbor discovery roaming

I have it configured as an L3 switch (ve interfaces, static routes). There are two access points plugged into different ports, both serving the same SSID/VLAN. When a device roams from one AP to the other, the switch updates its MAC table (show mac-address vlan 10) with the new port, but the ND table still binds the IP/mac combo to the old port (show ipv6 neighbor ve 10). This makes the IP address unreachable.

Has anyone else encountered this bug? Do newer software releases fix this (without adding too many other IPv6-related bugs)? Is there a command to work around this? Or do I need to just stop trying to use L3 functionality on this switch?

 

[kevindd992002]

Based on the icx6450 documentation, without the external psu it only has 24 poe+ capable ports and 48 poe capable ports. Does that mean what it says it means? Only 24 ports can be used as poe+ even if the power budget is not exceeded?

@fohdeesha any ideas? Can I use more than 24 ports as PoE+ as long as the total power budget of 780W is not exceeded? At least that's what I know about PoE/PoE+ is.

 

[kpfleming]

Do newer software releases fix this (without adding too many other IPv6-related bugs)?

I had lots of strange IPv6 issues using 08.xx firmware, so I switched to 09.xx as soon as it was released (against advice in this thread, but I knew how to revert if I needed to do so). I haven't had any new problems with 09.xx, and those IPv6 issues went away.

If you don't want to go that far, moving up 08.0.95<current> would be an improvement over what you have now.

 

[juju]

I am having connectivity problems hooking up my ICX-7250 to a Dell Poweredge R740 server . The Dell has an Intel X520/I350 daughter card. My connection uses a DAC cable from one of the 10G ports on the Intel card to one 10G port on the ICX-7250.
My connection is very instable , with frequent loss of connectivity. Not sure how to troubleshoot this - change the DAC cable? I previously had stp setup on the brocade, so I turned it off because I thought the Dell Poweredge was bringing down the network with a broadcast storm. The network was rock solid until I included the Dell server. What should I be looking for?

 

[fohdeesha]

ICX 7250 / 08.0.90k / IPv6 neighbor discovery roaming

I have it configured as an L3 switch (ve interfaces, static routes). There are two access points plugged into different ports, both serving the same SSID/VLAN. When a device roams from one AP to the other, the switch updates its MAC table (show mac-address vlan 10) with the new port, but the ND table still binds the IP/mac combo to the old port (show ipv6 neighbor ve 10). This makes the IP address unreachable.

Has anyone else encountered this bug? Do newer software releases fix this (without adding too many other IPv6-related bugs)? Is there a command to work around this? Or do I need to just stop trying to use L3 functionality on this switch?

this was always an issue for me on the icx6xxx (v8030) series, never tried on the 7 series - I would at least update to the recommended stable from ruckus (8095g, it's what's on the guide in this thread), if that still doesn't fix it you can try the absolute latest (09.0.10c)

 

[pr09]

this was always an issue for me on the icx6xxx (v8030) series, never tried on the 7 series - I would at least update to the recommended stable from ruckus (8095g, it's what's on the guide in this thread), if that still doesn't fix it you can try the absolute latest (09.0.10c)

8095g seems to fix this bug and I haven't found other breakage yet.

(I tried 09 series a while back, ran into weirdness with multicast & host-to-host neighbor discovery, and it didn't *seem* to be fixed by going back to 8095, but 8090 did. But now 8095 is working. I don't know anymore...)

 

[fohdeesha]

8095g seems to fix this bug and I haven't found other breakage yet.

(I tried 09 series a while back, ran into weirdness with multicast & host-to-host neighbor discovery, and it didn't *seem* to be fixed by going back to 8095, but 8090 did. But now 8095 is working. I don't know anymore...)

like any new train 8095 was rocky for a while, but they finally hit "recommended" with F or G rev if I remember right. It's what I've been running in production in a few different DCs and haven't found any showstoppers yet

 

[Vesalius]

8095g seems to fix this bug and I haven't found other breakage yet.

(I tried 09 series a while back, ran into weirdness with multicast & host-to-host neighbor discovery, and it didn't *seem* to be fixed by going back to 8095, but 8090 did. But now 8095 is working. I don't know anymore...)

Multicast discovery is weird so far, at least with my combination of HomeKit, and Proxmox VM’s

 

[kevindd992002]

@fohdeesha any ideas? Can I use more than 24 ports as PoE+ as long as the total power budget of 780W is not exceeded? At least that's what I know about PoE/PoE+ is.

Anyone has any ideas about this question?

 

[LodeRunner]

Anyone has any ideas about this question?

FAQ document would seem to indicate that all ports are POE+, so as long as you don't cross the wattage limit (390 for 24, 780 for 48) then it should be fine. Images of the switch don't appear to have any special markings on the ports; for example, my 7450 has the first 8 ports specially marked to indicate they do PoH/PoE++ as apposed to the other 40 which are PoE+ only.

 

[seatrope]

i was trying to get my Sonos devices working with controller and Sonos devices in different VLANs. this is on a ICX-6610 with interVLAN routing done on the switch, and pfsense for FW only. There are a bunch of guide floating out there which are not specific for the Brocade.

at first I tried using a Rpi with interfaces in both VLAN, using udp-broadcast-relay:

GitHub - udp-redux/udp-broadcast-relay-redux: Small daemon to relay udp broadcast / multicast packages on a different subnet.

Small daemon to relay udp broadcast / multicast packages on a different subnet. - GitHub - udp-redux/udp-broadcast-relay-redux: Small daemon to relay udp broadcast / multicast packages on a differe...

github.com

this worked, but I wanted a native solution on the switch. After watching some Terry Henry videos, the solution was so ridiculously simple that I was kicking myself. I’m sure most here know this, but sharing for other noobs like myself

1. At the config level: router pim
2. For each ve that you need multicasts to be bridged, enter each vif and: ip pim

Sonos now works perfectly across VLANs. Apply ACLS as needed for security.

Terry Henry video: https://www.google.com/url?sa=t&rct...=PNsIbdXqHlI&usg=AOvVaw3W1AYaZ1JbclyuZ0jqjiOV

 

[Vesalius]

@seatrope glad that relatively simple fix worked for you. Let us know if you run into any issues going forward. Multicast at home on the ruckus hasn’t been the easiest to figure out for a native solution. Looks like pim might work well for those at layer 3 with established intervlan routing set-up.

I might give the new Multicast VLAN Registration (MVR) module a try on the 0.9 series firmware.

 

[seatrope]

Additional question for the gurus here.

I'm planning to do HA failover with the raspberry pi's that run piHole/DHCP using keepalived, which uses VRRP for failover between a VIP and 2 actual IPs for each RPi pair.

Is the VRRP implementation as detailed here:
GitHub - matayto/pihole-keepalived: Simple failover configurations for a multi-pihole infrastructure

independent of the VRRP stuff I see in the Brocade manual for our switches? I'm guessing that VRRP is more for router failover and has nothing to do with failover of host devices?

Seems like VRRP is handled via multicast through IP type 112:
Using Keepalived for managing simple failover in clusters | Enable Sysadmin (redhat.com)

So I assume from this, all I need to ensure is that the VRRP packets can get via multicast from one Rpi to the other pair of the HA, which shouldn't be an issue given they will both be in the same subnet (or PIM-Dense will take care of it if I perversely decide to have the pair in different subnets).

Does that sound right?

 

[kevindd992002]

FAQ document would seem to indicate that all ports are POE+, so as long as you don't cross the wattage limit (390 for 24, 780 for 48) then it should be fine. Images of the switch don't appear to have any special markings on the ports; for example, my 7450 has the first 8 ports specially marked to indicate they do PoH/PoE++ as apposed to the other 40 which are PoE+ only.

That's what I thought. It's just that the FAQ is saying 24 ports of PoE+ without the external PSU. I guess it just means 24 ports of full 30W PoE+.