Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

[fohdeesha]

also pushed a fix for the cases where some users with icx7xxx series experienced it rebooting into an old bootloader (which then couldn't boot the new OS) even though they just flashed a new one. The issue was caused by the fact that the bootloader flashing command only flashes the currently active bootloader slot, for these users that would be "slot secondary". The guide then specified using "boot_primary" to boot into the OS - however this bootloader command turned out to also be specifying which bootloader slot to boot from as well (not just OS slot), so for these users who had just updated the secondary boot slot, booting from the primary boot slot would be an old bootloader. using reset instead of boot_primary always reboots using the currently active boot slot (the one we just updated), then once fastiron fully boots, the UEFI image will update all boot slots

fix icx7xxx booting into wrong bootloader · Fohdeesha/lab-docu@b96f01c

fix for users whose switches were booting into old bootloaders, "reset" will always reboot using the current bootloader (the new one we just updated)

github.com

 

[kimbo]

see example command: ICX6xxx Advanced - Fohdeesha Docs

Thank you for the link!

Everything is reporting as normal (see below), but the Tx Bias seems high to me. Is there anyway to see what the normal range is? I'm using LC UPC OS2 fibre if that makes a difference.

 Port  Temperature   Tx Power     Rx Power       Tx Bias Current
+----+-----------+--------------+--------------+---------------+
1/2/3   30.0703 C  -001.5187 dBm -005.5067 dBm   42.718 mA
        Normal      Normal        Normal         Normal

 

[pr09]

ICX 7250 / 08.0.90k / IPv6 neighbor discovery roaming

I have it configured as an L3 switch (ve interfaces, static routes). There are two access points plugged into different ports, both serving the same SSID/VLAN. When a device roams from one AP to the other, the switch updates its MAC table (show mac-address vlan 10) with the new port, but the ND table still binds the IP/mac combo to the old port (show ipv6 neighbor ve 10). This makes the IP address unreachable.

Has anyone else encountered this bug? Do newer software releases fix this (without adding too many other IPv6-related bugs)? Is there a command to work around this? Or do I need to just stop trying to use L3 functionality on this switch?

 

[kevindd992002]

Based on the icx6450 documentation, without the external psu it only has 24 poe+ capable ports and 48 poe capable ports. Does that mean what it says it means? Only 24 ports can be used as poe+ even if the power budget is not exceeded?

@fohdeesha any ideas? Can I use more than 24 ports as PoE+ as long as the total power budget of 780W is not exceeded? At least that's what I know about PoE/PoE+ is.

 

[kpfleming]

Do newer software releases fix this (without adding too many other IPv6-related bugs)?

I had lots of strange IPv6 issues using 08.xx firmware, so I switched to 09.xx as soon as it was released (against advice in this thread, but I knew how to revert if I needed to do so). I haven't had any new problems with 09.xx, and those IPv6 issues went away.

If you don't want to go that far, moving up 08.0.95<current> would be an improvement over what you have now.

 

[juju]

I am having connectivity problems hooking up my ICX-7250 to a Dell Poweredge R740 server . The Dell has an Intel X520/I350 daughter card. My connection uses a DAC cable from one of the 10G ports on the Intel card to one 10G port on the ICX-7250.
My connection is very instable , with frequent loss of connectivity. Not sure how to troubleshoot this - change the DAC cable? I previously had stp setup on the brocade, so I turned it off because I thought the Dell Poweredge was bringing down the network with a broadcast storm. The network was rock solid until I included the Dell server. What should I be looking for?

 

[fohdeesha]

ICX 7250 / 08.0.90k / IPv6 neighbor discovery roaming

I have it configured as an L3 switch (ve interfaces, static routes). There are two access points plugged into different ports, both serving the same SSID/VLAN. When a device roams from one AP to the other, the switch updates its MAC table (show mac-address vlan 10) with the new port, but the ND table still binds the IP/mac combo to the old port (show ipv6 neighbor ve 10). This makes the IP address unreachable.

Has anyone else encountered this bug? Do newer software releases fix this (without adding too many other IPv6-related bugs)? Is there a command to work around this? Or do I need to just stop trying to use L3 functionality on this switch?

this was always an issue for me on the icx6xxx (v8030) series, never tried on the 7 series - I would at least update to the recommended stable from ruckus (8095g, it's what's on the guide in this thread), if that still doesn't fix it you can try the absolute latest (09.0.10c)

 

[pr09]

this was always an issue for me on the icx6xxx (v8030) series, never tried on the 7 series - I would at least update to the recommended stable from ruckus (8095g, it's what's on the guide in this thread), if that still doesn't fix it you can try the absolute latest (09.0.10c)

8095g seems to fix this bug and I haven't found other breakage yet.

(I tried 09 series a while back, ran into weirdness with multicast & host-to-host neighbor discovery, and it didn't *seem* to be fixed by going back to 8095, but 8090 did. But now 8095 is working. I don't know anymore...)

 

[fohdeesha]

8095g seems to fix this bug and I haven't found other breakage yet.

(I tried 09 series a while back, ran into weirdness with multicast & host-to-host neighbor discovery, and it didn't *seem* to be fixed by going back to 8095, but 8090 did. But now 8095 is working. I don't know anymore...)

like any new train 8095 was rocky for a while, but they finally hit "recommended" with F or G rev if I remember right. It's what I've been running in production in a few different DCs and haven't found any showstoppers yet

 

[Vesalius]

8095g seems to fix this bug and I haven't found other breakage yet.

(I tried 09 series a while back, ran into weirdness with multicast & host-to-host neighbor discovery, and it didn't *seem* to be fixed by going back to 8095, but 8090 did. But now 8095 is working. I don't know anymore...)

Multicast discovery is weird so far, at least with my combination of HomeKit, and Proxmox VM’s

 

[kevindd992002]

@fohdeesha any ideas? Can I use more than 24 ports as PoE+ as long as the total power budget of 780W is not exceeded? At least that's what I know about PoE/PoE+ is.

Anyone has any ideas about this question?

 

[LodeRunner]

Anyone has any ideas about this question?

FAQ document would seem to indicate that all ports are POE+, so as long as you don't cross the wattage limit (390 for 24, 780 for 48) then it should be fine. Images of the switch don't appear to have any special markings on the ports; for example, my 7450 has the first 8 ports specially marked to indicate they do PoH/PoE++ as apposed to the other 40 which are PoE+ only.

 

[seatrope]

i was trying to get my Sonos devices working with controller and Sonos devices in different VLANs. this is on a ICX-6610 with interVLAN routing done on the switch, and pfsense for FW only. There are a bunch of guide floating out there which are not specific for the Brocade.

at first I tried using a Rpi with interfaces in both VLAN, using udp-broadcast-relay:

GitHub - udp-redux/udp-broadcast-relay-redux: Small daemon to relay udp broadcast / multicast packages on a different subnet.

Small daemon to relay udp broadcast / multicast packages on a different subnet. - udp-redux/udp-broadcast-relay-redux

github.com

this worked, but I wanted a native solution on the switch. After watching some Terry Henry videos, the solution was so ridiculously simple that I was kicking myself. I’m sure most here know this, but sharing for other noobs like myself

1. At the config level: router pim
2. For each ve that you need multicasts to be bridged, enter each vif and: ip pim

Sonos now works perfectly across VLANs. Apply ACLS as needed for security.

Terry Henry video: https://www.google.com/url?sa=t&rct...=PNsIbdXqHlI&usg=AOvVaw3W1AYaZ1JbclyuZ0jqjiOV

 

[Vesalius]

@seatrope glad that relatively simple fix worked for you. Let us know if you run into any issues going forward. Multicast at home on the ruckus hasn’t been the easiest to figure out for a native solution. Looks like pim might work well for those at layer 3 with established intervlan routing set-up.

I might give the new Multicast VLAN Registration (MVR) module a try on the 0.9 series firmware.

 

[seatrope]

Additional question for the gurus here.

I'm planning to do HA failover with the raspberry pi's that run piHole/DHCP using keepalived, which uses VRRP for failover between a VIP and 2 actual IPs for each RPi pair.

Is the VRRP implementation as detailed here:
GitHub - matayto/pihole-keepalived: Simple failover configurations for a multi-pihole infrastructure

independent of the VRRP stuff I see in the Brocade manual for our switches? I'm guessing that VRRP is more for router failover and has nothing to do with failover of host devices?

Seems like VRRP is handled via multicast through IP type 112:
Using Keepalived for managing simple failover in clusters | Enable Sysadmin (redhat.com)

So I assume from this, all I need to ensure is that the VRRP packets can get via multicast from one Rpi to the other pair of the HA, which shouldn't be an issue given they will both be in the same subnet (or PIM-Dense will take care of it if I perversely decide to have the pair in different subnets).

Does that sound right?

 

[kevindd992002]

FAQ document would seem to indicate that all ports are POE+, so as long as you don't cross the wattage limit (390 for 24, 780 for 48) then it should be fine. Images of the switch don't appear to have any special markings on the ports; for example, my 7450 has the first 8 ports specially marked to indicate they do PoH/PoE++ as apposed to the other 40 which are PoE+ only.

That's what I thought. It's just that the FAQ is saying 24 ports of PoE+ without the external PSU. I guess it just means 24 ports of full 30W PoE+.

 

[kpfleming]

HA failover with the raspberry pi's that run piHole/DHCP using keepalived

Please note that DHCP is not a normal application protocol which can be handled by VRRP or similar techniques.

Clients locate the DHCP server using broadcast messages, which don't even have normal IP addresses in them (they can't since the client doesn't have an address at that point). Once the DHCP transaction has been completed the client will use unicast messages for renewal/release (unless that fails, in which case it will go back to broadcast). if the IP address of the DHCP server is handled via VRRP, the secondary/failover DHCP server will need to have all of the lease state from the primary one in order to be able to respond to these messages properly. It also needs that state in order to ensure that it doesn't hand out duplicate addresses.

HA DHCP is really quite off-topic for this thread (although not this forum!) but it certainly can be done. I've got ISC Kea DHCP running on two boxes on my LAN, using ICX 7150s for traffic, and it works really well. The ICXs are configured with two helper addresses for forwarding DHCP broadcast traffic to both Kea boxes in parallel, and then Kea handles the HA aspects itself.

 

[LodeRunner]

Additional question for the gurus here.

I'm planning to do HA failover with the raspberry pi's that run piHole/DHCP using keepalived, which uses VRRP for failover between a VIP and 2 actual IPs for each RPi pair.

Is the VRRP implementation as detailed here:
GitHub - matayto/pihole-keepalived: Simple failover configurations for a multi-pihole infrastructure

independent of the VRRP stuff I see in the Brocade manual for our switches? I'm guessing that VRRP is more for router failover and has nothing to do with failover of host devices?

Seems like VRRP is handled via multicast through IP type 112:
Using Keepalived for managing simple failover in clusters | Enable Sysadmin (redhat.com)

So I assume from this, all I need to ensure is that the VRRP packets can get via multicast from one Rpi to the other pair of the HA, which shouldn't be an issue given they will both be in the same subnet (or PIM-Dense will take care of it if I perversely decide to have the pair in different subnets).

Does that sound right?

This sounds like complexity for the sake of complexity. DNS is already redundant without VRRP, and ISC DHCP has its own failover support (A Basic Guide to Configuring DHCP Failover). Just setup two PiHoles for DNS only, then separately configure ISC DHCP. Use something like Gravity Sync to keep the PiHole DNS configuration synced between the two.

Then you have two unused rPis to do other things with. Or if you have a virtualization platform or Docker, skip the hardware entirely then you have 4 rPis for other projects.

 

[kpfleming]

DNS is already redundant without VRRP

If you're able to supply both DNS resolver addresses to the clients, this is certainly true.

 

[seatrope]

This sounds like complexity for the sake of complexity. DNS is already redundant without VRRP, and ISC DHCP has its own failover support (A Basic Guide to Configuring DHCP Failover). Just setup two PiHoles for DNS only, then separately configure ISC DHCP. Use something like Gravity Sync to keep the PiHole DNS configuration synced between the two.

Then you have two unused rPis to do other things with. Or if you have a virtualization platform or Docker, skip the hardware entirely then you have 4 rPis for other projects.

@LodeRunner I have pihole and DHCP (dnsmasq) on the same pi. So only 2pi not 4pi lol.

I did look into using ISC DHCP. The downside of ISC DHCP is that pihole does not resolve hostnames automatically if you use ISC DHCP (is what I read). If you use the built in dnsmasq for DHCP it will. I know, first world problems...