Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

[kpfleming]

HA failover with the raspberry pi's that run piHole/DHCP using keepalived

Please note that DHCP is not a normal application protocol which can be handled by VRRP or similar techniques.

Clients locate the DHCP server using broadcast messages, which don't even have normal IP addresses in them (they can't since the client doesn't have an address at that point). Once the DHCP transaction has been completed the client will use unicast messages for renewal/release (unless that fails, in which case it will go back to broadcast). if the IP address of the DHCP server is handled via VRRP, the secondary/failover DHCP server will need to have all of the lease state from the primary one in order to be able to respond to these messages properly. It also needs that state in order to ensure that it doesn't hand out duplicate addresses.

HA DHCP is really quite off-topic for this thread (although not this forum!) but it certainly can be done. I've got ISC Kea DHCP running on two boxes on my LAN, using ICX 7150s for traffic, and it works really well. The ICXs are configured with two helper addresses for forwarding DHCP broadcast traffic to both Kea boxes in parallel, and then Kea handles the HA aspects itself.

 

[LodeRunner]

Additional question for the gurus here.

I'm planning to do HA failover with the raspberry pi's that run piHole/DHCP using keepalived, which uses VRRP for failover between a VIP and 2 actual IPs for each RPi pair.

Is the VRRP implementation as detailed here:
GitHub - matayto/pihole-keepalived: Simple failover configurations for a multi-pihole infrastructure

independent of the VRRP stuff I see in the Brocade manual for our switches? I'm guessing that VRRP is more for router failover and has nothing to do with failover of host devices?

Seems like VRRP is handled via multicast through IP type 112:
Using Keepalived for managing simple failover in clusters | Enable Sysadmin (redhat.com)

So I assume from this, all I need to ensure is that the VRRP packets can get via multicast from one Rpi to the other pair of the HA, which shouldn't be an issue given they will both be in the same subnet (or PIM-Dense will take care of it if I perversely decide to have the pair in different subnets).

Does that sound right?

This sounds like complexity for the sake of complexity. DNS is already redundant without VRRP, and ISC DHCP has its own failover support (A Basic Guide to Configuring DHCP Failover). Just setup two PiHoles for DNS only, then separately configure ISC DHCP. Use something like Gravity Sync to keep the PiHole DNS configuration synced between the two.

Then you have two unused rPis to do other things with. Or if you have a virtualization platform or Docker, skip the hardware entirely then you have 4 rPis for other projects.

 

[kpfleming]

DNS is already redundant without VRRP

If you're able to supply both DNS resolver addresses to the clients, this is certainly true.

 

[seatrope]

This sounds like complexity for the sake of complexity. DNS is already redundant without VRRP, and ISC DHCP has its own failover support (A Basic Guide to Configuring DHCP Failover). Just setup two PiHoles for DNS only, then separately configure ISC DHCP. Use something like Gravity Sync to keep the PiHole DNS configuration synced between the two.

Then you have two unused rPis to do other things with. Or if you have a virtualization platform or Docker, skip the hardware entirely then you have 4 rPis for other projects.

@LodeRunner I have pihole and DHCP (dnsmasq) on the same pi. So only 2pi not 4pi lol.

I did look into using ISC DHCP. The downside of ISC DHCP is that pihole does not resolve hostnames automatically if you use ISC DHCP (is what I read). If you use the built in dnsmasq for DHCP it will. I know, first world problems...

 

[seatrope]

If you're able to supply both DNS resolver addresses to the clients, this is certainly true.

@kpfleming My understanding is that if you supply two DNS addresses, if the first is down, it will take some timeout period before it tries the second? So it is not a seamless failover? Better to have a VIP through VRRP that is the single failsafe no?

 

[kpfleming]

That's very much dependent on client behavior, indeed. In my network I use anycast addresses for DNS and NTP, so clients don't have to implement any failover at all. VRRP could provide a similar result.

 

[LodeRunner]

@LodeRunner I have pihole and DHCP (dnsmasq) on the same pi. So only 2pi not 4pi lol.

I did look into using ISC DHCP. The downside of ISC DHCP is that pihole does not resolve hostnames automatically if you use ISC DHCP (is what I read). If you use the built in dnsmasq for DHCP it will. I know, first world problems...

Sorry the way your first post was phrased I misread it as you having more than 1 pair.

As far as DNS client failover time, it’s short. Like 2 seconds for Windows? I practically never notice when rebooting my primary DNS (which is Active Directory that forwards to a pair of Pihole VMs).

Getting Dynamic DNS to work from ISC DHCP into Pihole appears to be an issue, yes. I've seen some people solve it by using some sort of conditional rules in the Pihole. And I guess a Bind server holding the local zone? The Pihole forwards local requests to the Bind server.

The Bind server and DDNS in ISC-DHCP obviously is extra complexity, but I think less so than configuring keepalived or other HA software solution.

 

[jasonwc]

A user on Reddit posted an interesting problem with his ICX6610-24 (non PoE). He said his switch idles at 180-200W and when under load, can hit 400W. I told him this makes no sense given that the spec sheet indicates it only requires a single 250W power supply. The specs say the second power supply is optional, for redundancy. Also, this thread indicates it should idle at 80W or so.

I asked him his OS version and power supply revision. He is on 08.0.10c. His output from “show chassis” is really strange:

Power supply 1 (AC - Regular) present, status ok
Model Number: 00-0000000-00
Serial Number: FFFF
Firmware Ver:
Power supply 1 Fan Air Flow Direction: Front to Back
Power supply 2 (AC - Regular) present, status ok
Model Number: 00-0000000-00
Serial Number: FFFF
Firmware Ver:
Power supply 2 Fan Air Flow Direction: Front to Back

Fan 1 ok, speed (auto): [[1]]<->2
Fan 2 ok, speed (auto): [[1]]<->2

So, all zeroes model number, FFFF serial number, and blank firmware version. He mentioned he has three of these power supplies, marked “S5/S2.”

I then suggested that he start fresh using the guide to upgrade his bootloader and firmware to the latest versions.

He indicated the switch shut down after upgrading the boot loader and now appears bricked.

Based on this information, does the switch appear to be genuine? Why would the power supply show a blank model number with no revision?

 

[kpfleming]

When a device roams from one AP to the other, the switch updates its MAC table (show mac-address vlan 10) with the new port, but the ND table still binds the IP/mac combo to the old port (show ipv6 neighbor ve 10). This makes the IP address unreachable.

For what it's worth, I just experienced a very similar symptom on 09.0.10c firmware. I shut down a machine, moved it a different location temporarily, and connected it via another switch that is connected to the ICX stack. When the machine came up, it was reachable via IPv4 but not IPv6. After I moved it back to its original port on the ICX, it became reachable over both protocols.

I didn't leave it in that condition very long, so I don't know how long it would have taken for the NDP cache to time out.

 

[fohdeesha]

A user on Reddit posted an interesting problem with his ICX6610-24 (non PoE). He said his switch idles at 180-200W and when under load, can hit 400W. I told him this makes no sense given that the spec sheet indicates it only requires a single 250W power supply. The specs say the second power supply is optional, for redundancy. Also, this thread indicates it should idle at 80W or so.

I asked him his OS version and power supply revision. He is on 08.0.10c. His output from “show chassis” is really strange:

Power supply 1 (AC - Regular) present, status ok
Model Number: 00-0000000-00
Serial Number: FFFF
Firmware Ver:
Power supply 1 Fan Air Flow Direction: Front to Back
Power supply 2 (AC - Regular) present, status ok
Model Number: 00-0000000-00
Serial Number: FFFF
Firmware Ver:
Power supply 2 Fan Air Flow Direction: Front to Back

Fan 1 ok, speed (auto): [[1]]<->2
Fan 2 ok, speed (auto): [[1]]<->2

So, all zeroes model number, FFFF serial number, and blank firmware version. He mentioned he has three of these power supplies, marked “S5/S2.”

I then suggested that he start fresh using the guide to upgrade his bootloader and firmware to the latest versions.

He indicated the switch shut down after upgrading the boot loader and now appears bricked.

Based on this information, does the switch appear to be genuine? Why would the power supply show a blank model number with no revision?

these switches were never counterfeited (that I'm aware of), so it's certainly genuine, but it sounds like it had a pretty serious fault to begin with. if a 250w PSU is pulling 400w, I'm assuming the PSU(s) themselves had a pretty bad fault, which could also explain why their manuf data EEPROM couldn't be read (reporting all FFF)

 

[fohdeesha]

A user on Reddit posted an interesting problem with his ICX6610-24 (non PoE). He said his switch idles at 180-200W and when under load, can hit 400W. I told him this makes no sense given that the spec sheet indicates it only requires a single 250W power supply. The specs say the second power supply is optional, for redundancy. Also, this thread indicates it should idle at 80W or so.

I asked him his OS version and power supply revision. He is on 08.0.10c. His output from “show chassis” is really strange:

Power supply 1 (AC - Regular) present, status ok
Model Number: 00-0000000-00
Serial Number: FFFF
Firmware Ver:
Power supply 1 Fan Air Flow Direction: Front to Back
Power supply 2 (AC - Regular) present, status ok
Model Number: 00-0000000-00
Serial Number: FFFF
Firmware Ver:
Power supply 2 Fan Air Flow Direction: Front to Back

Fan 1 ok, speed (auto): [[1]]<->2
Fan 2 ok, speed (auto): [[1]]<->2

So, all zeroes model number, FFFF serial number, and blank firmware version. He mentioned he has three of these power supplies, marked “S5/S2.”

I then suggested that he start fresh using the guide to upgrade his bootloader and firmware to the latest versions.

He indicated the switch shut down after upgrading the boot loader and now appears bricked.

Based on this information, does the switch appear to be genuine? Why would the power supply show a blank model number with no revision?

also, if his switches were really pulling 200w or above like he states, the fans would NOT be on fan speed 1 still as his output indicates - that is an insane amount of heat for these fans to exhaust out of 1ru, they would have ramped up to fan speed 2. How is he measuring power draw?

 

[jasonwc]

also, if his switches were really pulling 200w or above like he states, the fans would NOT be on fan speed 1 still as his output indicates - that is an insane amount of heat for these fans to exhaust out of 1ru, they would have ramped up to fan speed 2. How is he measuring power draw?

I asked him the same question. He said he used this meter -https://www.amazon.com/Digital-Wattmeter-Consumption-Frequency-Electricity/dp/B0828QWQZP

He noted he’s using a 230V Shucko plug (Poland).

It didn’t make sense to me that the switch would really pull that much power so I told him to test by running with a single power supply. He said it wouldn’t boot in that configuration.

 

[jasonwc]

these switches were never counterfeited (that I'm aware of), so it's certainly genuine, but it sounds like it had a pretty serious fault to begin with. if a 250w PSU is pulling 400w, I'm assuming the PSU(s) themselves had a pretty bad fault, which could also explain why their manuf data EEPROM couldn't be read (reporting all FFF)

He says he has three of the 250W power supplies. Could all be defective?

 

[fohdeesha]

He says he has three of the 250W power supplies. Could all be defective?

if he's tried three separate supplies then there's a 99% chance the failure mode is inside the switch itself. especially if it only boots with two power supplies

 

[jasonwc]

if he's tried three separate supplies then there's a 99% chance the failure mode is inside the switch itself. especially if it only boots with two power supplies

So, it sounds like the switch itself is defective, and his best bet is simply to buy a new ICX6610-24. Is there any chance old firmware or a bad configuration could cause these issues? He’s looking at paying over $200 in shipping to buy a new switch from the US due to limited availability in Poland.

 

[RobstarUSA]

Has anyone got RANCID workin with backups for the 6610s? It fails for me when configured as type foundry. I've seen a few other posts like that on the RANCID mailing list but no solution. If anyone has an ieas......(I have a stack of 2)

 

[danb35]

I'm using Oxidized to back up mine, and it's working fine--don't know if information there would help though.

 

[fohdeesha]

So, it sounds like the switch itself is defective, and his best bet is simply to buy a new ICX6610-24. Is there any chance old firmware or a bad configuration could cause these issues? He’s looking at paying over $200 in shipping to buy a new switch from the US due to limited availability in Poland.

if he's tried 3 different PSUs and the switch always shows this behavior I'm not sure what else it could be. no way a config or firmware issue is going to make it draw 3x it's rated power

 

[seatrope]

Please note that DHCP is not a normal application protocol which can be handled by VRRP or similar techniques.

Clients locate the DHCP server using broadcast messages, which don't even have normal IP addresses in them (they can't since the client doesn't have an address at that point). Once the DHCP transaction has been completed the client will use unicast messages for renewal/release (unless that fails, in which case it will go back to broadcast). if the IP address of the DHCP server is handled via VRRP, the secondary/failover DHCP server will need to have all of the lease state from the primary one in order to be able to respond to these messages properly. It also needs that state in order to ensure that it doesn't hand out duplicate addresses.

HA DHCP is really quite off-topic for this thread (although not this forum!) but it certainly can be done. I've got ISC Kea DHCP running on two boxes on my LAN, using ICX 7150s for traffic, and it works really well. The ICXs are configured with two helper addresses for forwarding DHCP broadcast traffic to both Kea boxes in parallel, and then Kea handles the HA aspects itself.

@kpfleming thanks for this valuable nugget. I've been trying to understand the underpinnings of this, and went and learned a bit more about how DHCP is performed. As DHCPREQUEST is being broadcast (I assume across only the level 2 subnet and not further unless we enable some form of multicast forwarding) - will VRRP with keepalived work IF:
- the pair of HA (active-passive, not active-active) raspi's are on a separate subnet containing only those two hosts; and
- iphelper is configured in the ICX VLANs to point to the VIP in that VLAN; and
- no multicast passthrough to that "DHCP server" VLAN is active?

If so, the DHCPREQUEST broadcast should only be directed by the 6610 to the iphelper addess which should be the VIP (and not the specific IPs of the primary or secondary DHCP servers?

Thoughts before I go down this rabbit hole? I started configuring isc-dhcp then got sucked back into thinking about this. I really like having everything under the pihole interface for ease of use

 

[kpfleming]

Yes, that configuration should work, although you will need to be certain to configure the DHCP server instance to use the VIP as their identity address, so that unicast replies from them will use the proper source address.