Alright, I've got a weird one and it's driving me absolutely bonkers. This is ALL in the context of IPv6
Problem
I'm having an issue with wireless IPv6 connectivity to the internet. When a wireless client roams from the initial association WAP (access point) it loses IPv6 connectivity to the internet.
If I re-associate (reset wifi on the device) after roaming IPv6 comes back for the new WAP, roaming again will break connectivity to the internet.
Environment
Router
I have an OpnSense router with a direct "transit" connection down to a pair of stacked ICX6610's. I turned off all dynamic routing protocols and I'm using only static routing on the ICX's and the opnSense.
Switches
The ICX 6610's are running the latest firmware:
Copyright (c) Ruckus Networks, Inc. All rights reserved.
UNIT 1: compiled on Aug 8 2023 at 23:06:54 labeled as SPR08095m
(33554432 bytes) from Primary SPR08095m.bin (UFI)
SW: Version 08.0.95mT213
Compressed Primary Boot Code size = 786944, Version:10.1.26T215 (spz10126)
Compiled on Tue Nov 29 04:43:15 2022
UNIT 2: compiled on Aug 8 2023 at 23:06:54 labeled as SPR08095m
(33554432 bytes) from Primary SPR08095m.bin (UFI)
SW: Version 08.0.95mT213
Compressed Primary Boot Code size = 786944, Version:10.1.26T215 (spz10126)
HW: Stackable ICX7250-24-HPOE
The ICX's are running as L3 gateways for their respective VLANs, each vlan is using SLAAC for IPv6 address allocation.
The ICX routing table has the upstream opnSense router as it's default route.
Access Points
I trunk 3 VLANs to three different Aruba InstantOn AP22 access points, the 3 access points are split accross multiple switch ports in the stack.
Debugging Attempts
Removed all dynamic routing protocols and moved to static routes
*after* roaming to a different access point
Ping from OpnSense to wireless device - fails
Ping from hardwired device to wireless device in the same VLAN (ie no routing) - succeeds
Ping from hardwired device to wireless device in a different VLAN (ie routed through 6610) - succeeds
Here is a tcpdump from the perspective of the OpnSense router:
OpnSense: 2001:db8:2d00:24f0::1
Wireless Client: 2001:db8:2d00:2405:207a:305e:39d9:b385
Associated with WAP1
19:57:46.434682 IP6 2001:db8:2d00:2405:207a:305e:39d9:b385 > 2001:db8:2d00:24f0::1: ICMP6, echo reply, seq 994, length 16
19:57:47.427977 IP6 2001:db8:2d00:24f0::1 > 2001:db8:2d00:2405:207a:305e:39d9:b385: ICMP6, echo request, seq 995, length 16
19:57:47.434530 IP6 2001:db8:2d00:2405:207a:305e:39d9:b385 > 2001:db8:2d00:24f0::1: ICMP6, echo reply, seq 995, length 16
19:57:48.430550 IP6 2001:db8:2d00:24f0::1 > 2001:db8:2d00:2405:207a:305e:39d9:b385: ICMP6, echo request, seq 996, length 16
19:57:48.438575 IP6 2001:db8:2d00:2405:207a:305e:39d9:b385 > 2001:db8:2d00:24f0::1: ICMP6, echo reply, seq 996, length 16
19:57:49.433206 IP6 2001:db8:2d00:24f0::1 > 2001:db8:2d00:2405:207a:305e:39d9:b385: ICMP6, echo request, seq 997, length 16
19:57:49.499337 IP6 2001:db8:2d00:2405:207a:305e:39d9:b385 > 2001:db8:2d00:24f0::1: ICMP6, echo reply, seq 997, length 16
Roam to WAP2
19:57:50.437464 IP6 2001:db8:2d00:24f0::1 > 2001:db8:2d00:2405:207a:305e:39d9:b385: ICMP6, echo request, seq 998, length 16
19:57:51.438580 IP6 2001:db8:2d00:24f0::1 > 2001:db8:2d00:2405:207a:305e:39d9:b385: ICMP6, echo request, seq 999, length 16
19:57:52.440642 IP6 2001:db8:2d00:24f0::1 > 2001:db8:2d00:2405:207a:305e:39d9:b385: ICMP6, echo request, seq 1000, length 16
19:57:53.441971 IP6 2001:db8:2d00:24f0::1 > 2001:db8:2d00:2405:207a:305e:39d9:b385: ICMP6, echo request, seq 1001, length 16
19:57:54.443574 IP6 2001:db8:2d00:24f0::1 > 2001:db8:2d00:2405:207a:305e:39d9:b385: ICMP6, echo request, seq 1002, length 16
19:57:55.444589 IP6 2001:db8:2d00:24f0::1 > 2001:db8:2d00:2405:207a:305e:39d9:b385: ICMP6, echo request, seq 1003, length 16
19:57:56.448761 IP6 2001:db8:2d00:24f0::1 > 2001:db8:2d00:2405:207a:305e:39d9:b385: ICMP6, echo request, seq 1004, length 16
19:57:57.450438 IP6 2001:db8:2d00:24f0::1 > 2001:db8:2d00:2405:207a:305e:39d9:b385: ICMP6, echo request, seq 1005, length 16
19:57:58.452506 IP6 2001:db8:2d00:24f0::1 > 2001:db8:2d00:2405:207a:305e:39d9:b385: ICMP6, echo request, seq 1006, length 16
19:57:59.457450 IP6 2001:db8:2d00:24f0::1 > 2001:db8:2d00:2405:207a:305e:39d9:b385: ICMP6, echo request, seq 1007, length 16
19:58:00.458815 IP6 2001:db8:2d00:24f0::1 > 2001:db8:2d00:2405:207a:305e:39d9:b385: ICMP6, echo request, seq 1008, length 16
19:58:01.463074 IP6 2001:db8:2d00:24f0::1 > 2001:db8:2d00:2405:207a:305e:39d9:b385: ICMP6, echo request, seq 1009, length 16
19:58:02.464345 IP6 2001:db8:2d00:24f0::1 > 2001:db8:2d00:2405:207a:305e:39d9:b385: ICMP6, echo request, seq 1010, length 16
19:58:03.465588 IP6 2001:db8:2d00:24f0::1 > 2001:db8:2d00:2405:207a:305e:39d9:b385: ICMP6, echo request, seq 1011, length 16
19:58:04.467961 IP6 2001:db8:2d00:24f0::1 > 2001:db8:2d00:2405:207a:305e:39d9:b385: ICMP6, echo request, seq 1012, length 16
19:58:05.468944 IP6 2001:db8:2d00:24f0::1 > 2001:db8:2d00:2405:207a:305e:39d9:b385: ICMP6, echo request, seq 1013, length 16
19:58:06.470789 IP6 2001:db8:2d00:24f0::1 > 2001:db8:2d00:2405:207a:305e:39d9:b385: ICMP6, echo request, seq 1014, length 16
19:58:07.478637 IP6 2001:db8:2d00:24f0::1 > 2001:db8:2d00:2405:207a:305e:39d9:b385: ICMP6, echo request, seq 1015, length 16
19:58:08.479740 IP6 2001:db8:2d00:24f0::1 > 2001:db8:2d00:2405:207a:305e:39d9:b385: ICMP6, echo request, seq 1016, length 16
19:58:09.484910 IP6 2001:db8:2d00:24f0::1 > 2001:db8:2d00:2405:207a:305e:39d9:b385: ICMP6, echo request, seq 1017, length 16
19:58:10.492031 IP6 2001:db8:2d00:24f0::1 > 2001:db8:2d00:2405:207a:305e:39d9:b385: ICMP6, echo request, seq 1018, length 16
19:58:11.494773 IP6 2001:db8:2d00:24f0::1 > 2001:db8:2d00:2405:207a:305e:39d9:b385: ICMP6, echo request, seq 1019, length 16
19:58:12.502221 IP6 2001:db8:2d00:24f0::1 > 2001:db8:2d00:2405:207a:305e:39d9:b385: ICMP6, echo request, seq 1020, length 16
19:58:13.508098 IP6 2001:db8:2d00:24f0::1 > 2001:db8:2d00:2405:207a:305e:39d9:b385: ICMP6, echo request, seq 1021, length 16
19:58:14.514172 IP6 2001:db8:2d00:24f0::1 > 2001:db8:2d00:2405:207a:305e:39d9:b385: ICMP6, echo request, seq 1022, length 16
19:58:15.516477 IP6 2001:db8:2d00:24f0::1 > 2001:db8:2d00:2405:207a:305e:39d9:b385: ICMP6, echo request, seq 1023, length 16
19:58:16.519838 IP6 2001:db8:2d00:24f0::1 > 2001:db8:2d00:2405:207a:305e:39d9:b385: ICMP6, echo request, seq 1024, length 16
19:58:17.521306 IP6 2001:db8:2d00:24f0::1 > 2001:db8:2d00:2405:207a:305e:39d9:b385: ICMP6, echo request, seq 1025, length 16
19:58:18.523018 IP6 2001:db8:2d00:24f0::1 > 2001:db8:2d00:2405:207a:305e:39d9:b385: ICMP6, echo request, seq 1026, length 16
19:58:19.529879 IP6 2001:db8:2d00:24f0::1 > 2001:db8:2d00:2405:207a:305e:39d9:b385: ICMP6, echo request, seq 1027, length 16
19:58:20.532896 IP6 2001:db8:2d00:24f0::1 > 2001:db8:2d00:2405:207a:305e:39d9:b385: ICMP6, echo request, seq 1028, length 16
19:58:21.534109 IP6 2001:db8:2d00:24f0::1 > 2001:db8:2d00:2405:207a:305e:39d9:b385: ICMP6, echo request, seq 1029, length 16
19:58:22.535387 IP6 2001:db8:2d00:24f0::1 > 2001:db8:2d00:2405:207a:305e:39d9:b385: ICMP6, echo request, seq 1030, length 16
19:58:23.536953 IP6 2001:db8:2d00:24f0::1 > 2001:db8:2d00:2405:207a:305e:39d9:b385: ICMP6, echo request, seq 1031, length 16
19:58:24.537997 IP6 2001:db8:2d00:24f0::1 > 2001:db8:2d00:2405:207a:305e:39d9:b385: ICMP6, echo request, seq 1032, length 16
19:58:25.541886 IP6 2001:db8:2d00:24f0::1 > 2001:db8:2d00:2405:207a:305e:39d9:b385: ICMP6, echo request, seq 1033, length 16
19:58:26.546915 IP6 2001:db8:2d00:24f0::1 > 2001:db8:2d00:2405:207a:305e:39d9:b385: ICMP6, echo request, seq 1034, length 16
19:58:27.555055 IP6 2001:db8:2d00:24f0::1 > 2001:db8:2d00:2405:207a:305e:39d9:b385: ICMP6, echo request, seq 1035, length 16
19:58:28.563192 IP6 2001:db8:2d00:24f0::1 > 2001:db8:2d00:2405:207a:305e:39d9:b385: ICMP6, echo request, seq 1036, length 16
19:58:29.567090 IP6 2001:db8:2d00:24f0::1 > 2001:db8:2d00:2405:207a:305e:39d9:b385: ICMP6, echo request, seq 1037, length 16
19:58:30.568384 IP6 2001:db8:2d00:24f0::1 > 2001:db8:2d00:2405:207a:305e:39d9:b385: ICMP6, echo request, seq 1038, length 16
19:58:31.583843 IP6 2001:db8:2d00:24f0::1 > 2001:db8:2d00:2405:207a:305e:39d9:b385: ICMP6, echo request, seq 1039, length 16
19:58:32.595609 IP6 2001:db8:2d00:24f0::1 > 2001:db8:2d00:2405:207a:305e:39d9:b385: ICMP6, echo request, seq 1040, length 16
19:58:33.604105 IP6 2001:db8:2d00:24f0::1 > 2001:db8:2d00:2405:207a:305e:39d9:b385: ICMP6, echo request, seq 1041, length 16
19:58:34.610626 IP6 2001:db8:2d00:24f0::1 > 2001:db8:2d00:2405:207a:305e:39d9:b385: ICMP6, echo request, seq 1042, length 16
19:58:35.618694 IP6 2001:db8:2d00:24f0::1 > 2001:db8:2d00:2405:207a:305e:39d9:b385: ICMP6, echo request, seq 1043, length 16
19:58:36.622297 IP6 2001:db8:2d00:24f0::1 > 2001:db8:2d00:2405:207a:305e:39d9:b385: ICMP6, echo request, seq 1044, length 16
19:58:37.627183 IP6 2001:db8:2d00:24f0::1 > 2001:db8:2d00:2405:207a:305e:39d9:b385: ICMP6, echo request, seq 1045, length 16
19:58:38.629207 IP6 2001:db8:2d00:24f0::1 > 2001:db8:2d00:2405:207a:305e:39d9:b385: ICMP6, echo request, seq 1046, length 16
19:58:39.630363 IP6 2001:db8:2d00:24f0::1 > 2001:db8:2d00:2405:207a:305e:39d9:b385: ICMP6, echo request, seq 1047, length 16
19:58:40.633129 IP6 2001:db8:2d00:24f0::1 > 2001:db8:2d00:2405:207a:305e:39d9:b385: ICMP6, echo request, seq 1048, length 16
19:58:41.634566 IP6 2001:db8:2d00:24f0::1 > 2001:db8:2d00:2405:207a:305e:39d9:b385: ICMP6, echo request, seq 1049, length 16
19:58:42.636545 IP6 2001:db8:2d00:24f0::1 > 2001:db8:2d00:2405:207a:305e:39d9:b385: ICMP6, echo request, seq 1050, length 16
19:58:43.637942 IP6 2001:db8:2d00:24f0::1 > 2001:db8:2d00:2405:207a:305e:39d9:b385: ICMP6, echo request, seq 1051, length 16
19:58:44.638553 IP6 2001:db8:2d00:24f0::1 > 2001:db8:2d00:2405:207a:305e:39d9:b385: ICMP6, echo request, seq 1052, length 16
19:58:45.641411 IP6 2001:db8:2d00:24f0::1 > 2001:db8:2d00:2405:207a:305e:39d9:b385: ICMP6, echo request, seq 1053, length 16
19:58:46.642567 IP6 2001:db8:2d00:24f0::1 > 2001:db8:2d00:2405:207a:305e:39d9:b385: ICMP6, echo request, seq 1054, length 16
19:58:47.643843 IP6 2001:db8:2d00:24f0::1 > 2001:db8:2d00:2405:207a:305e:39d9:b385: ICMP6, echo request, seq 1055, length 16
19:58:48.645237 IP6 2001:db8:2d00:24f0::1 > 2001:db8:2d00:2405:207a:305e:39d9:b385: ICMP6, echo request, seq 1056, length 16
19:58:49.646044 IP6 2001:db8:2d00:24f0::1 > 2001:db8:2d00:2405:207a:305e:39d9:b385: ICMP6, echo request, seq 1057, length 16
19:58:50.649867 IP6 2001:db8:2d00:24f0::1 > 2001:db8:2d00:2405:207a:305e:39d9:b385: ICMP6, echo request, seq 1058, length 16
19:58:51.650661 IP6 2001:db8:2d00:24f0::1 > 2001:db8:2d00:2405:207a:305e:39d9:b385: ICMP6, echo request, seq 1059, length 16
19:58:52.651403 IP6 2001:db8:2d00:24f0::1 > 2001:db8:2d00:2405:207a:305e:39d9:b385: ICMP6, echo request, seq 1060, length 16
19:58:53.652821 IP6 2001:db8:2d00:24f0::1 > 2001:db8:2d00:2405:207a:305e:39d9:b385: ICMP6, echo request, seq 1061, length 16
19:58:54.653917 IP6 2001:db8:2d00:24f0::1 > 2001:db8:2d00:2405:207a:305e:39d9:b385: ICMP6, echo request, seq 1062, length 16
19:58:55.657144 IP6 2001:db8:2d00:24f0::1 > 2001:db8:2d00:2405:207a:305e:39d9:b385: ICMP6, echo request, seq 1063, length 16
Return to WAP1
19:58:55.665094 IP6 2001:db8:2d00:2405:207a:305e:39d9:b385 > 2001:db8:2d00:24f0::1: ICMP6, echo reply, seq 1063, length 16
19:58:56.657742 IP6 2001:db8:2d00:24f0::1 > 2001:db8:2d00:2405:207a:305e:39d9:b385: ICMP6, echo request, seq 1064, length 16
19:58:56.665141 IP6 2001:db8:2d00:2405:207a:305e:39d9:b385 > 2001:db8:2d00:24f0::1: ICMP6, echo reply, seq 1064, length 16
19:58:57.659884 IP6 2001:db8:2d00:24f0::1 > 2001:db8:2d00:2405:207a:305e:39d9:b385: ICMP6, echo request, seq 1065, length 16
19:58:57.664363 IP6 2001:db8:2d00:2405:207a:305e:39d9:b385 > 2001:db8:2d00:24f0::1: ICMP6, echo reply, seq 1065, length 16
During that entire test, pings from a device on the same VLAN and another device in a different VLAN experienced zero packet loss.
P.S. I'm seeing these issues ONLY with my wireless clients, my hard wired IPv6 client works great.