...like STUCK stuck.
So, I'm not exactly a beginner to the world of networking, switching and VLAN's. Been playing with this stuff successfully since the late 2000's. RouterOS is - however - a different beast. Where most networking products I've used (ProCurve, HP, Aruba, ESXi, Linux, pfSense, OPNSense, Proxmox, KVM, LXC, Unifi, and even Mikrotik switches under SwOS etc.) were fairly straight forward and intuitive, holy shit, RouterOS is a different beast. It may be powerful, but all that power comes at a significant cost of complexity.
While I have 5 Mikrotik switches in the house (CRS326-24S+2Q+, CRS317-1G-16S+, two CSS326-24G-2S+ and a little CRS305-1G-4S+) I always opted to just use these in SwOS mode, as all I was doing was some layer 2 switching with VLAN's, no need for routing, so in retrospect maybe I should not have taken on my first RouterOS project and my first Mikrotik Wifi experience at the same time, but it is what it is. Trial by fire I guess.
Background (optional reading):
I have used Unifi wireless Access points since ~2010. In that time Ubiquiti Networks has morphed from a cool startup with really neat implementations that were upsetting the industry and were great for "pro-sumers" like myself, to being just another evil Enterprise vendor trying to twist users arms into doing things their way only, pushing more and more cloud nonsense, etc. I started getting annoyed when they decided to not just be an access point company, but also push their routers and switches, and integrate everything into the same controller, and show big red failure notifications when it couldn't find the Unifi routing and switching hardware (because I was using my own router and switches). The last straw came a few years ago when at the same time as they were having a highly publicized attack on their cloud services, they decided to discontinue the local server version of their Unifi Video product and force users onto their cloud.
I decided then and there I was going to get off of Unifi products as soon as I can, before they decided to force my local Unifi controller onto the cloud as well. I just didn't get around to it until now, because quite frankly, the 802.11ac WAP's I was using continued to work fine, and I really didn't have the need for more wireless bandwidth than that, as I am really more of a wired guy, and prefer all things wired.
But I finally got around to the "getting rid of Unifi AP's" project, in part because the Linux Unifi controller install I had broke and went into MongoDB dependency hell preventing me from updating it without breaking the controller, when the recent xz utils vulnerability came out, forcing me to either patch and break the controller spending hours trying to fix it again, or just make now the time I finally move away from Unifi. I chose the latter.
Since I already had (and liked) a bunch of Mikrotik switches, I decided to give Mikrotik AP's a try this time around and ordered two CAP ax units.
The Project:
In the current configuration I had three Unifi UAP-AC-LR WAP's. I didn't need that many (used to live in a larger house where I needed them for coverage) so I decided to get two Mikrotik CAP ax WAP's and just put one on each side of the house, as drop in replacements for two of the Unifi AP's (the third is unnecessary)
the VLAN trunks were already set up and working on the Unifi WAP's, so I figured it would just be a drop in replacement and off to the races.
I did decide that I kind of liked the "manage all of my Wifi from a central location" experience, and since I didn't want asymmetrical configuration with CAPSMAN running on one of the CAP's but not on the other, I just decided to install CHR in a VM on my KVM box, replacing the Unifi controller, and serving as a simple central controller for the two AP's.
When I went to set these up, I found out the hard way that by default these access points are set up for routing (which I find really weird, as I've never once wanted to have a WIFI access point do any routing, but whatever)
I already have a custom built 2U Xeon server running OPNSense for 10gig routing and firewall services which I am able to push gigabit speeds over WireGuard with, so I kind of wanted to keep it as the main router, and not worry about anything else.
Just like the existing Unifi setup the Wireless network needs to have three SSID's:
1.) Main LAN (VLAN 1)
2.) Untrusted IoT (VLAN 22)
3.) Guest (VLAN 5)
Initial Setup:
For this step, I relied on the Mikrotik Youtube video for CAPSMAN here:
Things that are known to work are the VLAN's on the switches and OPNSense router, as they worked just fine in the previous configuration with all the VLAN's already configured.
Here is a high level diagram (with non-relevant servers and clients omitted)
Before plugging in the CAP ax units for the first time, I had already configured my CHR VM install to work in bridge mode, assigned it a static IP and enabled CAPSMAN in it.
I initially struggled with getting the CHR install CAPSMAN to find and be able to control the two CAP ax units because I wasn't expecting an AP to be set up as a Router, with Firewall and NAT enabled.
Once I got past that hurdle though, I was able to log into the devices, and on each of them:
1.) Update to latest RouterOS (7.14.2)
2.) Disable NAT and all services (DHCP, etc.)
3.) Expand the default bridge1 (encompassing the two wireless devices, wifi1 and wifi2 and ether2) to include ether1 as well, as this is the port I need to use without NAT as it is the port that supports incoming PoE.
4.) Change IP to static and set it to an IP matching my Main LAN range (10.0.1.0/24)
5.) Enable CAP, and point it towards the IP for my CHR Capsman
6.) configure each of the wireless devices to be managed by CAPSMAN, and set the datapath to use bridge1
Success. The four wireless devices now showed up in CAPSMAN. Time to start configuring SSID's.
There are a lot more configuration options than I am used to for wifi. I decided to start with a rather basic configuration to get things up and running quickly, figuring I can always fine tune things later. No one likes downtime. (especially my untrusted wifi thermostats while we are still in heating season)
First created a security profile named "Main". It is configured with WPA2 PSK, and WPA3 PSKm with CCMP, GCMP, CCMP 256 and GCMP 256 enabled. Additionally a passphrase has been assigned.
Secondly I created a configuration profile for the main LAN, also named "Main", and use my previously configured security provfile. Set it to "ap" mode, assigned an SSID, and did not assign a VLAN. I also assigned the country to the United States. (Strangely enough, the configuration keeps forgetting my country setting. Not sure why, or if it is important)
I then set all four of the wifi devices to use the "main" configuration profile.
And it worked. The SSID appeared immediately. Well.. Sortof. My Android phone is able to see and log into the SSDI's. As is my Linux laptop. The same laptop when dual booted to Windows 10, does not see the Mikrotik SSID's at all for some reason. At first I thought maybe it was because the aging Intel Wireless-AC 7260 Mini-PCie WLAN card couldn't handle WPA3, but then why did it work in Linux? And I also have WPA2 enabled. It also shouldn't be a cipher problem, as I have everything except TKIP enabled (which I will never enable) so it should be working. Either way, I digress. Might be an outdated driver issue or something. It's not a big deal, I can troubleshoot that later.
The Problem
This is where I was starting to feel pretty confident. Things were working! (And damn are these Mikrotik CAP ax units fast compared to the old Unifi AC LR units I used to use.)
Part of the issue I think I am having is that most guides/posts I find when googling are for the old CAPSMAN used with 802.11ac WAPs, not the new CAPSMAN for ax devices.
Anyway, I started clicking around looking for a way to add my second and third SSID's (to be configured with VLANs 22 and 5) and I got stuck.
Some googling later, and it turns out that in order to have multiple SSID's on the same radio, I have to create slave devices subordinate to the main wireless devices. So in the CHR management interface, I created a slave for each of the four wifi devices.
Then I created a second security profile (same authentication/cipher options as previously known to work in the "main" profile, and a second configuration profile using that security profile, a different SSID and also adding VLAN 22 to datapath settings, and assigned it to the slaves.
I got it partially right, because almost immediately I saw my pesky untrusted thermostats show up under registration. This is were things broke down though. The DHCP server on my OPNSense router (set to VLAN 22 and previously working with Unifi on this VLAN) never sees the two thermostats. I also cannot log in to the SSID for VLAN22 from my laptop in Linux. While the main LAN works, it just tries and tries and tries and then fails.
Things I have tried:
1.) Enabling VLAN filtering on the bridges set up on the CAP's to make sure they can transfer the VLAN data from the wifi devices out to the switches. I configured them as follows: EtherType 0x8100, PVID 1 (matching main LAN). Frame Types admit all, Ingress Filtering, Yes. This appears to ahve had no effect. It doesn't help VLAN22 get through to the upstream CSS326 switch, but it also doesn't appear to break main lan traffic, despite it not being tagged with VLAN 1.
2.) I tried changing the configuration profile on the main LAN to tag it with VLAN 1, and that just resulted in the main LAN wifi becoming unusable. So VLAN traffic is just not getting through to the upstream CSS326 switches.
3.) I tried creating VLAN22 devices on the CAPs linked to the bridge, but this doesn't seem to have done anyhting (and I am not even sure it is supposed to be necessary, unless I want to assign the CAPs an IP on the VLAN for DHCP or something like that, which I don't.)
4.) While it shouldn't be necessary (as no wifi traffic hits the CHR, as it only communicates configuration data to the CAPS over the main LAN) I also did #2 and #3 above to the CHR install as well, with no effect.
5.) When setting up the second SSID's I was a little bit during the configuration as notably in the configuration in CAPSMAN, in the dropdowns under datapath, the only bridge I see is the bridge configured on the CHR VM, not the bridges on the CAP devices which the wifi devices actually need to use to reach the CSS326 switch. This seems like a potential issue. To test tthis I ran some high bandwidth activity over wifi, and looked for traffic activity and found that while I observed high traffic on the wifi devices during my test, I saw no traffic at all on the configured bridge, which is very strange, as the traffic would have HAD TO cross the bridge to reach the switch. I have no idea what is going on there, but suspect this may be a contributing cause for my issues.
6.) I did some more googling, and reading of the Mikrotik Wiki. I came across the "Layer2 Misconfiguration" page, and notably the subsection for "VLAN Interface on a Slave Device"
Essentially what this article says is that a slave device cannot carry VLAN, as that needs to be on the master.
I can't for the life of me figure out how to fix it though.
As mentioned above, I tried setting VLAN1 on the master Wifi device, and that just resulted in not being able to communicate with the upstream CSS326 switch.
It would seem (to me, unless I got something wrong) that the only way to configure additional SSID's is by using slave devices, but if slave devices can't assign VLAN's, then how is this supposed to work?
The most likely reason you'd want additional SSID's is if you are going to use VLAN's on them, and that would necessitate slave devices.
I am thus thoroughly confused about the RouterOS way of doing things, and would greatly appreciate anyone who could give me a few suggestions.
I appreciate anyone who would take the time to read through everything I have written. You are community forum heroes.
Thanks again, and appreciate any input at all!
So, I'm not exactly a beginner to the world of networking, switching and VLAN's. Been playing with this stuff successfully since the late 2000's. RouterOS is - however - a different beast. Where most networking products I've used (ProCurve, HP, Aruba, ESXi, Linux, pfSense, OPNSense, Proxmox, KVM, LXC, Unifi, and even Mikrotik switches under SwOS etc.) were fairly straight forward and intuitive, holy shit, RouterOS is a different beast. It may be powerful, but all that power comes at a significant cost of complexity.
While I have 5 Mikrotik switches in the house (CRS326-24S+2Q+, CRS317-1G-16S+, two CSS326-24G-2S+ and a little CRS305-1G-4S+) I always opted to just use these in SwOS mode, as all I was doing was some layer 2 switching with VLAN's, no need for routing, so in retrospect maybe I should not have taken on my first RouterOS project and my first Mikrotik Wifi experience at the same time, but it is what it is. Trial by fire I guess.
Background (optional reading):
I have used Unifi wireless Access points since ~2010. In that time Ubiquiti Networks has morphed from a cool startup with really neat implementations that were upsetting the industry and were great for "pro-sumers" like myself, to being just another evil Enterprise vendor trying to twist users arms into doing things their way only, pushing more and more cloud nonsense, etc. I started getting annoyed when they decided to not just be an access point company, but also push their routers and switches, and integrate everything into the same controller, and show big red failure notifications when it couldn't find the Unifi routing and switching hardware (because I was using my own router and switches). The last straw came a few years ago when at the same time as they were having a highly publicized attack on their cloud services, they decided to discontinue the local server version of their Unifi Video product and force users onto their cloud.
I decided then and there I was going to get off of Unifi products as soon as I can, before they decided to force my local Unifi controller onto the cloud as well. I just didn't get around to it until now, because quite frankly, the 802.11ac WAP's I was using continued to work fine, and I really didn't have the need for more wireless bandwidth than that, as I am really more of a wired guy, and prefer all things wired.
But I finally got around to the "getting rid of Unifi AP's" project, in part because the Linux Unifi controller install I had broke and went into MongoDB dependency hell preventing me from updating it without breaking the controller, when the recent xz utils vulnerability came out, forcing me to either patch and break the controller spending hours trying to fix it again, or just make now the time I finally move away from Unifi. I chose the latter.
Since I already had (and liked) a bunch of Mikrotik switches, I decided to give Mikrotik AP's a try this time around and ordered two CAP ax units.
The Project:
In the current configuration I had three Unifi UAP-AC-LR WAP's. I didn't need that many (used to live in a larger house where I needed them for coverage) so I decided to get two Mikrotik CAP ax WAP's and just put one on each side of the house, as drop in replacements for two of the Unifi AP's (the third is unnecessary)
the VLAN trunks were already set up and working on the Unifi WAP's, so I figured it would just be a drop in replacement and off to the races.
I did decide that I kind of liked the "manage all of my Wifi from a central location" experience, and since I didn't want asymmetrical configuration with CAPSMAN running on one of the CAP's but not on the other, I just decided to install CHR in a VM on my KVM box, replacing the Unifi controller, and serving as a simple central controller for the two AP's.
When I went to set these up, I found out the hard way that by default these access points are set up for routing (which I find really weird, as I've never once wanted to have a WIFI access point do any routing, but whatever)
I already have a custom built 2U Xeon server running OPNSense for 10gig routing and firewall services which I am able to push gigabit speeds over WireGuard with, so I kind of wanted to keep it as the main router, and not worry about anything else.
Just like the existing Unifi setup the Wireless network needs to have three SSID's:
1.) Main LAN (VLAN 1)
2.) Untrusted IoT (VLAN 22)
3.) Guest (VLAN 5)
Initial Setup:
For this step, I relied on the Mikrotik Youtube video for CAPSMAN here:
Things that are known to work are the VLAN's on the switches and OPNSense router, as they worked just fine in the previous configuration with all the VLAN's already configured.
Here is a high level diagram (with non-relevant servers and clients omitted)
Before plugging in the CAP ax units for the first time, I had already configured my CHR VM install to work in bridge mode, assigned it a static IP and enabled CAPSMAN in it.
I initially struggled with getting the CHR install CAPSMAN to find and be able to control the two CAP ax units because I wasn't expecting an AP to be set up as a Router, with Firewall and NAT enabled.
Once I got past that hurdle though, I was able to log into the devices, and on each of them:
1.) Update to latest RouterOS (7.14.2)
2.) Disable NAT and all services (DHCP, etc.)
3.) Expand the default bridge1 (encompassing the two wireless devices, wifi1 and wifi2 and ether2) to include ether1 as well, as this is the port I need to use without NAT as it is the port that supports incoming PoE.
4.) Change IP to static and set it to an IP matching my Main LAN range (10.0.1.0/24)
5.) Enable CAP, and point it towards the IP for my CHR Capsman
6.) configure each of the wireless devices to be managed by CAPSMAN, and set the datapath to use bridge1
Success. The four wireless devices now showed up in CAPSMAN. Time to start configuring SSID's.
There are a lot more configuration options than I am used to for wifi. I decided to start with a rather basic configuration to get things up and running quickly, figuring I can always fine tune things later. No one likes downtime. (especially my untrusted wifi thermostats while we are still in heating season)
First created a security profile named "Main". It is configured with WPA2 PSK, and WPA3 PSKm with CCMP, GCMP, CCMP 256 and GCMP 256 enabled. Additionally a passphrase has been assigned.
Secondly I created a configuration profile for the main LAN, also named "Main", and use my previously configured security provfile. Set it to "ap" mode, assigned an SSID, and did not assign a VLAN. I also assigned the country to the United States. (Strangely enough, the configuration keeps forgetting my country setting. Not sure why, or if it is important)
I then set all four of the wifi devices to use the "main" configuration profile.
And it worked. The SSID appeared immediately. Well.. Sortof. My Android phone is able to see and log into the SSDI's. As is my Linux laptop. The same laptop when dual booted to Windows 10, does not see the Mikrotik SSID's at all for some reason. At first I thought maybe it was because the aging Intel Wireless-AC 7260 Mini-PCie WLAN card couldn't handle WPA3, but then why did it work in Linux? And I also have WPA2 enabled. It also shouldn't be a cipher problem, as I have everything except TKIP enabled (which I will never enable) so it should be working. Either way, I digress. Might be an outdated driver issue or something. It's not a big deal, I can troubleshoot that later.
The Problem
This is where I was starting to feel pretty confident. Things were working! (And damn are these Mikrotik CAP ax units fast compared to the old Unifi AC LR units I used to use.)
Part of the issue I think I am having is that most guides/posts I find when googling are for the old CAPSMAN used with 802.11ac WAPs, not the new CAPSMAN for ax devices.
Anyway, I started clicking around looking for a way to add my second and third SSID's (to be configured with VLANs 22 and 5) and I got stuck.
Some googling later, and it turns out that in order to have multiple SSID's on the same radio, I have to create slave devices subordinate to the main wireless devices. So in the CHR management interface, I created a slave for each of the four wifi devices.
Then I created a second security profile (same authentication/cipher options as previously known to work in the "main" profile, and a second configuration profile using that security profile, a different SSID and also adding VLAN 22 to datapath settings, and assigned it to the slaves.
I got it partially right, because almost immediately I saw my pesky untrusted thermostats show up under registration. This is were things broke down though. The DHCP server on my OPNSense router (set to VLAN 22 and previously working with Unifi on this VLAN) never sees the two thermostats. I also cannot log in to the SSID for VLAN22 from my laptop in Linux. While the main LAN works, it just tries and tries and tries and then fails.
Things I have tried:
1.) Enabling VLAN filtering on the bridges set up on the CAP's to make sure they can transfer the VLAN data from the wifi devices out to the switches. I configured them as follows: EtherType 0x8100, PVID 1 (matching main LAN). Frame Types admit all, Ingress Filtering, Yes. This appears to ahve had no effect. It doesn't help VLAN22 get through to the upstream CSS326 switch, but it also doesn't appear to break main lan traffic, despite it not being tagged with VLAN 1.
2.) I tried changing the configuration profile on the main LAN to tag it with VLAN 1, and that just resulted in the main LAN wifi becoming unusable. So VLAN traffic is just not getting through to the upstream CSS326 switches.
3.) I tried creating VLAN22 devices on the CAPs linked to the bridge, but this doesn't seem to have done anyhting (and I am not even sure it is supposed to be necessary, unless I want to assign the CAPs an IP on the VLAN for DHCP or something like that, which I don't.)
4.) While it shouldn't be necessary (as no wifi traffic hits the CHR, as it only communicates configuration data to the CAPS over the main LAN) I also did #2 and #3 above to the CHR install as well, with no effect.
5.) When setting up the second SSID's I was a little bit during the configuration as notably in the configuration in CAPSMAN, in the dropdowns under datapath, the only bridge I see is the bridge configured on the CHR VM, not the bridges on the CAP devices which the wifi devices actually need to use to reach the CSS326 switch. This seems like a potential issue. To test tthis I ran some high bandwidth activity over wifi, and looked for traffic activity and found that while I observed high traffic on the wifi devices during my test, I saw no traffic at all on the configured bridge, which is very strange, as the traffic would have HAD TO cross the bridge to reach the switch. I have no idea what is going on there, but suspect this may be a contributing cause for my issues.
6.) I did some more googling, and reading of the Mikrotik Wiki. I came across the "Layer2 Misconfiguration" page, and notably the subsection for "VLAN Interface on a Slave Device"
Essentially what this article says is that a slave device cannot carry VLAN, as that needs to be on the master.
I can't for the life of me figure out how to fix it though.
As mentioned above, I tried setting VLAN1 on the master Wifi device, and that just resulted in not being able to communicate with the upstream CSS326 switch.
It would seem (to me, unless I got something wrong) that the only way to configure additional SSID's is by using slave devices, but if slave devices can't assign VLAN's, then how is this supposed to work?
The most likely reason you'd want additional SSID's is if you are going to use VLAN's on them, and that would necessitate slave devices.
I am thus thoroughly confused about the RouterOS way of doing things, and would greatly appreciate anyone who could give me a few suggestions.
I appreciate anyone who would take the time to read through everything I have written. You are community forum heroes.
Thanks again, and appreciate any input at all!