What would you recommend I do about a congested network segment?

AveryFreeman

consummate homelabber
Mar 17, 2017
254
23
18
39
Near Seattle
averyfreeman.com
Hey,

I manage a network at an outdoor restaurant we have a single subnet with a ton of devices. We have occasional gremlins and I am trying to figure out why. It's not terrible, but there's issues with reliability now and then - mostly latency issues (I think) which usually get noticed when trying to do important things like process credit cards.

I have a few questions about what we can do to improve performance.

The owner is a vehement DIY-er so he's unlikely to hire someone who actually trained for what I'm doing. I am not a trained network technician, I have just picked up what I know from doing my own projects in homelab and installing equipment for end users. I'm not terrible, but there are some definite holes in my knowledge am hoping other people would help me with. One is VLANs, the other is setting up additional NAT within the LAN for additional subnets.

I have a feeling one, the other, or both might help with the restaurant traffic, but I'm not entirely sure, so I thought I'd ask you nice people if my inklings have any shred of relevancy.

So here's the questions:

1) Big picture: Is having a ton of devices constantly communicating on a single lan segment going to cause collisions, higher latency, etc? The restaurant owner loves his IP cams, so there is constant streaming of about 50 devices, all day, all night. The DHCP range of about 130 addresses is nearly full during peak hours with all the employee's phones, etc.

2) If having all the traffic on a single subnet can cause latency issues, would moving them into a different address space or isolating them with VLAN tags benefit traffic for other devices in any meaningful way? Like, is there a good argument to be concerned about packet collisions, or do we just need greater bandwidth?

3) If the number of devices on a single subnet IS worth being concerned about, how would you recommend I isolate traffic? Would VLAN tagging help if they're still all on the same subnet, or should I NAT different types of traffic? Some other method I am not thinking of?

More about the environment:

All 1Gbps
Mostly Unifi equipment
7 WIFI APs - 4 are dual band, 3 are 2.4GHz only (wireless N) - various Unifi models
Gateway is a Gen1 Unifi Security Gateway
Switches: Started w/ 8 port Unifi switch, have added a Cisco SG110-16P directly after it, and there's a couple cheap 5-port switches near end points at various places on the property for the devices in close proximity
At least 8 Ring cameras (all 2.4GHz)
At least 30 Wyse cameras (all 2.4GHz)
3 Dahua IP cams about 150 ft from the gateway (POE)
A 32 Channel analog security DVR that doesn't use bandwidth for cameras, but people off site watch on their phones, computers, etc.
Oh, and most importantly, 7 credit card machines - 3 of which are handhelds so have to use WIFI

Thanks!
 
  • Like
Reactions: SamuelIsaac

gregsachs

Active Member
Aug 14, 2018
380
110
43
Probably 2 different issues, but that is just my gut;
1: Latency is probably showing when devices are roaming between AP, positioning and signal strength may need some tuning. This can be done via Unifi AP on IOS with some success.
2: I would definitely establish at least 3 different VLANS to segment the network into 3 different zones that are mostly isolated from each other.
First VLAN: Guest/Non-business network for all the employee and/or customer devices. Unifi allows you to make this a network where the devices are isolated from each other.
Second VLAN: "Cameras". Get them on their own segment, .
Third VLAN: "Business Applications": Put the card readers and other business functionality on this. (printers, office pc, etc....)
You will need to replace the Cisco switch and the 5 port switches with managed units; for the 5 port devices the unifi flex mini is really nice at $29. Primary rationale for doing this segmentation is security, to isolate your business traffic from everything else.
Fourth VLAN, optional: customer guest network, can do vouchers for this as well with UNifi.
The USG can do firewall rules between VLANS to allow/block traffic as you want.
The only "sticky" part of this where cost is having to replace with Cisco switch, you can get a netgear managed switch for $99, or look at unifi or similar.
You could also move the dumb 5 port switches to the location where the cisco is, and use them to give 4 extra ports on a given VLAN off of the Unifi switch-IE set port switch 1 to port Profile Business Applications, then plug a dumb 5 port switch into it and you have 4 extra business application ports. Not a great solution, but functional if he doesn't want to spend any $$$.

Primary argument for doing this type of layout is segmentation and security, you don't want the busboy who plays with computers trying to get into your office PC.
 
  • Like
Reactions: AveryFreeman

MBastian

Active Member
Jul 17, 2016
135
32
28
Düsseldorf, Germany
>Oh, and most importantly, 7 credit card machines - 3 of which are handhelds so have to use WIFI

This sounds to me like putting your hand into the proverbial meat grinder. If anything untoward happens you could be the one that gets blamed. I am in agreement with @Tom5051 on this.
Another thing you should consider. The majority of restaurants tend to be open in the evenings and weekends. Personally I don't want to have angry customers on the phone at that time, especially if they are prone to messing around with their infrastructure without me knowing it.
 
Last edited:

AveryFreeman

consummate homelabber
Mar 17, 2017
254
23
18
39
Near Seattle
averyfreeman.com
Probably 2 different issues, but that is just my gut;
1: Latency is probably showing when devices are roaming between AP, positioning and signal strength may need some tuning. This can be done via Unifi AP on IOS with some success.
2: I would definitely establish at least 3 different VLANS to segment the network into 3 different zones that are mostly isolated from each other.
First VLAN: Guest/Non-business network for all the employee and/or customer devices. Unifi allows you to make this a network where the devices are isolated from each other.
Second VLAN: "Cameras". Get them on their own segment, .
Third VLAN: "Business Applications": Put the card readers and other business functionality on this. (printers, office pc, etc....)
You will need to replace the Cisco switch and the 5 port switches with managed units; for the 5 port devices the unifi flex mini is really nice at $29. Primary rationale for doing this segmentation is security, to isolate your business traffic from everything else.
Fourth VLAN, optional: customer guest network, can do vouchers for this as well with UNifi.
The USG can do firewall rules between VLANS to allow/block traffic as you want.
The only "sticky" part of this where cost is having to replace with Cisco switch, you can get a netgear managed switch for $99, or look at unifi or similar.
You could also move the dumb 5 port switches to the location where the cisco is, and use them to give 4 extra ports on a given VLAN off of the Unifi switch-IE set port switch 1 to port Profile Business Applications, then plug a dumb 5 port switch into it and you have 4 extra business application ports. Not a great solution, but functional if he doesn't want to spend any $$$.

Primary argument for doing this type of layout is segmentation and security, you don't want the busboy who plays with computers trying to get into your office PC.
This is super comprehensive and helpful. I probably should have done more reading before posing my question, I didn't realize VLANs were also used to reduce congestion in addition to isolation for security purposes. Also learned that VLANs are layer-2 which I did not know. So, like I said, definite holes in knowledge. In re: to people saying we should hire out, you're probably right, but it's not going to happen. Besides, the owner likes giving me learning experiences, it's definitely not the first.

Luckily, I'm the only one who really has access to any of the networking setup (not even the owner), and employees don't ever use our computers, so at least we don't have that to worry about.

I'm guessing you're recommending we get rid of the Cisco SG110-16P because of VLAN tagging - we just bought it so I'm hoping it can at least be used for the IP cams. I am going to do some reading and reply back once I can do a proper diagram. I really like your recommendations, though.
 

gregsachs

Active Member
Aug 14, 2018
380
110
43
This is super comprehensive and helpful. I probably should have done more reading before posing my question, I didn't realize VLANs were also used to reduce congestion in addition to isolation for security purposes. Also learned that VLANs are layer-2 which I did not know. So, like I said, definite holes in knowledge. In re: to people saying we should hire out, you're probably right, but it's not going to happen. Besides, the owner likes giving me learning experiences, it's definitely not the first.

Luckily, I'm the only one who really has access to any of the networking setup (not even the owner), and employees don't ever use our computers, so at least we don't have that to worry about.

I'm guessing you're recommending we get rid of the Cisco SG110-16P because of VLAN tagging - we just bought it so I'm hoping it can at least be used for the IP cams. I am going to do some reading and reply back once I can do a proper diagram. I really like your recommendations, though.
Without using VLAN segmentation, or physical segmentation, the network is the network. All devices on the network can see all other devices, and access all other devices.

If the cisco is used purely for IPcams, then you can reuse it. You can't have devices on two different VLANS plugged into it. Yes, a diagram would be useful. I would suggest that you may want to lay it out physically, and then use colors to indicate the application/possible VLAN.
I think you can get 4 ssid/access point without adversely impacting performace with most of the UAP, but not positive.
One other thought I just had; 7 AP is a lot; and you may have some of them tuned for higher power than you really want. Especially the 2.4g units are very possibly interfering with each other . Use the heat map tool in the Unifi APP to try and understand signal strength, or do similar with Acrylic Wifi.
 
Last edited:
  • Like
Reactions: AveryFreeman

AveryFreeman

consummate homelabber
Mar 17, 2017
254
23
18
39
Near Seattle
averyfreeman.com
Without using VLAN segmentation, or physical segmentation, the network is the network. All devices on the network can see all other devices, and access all other devices.

If the cisco is used purely for IPcams, then you can reuse it. You can't have devices on two different VLANS plugged into it. Yes, a diagram would be useful. I would suggest that you may want to lay it out physically, and then use colors to indicate the application/possible VLAN.
I think you can get 4 ssid/access point without adversely impacting performace with most of the UAP, but not positive.
One other thought I just had; 7 AP is a lot; and you may have some of them tuned for higher power than you really want. Especially the 2.4g units are very possibly interfering with each other . Use the heat map tool in the Unifi APP to try and understand signal strength, or do similar with Acrylic Wifi.
Will do! Our camera guy is always asking me to turn the 2.4GHz radio up because it's the only band used by all the WIFI cameras. I generally would let the system determine what strength radios should be, but we were having either some interference or reception issues: I find it hard to believe it is reception, though, because the 7 APs we have on an approximately 6,000 sqft lot seems like it should be fine coverage-wise. Maybe I can get back to you with a heat map, too - great call.
 

ArmedAviator

Member
May 16, 2020
89
51
18
Ohio
You have 7 APs in the 2.4GHz band inside 6,000 square feet in a, likely, mostly open area? Yeah, that's a problem. Turning the signal power up will likely make reception worse, not better. RF isn't just "pump in more power to increase signal strength." RF bounces and intereres. Make sure you're using band steering to ideally get half of the WiFI devices on 5GHz and half on 2.4GHz.

VLANs are absolutely necessary as mentioned above. I have 4 WiFI networks just in my home:
IoT - limited WAN access, no access between VLANs
Cameras - no WAN access, extremely limited access between VLANs
Guests - WAN access, very limited access to other VLANs
Private - Full access for my personal devices

Firewall rules, RF planning, VLAN planning, etc. are no easy task but can be done by a DIYer. Be sure the owner understands you're not liable. Don't make promises you can't keep.
 
  • Like
Reactions: AveryFreeman

ccie4526

New Member
Jan 25, 2021
16
10
3
So I do this stuff for a living, for a producer of products that your restaurant probably sells.

First biggest problem I see here.... you have 3 non-overlapping 20MHz channels in 2.4GHz. And you've got 38 clients (cameras) that are streaming video back to the APs. Regardless of all the VLAN-related things we can do to segment the network, you're still going to run into the limit of physics.... there is only so much RF (radio frequency) bandwidth available, and it's getting absolutely flooded.

We don't put ANY cameras on wi-fi. Only wireless POS terminals and authorized corporate clients have unrestricted bandwidth access. We do allow visitors/customers access but we restrict the per-client bandwidth so no one guest user can flood the network.

In your case, it's going to be extremely unpopular, but I would strongly recommend replacing all the wireless cameras with wired cameras. That will resolve 98% of your problems.

Edit to add: I run a Wyze Cam at home. It's nice, but I'm trying to figure out how to make it a wired device.
 
  • Like
Reactions: AveryFreeman

AveryFreeman

consummate homelabber
Mar 17, 2017
254
23
18
39
Near Seattle
averyfreeman.com
You have 7 APs in the 2.4GHz band inside 6,000 square feet in a, likely, mostly open area? Yeah, that's a problem. Turning the signal power up will likely make reception worse, not better. RF isn't just "pump in more power to increase signal strength." RF bounces and intereres. Make sure you're using band steering to ideally get half of the WiFI devices on 5GHz and half on 2.4GHz.

VLANs are absolutely necessary as mentioned above. I have 4 WiFI networks just in my home:
IoT - limited WAN access, no access between VLANs
Cameras - no WAN access, extremely limited access between VLANs
Guests - WAN access, very limited access to other VLANs
Private - Full access for my personal devices

Firewall rules, RF planning, VLAN planning, etc. are no easy task but can be done by a DIYer. Be sure the owner understands you're not liable. Don't make promises you can't keep.
Thanks, super helpful recommendations. Is wifi traffic essentially independent of VLAN encapsulation? Or do you think VLAN isolation will help with wifi somewhat, too?

The wireless cameras all operate in the 2.4 GHz band - would it be helpful to move as much traffic as possible to the 5 GHz band (guests, employees) even if cameras, guests and employees are isolated by VLANs separately?

From a raw numbers standpoint I think the cameras will be the primary issue to tackle. There's more cameras than employees, and they're always adding more. I'm not familiar with either Wyse or Ring, I do more conventional IP cams. But I think they might not actually transmit 24/7, only if they're activated by motion - unless someone is watching them. So that might be a bit of a saving grace.

Thinking out loud, I wonder if I would need to go as so far to isolate the cameras into groups based on some attribute like location, so no more than a certain number are isolated in a particular group - or would that be overkill?
 

ArmedAviator

Member
May 16, 2020
89
51
18
Ohio
Long story short....

You MUST either:
A) get wires to the IP cams if they can be wired or
B) Replace the cameras with models that can be wired, and then wire them.

WiFi is a whole different beast than wired networking. With wired networks, you may have a bottleneck, but you can just add more links, add higher capacity equipment, use a higher bandwidth module/switch, etc. Each connection has a determined bandwidth available (100Mbps, 1Gbps, etc.). WiFi is a spectrum and it's a narrow spectrum. The 2.4GHz spectrum has only 3 channels that do not overlap. Of you have 4 APs in a 6,000 square feet area, at least two of them are overlapping and interfering with each other before you even introduce a single client. Now add many wireless devices that are all fighting to communicate on those channels and you have serious issues. Each device basically has to get a time slot, like a car merging on a highway. Busy highway means not many cars can find a gap - latency, dropped packets, etc.

You MUST start with dropping the WiFi cameras, at least most of them.

Use a WiFi app on your phone and the Unifi software to identify the best channel options for your APs - each AP should be on a different channel that has minimal interference from other APs. This may actually require you to disable an AP. Use the phone app to map signals out around the place. Relocating the APs may be helpful for better coverage.

Use Band Steering so guest traffic is roughly split between 2.4 and 5 GHz - Unifi has this as an easy to select option.

Might want to upgrade the APs depending on models usedm. Consult Unifi website to see rough client limitations.

Regarding VLANs, it sounds like you don't know much about them and it might be a little toouch to tackle it all at the same time since you're already struggling with the spectrum being so busy.

When you do get to VLANs,.set up different SSIDs for each VLAN. Unifi supports up to 4 SSIDs with different VLANs on each from each AP, at least on my AP AC Pro. This requires a capable switch, and probably a capable router, too.

When running out of available IPs in the subnet, you can increase the subnet size. Say from /24 to a /22. Perhaps you'll want this on the Guest VLAN.
 
  • Like
Reactions: AveryFreeman

Sean Ho

seanho.com
Nov 19, 2019
45
18
8
Vancouver, BC
seanho.com
Some of the Amcrest PoE cams are rebranded Dahua and quite affordable. Not as throwaway cheap as the WiFi Wyze cams, but you get what you pay for. 2MP Sony StarVis sensors are amazing. If considering a bulk order, talk with Andy at EmpireTech (directly, not via Amazon/eBay). Lots more info at ipcamtalk.

A few well-placed cams are better than a gazillion randomly-placed ones; more pixels is not always better (particularly at night); and wider is definitely not always better. Plan out goals, use cases, and threat models. Design clear sight lines for broad overview shots, and choke points at points of ingress (doors) for eye-level cams for identification (see how Wal-Mart does this). Think about DORI distances and choose lens/sensor combinations accordingly.
 

BoredSysadmin

Active Member
Mar 2, 2019
564
172
43
I'd add strongly to moving ALL of your IP cameras to wired PoE - This is an absolute must thing to do, simply due to in case of fraud - you shouldn't have to rely on subpar image quality nor half of the cameras go berzerk every time microwave goes on.
I'd throw my hat with Reolink as super affordable and decent enough quality

After you did that, you have lots of great suggestions about the Wifi above. I'd add one more great read as to why we have "unexplained" wifi gremlins and high latency (seconds)
 
Last edited:

sleeper404

New Member
Jan 10, 2021
19
5
3
As pretty much everyone above has stated, Cameras on 2.4Ghz and only 3 non-overlapping channels limits the total throughput capacity significantly. You could probably heat-map out which three APs should be the only ones with 2.4Ghz enabled and shut it off on the remaining 4, but the latency issue will persist. There's only so many timeslots available on each of those radios and as the latency increases exponentially with the rate of utilization with the channel. Newer Wifi6(AX) will help with this, but it's not an appropriate fix since it would require all new APs and new cameras and at that point wiring them in and buying a reasonable PoE switch that's wired into it's own segment would still be a cheaper and better solution.

TLDR; No fix for your latency with constant timeslot congestion from the cameras and no amount of VLANs or segmentation will reduce this at the problem point, your AP radio. Although VLANs and firewalls zones to keep the devices of different business uses separated is a bare-minimum security must.
 

AveryFreeman

consummate homelabber
Mar 17, 2017
254
23
18
39
Near Seattle
averyfreeman.com
Long story short....

You MUST either:
A) get wires to the IP cams if they can be wired or
B) Replace the cameras with models that can be wired, and then wire them.

WiFi is a whole different beast than wired networking. With wired networks, you may have a bottleneck, but you can just add more links, add higher capacity equipment, use a higher bandwidth module/switch, etc. Each connection has a determined bandwidth available (100Mbps, 1Gbps, etc.). WiFi is a spectrum and it's a narrow spectrum. The 2.4GHz spectrum has only 3 channels that do not overlap. Of you have 4 APs in a 6,000 square feet area, at least two of them are overlapping and interfering with each other before you even introduce a single client. Now add many wireless devices that are all fighting to communicate on those channels and you have serious issues. Each device basically has to get a time slot, like a car merging on a highway. Busy highway means not many cars can find a gap - latency, dropped packets, etc.

You MUST start with dropping the WiFi cameras, at least most of them.

Use a WiFi app on your phone and the Unifi software to identify the best channel options for your APs - each AP should be on a different channel that has minimal interference from other APs. This may actually require you to disable an AP. Use the phone app to map signals out around the place. Relocating the APs may be helpful for better coverage.

Use Band Steering so guest traffic is roughly split between 2.4 and 5 GHz - Unifi has this as an easy to select option.

Might want to upgrade the APs depending on models usedm. Consult Unifi website to see rough client limitations.

Regarding VLANs, it sounds like you don't know much about them and it might be a little toouch to tackle it all at the same time since you're already struggling with the spectrum being so busy.

When you do get to VLANs,.set up different SSIDs for each VLAN. Unifi supports up to 4 SSIDs with different VLANs on each from each AP, at least on my AP AC Pro. This requires a capable switch, and probably a capable router, too.

When running out of available IPs in the subnet, you can increase the subnet size. Say from /24 to a /22. Perhaps you'll want this on the Guest VLAN.
You're presenting some pretty interesting considerations. We're not changing cameras, so that's out of the question. I set up the 3 IP cams we have now, and they're good for the application (monitoring a large parking lot from ~20ft up a pole), but the owner loves his little Wyze cams because they're tiny and he can stick them up himself. Has them watching things like stock levels in the walk-in fridge and stuff like that, and he likes their motion detection algorithm. So I'll have to come up with another solution. Thankfully, the algorithm probably reduces the amount they stream somewhat - I don't think they upload unless they detect something (not a Wyze expert, but I remember him mentioning something about that)

I don't think I need to lower the subnet resolution yet, but re: "time slots", there's a chance we could upgrade to a couple APs with more MIMO. The AP-LRs are 2x2, the UAP-AC-PROs are 3x3, there's some new ones (I forget which model - I wanna say Nano?) that are 4x4. I'll have to figure out which ones we have what number MIMO radios and which frequency band they're on (well, the LRs are only 2.4 Ghz, so that's easy).

I was actually working on the diagram right now, just came on to ask a quick question about switches...
 

sleeper404

New Member
Jan 10, 2021
19
5
3
MIMO only really helps with large data transfers. I have wyze cameras myself and the video stream to the cloud is constant because it's doing the motion detection server-side and not local to the camera. MIMO won't help you here because it's not a large amount of data, it's tiny bits of data eating up timeslots in a death by a thousand cuts sort of way. That timeslot with MIMO and AC and bonded channels could send MBs of data, but instead it's eaten up by a 38k frame of video and so are 80% of all the timeslots from each camera constantly sending tiny bits of data.
 
  • Like
Reactions: AveryFreeman

AveryFreeman

consummate homelabber
Mar 17, 2017
254
23
18
39
Near Seattle
averyfreeman.com
Some of the Amcrest PoE cams are rebranded Dahua and quite affordable. Not as throwaway cheap as the WiFi Wyze cams, but you get what you pay for. 2MP Sony StarVis sensors are amazing. If considering a bulk order, talk with Andy at EmpireTech (directly, not via Amazon/eBay). Lots more info at ipcamtalk.

A few well-placed cams are better than a gazillion randomly-placed ones; more pixels is not always better (particularly at night); and wider is definitely not always better. Plan out goals, use cases, and threat models. Design clear sight lines for broad overview shots, and choke points at points of ingress (doors) for eye-level cams for identification (see how Wal-Mart does this). Think about DORI distances and choose lens/sensor combinations accordingly.
I agree with your strategy, and I prefer "real" ip cams, myself. But it's not my call. They love their Wyze and their Rings, and their methods of placement. there's nothing I could say that would change their mind. So I just have to make it work.

I do buy from Andy, I asked him what he recommended and he said the IPC-T5442TM-AS, so have bought a handful of them for myself and others. I absolutely love them. Restaurant owner has 3 IPC-HFW2831T-ZS about 20' up a pole in a parking lot, also from Andy.

Dahua camera aside: If you want the newest firmware, you have to VPN to the Dahua site from another country - Canada works fine. The firmware updates aren't available in the US - when the camera detects you're in the US, the auto-upgrade or new firmware alert features are useless.
 

AveryFreeman

consummate homelabber
Mar 17, 2017
254
23
18
39
Near Seattle
averyfreeman.com
MIMO only really helps with large data transfers. I have wyze cameras myself and the video stream to the cloud is constant because it's doing the motion detection server-side and not local to the camera. MIMO won't help you here because it's not a large amount of data, it's tiny bits of data eating up timeslots in a death by a thousand cuts sort of way. That timeslot with MIMO and AC and bonded channels could send MBs of data, but instead it's eaten up by a 38k frame of video and so are 80% of all the timeslots from each camera constantly sending tiny bits of data.
Damn, good to know.

We could do a separate network just for the WIFI cams, but then the radios from additional APs for an additional WIFI network would be interfering, I imagine...

This situation is evolving out of control. I need to tackle one thing at a time, otherwise it's too overwhelming.
 

ArmedAviator

Member
May 16, 2020
89
51
18
Ohio
Bandaids on top of bandaids will make a broken network a complicated, broken network.

You asked for others' guidance in resolving the experienced network issues. Very intelligent people gave you good information to get you moving in the right direction. If your boss doesn't want to accept the information that you acquired in your research and asking than walk away from the task and advise him to hire a professional - which will share a similar story and will happily take your employer's money.