Network hardware and topology for multi-gig, poe++, managed vlan?

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

BrockSamson

New Member
Dec 7, 2022
8
0
1
I'm setting up a new home network in order to add internet-disabled security cameras and to better isolate internet-enabled IOT devices. I'm tech savvy but new to networking so I've spent the past few weeks trying to learn and plan as much as I can. My understanding is that normal consumer WIFI routers don't support this and that I need a managed switch with VLAN support. In addition, now that a normal WIFI router is out of the picture (unless there is some way to still use one?) I need to replace that functionality with something like OPNsense and some APs. Based on this understanding I have created a diagram with the two best options I've been cable to come up with so far. On the left I have a single switch "primary" switch and on the right I have a nearly identical setup but with a pair of "primary" switches.
NetworkTopologyIdeas.png
* If you notice that the single switch diagram is using 9/8 ports that's because 1 or 2 good AP is likely sufficient and therefore I only need 7 or 8 ports. I wanted to show 3 APs which exceeds the available ports to help visualize alternate switch combinations that would support additional devices.

I think either of my plans would work but I have this nagging feeling that there is a better option right in front of me that I just can't see. It seems like a single higher end ~16 port switch would be better but I can't seem to find anything that is multi-gig, managed, AND has 2-4 poe++ for under $1k. Trying to find alternatives like only having multi-gig on uplinks or using poe injectors instead of poe switches adds up to about the same cost or has significantly reduced features for no real savings. I just can't seem to find any combination of devices that beats one or two MS108EUP at $280 each.

I'm also torn between a dedicated router box like the R86S or just hosting OPNsense in a Proxmox VM on my automation pc. It has plenty of resources but only a single 2.5g NIC. I'd either need to use a single cable (security concerns), add a USB NIC (I guess these are bad), or maybe swap the unnecessary M2 A+E WIFI module with a NIC (which may not be possible if the WIFI module also controls the 2.5 LAN port like they sometimes do). Or maybe there IS some way to use a normal WIFI router as the internet connection (and AP?) but still use an OPNsense VM and VLANs to get the isolation and security that I want?

At this point I'm not really sure and I'm going in circles. Anyone have any suggestions?
 

Craig Curtin

Member
Jun 18, 2017
101
20
18
59
I'm setting up a new home network in order to add internet-disabled security cameras and to better isolate internet-enabled IOT devices. I'm tech savvy but new to networking so I've spent the past few weeks trying to learn and plan as much as I can. My understanding is that normal consumer WIFI routers don't support this and that I need a managed switch with VLAN support. In addition, now that a normal WIFI router is out of the picture (unless there is some way to still use one?) I need to replace that functionality with something like OPNsense and some APs. Based on this understanding I have created a diagram with the two best options I've been cable to come up with so far. On the left I have a single switch "primary" switch and on the right I have a nearly identical setup but with a pair of "primary" switches.
View attachment 25997
* If you notice that the single switch diagram is using 9/8 ports that's because 1 or 2 good AP is likely sufficient and therefore I only need 7 or 8 ports. I wanted to show 3 APs which exceeds the available ports to help visualize alternate switch combinations that would support additional devices.

I think either of my plans would work but I have this nagging feeling that there is a better option right in front of me that I just can't see. It seems like a single higher end ~16 port switch would be better but I can't seem to find anything that is multi-gig, managed, AND has 2-4 poe++ for under $1k. Trying to find alternatives like only having multi-gig on uplinks or using poe injectors instead of poe switches adds up to about the same cost or has significantly reduced features for no real savings. I just can't seem to find any combination of devices that beats one or two MS108EUP at $280 each.

I'm also torn between a dedicated router box like the R86S or just hosting OPNsense in a Proxmox VM on my automation pc. It has plenty of resources but only a single 2.5g NIC. I'd either need to use a single cable (security concerns), add a USB NIC (I guess these are bad), or maybe swap the unnecessary M2 A+E WIFI module with a NIC (which may not be possible if the WIFI module also controls the 2.5 LAN port like they sometimes do). Or maybe there IS some way to use a normal WIFI router as the internet connection (and AP?) but still use an OPNsense VM and VLANs to get the isolation and security that I want?

At this point I'm not really sure and I'm going in circles. Anyone have any suggestions?
OK a couple of things

1) As a noobie to networking - you may be better off looking at moving into the Ubiquiti Unifi universe as it covers all of this and makes it a lot easier to administer on an ongoing basis - remember these sort of things evolve over time and as such need changes and fixes/updates - as such unless you are prepared to put in a fair amount of learning time - you will not end up with a perfect solution cobbled together from different vendors.

2) if you are doing this as a learning exercise and are willing to invest a fair whack of time to come upto speed then the world is your oyster.

3) Switch - i would look at either a used Cisco or better value a used Brocade 6450 with POE - either of which you will pick up off ebay for under $200 with a little patience - both of them in the 24 port range

4) OpnSense - would definitely not recommend running it virtualised as your first time effort - if your virtualisation box fails for anyone reason it will be a very stressful time trying to get the internet back and running and fixing up a virtualisation system - whilst running around looking for config documentation (which will probably be out of date) - start with a small cheap 2nd hand PC as your OpnSense firewall (even a laptop with USB 3) would be good enough - and then get fancy later.

I have previously run my OpnSense virtualised (and i do this stuff for a living) and have gone back to one of the Mini PCs that Patrick features on the YT channel all the time - as i do not need grief from the family when i am playing around with stuff and the internet goes down

Craig
 

LodeRunner

Active Member
Apr 27, 2019
540
227
43
Multi-gig switches are expensive, apart from a few low-port count ones. So you'd probably wind up with a 16-24 port enterprise 1/10G switch and a 5/8 port multigig capable switch connected on a 10G port to your core switch(es).

I run OPNsense as a VM; I have a 2-node cluster so I can lose a host and still stay online. The required VLANs exist as separate VNICs in my environment, and the OPNsense is completely unaware of any tagging, it sees each as a physical interface. In a single-host environment you are at risk of downtime for a variety of reasons ranging from planned maintenance to any number of failures. A R86S or equivalent would be fine. Or look at MicroTik's offerings.

Or as Craig, said, do all Unifi. It's a solid pick with a single pain of glass. The UDM-Pro-SE has 2.5 and 10G interfaces on it, and if you can get the new UI Enterprise series switches, they're 2.5Gbe. Otherwise the 24p Switch Pro is 16 PoE+ and 8 PoE++ with 2 10G SFP+ uplink ports.

I'll be honest, given the 2.5/5Gbe price premium, you're better off going with 10G equipment where you need that kind of throughput and 1G everywhere else from a budget perspective. It will require more admin effort.

It comes down to a short list of questions:
  • What's your budget cap?
  • Are you willing to dig into the guts of switching and multi-VLAN routing at the CLI level?
  • How much faith do you have in your VM platform and your ability to quickly troubleshoot an outage (in the case of a virtualized router/firewall)
  • Do you care if everything is from a single vendor?
  • Is a single pane of glass management UI preferred, or merely a would be nice feature?
 

BrockSamson

New Member
Dec 7, 2022
8
0
1
OK a couple of things

1) As a noobie to networking - you may be better off looking at moving into the Ubiquiti Unifi universe as it covers all of this and makes it a lot easier to administer on an ongoing basis - remember these sort of things evolve over time and as such need changes and fixes/updates - as such unless you are prepared to put in a fair amount of learning time - you will not end up with a perfect solution cobbled together from different vendors.

2) if you are doing this as a learning exercise and are willing to invest a fair whack of time to come upto speed then the world is your oyster.

3) Switch - i would look at either a used Cisco or better value a used Brocade 6450 with POE - either of which you will pick up off ebay for under $200 with a little patience - both of them in the 24 port range

4) OpnSense - would definitely not recommend running it virtualised as your first time effort - if your virtualisation box fails for anyone reason it will be a very stressful time trying to get the internet back and running and fixing up a virtualisation system - whilst running around looking for config documentation (which will probably be out of date) - start with a small cheap 2nd hand PC as your OpnSense firewall (even a laptop with USB 3) would be good enough - and then get fancy later.

I have previously run my OpnSense virtualised (and i do this stuff for a living) and have gone back to one of the Mini PCs that Patrick features on the YT channel all the time - as i do not need grief from the family when i am playing around with stuff and the internet goes down

Craig
I've looked at Unifi and plan on using the Switch Flex (USW-Flex) poe passthrough switches for the cameras. Those makes a lot of sense to me because I can run a single 5e cable to a place with no power AND the cameras would be on battery backup. For the rest of the network I'm not sure. The best I can come up with is a Dream Machine Pro or SE combined with a Switch XG 6 PoE. The DM would really only have the automation pc and nas connected to it, maybe some poe++ injectors for the camera legs. The XG would connect to the DM with 10g SFP+ and would then supply the 10g RJ45 poe++ that the AP(s) need as well as a 10g up to the office. For the office, the only Unifi switches with multi-gig uplink capabilities seem to be the Switch Flex XG or Enterprise 8 PoE.

So if I have that right(yes/no?) it's all great and gives me 10g links between the switches and AP(s) but it costs $1,277-$1,577 plus potentially an additional poe++ injector or two. It would only give me 2.5g but the MS108EUP, R86S-G1, and unmanaged office switch combo would only cost $725-1,005 depending on one or two main switches. I guess my question to all of you then is whether it's worth the extra $300-900 for Unifi. Or maybe mix the options and get the best of both worlds with a DM Pro and MS108EUP?
 

BrockSamson

New Member
Dec 7, 2022
8
0
1
Multi-gig switches are expensive, apart from a few low-port count ones. So you'd probably wind up with a 16-24 port enterprise 1/10G switch and a 5/8 port multigig capable switch connected on a 10G port to your core switch(es).

I run OPNsense as a VM; I have a 2-node cluster so I can lose a host and still stay online. The required VLANs exist as separate VNICs in my environment, and the OPNsense is completely unaware of any tagging, it sees each as a physical interface. In a single-host environment you are at risk of downtime for a variety of reasons ranging from planned maintenance to any number of failures. A R86S or equivalent would be fine. Or look at MicroTik's offerings.

Or as Craig, said, do all Unifi. It's a solid pick with a single pain of glass. The UDM-Pro-SE has 2.5 and 10G interfaces on it, and if you can get the new UI Enterprise series switches, they're 2.5Gbe. Otherwise the 24p Switch Pro is 16 PoE+ and 8 PoE++ with 2 10G SFP+ uplink ports.

I'll be honest, given the 2.5/5Gbe price premium, you're better off going with 10G equipment where you need that kind of throughput and 1G everywhere else from a budget perspective. It will require more admin effort.

It comes down to a short list of questions:
  • What's your budget cap?
  • Are you willing to dig into the guts of switching and multi-VLAN routing at the CLI level?
  • How much faith do you have in your VM platform and your ability to quickly troubleshoot an outage (in the case of a virtualized router/firewall)
  • Do you care if everything is from a single vendor?
  • Is a single pane of glass management UI preferred, or merely a would be nice feature?
Does having the VM host handle the VLAN->VNIC stuff cause any security concerns or performance issues? For LAN it seems fine but what about bringing internet WAN in this way? The external traffic would have to hit your switch and VLAN first, then VM host physical hardware, then VM host software, and finally get to OPNsense. That's my biggest concern really, missing something and leaving things wide open on accident. Otherwise this option is free and has easy snapshots and restore which are both great things.

For Unifi I'm going to do more research into it. The rest of my night is going to be "UDM Pro vs pfSense/OPNsense". The biggest hiccup for Unifi (or anything else) seems to be getting multi-gig poe++ for the AP(s). The Switch XG 6 PoE really seems to be the only option that Unifi has for this. Mixing a UDM Pro and Netgear MS108EUP is becoming oddly appealing though. Maybe not a terrible idea?

For the rest:
  • I'd like to say that $400-700 is a good budget but I suspect I'm looking more at $600-1000 to do what I want. This is for the router and switch(es) but NOT including the poe passthrough switches for the cameras which I'm including in the camera budget.
  • It's not a dealbreaker but I'd prefer to avoid CLI.
  • VM outages are not a concern. I'm decent enough at managing this and if I have any problems I can keep my family happy by connecting a backup wifi router directly to the cable modem.
  • No concerns about single vendor or multiple management UIs as long as it's not absolute torture. I doubt anything is that bad so it mostly just has to work and be secure.
 

Craig Curtin

Member
Jun 18, 2017
101
20
18
59
Does having the VM host handle the VLAN->VNIC stuff cause any security concerns or performance issues? For LAN it seems fine but what about bringing internet WAN in this way? The external traffic would have to hit your switch and VLAN first, then VM host physical hardware, then VM host software, and finally get to OPNsense. That's my biggest concern really, missing something and leaving things wide open on accident. Otherwise this option is free and has easy snapshots and restore which are both great things.
No it is not performance issue (within reason) but it can become a security issue - how big a one is really up to you to decide. The issue is that everything starts out all good - and then you get complacement and don't start applying updates and/or you are not a security professional so do not really know which ones are needed - and probably do not subscribe to all the advisories etc - and then one day your system gets attacked

I personally prefer discrete units that i can easily replace and upgrade and implement security in depth on - also even though i do this sort of stuff for a living - last thing i want to do is come home all weekend and fight a series of upgrades/patches so things will slip as will documentation

Then you end up with teenage kids who invite their friends over and give them the main wifi password etc etc.

Craig
 

LodeRunner

Active Member
Apr 27, 2019
540
227
43
Does having the VM host handle the VLAN->VNIC stuff cause any security concerns or performance issues? For LAN it seems fine but what about bringing internet WAN in this way? The external traffic would have to hit your switch and VLAN first, then VM host physical hardware, then VM host software, and finally get to OPNsense. That's my biggest concern really, missing something and leaving things wide open on accident. Otherwise this option is free and has easy snapshots and restore which are both great things.
I do it that way because other VMs may also need those VLANs so defining the VNIC once at the host level is more efficient and less error prone that having to deal with VLAN tagging and sub interfaces. For example, I have a VM for running various game servers; it's in its own VLAN and subnet and the firewall has strict rules about what traffic can transit between that LAN and the main home LAN.

For WAN, I have a Layer 2 only access (not trunked) port that goes to my ONT; you can't hit the switch management interface from that port/VLAN. The host interface that handles the VLAN trunk is not shared for management and doesn't have an IP address; none of the host management protocols are reachable on this port. So if someone can escape that, then all the big guys who do hosted firewalls are screwed, so what hope do I have. I don't lose sleep over it. Someone trying something like a VLAN hopping attack would have to first know enough about my ISPs internal networking and the GPON setup, then somehow have intelligence on the VLAN setup of my network. Nothing I have is running on default VLAN 1.

Anyone willing to do that much work to get me is probably someone I can't defend against without a complete air-gap anyway. Keep OPNsense up to date and watch the CVEs for your host and call it a day.

I have Veeam backing up the cluster, so in the event (and I've had cause to test this) that the firewall VM gets borked, I can restore it in 5-10 minutes, assuming I don't have a VM checkpoint to rollback. I keep 30 days of RPs in Veeam, so I'll always have a good RP for the firewall available, as the chances of me not noticing broken internet for 30 days are pretty low. That said, in an absolute worst case scenario, I have hardware that could be brought online as a firewall box to pull the Veeam backups from S3 storage in a 'house burned down' situation. At that point though, my time to recovery doesn't really matter as much as the fact that I can recover.

OK, now to the actual topic at hand:
  • What APs are you using (or planning to use)? Bear in mind that even if your client can actually sync at whatever max rate the AP offers, that's going to be much higher than the actual achievable throughput.
  • Do you know what your RF environment is like? For example, I know, that thanks to some very noisy neighboring APs, I have to use 20 MHz 2.4 channels (trying to get rid of my remaining 2.4Ghz devices) and I can only use 40MHz 5G channels even though all my 5G stuff supports 80 MHz.
  • How many wireless clients are you expecting, and what is the use case? If it's a bunch of phones or streaming boxes, or even a heavily used laptop, the possibility of saturating even a gigabit link to a given AP is low given the shared nature of RF spectrum.

For the price, it's hard to do better than the UniFi Flex switch for that particular use case.

The GUIs for most switches are trash, IMO, no matter how pretty they are. They almost always elide or obfuscate some part of the configuration. UniFi is decent, though last time I managed switching on one for a client, the way they did VLAN trunking was obtuse, compared to Cisco CLI syntax. Maybe it's better now.

Budget:
Retail for the MS108EUP is $440 each, looks like a few on eBay for $380-400 right now. What I'd do is count up how many multigig clients you have right now, or are planning to have. If the potential multigig clients don't already have a 2.5/5G NIC, then I'd take a good hard look at doing 10G instead (or accepting 1G speed); if you're pulling new runs, it's not that hard to do a few pre-terminated fiber runs; I used a unneeded CAT-5 cable as my pull string, taped two fiber cables to it and pulled them back to my rack (fs.com is a good source for fiber and transceivers, or RJ-45 SFP+ modules if you get a SFP core switch and copper 10G NICs).

The UDM Pro (1G/10G) is $380 new, the SE which has 2.5G ports is $500 new. And $100 per Flex. So going by your right-hand plan, 2x MS108EUP, 2x Flex, and a UDM Pro SE would be $1580 at MSRP, pre-tax and shipping. Going the same but with a GW-R86S-G1 instead of the UDMP-SE gets you $1395 minimum (price I see for GW-R86S-G1 is ~$315).

The left hand plan pencils out to $1140 or $955 since it only has one MS108EUP.
 

BrockSamson

New Member
Dec 7, 2022
8
0
1
I do it that way because other VMs may also need those VLANs so defining the VNIC once at the host level is more efficient and less error prone that having to deal with VLAN tagging and sub interfaces. For example, I have a VM for running various game servers; it's in its own VLAN and subnet and the firewall has strict rules about what traffic can transit between that LAN and the main home LAN.

For WAN, I have a Layer 2 only access (not trunked) port that goes to my ONT; you can't hit the switch management interface from that port/VLAN. The host interface that handles the VLAN trunk is not shared for management and doesn't have an IP address; none of the host management protocols are reachable on this port. So if someone can escape that, then all the big guys who do hosted firewalls are screwed, so what hope do I have. I don't lose sleep over it. Someone trying something like a VLAN hopping attack would have to first know enough about my ISPs internal networking and the GPON setup, then somehow have intelligence on the VLAN setup of my network. Nothing I have is running on default VLAN 1.

Anyone willing to do that much work to get me is probably someone I can't defend against without a complete air-gap anyway. Keep OPNsense up to date and watch the CVEs for your host and call it a day.

I have Veeam backing up the cluster, so in the event (and I've had cause to test this) that the firewall VM gets borked, I can restore it in 5-10 minutes, assuming I don't have a VM checkpoint to rollback. I keep 30 days of RPs in Veeam, so I'll always have a good RP for the firewall available, as the chances of me not noticing broken internet for 30 days are pretty low. That said, in an absolute worst case scenario, I have hardware that could be brought online as a firewall box to pull the Veeam backups from S3 storage in a 'house burned down' situation. At that point though, my time to recovery doesn't really matter as much as the fact that I can recover.

OK, now to the actual topic at hand:
  • What APs are you using (or planning to use)? Bear in mind that even if your client can actually sync at whatever max rate the AP offers, that's going to be much higher than the actual achievable throughput.
  • Do you know what your RF environment is like? For example, I know, that thanks to some very noisy neighboring APs, I have to use 20 MHz 2.4 channels (trying to get rid of my remaining 2.4Ghz devices) and I can only use 40MHz 5G channels even though all my 5G stuff supports 80 MHz.
  • How many wireless clients are you expecting, and what is the use case? If it's a bunch of phones or streaming boxes, or even a heavily used laptop, the possibility of saturating even a gigabit link to a given AP is low given the shared nature of RF spectrum.

For the price, it's hard to do better than the UniFi Flex switch for that particular use case.

The GUIs for most switches are trash, IMO, no matter how pretty they are. They almost always elide or obfuscate some part of the configuration. UniFi is decent, though last time I managed switching on one for a client, the way they did VLAN trunking was obtuse, compared to Cisco CLI syntax. Maybe it's better now.

Budget:
Retail for the MS108EUP is $440 each, looks like a few on eBay for $380-400 right now. What I'd do is count up how many multigig clients you have right now, or are planning to have. If the potential multigig clients don't already have a 2.5/5G NIC, then I'd take a good hard look at doing 10G instead (or accepting 1G speed); if you're pulling new runs, it's not that hard to do a few pre-terminated fiber runs; I used a unneeded CAT-5 cable as my pull string, taped two fiber cables to it and pulled them back to my rack (fs.com is a good source for fiber and transceivers, or RJ-45 SFP+ modules if you get a SFP core switch and copper 10G NICs).

The UDM Pro (1G/10G) is $380 new, the SE which has 2.5G ports is $500 new. And $100 per Flex. So going by your right-hand plan, 2x MS108EUP, 2x Flex, and a UDM Pro SE would be $1580 at MSRP, pre-tax and shipping. Going the same but with a GW-R86S-G1 instead of the UDMP-SE gets you $1395 minimum (price I see for GW-R86S-G1 is ~$315).

The left hand plan pencils out to $1140 or $955 since it only has one MS108EUP.
I think I get the first part but will take some time to digest it to make sure my understanding is correct.

Regarding the rest, I currently have an Archer C7 located in the corner of my house. It reaches everywhere in the house but the farthest rooms are a bit slow. My driveway and yard have a poor signal and may or may not work at random.

  • APs: Based on my current network performance I think a single centrally located AP with strong signal will probably be all I need. The upcoming EAP770/780 are at the top of my list right now and I've also been looking at the EAP660HD, WAX630(E), and similar models. I'm more concerned with range than speed but many spec sheets and reviews don't cover this well so I'm making what is perhaps an incorrect assumption that based on size, power requirements, and suggested use case that these models will have stronger signals and better sensitivity. How any of these compare to my C7 I honestly don't know. I was going to try 1 and if it's not enough I can reposition it and use 2 or 3.
  • Neighbor signals are reasonably low and not intrusive. I have a Zigbee network for automation and have selected appropriate channels for zigbee and 2.4ghz wifi to avoid frequency overlap.
  • The majority of usage is phones, tablets, laptops, and streaming TVs. The heaviest load would be if I had a large file download or NAS transfer to a laptop while the kids had all the TVs and other stuff going on at the same time.
  • Most of my devices are gigabit and that is likely to stay the same but I definitely want multigig between switches. I can and do saturate gigabit LAN now. 1g/10g is an option but if I'm spending a lot I also want to consider that it's very likely my next computer, NAS, or other hardware will be 2.5g or higher.
  • Normal price of the MS108EUP is $440 but it's widely available right now for just $280. Subtract the cost of a couple poe++ injectors and it's a steal.
 

Sean Ho

seanho.com
Nov 19, 2019
768
352
63
Vancouver, BC
seanho.com
Part of LodeRunner's point is that it's fairly unlikely that your APs can actually pull more than a gigabit over the wire, so you can feed them with a $120 gigabit PoE switch like ICX6450 or Aruba S2500.

You have (or plan for) 6x cams, 3x APs, 4x wired clients, and a router. You could do all this with a single 6450, or a pair if you need to have a separate switch for the office. The cams and APs would get gigabit PoE+ (30W); the wired clients and router would get SFP+ 10GbE.

SFP+ 10GbE switches and NICs are much cheaper and more abundant than 10GbaseT, let alone 2.5GbaseT. I have several machines with onboard 10GbaseT; I still use SFP+/QSFP+ add-on NICs with them. I do empathise that many consumer boards are limited on PCIe slots.
 

BrockSamson

New Member
Dec 7, 2022
8
0
1
Yes, but the real issue is that both passthrough switches for the cameras and many of the higher end APs like the EAP770/780 APs require 60w 802.3bt poe++ and not 30w 802.3at poe+. So at $50+ per gigabit injector any switch or switches that don't have poe++ automatically add $200 to the price to make it work.

Otherwise regular gigabit switches are fine as long as I have a multi-gig link between the primary switch in the equipment room and my office. There are enough devices on both sides of that which talk to each other that a gigabit link would not always be sufficient. It only works now because everything is in the same room and a few things are directly connected.