Home network topology suggestions?

[phekno]

My current setup, is one "server", around 30 IoT devices (e.g. switches, outlets, TVs, game consoles, etc.), 10-15 "trusted" devices (laptops, phones, tablets), and 4 cameras (all for 5 people). Right now, the various devices are all on their own, respective VLAN ("Main", IoT, Video), and I'm doing router-on-a-stick from my Ubiquiti EdgeSwitch 16 (which apparently doesn't do L3 very well) to a Ubiquiti EdgeRouter 4, over a 1-gig connection between the two. Thus far, the router doesn't seem to be bogged down or anything.

I've recently added 4 actual servers, a 15-bay enclosure and drives, and a Brocade ICX6610. One server is TrueNAS hooked to the external enclosure, one is for serving things (e.g. Plex/Jellyfin, NZBGet, Sonarr, Radarr, etc.). Of the other two, ONE is going to be an OPNSense machine, and I'm not sure what to do with the other one (Windows AD?). The EdgeSwitch 16 will be replaced by the Brocade ICX6610, and the EdgeRouter 4 will be replaced by the OPNSense machine, with a 10-gig connection between it and the Brocade.

Part of me wants to keep router-on-a-stick, like I have it now, but I realize that I won't learn anything new, nor will I be taking full advantage of what my newly-acquired switch is "good" at (better than my old one, anyway). The other part of me wants to lean on the capabilities of the switch. So, do I go full inter-VLAN, or keep with router-on-a-stick?

The other thing I'm wondering about, is, if I go the inter-VLAN route, how much do I punt to the switch? Do I run DHCP on the switch? Are there any good guides to doing this sort of thing?

 

[Drewy]

Don’t run dhcp on the switch, it sucks. Look at something like isc-dhcp. I run it as a container. You also don’t have to punt all the l3 routing to the switch. You could still use your existing router (and I assume fw) to take care of the untrusted iot type clients.
you can setup acl’s on the switch and (probably) advanced acl’s but these may not offer the protection you currently have in place.
I’ve got all “trusted” device l3 happening on the switch (stack of 7250’s) and the untrusted or not trusted as much still going through opnsense - which is the default router for the l3 config.
which ever way you go ensure you have a simply (as possible) way back. undoubtedly you’ll have a few challenges along the way and may find yourself in a bit of a pickle, for a while

you should find all the info you need in the monster brocade thread.

 

[Blue)(Fusion]

My humble recommendation:

• Avoid router-on-a-stick.
• Use an ISC or AD DHCP server that works across VLANs appropriately (pf/OPNSense's DHCP server does not).
• Don't overdo it with your router hardware. I'm running a 100/10Mbps edge link on OPNSense for NAT, firewall, fq_codel shaping, and Sensei on 4GB and a dual-core Pentium E6500, and 1Gbps link to ICX6610 no problem (specifically a Check Point T-160).
• Learn the capabilities of the switch.

 

[phekno]

My humble recommendation:

• Avoid router-on-a-stick.
• Use an ISC or AD DHCP server that works across VLANs appropriately (pf/OPNSense's DHCP server does not).
• Don't overdo it with your router hardware. I'm running a 100/10Mbps edge link on OPNSense for NAT, firewall, fq_codel shaping, and Sensei on 4GB and a dual-core Pentium E6500, and 1Gbps link to ICX6610 no problem (specifically a Check Point T-160).
• Learn the capabilities of the switch.

Yeah, I believe you replied to me in the Brocade mega-thread saying "don't use router-on-a-stick", which prompted this post.

I have some research to do...

 

[phekno]

Don’t run dhcp on the switch, it sucks. Look at something like isc-dhcp. I run it as a container. You also don’t have to punt all the l3 routing to the switch. You could still use your existing router (and I assume fw) to take care of the untrusted iot type clients.
you can setup acl’s on the switch and (probably) advanced acl’s but these may not offer the protection you currently have in place.
I’ve got all “trusted” device l3 happening on the switch (stack of 7250’s) and the untrusted or not trusted as much still going through opnsense - which is the default router for the l3 config.
which ever way you go ensure you have a simply (as possible) way back. undoubtedly you’ll have a few challenges along the way and may find yourself in a bit of a pickle, for a while

you should find all the info you need in the monster brocade thread.

The router hardware I've acquired is a Supermicro based board with an E5-1270v2 and 8GB RAM. I have a CenturyLink symmetric gigabit connection to the internet. The machine is a little low on RAM, but from a compute standpoint should be OK.

Either way, I have a lot of learning to do.

 

[Blue)(Fusion]

That Supermicro server should be plenty for OPNSense with IDS/IPS, firewall, NAT routing, and traffic shaping.

For starters, plan out your network before you start building.

Which VLANs, which devices on each VLAN, which subnet on which VLAN, and which device will be on which switch port.

A simple ICX6610 VLAN config for example:

vlan 1000 name Management
  tagged eth 1/1/24 eth 1/3/1
  untagged eth 1/1/48
  router-interface ve 1000
  exit

vlan 1010 name Trusted
  tagged eth 1/1/2 to 1/1/47 eth 1/3/1
  router-interface ve 1010
  exit

vlan 1020 name IoT
  tagged eth 1/1/24 eth 1/3/1
  router-interface ve 1020
  exit

vlan 1030 name Servers
  untagged eth 1/3/2 to 1/3/5
  router-interface ve 1030
  exit

ip route 0.0.0.0/0 10.199.199.1
ipv6 route ::/0 eth 1/1/1 fe80:xxxx:yyyy:zzzz

interface eth 1/1/1
  port-name Router Uplink
  route-only
  ip address 10.199.199.2/30
  exit

interface eth 1/1/2 to 1/1/47
  dual-mode 1010
  exit

interface eth 1/1/24
  port-name WiFi
  dual-mode 1000
  exit

interface eth 1/1/48
  port-name Emergency Mgmt
  exit

interface eth 1/3/1
  port-name Uplink-to-another-switch
  exit

interface ve  1000
  ip address 10.0.0.1/24
  ip helper-address 1 10.0.30.5
  exit

interface ve 1010
  ip address 10.0.10.1/24
  ip helper-address 1 10.0.30.5
  exit

interface ve 1020
  ip address 10.0.20.1/24
  ip helper-address 1 10.0.30.5
  exit

interface ve 1030
  ip address 10.0.30.1/24
  ip helper-address 1 10.0.30.5
  exit

This is a very basic example missing quite a bit and just making some random assumptions. The ip helper-address is forwarding DHCP requests to a specific DHCP server that will hand out IPv4 addresses, gateway and DNS information for all hosts on all VLANs.

In this scenario, port 1/1/1 would be the only connection to OPNSense and OPNSense's LAN port should be configured with the address 10.199.199.1/30. In OPNSense, you would add 10.199.199.2 as a gateway, and then route 10.0.0.0/16 (or as determined best for your IP subnet range) via 10.199.199.2 (the ICX). All hosts should be assigned the address of the appropriate VE address of the VLAN the device lies in. For example, a "smart fridge" might should be in the 10.0.20.0/24 subnet and have a default gateway of 10.0.20.1 (the ICX).

Getting IPv6 working takes a bit of work but is definitely possible. I'm still learning more about IPv6's gotchas but pretty happy with the results I've gotten thus far.

Getting ACLs to block traffic between VLANs is also very possible, and again takes a bunch of trial and error. Do not even bother with thus until you get the network up and running well!

The most common thing forgotten with this L3-on-switch setup and pfSense or OPNSense is setting up NAT from the extra IP subnets that the firewall is not aware of. Remember, you only set the address of 10.199.199.1/30? As far as it is concerned, it only needs to NAT 2 possible addresses to the WAN IP until you clone and add Outbound NAT rules for 10.0.0.0/16 (or as appropriate).

 

[phekno]

That Supermicro server should be plenty for OPNSense with IDS/IPS, firewall, NAT routing, and traffic shaping.

For starters, plan out your network before you start building.

Which VLANs, which devices on each VLAN, which subnet on which VLAN, and which device will be on which switch port.

A simple ICX6610 VLAN config for example:

vlan 1000 name Management
  tagged eth 1/1/24 eth 1/3/1
  untagged eth 1/1/48
  router-interface ve 1000
  exit

vlan 1010 name Trusted
  tagged eth 1/1/2 to 1/1/47 eth 1/3/1
  router-interface ve 1010
  exit

vlan 1020 name IoT
  tagged eth 1/1/24 eth 1/3/1
  router-interface ve 1020
  exit

vlan 1030 name Servers
  untagged eth 1/3/2 to 1/3/5
  router-interface ve 1030
  exit

ip route 0.0.0.0/0 10.199.199.1
ipv6 route ::/0 eth 1/1/1 fe80:xxxx:yyyy:zzzz

interface eth 1/1/1
  port-name Router Uplink
  route-only
  ip address 10.199.199.2/30
  exit

interface eth 1/1/2 to 1/1/47
  dual-mode 1010
  exit

interface eth 1/1/24
  port-name WiFi
  dual-mode 1000
  exit

interface eth 1/1/48
  port-name Emergency Mgmt
  exit

interface eth 1/3/1
  port-name Uplink-to-another-switch
  exit

interface ve  1000
  ip address 10.0.0.1/24
  ip helper-address 1 10.0.30.5
  exit

interface ve 1010
  ip address 10.0.10.1/24
  ip helper-address 1 10.0.30.5
  exit

interface ve 1020
  ip address 10.0.20.1/24
  ip helper-address 1 10.0.30.5
  exit

interface ve 1030
  ip address 10.0.30.1/24
  ip helper-address 1 10.0.30.5
  exit

This is a very basic example missing quite a bit and just making some random assumptions. The ip helper-address is forwarding DHCP requests to a specific DHCP server that will hand out IPv4 addresses, gateway and DNS information for all hosts on all VLANs.

In this scenario, port 1/1/1 would be the only connection to OPNSense and OPNSense's LAN port should be configured with the address 10.199.199.1/30. In OPNSense, you would add 10.199.199.2 as a gateway, and then route 10.0.0.0/16 (or as determined best for your IP subnet range) via 10.199.199.2 (the ICX). All hosts should be assigned the address of the appropriate VE address of the VLAN the device lies in. For example, a "smart fridge" might should be in the 10.0.20.0/24 subnet and have a default gateway of 10.0.20.1 (the ICX).

Getting IPv6 working takes a bit of work but is definitely possible. I'm still learning more about IPv6's gotchas but pretty happy with the results I've gotten thus far.

Getting ACLs to block traffic between VLANs is also very possible, and again takes a bunch of trial and error. Do not even bother with thus until you get the network up and running well!

The most common thing forgotten with this L3-on-switch setup and pfSense or OPNSense is setting up NAT from the extra IP subnets that the firewall is not aware of. Remember, you only set the address of 10.199.199.1/30? As far as it is concerned, it only needs to NAT 2 possible addresses to the WAN IP until you clone and add Outbound NAT rules for 10.0.0.0/16 (or as appropriate).

This is incredibly helpful. Through my reading, I had managed to piece a lot of it together, but IMO sometimes it's easier to conceptualize when there's an actual code or config.

As far as my VLANing is concerned, I already have (with my Ubiquiti setup):

• Management - VLAN1 - 10.100.1.1/24
• Main - VLAN10 - 10.100.10.1/24
• Guest - VLAN20 - 10.100.20.1/24
• IoT (i.e. untrusted) - VLAN30 - 10.100.30.1/24
• Video (i.e. security cameras) - VLAN40 - 10.100.40.1/24

(I don't know why I picked 10.100.0.0/16, I just did, I guess)
I'll probably stick with that schema, but I may add the "servers" VLAN, like you suggested. I was struggling with the question of if they belong on management or main, anyway, so putting them on their own VLAN makes sense. I wonder, though, does the Ubiquiti Unifi CloudKeyG2+ (the thing that manages the APs and cameras) belong on the management VLAN or the server VLAN?

Anyway, I remember reading somewhere that you shouldn't leave the management VLAN as VLAN1, which you've shown here. Is that true?

Presumably, any port connected to a WiFi AP will need to be dual-mode for all VLANs it's carrying traffic for, right (so, in my case, 10, 20, 30)?

For now, I'll probably leave ACLs alone, until I can figure out whether or not I really need them. I think the biggest problem with putting an ACL between IoT and everything else, is that I have stuff on main that needs to talk to IoT (e.g. a Denon receiver) and stuff that's on IoT that needs to talk to Main (e.g. Nvidia Shield TV). With my current setup I have that working (involving mDNS repeater and some firewall rules). But again, this kinda brings up the question of whether or not some of those devices belong on that particular VLAN. Stuff like smart switches, outlets, thermostat, fridge, etc. I can totally see belonging on there, but there are some "smarter" devices that maybe belong on Main? That's one of my big hangups, is probably second-guessing where things belong in the network.

THANK YOU A TON FOR YOUR HELP! I'm definitely still open to suggestions. My basement (where everything will land in a rack) is in the process of being finished, so I haven't completely gutted the network yet. For now I'm just testing and trying to figure out where I want things.

 

[Blue)(Fusion]

What devices you put on what VLAN comes down to what works best for you. I ended up with a policy of anything with an embedded OS goes on the IoT VLAN and pinholes are setup as required in ACLs. I left only devices that were necessary to operate the network and homelab on the Management VLAN (switches, DRACs, BMC, Unifi Controller and APs). It is recommended to not use VLAN 1 for anything. ACLs will filter inter-VLAN stuff and anything internet-bound is firewalled by your OPNSense.

Since you're not ready for deployment yet, it is a good time to load OPNSense on your hardware and connect your ICX6610, ensure it's running the latest firmware, licensed, and configure the routing and VLANs well before deployment to iron out issues.

 

[phekno]

So, I did actually follow "the guide" and have my 6610 all upgraded, licensed, etc.

Does anyone actually use the "management" interface? Or is that just kinda left alone? I mean...during the guide, it has you originally connect to it and the serial port, but then assign a static IP to port 1 and use that once you've upgraded. Is that normal?

 

[Blue)(Fusion]

As with most things, it's up to you to decide how to use the Management port. I set a static IP to each of my switches' Management ports and put a sticker on each indicating the IP (192.168.69.0/24 subnet). This allows me to connect directly to the Management port if I need to from a laptop. Additionally, if I decide on OOB management in the future, I can connect all the management ports to a dumb switch and have access to them all on an entirely different network not relying upon working configurations of the switches in question. Personally, I don't have a need for that option in my home network. I haven't used the Management ports since initial configuration.

 

[phekno]

WHEW! It took me a good part of yesterday, and most of today, but I managed to get it mostly working. Found what I thought were a lot of circular dependencies with regard to OPNSense and trying to configure it's interfaces. I only have 2 RJ45 1-gig ports, and 2 SFP+ 10-gig ports. The idea is/was to use ONE of the 1-gig ports as the WAN connection, and ONE of the 10-gig ports as the LAN connection...but then I couldn't configure the firewall because of things...it was a mess. I finally did get it figured out, though.

I have my ICX6610 doing inter-VLAN routing, my OPNSense machine (tho it's not connected to the Internet yet) is on firewall/NAT duties, and there's a Windows AD machine doing DHCP from the "Servers" VLAN. I can connect a client to a port that's assigned as untagged to a particular VLAN, and it gets an IP address from the correct scope. That client can even manage the firewall from there (which is something that I'll have to figure out, because I really only want things on any VLAN OTHER than "Main" to be able to manage the firewall).

I'm not sure I really want to be running Windows and AD, because I'm much more a Linux person, so that whole bit might be changing. If it does, then I have to figure out a good ad-blocking DNS server to run on Docker, and I'd probably lean in to ISC-DHCP for DHCP.

There's still a ton to figure out, specifically how my Ubiquiti crap is going to fit in, but I'll burn that bridge when I get to it.

Anyway, thanks for all your help! It definitely got me on the right track!

 

[Blue)(Fusion]

I run the Unifi Controller software on a barebones Ubuntu VM on Proxmox. It does nothing but run the Unifi controlle software. I'm also running 4 wireless networks - IoT, CCT, Guest, and Trusted which means in my case 5 total VLANs need to be tagged to the Unifi AP ports (management, plus the previously listed). The Management VLAN is the dual-mode VLAN on the AP switchports. Likewise the Unifi Controller VM is on the Management VLAN.

 

[phekno]

I run the Unifi Controller software on a barebones Ubuntu VM on Proxmox. It does nothing but run the Unifi controlle software. I'm also running 4 wireless networks - IoT, CCT, Guest, and Trusted which means in my case 5 total VLANs need to be tagged to the Unifi AP ports (management, plus the previously listed). The Management VLAN is the dual-mode VLAN on the AP switchports. Likewise the Unifi Controller VM is on the Management VLAN.

I had done some previous tinkering, trying to figure out how to "trunk" (in Cisco parlance) a port (i.e. pass more than 1 VLAN over it) and had managed to stumble my way through figuring out "dual-mode" and whatnot. Right now I have a similar set up for WiFi as far as VLANs are concerned (IoT, Guest, and Main). I think I'll get it all figured out.