Router on a Stick -vs- Layer 3 Routing with Switch

[PGlover]

I am a newbie to the world of networking; however, I have been educating myself by reading a lot of articles on the internet. I am in the process of redesigning my home network with the attempt to design it like a small business network with different VLANs on the local network as well as on the DMZ network. Currently I am using Pfsense as my router/firewall/internet gateway/and DHCP server.

I have 3 Juniper EX3300-24T Layer 3 switches, 2 Quanta LB4M, and 1 Quanta LB6M switch. The burning question is whether to use my Layer 3 switches for inter-VLAN routing and DHCP or use the Pfsense router to route between VLANs and to the internet (Router on a Stick). The goal is to achieve the best performance possible with the least amount of configuration.

I plan to post my proposed network design in the next couple of days but I need to determine first how routing between VLANs will occur.

FYI... Pfsense is installed on WatchGuard XTM 515 appliance.

 

[CreoleLakerFan]

The best design for performance is to do your intra-VLA routing on your switches and use the firewall for external routing/filtering.

There are, however, plenty of people who use firewalls for l3 functions - I have done two separate network migrations in the last three months where L3 functions were moved off a 6509 core into a set of PA firewalls. I can certainly understand migrating away from legacy power hog 6509s, but I find moving routing and igp from dedicated silicon to firewalls infuriating. I just don't see why anyone would want to cripple the performance of their network like that while reducing the efficacy of their security by using firewalls as a router.

 

[Scott Laird]

For small networks, there's something to be said for reducing complexity by putting all routing configuration in one place. Especially if you have a mix of networks with different security policies. Complexity is quite likely a bigger issue than performance in this sort of case.

I have a similar setup to the poster, with Juniper EX4200 switches and an SRX for a firewall. I'm routing on both devices, with OSPF between them, but that's (a) probably not a great choice for someone without a lot of network experience and (b) expensive with EX3300s, where OSPF is an extra-cost license.

 

[Drewy]

I do all my routing on pfsense, but then I use vlans to segregate traffic types. So where I do route I also firewall and run snort.

 

[PGlover]

Attached is my initial network design based on the Layer 3 switch performing inter-VLAN routing. Once again, I am not a network guru at all, so please don't laugh at the design. I welcome all feedback.

The one question I do have is how to handle vlan routing between vlans on the DMZ switch and LAN switches. Does the pfsense router handle the routing or do I need to have a trunk port between the 2 switches. Is it best practice to allow communication between the DMZ and LAN network?

 

[CreoleLakerFan]

The one question I do have is how to handle vlan routing between vlans on the DMZ switch and LAN switches. Does the pfsense router handle the routing or do I need to have a trunk port between the 2 switches. Is it best practice to allow communication between the DMZ and LAN network?

You want to filter traffic between your DMZ and internal LAN using a firewall. If your DMZ is compromised you want to minimize your risk - a firewall will do that for you, provided it is properly configured to filter traffic. A trunk between DMZ and LAN is a security "no-no."

 

[PGlover]

You want to filter traffic between your DMZ and internal LAN using a firewall. If your DMZ is compromised you want to minimize your risk - a firewall will do that for you, provided it is properly configured to filter traffic. A trunk between DMZ and LAN is a security "no-no."

Thank you so much... I will use the firewall capability in the pfsense box to filter traffic between the DMZ and internal LAN...

Any other feedback on the design so far?

 

[CreoleLakerFan]

Thank you so much... I will use the firewall capability in the pfsense box to filter traffic between the DMZ and internal LAN...

Any other feedback on the design so far?

If all your DMZ hosts are VMs running on your ESX hypervisor they can access the storage network through the same backend path your LAN VMs are using. You will need a separate physical uplink for your Internet traffic from your DMZ switch to a virtual switch on your ESX host, but your 10GbE network is massive overkill. As far as I can tell, you only really need four 10GbE connections in your network - two at your ESX host and two at your storage server.

It looks like you are planning on using two 10GbE SFP's on each of your EX3300's to link them together. Instead, you should configure them as a virtual chassis (stack) which will increase bandwidth between your EX3300's (80Gb per VC cable - 2 max in your configuration), simplify management, and save you four 10GbE edge ports for later use. If all you have is these two hosts, you don't really need the Quanta.

Juniper VC cables run about $35 each on eBay.

 

[PGlover]

It looks like you are planning on using two 10GbE SFP's on each of your EX3300's to link them together. Instead, you should configure them as a virtual chassis (stack) which will increase bandwidth between your EX3300's (80Gb per VC cable - 2 max in your configuration), simplify management, and save you four 10GbE edge ports for later use.

For the EX3300, there is no backplane cabling to create the virtual chassis. It instead uses the last two 10GbE SFP+ ports to connect the 2 switches together. So I will only have a total of four 10GbE SFP+ ports that can used in the virtual chassis configuration.

Virtual Chassis Cabling Configuration Examples for EX3300 Switches - Technical Documentation - Support - Juniper Networks

Juniper EX3300 Virtual Chassis Question • /r/networking

 

[PGlover]

If all your DMZ hosts are VMs running on your ESX hypervisor they can access the storage network through the same backend path your LAN VMs are using. You will need a separate physical uplink for your Internet traffic from your DMZ switch to a virtual switch on your ESX host, but your 10GbE network is massive overkill. As far as I can tell, you only really need four 10GbE connections in your network - two at your ESX host and two at your storage server.

The SAN server (Windows Server 2012 R2 with Storage Spaces configured) is used to serve up ISCSI LUNs/targets as datastores for my VMs in the DMZ and internal network as well as serve up Window file shares for physical and virtual computers connected to the DMZ and internal network. I have ripped a lot of movies and stored them on the SAN server and use Kodi (XMBC) media streamers connected to the internal network to view the content. I wanted to isolate the iSCSI traffic on its own storage network as well.

Let me know if the design needs to change based on my comments. If I have too many 10GbE connections, please let me know where so I can adjust my design..

I am learning a lot about networks. Once again, thanks for the input.

 

[Blinky 42]

Unless your ex3300's are in different locations or you need a lot of 1G ports, you could also cut down to just one or 2 there. You can let the VLANs provide the separation of networks and not dedicate physical hardware to each purpose.

I would concur with the sentiment of let the switches do L3 between VLANs as long as it is just routing and you are not trying to do packet filtering, firewall or NAT. Do those functions on your pfsense box.

The stacking on ex3300's is ok if you need it for something specific but there is no great speed advantage as the ports are still 10G (they can't go up to 25G or anything since they are older Broadcom silicon). The stacking would be helpful if you were going to spread your LAGs across multiple physical switches to keep the links alive if you loose a switch, but that is overkill unless you just want to do it for experience with Junos.
If you want higher throughput, make your quanta the core switch and then connect the ex3300's to that acting as edge switches breaking out your 10G to 1G.

If you only have 2 boxes with > 1G ports, then all of this is overkill until you add more. If you want to save power just get a pair of 40G nics and connect the VM and SAN servers directly and omit the Quanta and only have enough ex3300's to meet your 1G port needs.

 

[PGlover]

If you want higher throughput, make your quanta the core switch and then connect the ex3300's to that acting as edge switches breaking out your 10G to 1G.

As far as the core switch, are you talking about making the Quanta LB4M or LB6M the core switch? The Quanta switches do not have layer 3 routing capability.

 

[CreoleLakerFan]

For the EX3300, there is no backplane cabling to create the virtual chassis. It instead uses the last two 10GbE SFP+ ports to connect the 2 switches together. So I will only have a total of four 10GbE SFP+ ports that can used in the virtual chassis configuration.

Virtual Chassis Cabling Configuration Examples for EX3300 Switches - Technical Documentation - Support - Juniper Networks

Juniper EX3300 Virtual Chassis Question • /r/networking

Oh.

 

[gigatexal]

Gah I am so noobish here all of this sounds like gibberish to me after the vlan stuff.

 

[CreoleLakerFan]

The SAN server (Windows Server 2012 R2 with Storage Spaces configured) is used to serve up ISCSI LUNs/targets as datastores for my VMs in the DMZ and internal network as well as serve up Window file shares for physical and virtual computers connected to the DMZ and internal network. I have ripped a lot of movies and stored them on the SAN server and use Kodi (XMBC) media streamers connected to the internal network to view the content. I wanted to isolate the iSCSI traffic on its own storage network as well.

Let me know if the design needs to change based on my comments. If I have too many 10GbE connections, please let me know where so I can adjust my design..

I am learning a lot about networks. Once again, thanks for the input.

Typically people use "storage networks" for segments where high performance, low-latency access is desirable. For example, if you are hosting flash based-ESX datastores on a NAS, you would want to use 10Gbps between your ESX host and your storage platform. Other good candidates would be for replication between NAS or high availability (HA/vmotion/DRS) between ESX hosts.

But you don't need 10Gbps to serve media to your kodi clients, 1Gbps is more than enough for that.How many edge/client ports do you need in total?

 

[mumford]

This is a "home" network? I am doing all my firewalling, routing, QoS, Vlans, subnets, switching on a single Mikrotik CCR router. Simplify, I say.

 

[PGlover]

Typically people use "storage networks" for segments where high performance, low-latency access is desirable. For example, if you are hosting flash based-ESX datastores on a NAS, you would want to use 10Gbps between your ESX host and your storage platform. Other good candidates would be for replication between NAS or high availability (HA/vmotion/DRS) between ESX hosts.

But you don't need 10Gbps to serve media to your kodi clients, 1Gbps is more than enough for that.How many edge/client ports do you need in total?

I need about twenty eight 1Gbps ports.

My goal was to design the network like a small business network with the ability to scale up easily to a med size business network.

 

[CreoleLakerFan]

This is a "home" network? I am doing all my firewalling, routing, QoS, Vlans, subnets, switching on a single Mikrotik CCR router. Simplify, I say.

For my home I have an ESXi box with everything virtualized: NAS, firewall, Ubuquiti management, media servers, media capture - the works. A 24 port switch and a Unifi AC AP make up my network infrastructure. Simple.

My lab, however, is a different story. I've got a full 22U rack in the garage and a full 12U rack in the office. I had about 1/3 of this crap before I discovered STH.

 

[PigLover]

I think you are generally on the right track - but you need to ask yourself a question: why does "routing between VLANs" become such an interesting question? Getting to the heart of that question will help you self-critique your network design. And since you appear to be doing this for the learning the self-critique will be more valuable than just specifically commenting on details.

I believe that in most cases there needs to be a good reason behind each VLAN, a reason that it exists. And if you get these reasons "right" there will actually be limited traffic flowing between VLANs at all - and the traffic that does cross will almost always be subject to filtering/firewall/IDS needs. So, at the end of the day, if you find enough traffic flowing between VLANs that switch-level performance is even interesting to consider then you might need to re-consider the design itself. While there are exceptions to everything, in most cases, you probably want to do the inter-VLAN routing in your router/firewall rather than using a lot of layer-3 functions in the switches themselves.

Again - every rule has its exceptions and there are good reasons you might want to do some layer-3 work in your switch. But IMNSHO, if you find yourself doing this you should know and really understand the reason for it. Because if the reason is simply that "I have a lot of traffic between these two VLANs" you really ought to rethink why they are segregated into VLANs in the first place.

In well thought out networks there are a few good reasons to use VLAN:

• To segregate traffic into security zones (i.e., DMZ vs internal or RED zone/Black zone designs)
• To simulate multiple physical switches when resources don't allow physical separation (i.e., separating your "storage" network from your "workstation" network or your "VM Migration" network)
• To separate the network into functional zones (the "front side" web access vs "back side" database access or to separate "bearer path" from "control plane" from "administrative access")
• To separate different "tenants" from seeing each other's traffic (i.e., tenant networking in an openstack cluster)
• etc.

When constructed for these reasons it quickly becomes obvious that (1) there is limited traffic that should ever be crossing between VLANs and (2) the traffic that does cross really needs filter, firewall, IDS and perhaps logging.

Again - be self-critical of your approach. Ask yourself why this is even an interesting question (e.g., instead of asking yourself "whats the best way to do inter-VLAN routing?" perhaps you should be asking yourself "why do I need so much inter-VLAN routing...is this even a good use of VLAN?"). Perhaps there are good reasons for doing so much inter-VLAN traffic - but you should really understand those reasons and be able to defend them (at least to yourself) before you get too gung-ho on how ti implement it.

 

[PGlover]

I think you are generally on the right track - but you need to ask yourself a question: why does "routing between VLANs" become such an interesting question? Getting to the heart of that question will help you self-critique your network design. And since you appear to be doing this for the learning the self-critique will be more valuable than just specifically commenting on details.

I believe that in most cases there needs to be a good reason behind each VLAN, a reason that it exists. And if you get these reasons "right" there will actually be limited traffic flowing between VLANs at all - and the traffic that does cross will almost always be subject to filtering/firewall/IDS needs. So, at the end of the day, if you find enough traffic flowing between VLANs that switch-level performance is even interesting to consider then you might need to re-consider the design itself. While there are exceptions to everything, in most cases, you probably want to do the inter-VLAN routing in your router/firewall rather than using a lot of layer-3 functions in the switches themselves.

Again - every rule has its exceptions and there are good reasons you might want to do some layer-3 work in your switch. But IMNSHO, if you find yourself doing this you should know and really understand the reason for it. Because if the reason is simply that "I have a lot of traffic between these two VLANs" you really ought to rethink why they are segregated into VLANs in the first place.

In well thought out networks there are a few good reasons to use VLAN:

• To segregate traffic into security zones (i.e., DMZ vs internal or RED zone/Black zone designs)
• To simulate multiple physical switches when resources don't allow physical separation (i.e., separating your "storage" network from your "workstation" network or your "VM Migration" network)
• To separate the network into functional zones (the "front side" web access vs "back side" database access or to separate "bearer path" from "control plane" from "administrative access")
• To separate different "tenants" from seeing each other's traffic (i.e., tenant networking in an openstack cluster)
• etc.

When constructed for these reasons it quickly becomes obvious that (1) there is limited traffic that should ever be crossing between VLANs and (2) the traffic that does cross really needs filter, firewall, IDS and perhaps logging.

Again - be self-critical of your approach. Ask yourself why this is even an interesting question (e.g., instead of asking yourself "whats the best way to do inter-VLAN routing?" perhaps you should be asking yourself "why do I need so much inter-VLAN routing...is this even a good use of VLAN?"). Perhaps there are good reasons for doing so much inter-VLAN traffic - but you should really understand those reasons and be able to defend them (at least to yourself) before you get too gung-ho on how ti implement it.

OK... I'm not going to get hung up on the "inter-VLAN routing" question. I will focus on the network design and truly what traffic separation I really need.

For example I need I need the following zones:
1. DMZ
2. Home Network
3. Home Lab
4. Management
5. Wi-Fi (maybe, I have a lot of Wi-Fi devices, 12 Sonos devices, etc.)
6. Guest Wi-Fi (maybe)

As you can see, I have a lot of equipment already including 2 Quanta LB4M switches and a SuperMicro JBOD server with 16 empty slots that is not shown on the drawing.

I probably have some opportunity to consolidate and sell some of the equipment.

Keep the feedback coming...