Need Advise on Network & Hardware

[ambad4u]

greetings to all,

first, i have a hard time thinking of a proper title name...

i have this idea that has been bugging me for a long time...
i'm planning to create a home network (fruition might be a problem)
below image should provide adequate visuals on my 'networking' question(s)


we'll just 'assume' that the switch is capable of 10GbE, 2.5GbE and 1GbE
lets put an example that the NAS is 10GbE capable (green arrow)
some server(s) are also 10GbE capable (green arrow)
other server(s) might be 1GbE or 2.5GbE and might be under a sub switch (those in white arrows)


the "blue arrow" is my main question
i'm trying to create a pfSense/OPNSense/OpenWRT (leaning towards OpenWRT)
i'm torn between choosing:
-: option #1: "n5105 w/ 4x/6x intel 2.5GbE ports"
-: option #2: "any SFF & fitting it w/ 2.5GbE(s) & 10GbE(s) on its pcie ports"

would there be any penalty (network wise) since it is using only 2.5GbE to the 'main' switch (option #1)? VS going with 'option #2' w/ a 10GbE connection?

example, if i choose option #1, would there be 'bottlenecks' if NAS and other servers talk to each other?

thank you for your time reading my post.

 

[Rand__]

First,
white arrows on a white background don't work really well.

Second,
If you set up your network flat (single ip range, no vlans, no different subnets) no local traffic should ever hit the firewall. In that case the interconnect in blue totally depends on your internet connection as that is the only traffic that should go back and forth. So basically blue needs to be equal or greater then internet connection.

However, if you set up vlans or different subnets *and* you set up the firewall as router for those then all traffic thats supposed to pass through would need to traverse the blue arrow which would require a 10g connection if you want 10g. Based on your diagram I don't think you want to do that, it only would apply if the NAS and Server1 were on different subnets/vlans.

 

[ambad4u]

First,
white arrows on a white background don't work really well.

ah, my apologies...
i'm fixated with dark mode things, updated the image..., fixed.

However, if you set up vlans or different subnets *and* you set up the firewall as router for those then all traffic thats supposed to pass through would need to traverse the blue arrow which would require a 10g connection if you want 10g. Based on your diagram I don't think you want to do that, it only would apply if the NAS and Server1 were on different subnets/vlans.

so as long as any 2 devices communicates on the same subnet then..., noted.

anyways, if in the event i'll have devices communicate on different subnets (but still in my internal network that has not left to the internet), would there be a way it wont bottleneck?

 

[sic0048]

If you set the switch up as a layer 3 switch (where the VLAN management, routing and rules are on the switch, not the firewall), then really the only data going through the firewall is data entering/exiting the local network (ie to the internet). Setting this up like this requires a little more networking skill, and not every layer3 switch is created equal (I'm looking at you Brocade with your non-authoritative DHCP service).

Long story short, personally I think the easier way for us "non-IT professionals" to set up our home networks is to have the firewall manage the DHCP service, VLANs, routing and firewall rules, etc. This means that any traffic that needs to move from one VLAN to another will have to flow into and out of the firewall and through that "blue connection". Please note however that any traffic moving within the same VLAN is routed at the switch and doesn't have to traverse to the firewall.

This isn't the end of the world in a typical home network. I suspect that unless you have some unique "home lab" situation going on, you'll find it hard to saturate a 10gb even if all your data had to pass through the firewall. But you can greatly reduce the amount of data being transferred through the firewall simply by thinking about how you set up your VLANs. If your media server is going to pass large amounts of data, then simply put the media players on the same VLAN and none of that traffic will have to traverse to/from the firewall. If you are a video editor and are moving large files to/from your NAS, then just make sure the NAS and your workstation are on the same VLAN, etc. etc. etc.

That being said, I would make sure my firewall device had the ability to add/include a 10gb (or faster) connection, even if I didn't plan on using it immediately. If the device as a PCIe x4 gen3 (or better) slot, then you can easily add a 10gb network card if/when needed. Obviously you should also make sure your network switch also has at least a few 10gb (or better) ports as well.

EDIT - it should also be noted that using multiple 2.5gb connections (four 2.5gb ports in a LAGG for example) is not the same as a 10gb connection. Each 2.5gb connection is still limited to a max of 2.5gb of data, so if you are moving large files, multiple 2.5gb connections will still bottleneck more than a 10gb connection. Therefore I would suggest that you stick with "option 2" when considering hardware for your firewall.

 

[gaidin123]

sic0048's answer above is a good explanation of what you should consider in this scenario. I do your option #2 and for the blue arrow I'm using a mellanox cx311a in a pcie 3.0 x4 slot in an HP SFF, and a 2.5Gb intel NIC for the link to the ISP.

If you want to do any firewalling between your internal vlans (probably a good idea if you have a guest network or sketchy IoT devices), then you'd want your pfsense/firewall box to do the routing of the vlans, rather than the managed switches as this is what will force the traffic to cross the firewall rather than stay local to the switch. And that's where the possibility of a bottleneck comes in if you have a 10Gb home network and intend to do a bunch of inter-vlan traffic (media, IP camera/NVR, backups, VM migration, etc.).

It all comes down to how you decide to logically segment your devices. You can mitigate bottlenecks through that process as long as the bandwidth and vlan structure make sense or just go for 10Gb for the blue arrow to start. The flexibility offered by a low power SFF with a few pcie slots won me over. Your option #1 is my backup plan if the SFF suddenly dies though. I'd go for an n100 one of those now if you don't already have an n5105 box lying around.

 

[ambad4u]

the 'clouded' things that i thought before are starting to clear up now due to the superb of insights given!!!
i have solid footing now to go with SFF 'ish builds and attach 10GbE pcie nic(s) on them.

thank you all!