New Router Suggestions (Multi-Gig/10 GbE)

ReturnedSword

Active Member
Jun 15, 2018
526
226
43
Santa Monica, CA
I switched to OPNsense from pfsense for my main router, still have both running in VM's though. OPNsense has been more stable for me. OPnsense latest is already running on freebsd 13 (no longer using hardenedBSD) while pfsense has not made that change yet on plus or CE. If CLI is a consideration VyOS should be considered. Linux based and better overall throughput than either of the *sense's on my equipment.

I believe AFAIK, that the main bonus of BSD is that it has a smaller codebase than Linux, and therefore provides a smaller attack surface. I'm not opposed to running Linux (all my non-main workstation stuff in my homelab is on Linux) if the feature set is similar/better.

Many years ago I was running Vyatta, but it was a bit complicated and I was unwilling to learn it at the time. For my SMB clients we had standardized around m0n0wall/pfSense. Is VyOS much better than what Vyatta was?
 

oneplane

Active Member
Jul 23, 2021
297
161
43
Keep in mind that at the end of the day, your threat model might not really be all that intense to the point where even a basic OpenWRT installation would be sufficient security-wise.
 
  • Like
Reactions: Vesalius

Vesalius

Active Member
Nov 25, 2019
224
160
43
Is VyOS much better than what Vyatta was?
Never ran Vyatta directly so can't comment other than to say that after reading posts from people that have run both they state it's better.

The attack vectors on either *sense or VyOS are not sufficiently different to worry about IMO. Both battle-tested in the enterprise on server, routers and switches, Linux more so than FreeBSD these days.
 

ReturnedSword

Active Member
Jun 15, 2018
526
226
43
Santa Monica, CA
Never ran Vyatta directly so can't comment other than to say that after reading posts from people that have run both they state it's better.

The attack vectors on either *sense or VyOS are not sufficiently different to worry about IMO. Both battle-tested in the enterprise on server, routers and switches, Linux more so than FreeBSD these days.
My first exposure to the Nixes was through university at Berkeley years ago (BSD), so I suppose it’s become habit. Since Solaris began to be less used in the enterprise, we have mostly switched to Linux aside from legacy Solaris systems. I have been concerned in recent years that development of the various BSD distros has mostly stalled. I’m open to Linux since I do use Debian and Arch on all my non-workstation systems. The main sticking point for me and I think @Parallax would agree, is that more admin load = less enjoyable experience with the homelab. I had high hopes for TNSR, but development there still seems shaky years later.
 

Vesalius

Active Member
Nov 25, 2019
224
160
43
That’s a shame. Losing the capabilities of a NG Firewall would be a big negative. Granted, the NG Firewall “features” in pfSense and OPNsense are a bit cobbled together and not at all an intuitive/cohesive product.
It will be available in 1.4 via a container approach. Doubt they go through the trouble of building it into VyOS itself with this podman paradigm shift in the works.

suricata-vyos
 
  • Like
Reactions: BoredSysadmin

Parallax

Active Member
Nov 8, 2020
260
113
43
London, UK
The main sticking point for me and I think @Parallax would agree, is that more admin load = less enjoyable experience with the homelab.
Yes, "this is the way."

I started an ISP in 1991 so I saw the pretty early days of the Internet; but nowadays inevitably I've ended up in senior management so I don't have the access to tinker at work I used to. I would love to be all leading-edge and my inner 20 year old would like to sneer at people who use CLIs, but I have enough to do at work that I want my home environment to be as simple and autonomous as possible. I don't need to come home from work to start a second job, so I swallow my pride and look for GUIs, simplicity of operations, automation, and flashy dashboards. ;) It's no bad thing, to be honest, it's taught me a heck of lot I would never otherwise have learned about.
 

zer0sum

Well-Known Member
Mar 8, 2013
719
387
63
Just as a point of reference for anyone looking my M920q with an i5-8500T running Proxmox and various firewalls is pretty quick.
The T740 with the Ryzen V1756B might even be a little bit quicker if you compare their respective passmark numbers.

I can get line rate speed on my 2G symmetric fiber link with some basic traffic inspection like antivirus, antispyware, url filtering, sandbox, etc.
1649006801960.png

And I can still get over 1Gbps with basic inspection and SSL decryption turned on
1649006914251.png
 
Last edited:

sko

Member
Jun 11, 2021
92
53
18
Ah, I wasn't aware OPNsense was doing that. Certainly having multiple services running provides a larger attack surface. How are you implementing this? In jails/VMs?
At home on my OpenBSD router I'm running the local services (unbound and dhcpd) in the 'local' rdomain; the external interface runs in a separate rdomain so all routing/interaction between external and internal rdomain needs to be explicitly configured via PF.
On my main server at home as well as the "non-smartOS-hosts" in our company network, all services (e.g. 2nd DNS, nginx reverse-proxy, postgresql, zabbix, CUPS print server...) and especially all larger "services" and stuff that drags in tons of packages (nextcloud, emby...) always run in jails. This makes maintenance easier by several magnutudes. E.g. for (larger) upgrades just clone the jail and leave the original one running and unmodified until the upgrade is done & confirmed working. No more waiting for maintenace windows - swapping out the jails after upgrades takes a few seconds at most and can be easily scripted.

On our branch routers we are running smartOS as hypervisor and the dedicated WAN interface(s) is/are only connected to the VM running the OpenBSD router/firewall instance. All zones for local services are only connected to the local interfaces/VLANs they need to have access to/be accessed from.
Because smartOS always runs the hypervisor (KVM or bhyve) within a zone, if someone would manage to infiltrate the VM and break out of the hypervisor, he'd be even more confined within this almost completely empty zone...

One point (and a major one) for me is the trade-off between administration time and time spent doing more enjoyable things (homelab tinkering). How admin-intensive is that approach for your work? It's one thing to have multiple resources keeping an eye on infrastructure, and another to have one/few people doing it all while keeping track of multiple pieces, which is where consolidation/fewer pieces of infrastructure becomes attractive. I completely agree with all your points though from a security standpoint though.
I've been using jails/zones and ZFS for many years now and it *vastly* simplified maintenance, backups and day-to-day work.
Jails/zones have almost no overhead; same goes for ZFS snapshots - so you can just use them extensively and thus always have an "escape plan" at hand (even for the host thanks to boot environments).
Splitting everything up in several, well contained instances not only increases security, but also makes it easy to manage upgrades more flexible. I.e. you can keep the host and public-facing services always up-to-date but e.g. update the purely local services as you come by (which is exceptionally convenient for stuff like nextcloud or horde groupware, which blow up on updates rather regularly).
It makes it also much easier to get rid of or just try out software packages without polluting the whole host and maybe even interfering with running services.
My jailhosts usually only directly connect to the management-VLAN - other VLANs are only connected to jails running services for those networks. So there is no direct attack surface to the host from any "easily" accessible network (guests are in a completely separate VLAN and rdomain anyways).
The jailhosts also only run a minimal set of packages (vim, iocell, zfsnap, zabbix-agent, ssmtp and *maybe* a few site-specific ones), so they can be very easily rebuilt and are very low-maintenance, especially because they are following the quarterly package-branch. SmartOS is an immutable install anyways - so just plug the drives and an USB-stick containing a smartOS image in a new host and you're back in business.
Jails and zones (which includes VMs on smartOS) can be easily transferred via their zfs snapshots, which are already used for backups anyways - so I couldn't care less if a jailhost dies, as all jails can be "rebuilt" by simply zfs send|recv from the backup system to a new/existing jailhost. If you use the same notation for vlan interfaces (or just attach them to loopback interfaces) you wouldn't even have to touch the network configuration of the jails. Again - MUCH easier and sane than trying to pull config and user files from a "one-for-all" bare-metal-host...

TL;DR: using jails (and ZFS) extensively makes life much easier, upgrades very safe and increases security. Administrative overhead is minimal, especially compared to the madness of "everything on the same host".
 

Parallax

Active Member
Nov 8, 2020
260
113
43
London, UK
This is all very fancy, and I can see that it's potentially easier to install and upgrade components since they're more loosely coupled. I also can see that it's potentially more secure than a monolithic entity in terms of each individual attack surface being smaller.

But over 70% of all enterprise breaches are through compromised credentials and much of the rest is through software issues like log4j, Solarwinds, etc. Hardly any are through the sort of hacking you see on movies where the attacker progressively breaks through the layers of defence in depth, it's not a good use of time and effort.

I'm not critiquing the edifice you've built here, but just saying to you from a security perspective there's minimal gain from it. I can see in your particular situation you get operational benefits from it.
 
Last edited:
  • Like
Reactions: oneplane

oneplane

Active Member
Jul 23, 2021
297
161
43
Yeah, I agree with your original comment that for almost any home scenario even OpenWRT (assuming it's up to date and correctly configured) would cover it.
On the other hand, I also agree that it can be very interesting to dive deep and make your own pf rules, run everything in BSD Jails etc. but the ROI on that, even in hobby projects has been in such steep decline that you almost need to have a day job where you can cycle that knowledge in order to make practical use of it. Personally I've stopped doing that since I'd be mixing BSD knowledge at home with Linux knowledge at work a lot and it just doesn't fit all that well together from a muscle memory point of view.
 

Parallax

Active Member
Nov 8, 2020
260
113
43
London, UK
It's been a long time since I had a job which required me to configure anything, so since my main responsibility is technical strategy, architecture, and governance I built and use my home lab over the last couple of years to work out how we can get from where we are now to where we need to be.

All that to say that I definitely see the value in putting the time in a home lab where it makes sense and you can learn from it, but I want as little admin overhead as possible and since my job revolves around risk appetite, RoI, security, etc I looked at the home perimeter and it just doesn't warrant that much effort, particularly as I don't run any outward facing services.
 
  • Like
Reactions: oneplane

ddaenen1

New Member
Jul 7, 2020
18
1
3
Definitely time to ditch pfsense and switch to opnsense :)
Why would you say that? Pfsense is great and works perfect. I am not jumping on the hype-train that one is better than the other and even less on the "frequenter updates and opensource" train. I also have looked at opnsense, which is just another pfsense fork in my view but i don't like the GUI very much and now i know my way around pfsense, i don't see any value add of switching over.
 

Sealside

Member
May 10, 2019
79
18
8
Stockholm/Sweden
I spent a lot of time setting up opnsense in the same way I have my pfsense set up, this was one year ago. I ran into several bugs, many related to dual wan with VPN. In the end I switched back.I'm very happy with pfsense but might give opnsense another try, I like the rest apI that is included, where as in pfsense you have to rely on another project.
 

zer0sum

Well-Known Member
Mar 8, 2013
719
387
63
Why would you say that? Pfsense is great and works perfect. I am not jumping on the hype-train that one is better than the other and even less on the "frequenter updates and opensource" train. I also have looked at opnsense, which is just another pfsense fork in my view but i don't like the GUI very much and now i know my way around pfsense, i don't see any value add of switching over.
If you do a few minutes of research you'll quickly see pfsense is not open source, and in fact I'd go so far as to say they have given the middle finger to opensource and are actually hostile.

Then there is the whole wireguard drama - Buffer overruns, license violations, and bad code: FreeBSD 13’s close call

If you want to keep supporting them, then go for it, but I'd rather spend my time and energy on something like opnsense
 

adman_c

Active Member
Feb 14, 2016
156
71
28
Chicago
If you do a few minutes of research you'll quickly see pfsense is not open source, and in fact I'd go so far as to say they have given the middle finger to opensource and are actually hostile.

Then there is the whole wireguard drama - Buffer overruns, license violations, and bad code: FreeBSD 13’s close call

If you want to keep supporting them, then go for it, but I'd rather spend my time and energy on something like opnsense
I've had nothing but good experiences with pfsense as a product. But as a company netgate seems pretty shitty. So for my new firewall I'm going to try opnsense.
 

oneplane

Active Member
Jul 23, 2021
297
161
43
I've had nothing but good experiences with pfsense as a product. But as a company netgate seems pretty shitty. So for my new firewall I'm going to try opnsense.
This is generally what it boils down to for most people. It's not that pfSense is suddenly going to drop your packets all over the floor or anything like that, and more about the community (or lack thereof) and the company that turned everything to shit.

It's not a true FOSS project, it mostly just wants you to buy their hardware, and it doesn't really care about home users, but keeps it patched and afloat for brand value. Not everyone has to care about any of that, but to be honest, if those things aren't important to a user, they might as well be running some random ZyXEL box or Cisco ASA, or their consumer CPE the home ISP hands out.