New Router Suggestions (Multi-Gig/10 GbE)

Parallax

Active Member
Nov 8, 2020
214
87
28
London, UK
Like we already said, security-wise a properly configured and patched almost anything will cover pretty much any cyber attack most homes will have.

So beyond that we're mostly really talking comfort, familiarity, extra polish, and non-functional requirements.

A perfectly valid non-functional requirement could be that you want to use something that you need to use at work or need to learn to get a better job. Or you'd like to support a project you respect. Maybe you need help, and the community around that particular product is great. Perhaps something is just very easy to use and you don't want to be messing with stuff all the time. All this is fine by me and I think all those reasons have value.
 

ddaenen1

New Member
Jul 7, 2020
17
1
3
If you do a few minutes of research you'll quickly see pfsense is not open source, and in fact I'd go so far as to say they have given the middle finger to open source and are actually hostile.
That is not even correct information. pfsense CE 2.6.0 is still opensource and whilst i am fully aware they would like to see people moving to pfsense+ (which is not opensource) they don't "make" you do it. As long as it serves my purpose and there are no fees attached to it, i will stay with it. In the end, we are all using software that is not open source because it is what we need or favor.
 

ReturnedSword

Active Member
Jun 15, 2018
520
221
43
Santa Monica, CA
Apologies for the late replies, I was quite busy in the last few days.

Just as a point of reference for anyone looking my M920q with an i5-8500T running Proxmox and various firewalls is pretty quick.
The T740 with the Ryzen V1756B might even be a little bit quicker if you compare their respective passmark numbers.

I can get line rate speed on my 2G symmetric fiber link with some basic traffic inspection like antivirus, antispyware, url filtering, sandbox, etc.
View attachment 22301

And I can still get over 1Gbps with basic inspection and SSL decryption turned on
View attachment 22302
I would expect any somewhat modern CPU to be able to handle NAT-only full line speed, though it's great to see pretty decent speed on IDS/IPS. I'm curious about what would be needed to get 5 Gbps with IDS/IPS...

At home on my OpenBSD router I'm running the local services (unbound and dhcpd) in the 'local' rdomain; the external interface runs in a separate rdomain so all routing/interaction between external and internal rdomain needs to be explicitly configured via PF.
On my main server at home as well as the "non-smartOS-hosts" in our company network, all services (e.g. 2nd DNS, nginx reverse-proxy, postgresql, zabbix, CUPS print server...) and especially all larger "services" and stuff that drags in tons of packages (nextcloud, emby...) always run in jails. This makes maintenance easier by several magnutudes. E.g. for (larger) upgrades just clone the jail and leave the original one running and unmodified until the upgrade is done & confirmed working. No more waiting for maintenace windows - swapping out the jails after upgrades takes a few seconds at most and can be easily scripted.

On our branch routers we are running smartOS as hypervisor and the dedicated WAN interface(s) is/are only connected to the VM running the OpenBSD router/firewall instance. All zones for local services are only connected to the local interfaces/VLANs they need to have access to/be accessed from.
Because smartOS always runs the hypervisor (KVM or bhyve) within a zone, if someone would manage to infiltrate the VM and break out of the hypervisor, he'd be even more confined within this almost completely empty zone...


I've been using jails/zones and ZFS for many years now and it *vastly* simplified maintenance, backups and day-to-day work.
Jails/zones have almost no overhead; same goes for ZFS snapshots - so you can just use them extensively and thus always have an "escape plan" at hand (even for the host thanks to boot environments).
Splitting everything up in several, well contained instances not only increases security, but also makes it easy to manage upgrades more flexible. I.e. you can keep the host and public-facing services always up-to-date but e.g. update the purely local services as you come by (which is exceptionally convenient for stuff like nextcloud or horde groupware, which blow up on updates rather regularly).
It makes it also much easier to get rid of or just try out software packages without polluting the whole host and maybe even interfering with running services.
My jailhosts usually only directly connect to the management-VLAN - other VLANs are only connected to jails running services for those networks. So there is no direct attack surface to the host from any "easily" accessible network (guests are in a completely separate VLAN and rdomain anyways).
The jailhosts also only run a minimal set of packages (vim, iocell, zfsnap, zabbix-agent, ssmtp and *maybe* a few site-specific ones), so they can be very easily rebuilt and are very low-maintenance, especially because they are following the quarterly package-branch. SmartOS is an immutable install anyways - so just plug the drives and an USB-stick containing a smartOS image in a new host and you're back in business.
Jails and zones (which includes VMs on smartOS) can be easily transferred via their zfs snapshots, which are already used for backups anyways - so I couldn't care less if a jailhost dies, as all jails can be "rebuilt" by simply zfs send|recv from the backup system to a new/existing jailhost. If you use the same notation for vlan interfaces (or just attach them to loopback interfaces) you wouldn't even have to touch the network configuration of the jails. Again - MUCH easier and sane than trying to pull config and user files from a "one-for-all" bare-metal-host...

TL;DR: using jails (and ZFS) extensively makes life much easier, upgrades very safe and increases security. Administrative overhead is minimal, especially compared to the madness of "everything on the same host".
I think for the immediate situation, I'll probably stick with running a "fat" firewall instance. It would be a big hassle to lose a core service due to tinkering if I hadn't been able to get it "right" yet. The scenario you shared is certainly intriguing, and I'm planning on testing that out in my homelab first before deploying a similar solution to make sure I have all the dependencies correct.
 

ReturnedSword

Active Member
Jun 15, 2018
520
221
43
Santa Monica, CA
Threat model.
I find a need to be realistic as much as possible, both at work and in the homelab. You're correct, even on my current pfSense setup, I am fairly confident things are locked down sufficiently that satisfies the threat model.

On the other hand, I also agree that it can be very interesting to dive deep and make your own pf rules, run everything in BSD Jails etc. but the ROI on that, even in hobby projects has been in such steep decline that you almost need to have a day job where you can cycle that knowledge in order to make practical use of it. Personally I've stopped doing that since I'd be mixing BSD knowledge at home with Linux knowledge at work a lot and it just doesn't fit all that well together from a muscle memory point of view.
I totally agree. As I get older, I have less and less time, and this is before having kids, which are in the plans. It used to be that weekends were completely free to do whatever I'd like. For a long time now, I have to deal with maintaining the home, gardening, handyman stuff for the rental property, etc. I have vastly less time, unfortunately.

I've had less and less exposure to BSD over the years coming from the peak in the university days and work immediately thereafter. It's just not really used anymore unfortunately in the enterprise. At the end of the day the purpose of the homelab is not only to be a hobby, but to develop/maintain skills for work purposes.

This is generally what it boils down to for most people. It's not that pfSense is suddenly going to drop your packets all over the floor or anything like that, and more about the community (or lack thereof) and the company that turned everything to shit.

It's not a true FOSS project, it mostly just wants you to buy their hardware, and it doesn't really care about home users, but keeps it patched and afloat for brand value. Not everyone has to care about any of that, but to be honest, if those things aren't important to a user, they might as well be running some random ZyXEL box or Cisco ASA, or their consumer CPE the home ISP hands out.
This is my understanding as well too, following pfSense and similar projects over the years. I don't really consider the "Plus" product to be that different from the CE product. The main interesting Plus offering to me is QAT capability, though that's one of those things that I have been lusting over for years but never got homelab hardware capable of QAT, so it's a moot point for the most part.

Like we already said, security-wise a properly configured and patched almost anything will cover pretty much any cyber attack most homes will have.

So beyond that we're mostly really talking comfort, familiarity, extra polish, and non-functional requirements.

A perfectly valid non-functional requirement could be that you want to use something that you need to use at work or need to learn to get a better job. Or you'd like to support a project you respect. Maybe you need help, and the community around that particular product is great. Perhaps something is just very easy to use and you don't want to be messing with stuff all the time. All this is fine by me and I think all those reasons have value.
My understanding is OPNsense's community is rather friendly/helpful, which is nice. On the other hand, I've been using pfSense for quite some time (since the project's initial build) and am familiar with it, so having a less friendly community doesn't matter to me.

The biggest values to me are a balance between familiarity, security, and features. I won't have any problem switching to OPNsense if there is a compelling "killer feature," but as it stands I probably will stick with pfSense.
 

ReturnedSword

Active Member
Jun 15, 2018
520
221
43
Santa Monica, CA
That is not even correct information. pfsense CE 2.6.0 is still opensource and whilst i am fully aware they would like to see people moving to pfsense+ (which is not opensource) they don't "make" you do it. As long as it serves my purpose and there are no fees attached to it, i will stay with it. In the end, we are all using software that is not open source because it is what we need or favor.
There are some who claim pfSense CE will be deprecated, though I haven't seen any concrete details about that. I'm aware of the bad blood between the pfSense/OPNsense devs, and tbh I find the perceived fanaticism of the OPNsense community to be as distasteful as the pfSense community's elitist mentality. At the end of the day it comes down to familiarity, features, and security.

What I'd really love to see is a NG firewall distro that's "unified" or "cohesive," rather than a firewall distro with cobbled together NG packages. A unified dashboard pane would be nice, for example, as would an API so I can push stats/logs elsewhere. Regretably, outside of the proprietary firewall products, I haven't seen such a distro yet in OSS.
 

ReturnedSword

Active Member
Jun 15, 2018
520
221
43
Santa Monica, CA
Is anyone running a X710-T4L in a TMM that can report if the temperatures are ok?

I'm not familiar with SFP+ NICs beyond connecting via DAC to a fiber switch. Many SFP+ switches can use copper transceivers to convert to copper multi-gig... can the same copper transceivers be used in SFP+ NICs such as a X520-DA2 or ConnectX-3?
 
Last edited:

ReturnedSword

Active Member
Jun 15, 2018
520
221
43
Santa Monica, CA
Here are some hardware scenarios I've come up with:

2.5 Gbps Only

$408 - Topton N6005 Quad i225V appliance [link]:
  • $277 - Topton barebones
  • - Intel N6005 2.0GHz/3.3GHz; 4513 PassMark
  • $81 - Crucial CT2K8G4SFS832A 2x8GB DDR4-3200
  • $50 - 500GB Western Digital Blue SN570
  • - Quad i225V (integrated)
Copper Multi-gig Quad Port TMM

$1,021 - HP T740 Plus w/ X710-T4L:
  • $300.00 - HP T740 (used)
  • - AMD V1756B 3.3GHz/3.6GHz; 8212 PassMark
  • $81 - Crucial CT2K8G4SFS832A 2x8GB DDR4-3200
  • $50 - 500GB Western Digital Blue SN570
  • $590 - Intel X710-T4L (new)

$1,171 - Lenovo M720q w/ X710-T4L:
  • $450.00 - Lenovo M720q (used)
  • - Intel 8700T 2.4GHz/4.0Ghz; 10466 PassMark
  • $81 - Crucial CT2K8G4SFS832A 2x8GB DDR4-3200
  • $50 - 500GB Western Digital Blue SN570
  • $590 - Intel X710-T4L (new)

$1,221 - Lenovo M920q w/ X710-T4L:
  • $500.00 - Lenovo M920q (used)
  • - Intel 8700T 2.4GHz/4.0Ghz; 10466 PassMark
  • $81 - Crucial CT2K8G4SFS832A 2x8GB DDR4-3200
  • $50 - 500GB Western Digital Blue SN570
  • $590 - Intel X710-T4L (new)

SFP+ Dual Port TMM (Requires immediate switch upgrades)

$851 - HP T740 Plus w/ X710-DA2:
  • $300.00 - HP T740 (used)
  • - AMD V1756B 3.3GHz/3.6GHz; 8212 PassMark
  • $81 - Crucial CT2K8G4SFS832A 2x8GB DDR4-3200
  • $50 - 500GB Western Digital Blue SN570
  • $150 - Intel X710-DA2 (used)
  • $270 - MikroTik CRS309-1G-8S+IN

$1,001 - Lenovo M720q w/ X710-DA2:
  • $450.00 - Lenovo M720q (used)
  • - Intel 8700T 2.4GHz/4.0Ghz; 10466 PassMark
  • $81 - Crucial CT2K8G4SFS832A 2x8GB DDR4-3200
  • $50 - 500GB Western Digital Blue SN570
  • $150 - Intel X710-DA2 (used)
  • $270 - MikroTik CRS309-1G-8S+IN

$1,051 - Lenovo M920q w/ X710-DA2:
  • $500.00 - Lenovo M920q (used)
  • - Intel 8700T 2.4GHz/4.0Ghz; 10466 PassMark
  • $81 - Crucial CT2K8G4SFS832A 2x8GB DDR4-3200
  • $50 - 500GB Western Digital Blue SN570
  • $150 - Intel X710-DA2 (used)
  • $270 - MikroTik CRS309-1G-8S+IN
 
Last edited:
  • Like
Reactions: spikeb and Parallax

zer0sum

Well-Known Member
Mar 8, 2013
713
368
63
Last edited:
  • Like
Reactions: Parallax

zer0sum

Well-Known Member
Mar 8, 2013
713
368
63
Apologies for the late replies, I was quite busy in the last few days.

I would expect any somewhat modern CPU to be able to handle NAT-only full line speed, though it's great to see pretty decent speed on IDS/IPS. I'm curious about what would be needed to get 5 Gbps with IDS/IPS...
It's hard to test, without something like a breaking point load testing appliance because when you run a speed test for instance it's just single stream, and single cpu core.

I'm not sure if any CPU will give you 5Gbps of single stream SSL decrypt and IDS/IPS throughput, but multi-stream would be a different story.

I've cobbled together a VERY rough spreadsheet, using single core passmark to get a guesstimate of performance of the TMM boxes that will take a PCIe card. - TinyMiniMicro CPU / throughput speeds
 
  • Like
Reactions: ReturnedSword

Vesalius

Active Member
Nov 25, 2019
206
151
43
This is my understanding as well too, following pfSense and similar projects over the years. I don't really consider the "Plus" product to be that different from the CE product. The main interesting Plus offering to me is QAT capability, though that's one of those things that I have been lusting over for years but never got homelab hardware capable of QAT, so it's a moot point for the most part.
If you get this hardware, QAT support is available via GUI menu selection in OPNsense 22.1. I don't think netgate intends to make it available in CE, per last I checked. By design, netgate stated that differences between CE and Plus would be relatively small at first with more customer-facing changes later. At the current pace of releases from them, it will likely be a while before things shift. If you are more comfortable there, then no reason to change now or really ever and you will have other alternatives available should you come to that conclusion later.

I expect CE to stagnate and slowly wither relative to Plus, but obviously, I may be dead wrong. No hurry for anyone using CE either way though.
 

zer0sum

Well-Known Member
Mar 8, 2013
713
368
63
If you get this hardware, QAT support is available via GUI menu selection in OPNsense 22.1. I don't think netgate intends to make it available in CE, per last I checked. By design, netgate stated that differences between CE and Plus would be relatively small at first with more customer-facing changes later. At the current pace of releases from them, it will likely be a while before things shift. If you are more comfortable there, then no reason to change now or really ever and you will have other alternatives available should you come to that conclusion later.

I expect CE to stagnate and slowly wither relative to Plus, but obviously, I may be dead wrong. No hurry for anyone using CE either way though.
As you said QAT is already available with OPNsense as well as a couple of other hardware accelerated options
1649603667414.png
 

oneplane

Active Member
Jul 23, 2021
238
114
43
A bit of a redundant reply but yet, QAT is available by default in OpnSense if you want to try that out. Works both in the netmap part of the system as well as in VPN connections. One of the issues with QAT and other acceleration techniques is that it's usually "useful" when your CPU would otherwise not be able to forward packets fast enough, but modern CPU cores can handle it quite well.

The topton boxes work really well, same quality as Qotom stuff. For the i225V, it is very important to get the B3 stepping, but I think they advertise that really clearly that they do in fact use the B3 chip version. Works well with BSD and Linux.

<rant>
The N6xxx series CPU cores are pretty fast, essentially i3 performance with a few instructions removed. I hope Intel just makes those based on binning with broken features fused off because at this point the whole CPU line-up they have is just bonkers and pointless. Atom, Celeron, Pentium, Core i3, i5, i7, i9, Xeon W-series, E-series, etc... Gold, Silver, Platinum. It's just dumb market segmentation at this point. If they would just roll i9 back into i7, Pentium back into i3, and Celeron back into Atom that would make more sense. Or just drop the whole name-based segmentation and make it a true gradient.</rant>
 

ReturnedSword

Active Member
Jun 15, 2018
520
221
43
Santa Monica, CA
It's hard to test, without something like a breaking point load testing appliance because when you run a speed test for instance it's just single stream, and single cpu core.

I'm not sure if any CPU will give you 5Gbps of single stream SSL decrypt and IDS/IPS throughput, but multi-stream would be a different story.

I've cobbled together a VERY rough spreadsheet, using single core passmark to get a guesstimate of performance of the TMM boxes that will take a PCIe card. - TinyMiniMicro CPU / throughput speeds
I can’t think of a near future scenario where I would need a single stream at 5 Gbps, so good point. Looks like squid has been multi-threaded for a long while now so that should be more than sufficient for my needs.
 

ReturnedSword

Active Member
Jun 15, 2018
520
221
43
Santa Monica, CA
If you get this hardware, QAT support is available via GUI menu selection in OPNsense 22.1. I don't think netgate intends to make it available in CE, per last I checked. By design, netgate stated that differences between CE and Plus would be relatively small at first with more customer-facing changes later. At the current pace of releases from them, it will likely be a while before things shift. If you are more comfortable there, then no reason to change now or really ever and you will have other alternatives available should you come to that conclusion later.

I expect CE to stagnate and slowly wither relative to Plus, but obviously, I may be dead wrong. No hurry for anyone using CE either way though.
Ah I wasn’t aware OPNsense has QAT support. It’s a nice to have though, because AFAIK it was mostly useful when network/edge appliances were on the older, lower performance Atom arches. Nowadays even with a modern Tremont coreQAT probably isn’t as useful.

NetGate has made it clear that the feature set between CE and Plus will diverge over time, but they also said that a long time ago. As of right now the feature set divergence isn’t really meaningful from what I’ve seen. It would be suicide if NetGate diverges too much though on core features as a big reason why they can sell Plus appliances to SMBs is the fact they have a large community of IT people who use CE personally at home. Otherwise a business would just go with proprietary firewalls from bigger vendors. Synology got too big for their britches in the last 2 years and have been trying to push first party hardware, to immense blowback for example.
 

ReturnedSword

Active Member
Jun 15, 2018
520
221
43
Santa Monica, CA
The topton boxes work really well, same quality as Qotom stuff. For the i225V, it is very important to get the B3 stepping, but I think they advertise that really clearly that they do in fact use the B3 chip version. Works well with BSD and Linux.

<rant>
The N6xxx series CPU cores are pretty fast, essentially i3 performance with a few instructions removed. I hope Intel just makes those based on binning with broken features fused off because at this point the whole CPU line-up they have is just bonkers and pointless. Atom, Celeron, Pentium, Core i3, i5, i7, i9, Xeon W-series, E-series, etc... Gold, Silver, Platinum. It's just dumb market segmentation at this point. If they would just roll i9 back into i7, Pentium back into i3, and Celeron back into Atom that would make more sense. Or just drop the whole name-based segmentation and make it a true gradient.</rant>
I’m actually considering buying the Topton box just to play with. I do believe newer manufacture date hardware uses the B3 stepping i225V controllers. Topton has it advertised as such.

I fully agree on Intel’s aggressive market segmentation, often to artificially segment feature sets.

That being said, the “11th gen” Atoms are actually a totally different architecture - Tremont, which was the “little” core of mobile Lakefield, Intel’s one off first attempt at hybrid CPUs. They are also present in Snow Ridge which is an edge optimized CPU for 5G sites. Tremont’s performance uplift is pretty large compared to “traditional” Atom, which ended with the immediate predecessor Goldmont/Goldmont Plus. The current Atom core is Gracemont, which is the little core in Alder Lake.

I was trying to hold out for full Gracemont, which apparently will be Alder Lake-N (no big cores, all little cores). It’s supposed to have up to 8
Gracemont cores. ADL-N was paper launched over a month ago, but no sight of it yet.
 

ReturnedSword

Active Member
Jun 15, 2018
520
221
43
Santa Monica, CA
I ended up buying the Topton N6005 box so that I can have something familiar to keep the network going.

Perhaps later this year, when I’m planning to do network upgrades I’ll transition to a 10G capable router as I’ll have the 10G switch ready to go by then. Possibly I’ll start buying stuff and playing around with a 10G capable router this summer.
 

oneplane

Active Member
Jul 23, 2021
238
114
43
That box should be powerful enough to allow you to switch firewall distributions on the fly in a virtual environment :D Heck, that's pretty much what was posted on the STH frontpage (using Proxmox).
 

tozmo

Active Member
Feb 1, 2017
101
64
28
72
An HP 290 would be a bit larger but still ecnomical and efficient. I use it with gigabit for my pfsense instance.
 

ReturnedSword

Active Member
Jun 15, 2018
520
221
43
Santa Monica, CA
Something to report here on the Topton box. I inquired why it’s taking so long to ship out, and Topton replied that N6005 supply is still insufficient, so the next batch they’ll get will be May 15, well over a month after my order. They have in stock quantities of the N5095 and N5105 versions, though.
 
  • Like
Reactions: Zinc64