New Router Suggestions (Multi-Gig/10 GbE)

ReturnedSword

Active Member
Jun 15, 2018
517
220
43
Santa Monica, CA
I started off my work day today with my entire network going down. Checked my pFSense physical console and it was still on/not throwing any errors but the web GUI was no longer responding. The display out was still fine but once at the keys, it was clear the output was frozen. Quick reboot. Oh no? It’s not rebooting at all.

I took the thing apart. Perhaps it was a power switch issue? I tried reseating the switch cable then manually booting via shorting the jumpers. No dice. Perhaps power brick issue? It wasn’t booting once plugging in a spare 20 pin ATX cable as well. Reseated memory, changed RTC battery. Nothing worked.

I guess it finally gave up the ghost after 8 years of service.

My current pFSense is built around a Jetway NF9N-N2930 mITX board with a quad 82574L daughter board. It was sufficient for my needs as I haven’t moved back to >1 GbE fiber yet. My intention was to do so soon, as up to 5 GbE service is available now, but I didn’t get around to it.

I’ve been trying to brainstorm some ideas for a “compact” router for up to multi-gig/10 GbE seeing as ISP speeds here will hit that within the lifetime of a new router. Preferably it will use standardized parts if possible, but I’m not against custom FF.
  • I like the HP T740, but I need at a minimum 4 multi-gig NICs, and worried even a X710-T4L will bake inside due to the T740 only having a single system fan.
  • Another consideration is a boutique mITX case such as the Velka 3/5, which will allow standard parts to be used. More than adequate cooling here.
  • “Custom” mini PC FF, e.g. Topton JSL quad i225V box. Major cons are NICs can’t be upgraded, and it will be stuck with 2.5 GbE.
My requirements:
  • Small(-ish). 5L and below would be nice.
  • Bare metal. I don’t virtualize for core networking.
  • 4 NICs minimum: WAN, LAN, GUEST, IoT, DMZ.
  • Multi-Gig up to 10 GbE would be ideal. Although I can probably live with 2.5 GbE WAN until the pricing side becomes more reasonable.
  • Decent OpenVPN (75-100 Mbps)/WireGuard (300-500 Mbps) performance.
  • IDS/IPS. I have been running Snort for many years but should probably move to Suricata.
  • DNSBL. Via PfBlockerNG.
  • Squid.
  • Other stuff should probably be able to run on any hardware newer than the N2930.
I’d appreciate not having to spend a ton of cash on this if possible, though I’m not opposed to doing so if warranted. Preliminary budget <$500 USD.

Closing thought: I’ve been running pFSense since the project’s inception, in 2006; 16 years. Prior to that I ran the project it was forked from, m0n0wall. I appreciate that as an appliance distribution changes to pFSense have been slow, but it certainly feels like that in the last couple years at least, development has stalled. Any other suggestions for an OSS router/firewall distro that is more modern than pFSense (needs to be stable)?
 
Last edited:

Sealside

Member
May 10, 2019
79
18
8
Stockholm/Sweden
If this is for home use, why not going the vlan route. You should be able to get 3 nics. Two sfp+ and one rj45 (although realtek) and use your suggested t740.

I'm doing this setup with mgmt lan.on the rj45.
 

RTM

Well-Known Member
Jan 26, 2014
868
325
63
If this is for home use, why not going the vlan route. You should be able to get 3 nics. Two sfp+ and one rj45 (although realtek) and use your suggested t740.

I'm doing this setup with mgmt lan.on the rj45.
If you extend this to use a L3 switch for internal traffic routing, you minimize the amount of bandwidth that you firewall has to be able to deal with.

As it is, I have a feeling that the weak point is the requirement to do 2.5Gbe minimum while also doing IPS within the budgetary and size constraints.
I am not saying that it can't be done, just that IPS is not really lightweight, especially if you don't tweak the rules deployed.

The weak point in all this is that i don't really think there are good benchmarks, that can give an idea of what kind of performance you can get with different hardware platforms, while also factoring in IPS and proxying.
 

adman_c

Active Member
Feb 14, 2016
136
59
28
Chicago
I haven't actually assembled and put it into use yet, but I got a Thinkstation Micro m720q with a i3-8100t and 8/128 for $250 off ebay. A PCIE riser and a compact 2x SFP+ NIC were another $70. Plus the little plate for the back and I'm looking at under $350 for something that almost meets your requirements (only 3 NICs instead of 4). But like RTM said, do your internal routing on your switch and there's no need for 4 NICs. Heck, even if you do your routing on your firewall, you're probably fine with a single 10GB connection between your switch and firewall.

Like I said, I have not assembled/tested, and heat could be a problem, per your concern with the t740. I'm hopeful that with the relatively light load I put on it it'll be fine.
 

sth2100

Member
Feb 22, 2022
36
16
8
I'm with RTM on this. Seems like you are looking for an all-in-one solution that checks all the boxes. I'm sure it's out there, but they are usually expensive and when you have a problem with an all-in-one solution, it usually means the "all" is broken - no routing for internet, no switching for LAN.

I know it eats a bit more power, but I personally have tried to keep things modular so you can get something that does a very good job at one thing and if something breaks, you only lose that function and could even keep a cheap backup around just in case.

A good switch that manages 90% of your traffic with SFP+ or QSFP, or whatever, and then a solid router that simply focuses on the internet side only. The switch will carry the heavy load and the router just needs to be secure.
 

zer0sum

Well-Known Member
Mar 8, 2013
712
368
63
I'm with RTM on this. Seems like you are looking for an all-in-one solution that checks all the boxes. I'm sure it's out there, but they are usually expensive and when you have a problem with an all-in-one solution, it usually means the "all" is broken - no routing for internet, no switching for LAN.

I know it eats a bit more power, but I personally have tried to keep things modular so you can get something that does a very good job at one thing and if something breaks, you only lose that function and could even keep a cheap backup around just in case.

A good switch that manages 90% of your traffic with SFP+ or QSFP, or whatever, and then a solid router that simply focuses on the internet side only. The switch will carry the heavy load and the router just needs to be secure.
This x1000 :D

Let your switches do switching.
Let your firewall protect the edge.
Let your AP's do wireless

Combing any and all of those into one box inevitably leads to heartbreak.

Definitely time to ditch pfsense and switch to opnsense :)
 

oneplane

Active Member
Jul 23, 2021
238
114
43
Indeed time to get rid of pfSense and go to OpnSense. They forked a while back then it became clear the people running pfSense became (or already were) a bunch of d*cks. (and then there's the whole Wireguard/anti-community behaviour thing)

Alternatives do exist, for example VyOS, but you'll be doing CLI and REST interfacing instead of working with a GUI so you might not like that idea in which case OpnSense is the way forward.

Like the rest wrote: separate the duties of the devices and you'll be better off for it.
 

ReturnedSword

Active Member
Jun 15, 2018
517
220
43
Santa Monica, CA
I’m aware of the disagreements that caused the pfSense/OPNsense fork. My other gripe with pfSense is even well before NetGate took over the project, development seemed to be slowing down. The main reason for continuing in the years since is a bit lame, but it boils down to “I’m used to it.”

I used to follow OPNsense development early on though, and it seemed like that project had a lot of stability issues. Have those been ironed out for the most part?

I also really like pfBlockerNG. AFAIK there’s no straight equivalent on OPNsense. I’ve seen others report they cobbled something together, or they run a PiHole instead. I do have a Docker environment so that won’t be a big deal aside from not having it all on the firewall appliance for easier admin.

Aside from my current, now deceased, router managing the various networks I don’t do any major switching on the router side. I agree switches should be running most of this stuff, however I personally feel iffy about RoS. I’d much rather have the physical interfaces if possible.

For wireless, I haven’t had the router providing the wireless AP since I believe the SmoothWall days (this is before I got on m0n0wall, so very long time ago). My APs are ASUS RT-AC68U units.

I believe something in my OP may have not been clear. For my case I use the router/firewall as mostly that. Core switching is separate. My networks are flat so the switching needs aren’t very complicated tbh. I do appreciate the features of a NG Firewall though.

@RTM: Can you please share thoughts on IDS/IPS? My dead router has a N2930, so hardly a speed demon even when I was new. Even running Snort, Squid, pfBlockerNG, and the occasional VPN tunnel the CPU never went above let’s say 70% ish. Most of the time the CPU averaged around 30%, with minimal RAM usage.

Do you have any thoughts to share about hardware sizing for a 2.5 GbE, and possibly up to 5 GbE/10 GbE? I have no doubt straight NAT will not be a problem on even a slow CPU. The issue I wrestle with is the IDS/IPS and VPN speed side. Throughout the years I haven’t found any conclusive recommendations aside from “throw more hardware at it.”
 
  • Like
Reactions: Amrhn

newabc

Active Member
Jan 20, 2019
253
99
28
Another thing, @bmeeks' Suricata/Snort GUI is pretty good if the users subscribe the Snort rules or use the free ones.
 

sko

Member
Jun 11, 2021
53
31
18
If you don't need the shiny GUI of OPNsense/pfSense and want to have that box *only* handle firewalling/routing anyways, I'd highly suggest taking a look at vanilla FreeBSD or even better OpenBSD.

My main selling point for OpenBSD on a router/firewall are still routingdomains. They are simply awesome. You can segragate the interfaces on a single host in multiple RDs and handle them all completely separated. Only via rules in PF will any packets be able to travel from one RD to another, but otherwise the RDs are behaving just like multiple separate routers with their own routing tables that know nothing about the interfaces/networks/routes and even services in other RDs on the system.

We run OpenBSD on our firewall- and routing-VMs (and BGP reflectors) and branch gateways and it is soooo easy and sane to manage.
PF syntax is one of the sanest and most readable/understandable out there, so its several orders of magnitudes easier to define even complex rules and spotting errors in the rule logic than on other platforms/firewalls (let alone that abomination that is iptables). Same goes for OpenBGPd which uses a similar syntax, so BGP filters are dead-simple.

I've tested OPNsense several times over the years and also still have to maintain one installation at a homeoffice. IMHO the GUI always feels way too complicated and clunky if you know what actually goes on underneath and could solve that task with a few lines in a config file. I recall the horrors of setting up (or rather trying to...) a simple nginx reverse proxy and acme.sh /w DNS API on it - a task that would have taken ~15 lines of nginx config at most and a one-liner for the initial certificate plus one cronjob for the cert; yet over the GUI this took over an hour and was working only halfways minus the DNS API. I ended up including a nginx.conf.local in the butchered nginx.conf and setting everything up by hand in a matter of minutes; including also running the OPNsense webserver behind the nginx proxy (that handles TLS) and the LE validation via DNS API. has been running ever since...
So to sum up that rather long story: If you feel comfortable in a shell, just use the vanilla OS and software packages instead of a (often restrictive) GUI that tries to handle text-base config files.

I think OPNsense shines if you have absolutely NO experience/affection to CLI and firewalls and want to run a multi-purpose host that does about everything you need in your network, including things like webservers and a small NAS (although the latter rather crudely compared to e.g. TrueNAS).
Don't do that - keep it simple. The edge router/firewall IMHO should have the lowest attack surface of all devices in a network and the most streamlined installation and configuration. ESPECIALLY if you have proper servers on site anyways where everything else can (and should) be handled in dedicated VMs/jails/zones/etc...


Regarding the routing part: I absolutely support the suggestion to handle local routing on the switch.
To get the same throughput from commodity hardware, you usually have to put up a pretty beefy machine with multiple times the power consumption just to do something an ASIC in your L3 switch can easily do at line-speed at a fraction of the power consumption and cost. This becomes even more true if you want to route at >1Gbit or even >10GBit speeds.
If you want to get fancy (and your switch supports it), use ACLs and policy-based routing to restrict routes from/to VLANs/subnets (and please use a default-deny policy if you care about security!). To get even more fancy, combine that with OSPF (or BGP) fed from e.g. a route reflector and you have a pretty resilient local routing that could be easily extended e.g. to multiple uplinks or whole new network segments.
I usually configure a small subset of my PF rules (usually the most used/matched local rules) from the router of that network as ACLs on the switch, so the major part of the local routing takes place there. Everything else (e.g. all egress traffic, routing to/from more restricted VLANs etc) simply gets forwarded to the router via a 'set next-hop' and is decided there.


My suggestion for the IDS/DPI: if you have beefy servers on site anyways, run the heavy lifting of the IDS on one of those, not at the edge. This way even an overwhelmed IDS won't interfere with your normal routing tasks. The easiest way would be a mirror port that connects to the IDS VM and the IDS is feeding back to the PF firewall (via a dedicated VLAN on another interface or e.g. through the management VLAN both are also connected to). Again: keep the edge systems slick and simple, especially if you are on a budget.

As for hardware suggestions: Maybe take a look on used Xeon-D15xx systems. They are often available for <500$ in various configurations, and a lot of them come with the dual 10GBit directly from the SoC.
E.g. the Supermicro SYS-5018D or E300-*D are targeted towards this very use case and most of them even come with 4-8 1GBit NICs on board.
 
Last edited:

ReturnedSword

Active Member
Jun 15, 2018
517
220
43
Santa Monica, CA
If you don't need the shiny GUI of OPNsense/pfSense and want to have that box *only* handle firewalling/routing anyways, I'd highly suggest taking a look at vanilla FreeBSD or even better OpenBSD.
The GUI is just a "nice-to-have." I don't have an issue with a CLI, but admittedly my focus in IT is a bit more on the architectural/design side (I'm a PM/Architect). My exposure to CLI networking, or for that matter, programming, is more of along the lines of general high-level knowledge, making a PoC if necessary, making simple tools for the QC team (to make my own life easier), and keeping the devs/analysts honest. The last time I seriously took a look at a CLI-based router/firewall was many years ago (Linux IPTables).

I don't often log into my pfSense. After I set up rules and IDS/IPS, I leave it alone unless it breaks or I receive an alert.

I think OPNsense shines if you have absolutely NO experience/affection to CLI and firewalls and want to run a multi-purpose host that does about everything you need in your network, including things like webservers and a small NAS (although the latter rather crudely compared to e.g. TrueNAS).
Don't do that - keep it simple. The edge router/firewall IMHO should have the lowest attack surface of all devices in a network and the most streamlined installation and configuration. ESPECIALLY if you have proper servers on site anyways where everything else can (and should) be handled in dedicated VMs/jails/zones/etc...
I know quite a few people that run everything on a single appliance. I don't do that though. I'd prefer the router/firewall to always be running on bare metal on a dedicated appliance, which is where hardware sizing gets tough. I can always "throw more hardware" at it, but as can be imagined, things get expensive rather quickly. I'm not above spending money for a purpose, but for example, the pfSense which died has a Celeron N2930 and it has no issue with a cable internet WAN (pretty pokey-slow), pfBlockerNG, Snort, Squid, and the occassional VPN tunnel when I'm road-warrioring. The main thing I'd like to take care of when planning the new router/firewall is to try to be able to upgrade my internet to 5 GbE (available now), and potentially 10 GbE (available in the near future).

I have both an ESXi and Proxmox host for VM-related stuff (including Docker environment). My NAS(es) are also on dedicated hardware.

Regarding the routing part: I absolutely support the suggestion to handle local routing on the switch.
To get the same throughput from commodity hardware, you usually have to put up a pretty beefy machine with multiple times the power consumption just to do something an ASIC in your L3 switch can easily do at line-speed at a fraction of the power consumption and cost. This becomes even more true if you want to route at >1Gbit or even >10GBit speeds.
If you want to get fancy (and your switch supports it), use ACLs and policy-based routing to restrict routes from/to VLANs/subnets (and please use a default-deny policy if you care about security!). To get even more fancy, combine that with OSPF (or BGP) fed from e.g. a route reflector and you have a pretty resilient local routing that could be easily extended e.g. to multiple uplinks or whole new network segments.
I usually configure a small subset of my PF rules (usually the most used/matched local rules) from the router of that network as ACLs on the switch, so the major part of the local routing takes place there. Everything else (e.g. all egress traffic, routing to/from more restricted VLANs etc) simply gets forwarded to the router via a 'set next-hop' and is decided there.
Are you referring to routing across subnets? In general my networks are rather flat -- yes, occassionally there's a need to route across subnets, which was done on the pfSense that died, but it's not very common in my use cases. I'm a bit ashamed to admit that I haven't really overhauled my network environment in the last 10 years or so. In fact I'm still running a Netgear 24 port unmanaged switch as my main core switch. It was "business-class" when I purchased it for about $900 many, many years ago. My eventual goal is to get a new core switch, perhaps a Mikrotik 8-12 port, or a used Brocade. I had actually started looking around late last year. I wasn't successful yet bidding on Brocade switches on eBay, and Mikrotik switches seem to be perpetually sold out on Amazon.

My suggestion for the IDS/DPI: if you have beefy servers on site anyways, run the heavy lifting of the IDS on one of those, not at the edge. This way even an overwhelmed IDS won't interfere with your normal routing tasks. The easiest way would be a mirror port that connects to the IDS VM and the IDS is feeding back to the PF firewall (via a dedicated VLAN on another interface or e.g. through the management VLAN both are also connected to). Again: keep the edge systems slick and simple, especially if you are on a budget.
This is a very intriguing proposal. Currently I'm familiar with a combination of Snort, pfBlockerNG, Squid for IDS/IPS. What software would you suggest for IDS/IPS that would be run on a separate machine (or VM)?

As for hardware suggestions: Maybe take a look on used Xeon-D15xx systems. They are often available for <500$ in various configurations, and a lot of them come with the dual 10GBit directly from the SoC.
E.g. the Supermicro SYS-5018D or E300-*D are targeted towards this very use case and most of them even come with 4-8 1GBit NICs on board.
A Xeon D seems a bit overkill for my use case, but I may be wrong on this.
 

sko

Member
Jun 11, 2021
53
31
18
The GUI is just a "nice-to-have." I don't have an issue with a CLI, but admittedly my focus in IT is a bit more on the architectural/design side (I'm a PM/Architect). My exposure to CLI networking, or for that matter, programming, is more of along the lines of general high-level knowledge, making a PoC if necessary, making simple tools for the QC team (to make my own life easier), and keeping the devs/analysts honest. The last time I seriously took a look at a CLI-based router/firewall was many years ago (Linux IPTables).

I don't often log into my pfSense. After I set up rules and IDS/IPS, I leave it alone unless it breaks or I receive an alert.
Seriously - take a look at the pf.conf syntax and a few example configs. It's nowhere near that obscure iptables notation with its chains and other (unnecessary complex/weird) abstractions:
Code:
pass quick proto tcp from $lan to !<localnets> port { http, https, ssh } tag EGRESS

I know quite a few people that run everything on a single appliance. I don't do that though. I'd prefer the router/firewall to always be running on bare metal on a dedicated appliance, which is where hardware sizing gets tough. I can always "throw more hardware" at it, but as can be imagined, things get expensive rather quickly. I'm not above spending money for a purpose, but for example, the pfSense which died has a Celeron N2930 and it has no issue with a cable internet WAN (pretty pokey-slow), pfBlockerNG, Snort, Squid, and the occassional VPN tunnel when I'm road-warrioring. The main thing I'd like to take care of when planning the new router/firewall is to try to be able to upgrade my internet to 5 GbE (available now), and potentially 10 GbE (available in the near future).

I have both an ESXi and Proxmox host for VM-related stuff (including Docker environment). My NAS(es) are also on dedicated hardware.
I wasn't suggesting to virtualize on that edge system (although depending on use case / budget this is also perfectly OK), although with FreeBSD you can (and should) contain services (e.g. local DHCP, DNS...) in separate jails without any significant overhead. IIRC that's even what OPNSense is doing for a lot of services by default and it is just a good practice.

This is a very intriguing proposal. Currently I'm familiar with a combination of Snort, pfBlockerNG, Squid for IDS/IPS. What software would you suggest for IDS/IPS that would be run on a separate machine (or VM)?
I've only tested Snort so far, so I might not be the most helpful source for this topic. But IIRC snort can directly hook into PF, so you can easily forward everything to PF at the edge.

A Xeon D seems a bit overkill for my use case, but I may be wrong on this.
True - depending on what you want to aggregate on that system. Although you can go the virtualization route and run e.g. an OpenBSD VM and several zones for other essential services like DHCP, local DNS resolver and RADIUS on it. That's what we are doing on our branch gateways with this hardware (SYS-5018D-FN8T) and smartOS, and it has been a perfect and cost-effective solution for us.
Overhead from bhyve on networking is minimal and well within acceptable ranges - i.e. we can easily saturate our 500/500Mbit uplink with traffic from HQ + 4 branches on such a setup.
 

RTM

Well-Known Member
Jan 26, 2014
868
325
63
Aside from my current, now deceased, router managing the various networks I don’t do any major switching on the router side. I agree switches should be running most of this stuff, however I personally feel iffy about RoS. I’d much rather have the physical interfaces if possible.
I don't really like router on a stick either, but as an example if you were to use a T740 with a 2 port SFP+ NIC, then one could go to the WAN and the other your L3 switch, the RJ45 could be used as management. This would not be Router on a stick.

In a configuration like this, you will never have to route more traffic than 10G (full duplex of course and yes, I am ignoring the management port).

@RTM: Can you please share thoughts on IDS/IPS? My dead router has a N2930, so hardly a speed demon even when I was new. Even running Snort, Squid, pfBlockerNG, and the occasional VPN tunnel the CPU never went above let’s say 70% ish. Most of the time the CPU averaged around 30%, with minimal RAM usage.

Do you have any thoughts to share about hardware sizing for a 2.5 GbE, and possibly up to 5 GbE/10 GbE? I have no doubt straight NAT will not be a problem on even a slow CPU. The issue I wrestle with is the IDS/IPS and VPN speed side. Throughout the years I haven’t found any conclusive recommendations aside from “throw more hardware at it.”
I am sorry, that is what I was trying to get at mentioning benchmarks, I don't have a good feel of how to size hardware for these requirements other than as you say "throw more hardware at it".

Part of the problem is that it is configuration dependent, do you use Snort or Suricata? (Suricata *should* be more performant), how many rules do you have deployed on the IDS/IPS? (more rules = greater load), have you configured Squid to do decrypt TLS traffic?

That said, some of the vendors of firewall software (Netgate/pfSense and Opnsense to name a few) post benchmark results of the appliances they sell, this usually includes traffic routing, but at least in the case of Opnsense includes "Threat protection" (I assume this covers IPS). Their High-end DEC850 appliance support ~2Gbps "Threat protection". The DEC850 is based on the AMD EPYC3201 SoC, so perhaps using something with a EPYC3251 will give you what you ask for.

If I were to build something like what you mentioned using new hardware, then I would probably use something like a Ryzen 7 (8 core and high clock frequency) + asrock rack x570 mobo as a base. That way if it was not enough (or too much), I could use it for something else anyway :cool:
Used hardware I would probably be looking at a skylake Xeon-D or better (I have a feeling that you want to have a high clock frequency for this, and not just raw cores).

Regarding software, I have a few points to make:
  1. Assuming you want all in one appliances, you may want to look at Untangle and Sophos solutions as alternatives to pfSense/Opnsense, they are based on Linux and should perform a bit better (so you can make do with less hardware)
    1. Note: I am not saying they are good or anything, just that they could be alternatives worth investigating.
  2. You (OP) mentioned pfblockerng functionality is not present in Opnsense, I think you want to look at the Sensei add-on to get this (paid though).
 

Vesalius

Active Member
Nov 25, 2019
205
151
43
Any other suggestions for an OSS router/firewall distro that is more modern than pFSense (needs to be stable)?
I switched to OPNsense from pfsense for my main router, still have both running in VM's though. OPNsense has been more stable for me. OPnsense latest is already running on freebsd 13 (no longer using hardenedBSD) while pfsense has not made that change yet on plus or CE. If CLI is a consideration VyOS should be considered. Linux based and better overall throughput than either of the *sense's on my equipment.

You (OP) mentioned pfblockerng functionality is not present in Opnsense, I think you want to look at the Sensei add-on to get this (paid though).
He can also consider the OPNsense package Adguard home. I like it better than pfblocker and it's free.
 

Vesalius

Active Member
Nov 25, 2019
205
151
43
Pfblocker can do a lot more than adguard or pihole though.

If you want more advanced protection on Opnsense you can always look at Sensei Zenarmour
For my use case, adguard has been great and better for my fam. Allows for isolating specific Mac addresses or IP to certain blocklist and very granular blocking of specific apps and services as well per IP/mac. Have not missed pfblocker or found a need for a paid service yet, not saying there isn’t one especially for others.
 

ReturnedSword

Active Member
Jun 15, 2018
517
220
43
Santa Monica, CA
Seriously - take a look at the pf.conf syntax and a few example configs. It's nowhere near that obscure iptables notation with its chains and other (unnecessary complex/weird) abstractions:

Code:
pass quick proto tcp from $lan to !<localnets> port { http, https, ssh } tag EGRESS

I'll check this out.


I wasn't suggesting to virtualize on that edge system (although depending on use case / budget this is also perfectly OK), although with FreeBSD you can (and should) contain services (e.g. local DHCP, DNS...) in separate jails without any significant overhead. IIRC that's even what OPNSense is doing for a lot of services by default and it is just a good practice.

Ah, I wasn't aware OPNsense was doing that. Certainly having multiple services running provides a larger attack surface. How are you implementing this? In jails/VMs?


I've only tested Snort so far, so I might not be the most helpful source for this topic. But IIRC snort can directly hook into PF, so you can easily forward everything to PF at the edge.

Snort/Suricata can be run as a service on a separate instance/VM/appliance. I can do that, though it complicates administration in the end, which is why a single appliance for Firewall/IDS/IPS is attractive to me (and many others). It's a trade-off I suppose in administration and further hardening the infrastructure.


True - depending on what you want to aggregate on that system. Although you can go the virtualization route and run e.g. an OpenBSD VM and several zones for other essential services like DHCP, local DNS resolver and RADIUS on it. That's what we are doing on our branch gateways with this hardware (SYS-5018D-FN8T) and smartOS, and it has been a perfect and cost-effective solution for us.

Overhead from bhyve on networking is minimal and well within acceptable ranges - i.e. we can easily saturate our 500/500Mbit uplink with traffic from HQ + 4 branches on such a setup.

One point (and a major one) for me is the trade-off between administration time and time spent doing more enjoyable things (homelab tinkering). How admin-intensive is that approach for your work? It's one thing to have multiple resources keeping an eye on infrastructure, and another to have one/few people doing it all while keeping track of multiple pieces, which is where consolidation/fewer pieces of infrastructure becomes attractive. I completely agree with all your points though from a security standpoint though.
 

ReturnedSword

Active Member
Jun 15, 2018
517
220
43
Santa Monica, CA
I don't really like router on a stick either, but as an example if you were to use a T740 with a 2 port SFP+ NIC, then one could go to the WAN and the other your L3 switch, the RJ45 could be used as management. This would not be Router on a stick.


In a configuration like this, you will never have to route more traffic than 10G (full duplex of course and yes, I am ignoring the management port).

You're right, since the maximum offered speed from the ISP at the moment is 5000/5000, even a single SFP+ port would have no issue, even at half-duplex, handling all upstream traffic from the downstream connected switch. My conundrum is as of now, I haven't upgraded my core switch yet and am stuck with a big 24-port unmanaged switch/smaller 8-port unmanaged switch. My future plan is to upgrade to a used Brocade/Mikrotik, but I haven't had the time to actively/daily take a look at eBay for the Brocade, and Mikrotik seems to be having supply issues on Amazon (and even official resale partners).


I am sorry, that is what I was trying to get at mentioning benchmarks, I don't have a good feel of how to size hardware for these requirements other than as you say "throw more hardware at it".


Part of the problem is that it is configuration dependent, do you use Snort or Suricata? (Suricata *should* be more performant), how many rules do you have deployed on the IDS/IPS? (more rules = greater load), have you configured Squid to do decrypt TLS traffic?


That said, some of the vendors of firewall software (Netgate/pfSense and Opnsense to name a few) post benchmark results of the appliances they sell, this usually includes traffic routing, but at least in the case of Opnsense includes "Threat protection" (I assume this covers IPS). Their High-end DEC850 appliance support ~2Gbps "Threat protection". The DEC850 is based on the AMD EPYC3201 SoC, so perhaps using something with a EPYC3251 will give you what you ask for.

Yes, this is the perennial problem for us homelabbers I suppose. It's even an issue at work for every company I've been with. It's either: Trust the vendor's word on performance (if they had tested it), or throw hardware at it (and have to test it ourselves anyway). I suspect I will be fine with a AMD V1756B, or any 4-core and up Skylake-derivative. The higher clock speed the better, though I'm willing to go with less if the power consumption is quite high. Electricity isn't exactly cheap here, especially after the nuclear plant nearby was decommissioned.


I do use Snort, though I've been meaning to move to Suricata since it's multi-threaded. I don't have a huge amount of Snort rules. Squid does decrypt TLS in my current config.


I do think the higher-end appliances sold by pfSense/OPNsense probably are overkill for me. Everyone appreciates more fancy hardware hah, but I do hope not to fall into the trap of throwing too much hardware at the thing (though from past experiences, I probably will, surely many here can sympathize with this).


If I were to build something like what you mentioned using new hardware, then I would probably use something like a Ryzen 7 (8 core and high clock frequency) + asrock rack x570 mobo as a base. That way if it was not enough (or too much), I could use it for something else anyway :cool:

Used hardware I would probably be looking at a skylake Xeon-D or better (I have a feeling that you want to have a high clock frequency for this, and not just raw cores).

This would be perfect, though I'd probably stick with 35W CPUs and below. That was my idea for using a Velkase (boutique) Velka 3 case, X710-T4L, and a Ryzen 7. I just checked Velkase the other day though and sadly they are not currently doing any production runs for that particular case. Skyreach also stopped production of the 4 Mini as well :(. Though It would be a big bigger than I'd like, using standard motherboards and PCIe cards is a huge plus in re-usability and upgrade ability later.


Regarding software, I have a few points to make:

Assuming you want all in one appliances, you may want to look at Untangle and Sophos solutions as alternatives to pfSense/Opnsense, they are based on Linux and should perform a bit better (so you can make do with less hardware)
Note: I am not saying they are good or anything, just that they could be alternatives worth investigating.
You (OP) mentioned pfblockerng functionality is not present in Opnsense, I think you want to look at the Sensei add-on to get this (paid though).

I looked into Sophos previously, and know a few people that swear by it, but I wasn't that impressed sadly. Same for Untangle.


As I'm considering OPNsense, I'd need to look into Sensei. I utilize PFBlockerNG quite a bit, so it'd be tough to part with that functionality.