5Gbps Routing - home internet

[Sardaan]

Long time forum lurker. Recently upgraded to 5Gbps home internet service (they buried fiber throughout our neighborhood) and I need a wired router to handle the load. The handoff from the ONT is 10Gbps RJ45 (not fiber), I have 10Gbps and 2.5Gbps switches but I need a router capable of the load. I am willing to build my own and use pfsense or openwrt although I have had trouble finding hardware requirements. I would like to keep it as low budget as possible, buying used hardware off ebay is fine. We work from home and need a reliable solution.

Some nice to have features:
Quiet
Low Power
Shelf mountable

Your thoughts are appreciated.

 

[Railgun]

TNSR Overview

TNSR software is high-performance router with industry-leading price-performance, scalability, and flexibility for edge, data center, and cloud deployments

www.netgate.com

 

[jdnz]

something x86 based, able to route that kind of load and small/quiet/shelf mountable is going to be very hard - maybe having a look at something like the mikrotik rb5009 instead

 

[Simius]

I think a Mikrotik RB5009UG+S+IN is probably as low as you can go, for an all in one package. You'll need a SFP+ RJ45 module though.
Why isn't the handoff fiber?

 

[Sardaan]

I think a Mikrotik RB5009UG+S+IN is probably as low as you can go, for an all in one package. You'll need a SFP+ RJ45 module though.
Why isn't the handoff fiber?

I have SFP's, the Adtran ONT is a SDX621 it only has a single 10Gbps RJ45 handoff. I like the price of the Microtik and the features of the netgate 6100 although I have heard finding a compatible copper SFP can be an issue. The only thing really holding me back from building my own is the long term power usage.

 

[jdnz]

I think a Mikrotik RB5009UG+S+IN is probably as low as you can go, for an all in one package. You'll need a SFP+ RJ45 module though.
Why isn't the handoff fiber?

also since the 5009 only has a single 10g sfp+ port you'll need to send the ont's 10g to your switch and use vlans/trunking to run the 5009 in a 'router on a stick' topology ( no problem doing this since the internet service is only 5gbps )

 

[jdnz]

I have SFP's, the Adtran ONT is a SDX621 it only has a single 10Gbps RJ45 handoff. I like the price of the Microtik and the features of the netgate 6100 although I have heard finding a compatible copper SFP can be an issue. The only thing really holding me back from building my own is the long term power usage.

building an opnsense/pfsense/tnsr appliance capable of routing 10gbps is quite do-able - however the NIC requirements plus the CPU requirements ( you'll need both decent single core speed AND a resonable core count ) means it's hard to achivie it in a small/low power/shelf-mountable system like you want

either you need to live with a larger/noisier/more power hungry system - or you need to look at off-the-shelf solutions like mikrotik or opnsense/pfsenses own-brand hardware

 

[newabc]

For 5Gbps, if routing, NAT, Suricata and VPN are needed, for lowing the hardware cost, noise and the cooling's cost, they should be separated into 3-4 individual machines like:
(1) routing+NAT
(2) Suricata IDS
(3) VPN server

 

[neb50]

I switched to the Google 5Gb service a few months ago and use the Mikrotik CCR2004-16G-2S+ with one of the lower power 10Gbe to SFP+ modules. I get around 5.7G down and 8G up most of the time to my main PC.

This is not the cheapest solution, but is available and works well (and is not Unifi)

 

[sic0048]

Personally I would use a SFF computer with something like an i5-8x00 CPU in it. Add a dual 10gb SFP+ card and for about $150 all in (edit - probably closer to $200 with an ethernet transceiver as they are more expensive than the optical versions), you'll have a very capable system. Sure it isn't going to be as energy efficient as some options, but the financial "break even" point is going to be pretty long given how expensive a lot of the "low power" options are.

 

[coxhaus]

I wish I had your problem. AT&T is close to me, only a few blocks away. Still waiting.

I use a Dell Optiplex i3-4130T PC, cost $75 used. I use a Broadcom 2 port 10 gig NIC off eBay. Pfsense has drivers for it using 23.01. TDP is 35 watts. I can run it for years before I could pay for $500 or router.

I am thinking it will get you close to 5 gigs but I have not tested it as I don't have a fiber option yet.

 

[blunden]

Depending on your need for IDS/IPS, you might get away with using something like one of the RS86S models.

 

[mattventura]

A little pricey, but I use an X10SDV board. I have the 8-core version and have benchmarked it at about 19gbps of plain NAT routing (using virtual interfaces). That would keep you going well into the future, or allow you to run a bunch of extra stuff on it at 5gbps.

Typical build cost would be $250 for the board, $50 for RAM, however much you want for the case/PSU.

 

[BoredSysadmin]

I don't think PF nor OpnSense could route at 10 gig speeds regardless of hardware (without breaking into fort nox).
TNSR or VyOS is the way do do "inexpensive" 10gig software routing.
here's one of the ways you could get it done:

Fast Home Lab Router - TNSR & AT&T 5Gbps Fiber

Update 11/20/23: This post is out of date, check the latest here: https://dogsinspace.dog/fast-home-lab-router-v2/ I have used MikroTik's RouterOS, pfSense, and OPNsense as edge devices to varying degrees of success. While these platforms all perform admirably within their designated use...

dogsinspace.dog

TNSR or VyOS would handle IPSec VPN, but for IDS/IPS you would need to roll another system.

 

[mach3.2]

I don't think PF nor OpnSense could route at 10 gig speeds regardless of hardware (without breaking into fort nox).

You can do "easy" (1500 byte packets) 10Gb/s with relatively recent hardware. "Hard" (64 byte packets) 10Gb/s, not a chance, not even close.

I personally tested iperf3 to 10Gb/s on a pfsense VM that is allocated 4 vCPU from an i5 12500. CPU utilisation reads around 30% from within the VM.

My setup with 4 vCPUs and VMXNET3 paravirtual NICs seem to top out at around 600-700k packets per second.

 

[fta]

You can do "easy" (1500 byte packets) 10Gb/s with relatively recent hardware.

To fully saturate 10GbE, you need to do 20Gbps since it is full duplex. I have a i5 1235u box that I am running as a linux router. My firewall rules have been tuned for efficiency (including using fastpath). A bidirectional iperf3 test where I am routing between a VLAN and my LAN, which goes through my firewall rules, does 12/12Gbps for a total of 24Gbps. It takes all 10 cores to do that. Note also that this is going between two VMs, and the router itself is a VM.

My setup with 4 vCPUs and VMXNET3 paravirtual NICs seem to top out at around 600-700k packets per second.

(700e3 * 1500 * 8) / 1e9 = 8.4Gbps?

 

[mach3.2]

To fully saturate 10GbE, you need to do 20Gbps since it is full duplex. I have a i5 1235u box that I am running as a linux router. My firewall rules have been tuned for efficiency (including using fastpath). A bidirectional iperf3 test where I am routing between a VLAN and my LAN, which goes through my firewall rules, does 12/12Gbps for a total of 24Gbps. It takes all 10 cores to do that. Note also that this is going between two VMs, and the router itself is a VM.

Fair, and true. I'll stretch the legs of my jank box again when I have some spare time and see what it can actually do with bidirectional traffic.

(700e3 * 1500 * 8) / 1e9 = 8.4Gbps?

I rounded down the number. I actually spy 755kpps at its peak on my pfsense RDD graph. The iperf3 print out shows 9.1Gb/s if memory serves me right.

 

[fta]

Here is my testing:

Command:

iperf3 -c 192.168.200.98 -Z --bidir

OPNSense:

[ 5][TX-C] 0.00-10.00 sec 4.50 GBytes 3.87 Gbits/sec 1095 sender
[ 5][TX-C] 0.00-10.00 sec 4.50 GBytes 3.86 Gbits/sec receiver
[ 7][RX-C] 0.00-10.00 sec 5.27 GBytes 4.52 Gbits/sec 1255 sender
[ 7][RX-C] 0.00-10.00 sec 5.26 GBytes 4.52 Gbits/sec receiver

Linux:

[ ID][Role] Interval Transfer Bitrate Retr
[ 5][TX-C] 0.00-10.00 sec 13.8 GBytes 11.8 Gbits/sec 609 sender
[ 5][TX-C] 0.00-10.00 sec 13.8 GBytes 11.8 Gbits/sec receiver
[ 7][RX-C] 0.00-10.00 sec 13.5 GBytes 11.6 Gbits/sec 198 sender
[ 7][RX-C] 0.00-10.00 sec 13.5 GBytes 11.6 Gbits/sec receiver

The test is going from a LAN VM and through a router/firewall VM to a VLAN VM. The router VMs are configured with logically identical configuration, which means I can do A/B testing. The OPNSense VM was my production router until I discovered its performance was so bad. I built the linux router VM to see if I could get faster routing, and it is my production router now. I don't know why the OPNSense router is so much slower. Perhaps because they throw in so many default firewall rules that you can't remove?

 

[blunden]

The test is going from a LAN VM and through a router/firewall VM to a VLAN VM. The router VMs are configured with logically identical configuration, which means I can do A/B testing. The OPNSense VM was my production router until I discovered its performance was so bad. I built the linux router VM to see if I could get faster routing, and it is my production router now. I don't know why the OPNSense router is so much slower. Perhaps because they throw in so many default firewall rules that you can't remove?

Interesting results. That Linux router VM, is it running a standard Linux distro or something like VyOS? By fastpath, do you mean the software based part of flowtable or hardware offload in the NIC?

Is OPNSense in your example using a fastpath equivalent? If not, perhaps that's why you get those lower results? I've seen other people get similarly low results in some cases though, despite BSD's supposedly better network stack. Not sure why.

 

[fta]

Interesting results. That Linux router VM, is it running a standard Linux distro or something like VyOS?

Standard debian bookworm.

By fastpath, do you mean the software based part of flowtable or hardware offload in the NIC?

The software based flowtable where established connections bypass all the netfilter processing. It turns out on linux it's actually bottlenecked elsewhere. Turning on/off fastpath (verified by watching the packet counter in the forward chain) makes no difference in the benchmark results.

Is OPNSense in your example using a fastpath equivalent?

I think FreeBSD 11+ has some sort of fastpath on by default.