10G Router options

Sealside

Member
May 10, 2019
50
9
8
Stockholm/Sweden
Hi!

I'm struggling trying to find a cost efficient 10G routing solution. I've been looking a lot in these forums
so hoping I can get some suggestions.
I want to run pfsense, either in a 1U-2U case or a small thin client.

Some basic requirements
- Should be able to handle 2x10G via PCI card (Like inte X520 DA2, which I have one spare)
- Should be able to push 10G speed
- OpenVPN performance should be ok right now I use a qotom i5 which can do around 40MB/s VPN
- Preferably not consume too much power (my current qotom box is drawing less than 20 W).

I've looked at the HP T730, but it's hard to find info of what it can do.
Some add 10G solarflare cards to this box, will the intel X520 work as well?
Will you actually get 10G throughput with this box (or is too weak?)

There are some Supermicro alternatives as well, are there other options?
The supermicro alternatives seems to be a bit overkill, I want to run the pfsense fw bare metal, I already have
a dedicated home server.

Thanks in advance, S
 

Sealside

Member
May 10, 2019
50
9
8
Stockholm/Sweden
Hi! Currently on 1G on WAN. From July I will be able to buy 10G, so feels good to be somewhat future safe. Running 10G across vlans with inter vlan routing in the switch.

Regards S
 

PigLover

Moderator
Jan 26, 2011
3,005
1,305
113
Do you want true 10Gbe line-rate routing? That can actually be a challenge with pfSense. It takes pretty high-end hardware to get it there.

The issue is that pfSense is built on top of the BSD Packet Filter (BPF, of the "pf" of "pfSense"). BPF does quite a lot of bit-copying as it processes packets. This tends to eat CPU cycles and memory bandwidth. Performance also tends to degrade non-linearly at high speeds as more and more of these bit copies result in a cache-miss. This is why Netgate has started working on TNSR for their next generation router/firewall. TNSR is build on a much more efficient network stack called VPP that makes packet pipelines much more efficient.

That said - you can build a router on pfSense that will work decently using 10Gbe links, especially if you don't expect to see continuous flows at/near line rate.

pfSense needs three things to get there (in order of importance): larger cache, clocks and then memory bandwidth. Cache size makes the bit copies faster, clocks let it handle the interrupt rate, memory bandwidth limits the pain of cache misses. It does not benefit a lot from having more cores - though there is some benefit if you are processing multiple IP flows where the IO handling can be distributed. You also don't need a lot of memory.

The T730 is probably a bit lightweight if you want 10Gbe routing. It may get to 2-3Gbe total sustained throughput before it starts dropping packets (depending on packet size - you can probably demonstrate 10Gbe throughput if pushing 100% jumboframes).

An I3-9100 or I5-9500 based system might do slightly better (50% more cache/core and a bit more clock). You can build one fairly cheap if you start with an HP 290-p0043w and upgrade it a bit (I3/I5 CPU, a second small ram stick and a cheap m.2 drive). It has a PCIe x16 slot ready to hold your NIC. (Oops - update - I just checked prices - these things used to be on eBay ~$100 but they seem much more expensive now).

If you want to come close to 10Gbe on a regular basis you really need to get bigger cache. The least expensive way isn't cheap at all. The D2100 series CPUs offer more cache/core and 4-way RAM, but slower clocks - low end is the X11SDV-4C-TLN2F but your up around $600 for the MB/CPU alone.

Other options to get certainty around 10Gbe routing start to get even more expensive...
 

Sealside

Member
May 10, 2019
50
9
8
Stockholm/Sweden
Thanks!

This was the kind of information I was looking for. At the end of the line it does not seem to be that many people running
10G pfsense (other that with the Netgate highend devices). I guess there is no really cheap option to get a 10G capable router.
I might go with the T730 or P 290-p0043w in the meantime and be happy getting slightly more than 1GPS.

Thanks for the write up!

Regards S
 

kapone

Well-Known Member
May 23, 2015
914
498
63
"many" people don't run 10g routers/pfSense because 10g just isn't practical/available in a home/homelab setting. And if it's commercial, there's many commercial routers available (not cheap) that will do it, and if you need that kind of bandwidth commercially, chances are, cost isn't your primary criteria.
 
  • Like
Reactions: Sealside

JSchuricht

Active Member
Apr 4, 2011
174
56
28
I have one setup in my home lab for fun but I haven't tried to really push the 10Gb routing hard. Mainly 2-3 clients with storage bottlenecks moving large files. I have no problem doing 6Gb across VLAN's (disk limit on one node) but if there was a large number of nodes hitting it things would be different.

Hardware is a bit old now but close to what you are looking for. Supermicro X10SLH-F, E3-1275 v3 (3.5Ghz 4 core Haswell with HT disabled), 2x 4GB 1333Mhz DDR3 and a Intel X520 DA2 in a 1u Supermicro case. I haven't measured power draw but I would guess 40w idle.

Couldn't find a good example of a long high Gb/s transfer but here is a taste of resource usage.



 
  • Like
Reactions: Sealside

PigLover

Moderator
Jan 26, 2011
3,005
1,305
113
one more good test on budget hardware:
That's a pretty interesting test. My only issue with it is that it doesn't specify the test method. From the screenshot snippets it appears to be using iperf. Iperf is a good test tool, but unless you set the segment size (--set-mss) you are sending max sized packets based on your MTU settings (and to get these speeds I'll jump to the conclusion that he has 9000 byte MTUs set up). It also doesn't stress the limiter of a software-based router/firewall like pfSense or Vyos. They are stress by Packets-per-second, not raw throughput.

This is a good benchmark for home users where the major use case for high speed networks is file services (SMB, NFS and file transfer).

Better benchmark tools vary the packet size transmitted throughout the test against some distribution. The typical distribution used is call "imix", which represents a mix of packet sizes that is typical of traffic seen on the Internet (well - to be fair - that was typically seen on the internet over 15 years ago when imix was published).
 

Skud

Active Member
Jan 3, 2012
115
54
28
I’ve run OPNSense, pfSense, Sophos UTM, Sophos XG, and Untangle on a Sophos SG330 box (i5-4570s, 12GB RAM, 240GB SSD). If you keep an eye open you can sometimes find them for a good price. The SG310 is a bit less powerful (i3, I think) but still has the network module spot.

It’s a Caswell unit and, as such, you can use any of the Caswell/Portwell network modules. For example, I’m using these dual-port 10Gb modules from a Checkpoint. Was cheap.


With OPNSense I was able to route 10Gb at around 85% CPU. With Sophos XG I can do the same with less than 30% CPU.

Riley
 

blinkenlights

Active Member
May 24, 2019
143
64
28
This may meet your requirements.
but power consumption won’t be 20W.

That is one cool 1U barebones kit. I have an E3-1230Lv3 (25W TDP) laying around that would probably be a good match. It was a special order from Intel, originally only sold to Juniper and friends for their high-end routers. If anyone is interested, shoot me a private message.
 

Sealside

Member
May 10, 2019
50
9
8
Stockholm/Sweden
Interesting links. I have actually eyed the X10SLH MB perviously, it interesting for sure.
I'll hang out some more on ebay looking for the X10SLH or a Sophos 330.
The current X10SLHs are all US-based, I'm in Europe so will wait and see if something pops up here.

Regards, S
 

fckruegel

New Member
May 27, 2020
4
0
1
I've used Mikrotik Routers in a project with several location linked via fiber, SFP(+) and encrypted (IPSEC) connections.
Look at this:

This is very special hardware. Each SFP+ port has its own CPU core, and there are a total of 72 cores so you have more than enough cpu power for filtering, nat, encryption. There are also cheaper models with less cpu cores.
 

Sealside

Member
May 10, 2019
50
9
8
Stockholm/Sweden
I've used Mikrotik Routers in a project with several location linked via fiber, SFP(+) and encrypted (IPSEC) connections.
I've played around with some of the Mikrotik routers. I was thinking about replacing my previous switch with a Mikrotik, and do both routing and switching. I have a friend who has pretty much thrown away all his previous hardware for Mikrotik. For me I miss too many features in Mikrotik (like geo-ip, pfblocker ng, openvpn and others), also lack of POE made me go another direction. I will probably extend my homelab with some Mikrotiks eventually, they are a lot of bang for the buck.

Regards, S
 

fckruegel

New Member
May 27, 2020
4
0
1
Mikrotik actually has several devices with POE. I've got an CRS328-24P-4S+RM with 24 POE 1G RJ45 ports and 500W POE power and four SFP+ ports. This however is a switch, not a router. It has not the cpu power to do routing and encryption, although it is capable of.
 

nickf1227

Member
Sep 23, 2015
74
46
18
30
As explained by @PigLover above, you aren't going to get anywhere near 10G with a low power system, especially running that 25 watt CPU in that super micro chassis.
You want a HIGH CLOCK speed CPU with a LARGE CACHE.
Otherwise, you are just wasting time.
Based on the graphs provided by @JSchuricht that gives us a good benchmark. I would say a 9th Gen Intel i5/i7 or Ryzen 3000 5/7 would actually get close to 10G routing in pfSense at line speed.
 

blinkenlights

Active Member
May 24, 2019
143
64
28
As explained by @PigLover above, you aren't going to get anywhere near 10G with a low power system, especially running that 25 watt CPU in that super micro chassis.
You want a HIGH CLOCK speed CPU with a LARGE CACHE.
Otherwise, you are just wasting time.
@nickf1227 you are correct that I have not tested that particular E3-1230Lv3 under pfSense at 10Gbps... but I also did not say that ;) As I am sure you know, the routing problem is not bytes per packet but packets per second (instructions/tick). The real bottleneck is the Linux or FreeBSD kernel - that is why Netflix had to heavily modify the FreeBSD kernel to support their CDN nodes and Netgate implemented the stack in software for TNSR. Jim T at Netgate posted a great explanation of this concept last year: https://www.reddit.com/r/homelab/comments/7gmn9f/_/dqla8zl
I have seen 10Gbps IMIX line rate solutions implemented at scale using commodity (Dell) hardware. In that case, though, the interface cards did all of the "heavy lifting" with offload processing and hashing to split the traffic across multiple general purpose cores. The operating system in that case acts as the supervisor/management engine and has little to do with the actual traffic routing.
 
  • Like
Reactions: Aluminat

nickf1227

Member
Sep 23, 2015
74
46
18
30
I have seen 10Gbps IMIX line rate solutions implemented at scale using commodity (Dell) hardware. In that case, though, the interface cards did all of the "heavy lifting" with offload processing and hashing to split the traffic across multiple general purpose cores. The operating system in that case acts as the supervisor/management engine and has little to do with the actual traffic routing.
I did not say it wasn't possible. But using TOE cards also have their own problems. Per pfsense documentation from netgate:
1590847917414.png

Basically, there is a reason Netgate is doing it in software :p

And as far as latency from memory, the quote you posted is correct, tyhat would be the biggest problem:
1590848261396.png
1590848534565.png

But that is why @PigLover mentioned CACHE SIZE as being of upmost importance.

Now, is 16kb going to be enough get you to where you are going in a homelab? I dont know.
 

Attachments

Last edited: