Vyos 10GbE Router

tsteine

Member
May 15, 2019
90
21
8
So, I figured it was time to retire my Edgerouter 8 Pro, and figured I would try my hand at Vyos as a 10GbE router. Seems like, at least with my particular setup, that Vyos is perfectly capable of routing at line rate for 10GbE, even across adapters.
Note that this is not a final setup in terms of vlans or connection, just for testing the router.

Does anyone have any experience with Vyos and 40GbE, and how well it performs >10gbe?

Machine specs:
Asus WS C246 Pro
WS C246 PRO | Servere og arbeidsstasjoner | ASUS Norge

Intel Xeon E-2134 (4-core 3.5 base, 4.5 turbo)
Intel® Xeon® E-2134 Processor (8M Cache, up to 4.50 GHz) Product Specifications

32gb 2400mhz ddr4 ECC memory (yes, I'm aware this is overkill for networking but it's what I had laying around)
https://www.kingston.com/datasheets/KVR24E17S8_8.pdf

3x Intel X710-DA2
Intel® Ethernet Converged Network Adapter X710-DA2 Product Specifications

Switches:
Edgeswitch XG 16port 10GbE
Arista 7050Q-16 16 port 40GbE
Mellanox SN2700 32 port 100GbE

Test:
Vlan 5 server (10.0.5.100)
vlan 10 server (10.0.10.100)
vlan 20 client (10.0.20.100)
vlan 100 client (10.0.100.100)

Vlan 20 client connects to vlan 5 server with single process
(iperf3 -c 10.0.5.100 -t 3600 -d)
vlan 100 client connects to vlan 10 server with single process
(iperf3 -c 10.0.10.100 -t 3600 -d)

Network Diagram
Network diagram.PNG

Janky setup:
jank.jpg

Throughput:
Throughput.PNG
 
Last edited:

tsteine

Member
May 15, 2019
90
21
8
Mistakenly wrote full duplex in original post.

Here is an actual full duplex test, though here it seems to cap out at 56 gbit without properly balancing receive to port 7
full duplex.JPG
 

tsteine

Member
May 15, 2019
90
21
8
So, i eliminated the ES-16-XG and connected the vyos router directly to the Arista 7050Q-16.

It seems I found the hard cap for throughput, which appears to be around 55-56 gbit, as was the case with the ER-16-XG

arista throughput.JPG
 

fohdeesha

Kaini Industries
Nov 20, 2016
2,087
1,949
113
30
fohdeesha.com
you might be interested in trying a related project, ATT's DanOS - it's the original source of vyos (vyatta) with several years of development from ATT, including functional DPDK forwarding - DANOS

Code:
DANOS is Vyatta + 7 years of development to make it usable in Telco networks. The high-level overview of the major work that has been done since the last Vyatta release:
1.) All traffic is now forwarded with a custom DPDK based dataplane.
2.) The configuration and operational infrastructure are both new compared to Vyatta.
3.) We have cleaned up the implementation of the CLI, formalized a scripting API, added NETCONF support, and opened up the REST API.
4.) The dataplane has support for programming hardware and a plugin is provided to support the UFiSpace S9500-30xs on top of Broadcom's OpenNSL.
5.) We support AAA using either on device RBAC rules or TACACS+ as is a common need in the networking industry.
6.) We sandbox operator and admin users so they can't change they system without going through the infrastructure.
7.) We added a new mechanism for integrating features that is cleaner than the old Vyatta mechanism.
I've had a poke around in a VM and a lot of the CLI changes definitely seem ATT/telco centric, but it seems functional at least. DPDK should yield several times higher PPS capability versus vyos
 

tsteine

Member
May 15, 2019
90
21
8
DANOS completely went under my radar, I need to try this out.

Hell, I might even throw some Connectx-3s in there and see how well it handles 40gbit
 
  • Like
Reactions: fohdeesha

tsteine

Member
May 15, 2019
90
21
8
Seems like it needs some time to mature for bare metal, when booting the danos image the machine hard locks completely.
 

manxam

Active Member
Jul 25, 2015
235
49
28
How costly is TNSR anyways? Their website mentions the following but then expects you to contact them for pricing :
"Simple straightforward subscription.
No hidden fees.
No complicated nickel and dime add-ons."
 

tsteine

Member
May 15, 2019
90
21
8
I'm not sure whether I'm allowed to disclose pricing i was quoted given that it's not visible on their site, but to ballpark it, expect between 3 to 4 digits, depending on your throughput.
Not what I would consider affordable for personal use
 

tsteine

Member
May 15, 2019
90
21
8
@PigLover

These were all 1500 mtu packets.

edit: I should also note that the above setup has been replaced with TNSR and an Intel XL710-QDA2 running 40gbit.
 

eduncan911

Active Member
Jul 27, 2015
139
66
28
66
eduncan911.com
Have you tried any DPI tools, Suricata, etc to see what impact it has on throughput on that little Xeon? What about L2TP or OpenVPN?

I've used OpenVPN w/AES on an N4200 SoC and could not eek out more than 650 Mbps bi-directional, and that was just openvpn - didn't have any other IPS or other tooling at that time.

I'm on the fence of building an LGA2011 v4 14-core server I have laying around for a router (have dual CPUs actually for 28C). Or, if an Xeon 1270 v3 (quad core) would be sufficient - which would require another purchase.

Currently my L3 ICX switch (thanks @fohdeesha !!) handles most ACLs for the 10 Gbps VLANs. However, I'd like to start monitoring cross-VLAN traffic at some point, which means I'd need 10 Gbps on the router.

Hence, the beast 14C CPUs I may need as 4 cores may not be enough for all of that on a single box, with 10 Gbps.
 
Last edited:
  • Like
Reactions: fohdeesha

tsteine

Member
May 15, 2019
90
21
8
Have you tried any DPI tools, Suricata, etc to see what impact it has on throughput on that little Xeon? What about L2TP or OpenVPN?
No, I would set up a separate VM to run tools such as Suricata and forward the traffic to it, to ensure that such workloads to not affect performance for routing.

It should be noted that TNSR is now free for home use and you only need to register to get the download link emailed to you. Even relatively modest configurations would get solid 10gbit performance with DPDK/VPP.

Of course, if you plan on running Suricata for 10gbit, and your ruleset is extensive, you will need a powerful server to keep up.
 

AveryFreeman

consummate homelabber
Mar 17, 2017
254
23
18
39
Near Seattle
averyfreeman.com
Have you tried any DPI tools, Suricata, etc to see what impact it has on throughput on that little Xeon? What about L2TP or OpenVPN?

I've used OpenVPN w/AES on an N4200 SoC and could not eek out more than 650 Mbps bi-directional, and that was just openvpn - didn't have any other IPS or other tooling at that time.

I'm on the fence of building an LGA2011 v4 14-core server I have laying around for a router (have dual CPUs actually for 28C). Or, if an Xeon 1270 v3 (quad core) would be sufficient - which would require another purchase.

Currently my L3 ICX switch (thanks @fohdeesha !!) handles most ACLs for the 10 Gbps VLANs. However, I'd like to start monitoring cross-VLAN traffic at some point, which means I'd need 10 Gbps on the router.

Hence, the beast 14C CPUs I may need as 4 cores may not be enough for all of that on a single box, with 10 Gbps.
From what I understand, if your 1270 v3's 3.5GHz is a faster freq than whatever processor your 14C 2011-3 is, my guess is it'd be a better choice - assuming the 2011-3 is one of the many high core-count, low frequency models

If you're choosing new hardware, throw one of those cheap i3-Es that are a base 4.0GHz at it, and the fastest memory you can find (likely non-ECC). Or one of the new 10nm processors that base at 3.5GHz but will turbo over 5GHz.

Unless you're doing DPDK or XDP, which is an entirely different story that may have other considerations I'm not privy to.

This is not particularly scientific, but one of the engineers for pfSense responds with a post detailing math+physics of PPS in the context of commodity hardware (with an aside about Ryzen registers being limited to bus speed when accessing L2/L3 cache): https://www.reddit.com/r/networking/comments/6upchy
Did you ever try DPDK on your N4200? I noticed OVS has a DPDK variant in Ubuntu universe repo now, eager to try it out.