Vyos 10GbE Router

[tsteine]

So, I figured it was time to retire my Edgerouter 8 Pro, and figured I would try my hand at Vyos as a 10GbE router. Seems like, at least with my particular setup, that Vyos is perfectly capable of routing at line rate for 10GbE, even across adapters.
Note that this is not a final setup in terms of vlans or connection, just for testing the router.

Does anyone have any experience with Vyos and 40GbE, and how well it performs >10gbe?

Machine specs:
Asus WS C246 Pro
WS C246 PRO | Servere og arbeidsstasjoner | ASUS Norge

Intel Xeon E-2134 (4-core 3.5 base, 4.5 turbo)
Intel® Xeon® E-2134 Processor (8M Cache, up to 4.50 GHz) Product Specifications

32gb 2400mhz ddr4 ECC memory (yes, I'm aware this is overkill for networking but it's what I had laying around)
https://www.kingston.com/datasheets/KVR24E17S8_8.pdf

3x Intel X710-DA2
Intel® Ethernet Converged Network Adapter X710-DA2 Product Specifications

Switches:
Edgeswitch XG 16port 10GbE
Arista 7050Q-16 16 port 40GbE
Mellanox SN2700 32 port 100GbE

Test:
Vlan 5 server (10.0.5.100)
vlan 10 server (10.0.10.100)
vlan 20 client (10.0.20.100)
vlan 100 client (10.0.100.100)

Vlan 20 client connects to vlan 5 server with single process
(iperf3 -c 10.0.5.100 -t 3600 -d)
vlan 100 client connects to vlan 10 server with single process
(iperf3 -c 10.0.10.100 -t 3600 -d)

Network Diagram


Janky setup:


Throughput:

 

[tsteine]

Mistakenly wrote full duplex in original post.

Here is an actual full duplex test, though here it seems to cap out at 56 gbit without properly balancing receive to port 7

 

[tsteine]

So, i eliminated the ES-16-XG and connected the vyos router directly to the Arista 7050Q-16.

It seems I found the hard cap for throughput, which appears to be around 55-56 gbit, as was the case with the ER-16-XG

 

[tsteine]

Managed to eke out a little more out of it.

 

[BoredSysadmin]

this is probably you already know/tried, but just in case:
Performance Tuning - VyOS Wiki

 

[fohdeesha]

you might be interested in trying a related project, ATT's DanOS - it's the original source of vyos (vyatta) with several years of development from ATT, including functional DPDK forwarding - DANOS

DANOS is Vyatta + 7 years of development to make it usable in Telco networks. The high-level overview of the major work that has been done since the last Vyatta release:
1.) All traffic is now forwarded with a custom DPDK based dataplane.
2.) The configuration and operational infrastructure are both new compared to Vyatta.
3.) We have cleaned up the implementation of the CLI, formalized a scripting API, added NETCONF support, and opened up the REST API.
4.) The dataplane has support for programming hardware and a plugin is provided to support the UFiSpace S9500-30xs on top of Broadcom's OpenNSL.
5.) We support AAA using either on device RBAC rules or TACACS+ as is a common need in the networking industry.
6.) We sandbox operator and admin users so they can't change they system without going through the infrastructure.
7.) We added a new mechanism for integrating features that is cleaner than the old Vyatta mechanism.

I've had a poke around in a VM and a lot of the CLI changes definitely seem ATT/telco centric, but it seems functional at least. DPDK should yield several times higher PPS capability versus vyos

 

[tsteine]

DANOS completely went under my radar, I need to try this out.

Hell, I might even throw some Connectx-3s in there and see how well it handles 40gbit

 

[tsteine]

Seems like it needs some time to mature for bare metal, when booting the danos image the machine hard locks completely.

 

[fohdeesha]

Seems like it needs some time to mature for bare metal, when booting the danos image the machine hard locks completely.

damn, that's strange. I recall booting it without issue on an intel system, granted it was an older xeon

 

[tsteine]

So, I'm getting a TNSR subscription from Netgate, in case anyone is interested in how the machine mentioned above handles it, even if it's not free software.

 

[manxam]

How costly is TNSR anyways? Their website mentions the following but then expects you to contact them for pricing :
"Simple straightforward subscription.
No hidden fees.
No complicated nickel and dime add-ons."

 

[tsteine]

I'm not sure whether I'm allowed to disclose pricing i was quoted given that it's not visible on their site, but to ballpark it, expect between 3 to 4 digits, depending on your throughput.
Not what I would consider affordable for personal use

 

[PigLover]

@tsteine Can you clarify the packet size used in the above tests? Was it all jumboframes or something more like imix?

 

[tsteine]

@PigLover

These were all 1500 mtu packets.

edit: I should also note that the above setup has been replaced with TNSR and an Intel XL710-QDA2 running 40gbit.

 

[eduncan911]

Have you tried any DPI tools, Suricata, etc to see what impact it has on throughput on that little Xeon? What about L2TP or OpenVPN?

I've used OpenVPN w/AES on an N4200 SoC and could not eek out more than 650 Mbps bi-directional, and that was just openvpn - didn't have any other IPS or other tooling at that time.

I'm on the fence of building an LGA2011 v4 14-core server I have laying around for a router (have dual CPUs actually for 28C). Or, if an Xeon 1270 v3 (quad core) would be sufficient - which would require another purchase.

Currently my L3 ICX switch (thanks @fohdeesha !!) handles most ACLs for the 10 Gbps VLANs. However, I'd like to start monitoring cross-VLAN traffic at some point, which means I'd need 10 Gbps on the router.

Hence, the beast 14C CPUs I may need as 4 cores may not be enough for all of that on a single box, with 10 Gbps.

 

[tsteine]

Have you tried any DPI tools, Suricata, etc to see what impact it has on throughput on that little Xeon? What about L2TP or OpenVPN?

No, I would set up a separate VM to run tools such as Suricata and forward the traffic to it, to ensure that such workloads to not affect performance for routing.

It should be noted that TNSR is now free for home use and you only need to register to get the download link emailed to you. Even relatively modest configurations would get solid 10gbit performance with DPDK/VPP.

Of course, if you plan on running Suricata for 10gbit, and your ruleset is extensive, you will need a powerful server to keep up.

 

[AveryFreeman]

Have you tried any DPI tools, Suricata, etc to see what impact it has on throughput on that little Xeon? What about L2TP or OpenVPN?

I've used OpenVPN w/AES on an N4200 SoC and could not eek out more than 650 Mbps bi-directional, and that was just openvpn - didn't have any other IPS or other tooling at that time.

I'm on the fence of building an LGA2011 v4 14-core server I have laying around for a router (have dual CPUs actually for 28C). Or, if an Xeon 1270 v3 (quad core) would be sufficient - which would require another purchase.

Currently my L3 ICX switch (thanks @fohdeesha !!) handles most ACLs for the 10 Gbps VLANs. However, I'd like to start monitoring cross-VLAN traffic at some point, which means I'd need 10 Gbps on the router.

Hence, the beast 14C CPUs I may need as 4 cores may not be enough for all of that on a single box, with 10 Gbps.

From what I understand, if your 1270 v3's 3.5GHz is a faster freq than whatever processor your 14C 2011-3 is, my guess is it'd be a better choice - assuming the 2011-3 is one of the many high core-count, low frequency models

If you're choosing new hardware, throw one of those cheap i3-Es that are a base 4.0GHz at it, and the fastest memory you can find (likely non-ECC). Or one of the new 10nm processors that base at 3.5GHz but will turbo over 5GHz.

Unless you're doing DPDK or XDP, which is an entirely different story that may have other considerations I'm not privy to.

This is not particularly scientific, but one of the engineers for pfSense responds with a post detailing math+physics of PPS in the context of commodity hardware (with an aside about Ryzen registers being limited to bus speed when accessing L2/L3 cache): https://www.reddit.com/r/networking/comments/6upchy
Did you ever try DPDK on your N4200? I noticed OVS has a DPDK variant in Ubuntu universe repo now, eager to try it out.

 

[Takrbark3]

Hi!
Can you measure the "pps: packets per second" instead of "bps: bits per second" ?

Measuring the "bps" is misleading and wrong like "Oranges vs Apples".
If you understand exactly the difference, you will know what is the "huge gap" between the "ASIC-Router vs PC-Router".

You can begin here:
https://kb.juniper.net/InfoCenter/index?page=content&id=kb14737

PC-Router:
1 Mpps - 15Mpps: using DPDK-NIC (connectx5) with many-core CPU

ASIC-Router:
120Mpps - HPE HSR6800
390Mpps - Cisco SUP2T
660Mpps - Juniper MX240

 

[tsteine]

@Takrbark3
Hi, I got around to testing this, finally.

so, the setup is as follows:
vpp router:
intel xeon e-2224 quad-core
16gb ram
mellanox connect-x5 100gbe dual port.
ips: 10.0.30.1, 10.0.40.1

trex server:
ryzen 3700x 8-core
32gb ram
mellanox connectx-5 100gbe dual port
ips: 10.0.30.230, 10.0.40.230

vpp runs using 2 cores.


trex server issues 64b packets from 10.0.30.230 to 10.0.40.230 using 10.0.30.1 as a gateway.

the result is a peak at about 33 MPPS, as soon as it passed 33MPPS, it started dropping packets.



from mellanox switch:

 

[BoredSysadmin]

I found today a very detailed series of articles on using/installing vpp on ubuntu by a small swiss ISP - IPng.
part 7 (includes install howto) and the rest of the article series are here:

Articles

IPng Networks GmbH provides networking consultancy, hosting, colocation, internet connectivity options primarily tailored for the Zurich metropolitan area.

ipng.ch