Addition of Layer 3 benchmarks to switch reviews

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

bryan_v

Active Member
Nov 5, 2021
135
64
28
Toronto, Ontario
www.linkedin.com
In shopping for switches for edge locations and I've notice that none of the vendors, except for Mikrotik publish L3 routing benchmarks, and almost all the review sites only test L2 switch performance. Especially for edge locations, Inter-VLAN routing and static routes are the most important because (a) you necissarily want to buy a dedicated edge router just to handle a handful of routing rules; and (b) you don't want to send traffic up to the core only to have it come back down to the edge again.

If Rohit or anyone else who is reviewing a switch when the hardware comes in can benchmark L3 performance right after they do an L2 test, it would be an amazing metric to have. The Mikrotik benchmarks of 20/25 VLAN and IP routes is a good place to start, since that seems to be the standard amount that the Broadcom chips should offload.

In my current scenario for example, I'm trying to figure out if a lower end Dell, Netgear, Mikrotik, EdgeSwitch or TP-LInk SFP+ switch will be able to get the job done, or if we need to spring for a higher end switch like a Brocade, Arista, Dell, HP, etc.. (which at that point I have to ask the guys if they want to throw Sonic on there as well, which also increases the management overhead and skill required).
 

Patrick

Administrator
Staff member
Dec 21, 2010
12,511
5,792
113
The two big challenges thus far basically come down to cost.

Doing this means we need 100GbE speeds for what we are testing. Beyond that, it adds extra configuration costs / time.

Dmitrij worked with T-Rex on the low end routers, but we started hitting a big cost cliff after the very low end devices and lower numbers of ports.
 

bryan_v

Active Member
Nov 5, 2021
135
64
28
Toronto, Ontario
www.linkedin.com
No worries Patrick. I guessed it was something along those lines since those edge switches usually are in the 960Gbps territory for a 48 port. I just wanted to mentioned it in hardcopy, since it seems no one had ever asked for it before in the forums.

If anything you can use this as a case to get a traffic generator test appliance sent to you, the same way FLIR sends out cameras to certain media outlets to use (and how LTT got their cable tester). I can't remember off the top of my head who makes them, though I'm sure you've stumbled across a few at the conferences. I think they run about $20k for a decent one (though that was like 6 years ago when I last checked).

Right now I had one of the juniors setup two Ryzen boxes with an Intel E810 at each end and a breakout cable on one end. It's not the greatest setup, but it was the only way to evaluate the switches since we only get to evaluate them for a few days now before we have to send them back to the supplier (if they even have the trial hardware that we're actually interested in). Sadly, I'm pretty sure the fully loaded cost of evaluating our top 5 vendors is going to wash out to about USD $12k.

Cheers,
Bryan
 
  • Like
Reactions: Patrick

Patrick

Administrator
Staff member
Dec 21, 2010
12,511
5,792
113
Hey @bryan_v what are you using to do the testing software-wise? I am half-considering getting an older Spirent now.
 

bryan_v

Active Member
Nov 5, 2021
135
64
28
Toronto, Ontario
www.linkedin.com
Hey @Patrick,

Last time I checked, we had two configurations on each end (dual boot NVMe):
  1. Bare-metal Ubuntu and just iperf3 just the test the raw single client L3 performance. If there's no wireline L3 handling on the ASIC, this initial test will reveal it without trying anything more complicated.
  2. Bare metal XCP-ng with the latest Intel drivers recompiled from source. We then create VFs for about 10 different Ubuntu VMs, passed through with SR-IOV, then confirm that each VM is getting wireline L2 performance with iperf3. Since iperf3 can't really push traffic faster than ~30Gbps per core, we use the multiple VMs with core affinity to push an aggregate traffic of closer to 100Gbps (or whatever is the max speed for the port/breakout cable config). Once we've confirmed L2 is working properly from a client perspective, we setup the L3 rules on the switch (mainly inter-VLAN routing) and try again with iperf3.
  3. We tried Bare-metal Ubuntu with TRex once to replay pcap traffic, however capturing the pcap and getting it to replay properly was very very hard, and we're pretty sure we did it wrong because the performance was a lot worse than iperf3 TCP tests. We haven't had a chance to test the setup with VFs and VMs instead to see if it is any better.
Right now we're using SFF cases, so they can only handle 1 PCIe x16 card. We haven't encountered the situation yet, but if the switch maxes out the E810/VFs config, we might grab either 4 of the XL710 or X710 and plug them into an EPYC ATX board to ensure we're not hitting PCIe bottlenecks. As of yet, the L3 data for initial entry-level gear was very underwhelming, and I'm still looking at late Jan to get an evaluation Extreme and Dell switch (haven't heard back from the HP supplier yet).

I'm totally sure iperf is not really the real way to test these switches, but at least we get an idea of what their performance is under very favourable synthetic conditions.

Hope that helps!

Cheers,
Bryan
 
Last edited:
  • Like
Reactions: Patrick