Looking for Feedback on Next-Gen STH Network Device Testing

[Patrick Kennedy]

Our new high-end network load tool is now up and running. This is your chance to shape how we review network devices on STH

The post Looking for Feedback on Next-Gen STH Network Device Testing appeared first on ServeTheHome.

Continue reading...

 

[siematos]

I'd put the target range of testing where the SMB sector would be in a 3-4 year time frame. My current situation is, that I find myself dealing with 100gbit switches donated to a local nonprofit I'm active in, because the donor's upgrading to faster networking. Talks with them already indicate, that we can expect hand-downs for 200gbit equipment within 3 years due to the recent super short replacement time in DC land. So that's where I see the upper end of SMB as well at that point in time. That said, in a homelab setting, I'm limited by efficiency. I'm not going to place a 300W arista in my cabinet, but I already am running 2x CRS510-8XS-2XQ-IN / (Q)SFP28/25gbit for ~almost~ 2 years now. Upgrading to SFP56 is possible, but only within a similar power consumption footprint. At this point, decking out 16 tranceivers is already roughly +100Watts though on top of the baseline switch power, a trade I'm not willing to take in a personal capacity.

If I base a suggestion for testing on that, I'd say, that it might be a good point in time to start testing (Q)SFP56 somewhat soon. I've often referenced and rewatched some older content when switching, so having a decent archive seems like a good place to be. Preceeding SMB and prosumer adoption by a decent timeframe seems to be a smart idea, especially since it should bring recurring views/reads.

I'd love to see content in more adjacent topics though. 60ghz point-to-point connections are becoming a thing, and I see them being used in metropolitan areas recently to connect branches. We placed a bunch of them for different organisations on roof tops and landmarks, with churches and large buildings like hospitals acting as a proxy for locations without direct line-of-sight. With 1° beams for little interference and thanks to channels which are twice as wide now, we're pushing 10gbit through those.

Another interesting topic would be, how to circumvent the limits imposed by energy consumption in ethernet settings. USB4.2 does specify an ethernet domain for example, which could be interesting for short distance connections, not just for the theoretically possible speeds but energy consumption. I could see that being a larger topic in a few years. With RJ45 reaching signal integrity limits, the question of what comes after seems fairly pressing, when looking beyond 10GBASE-T. I mean, there are Cat8 and Cat8.1 plugs for up to 40gbit, but what after?

I hope you can make something of my ramblings. Ty for all the good content.

 

[justincormack]

Reliability testing is really important, that sounds great if you can test high load scenarios and if devices just die thats really useful information.If it is reproducible hopefully they can fix it too, but thats a whole level of value.

 

[i386]

I think "the more the better" for benchmarks/tests? Similar to how there are multiple benchmark tools when you guys test/review ssds (crystal diskmark, atto disk mark, etc)
Especially because I like to be able to reproduce tests (without paying 1m every year)...

One thing that I would like to see is what software/firmware (drivers are less interesting, but still relevant) was used on the network devices. Sometimes manufactures mess things up in a version and the results are drastically different from reviews/tests/white papers...

Spoiler: Small rant about iperf(3):

Iperf is a very specific "workload" and is usually among the first results when somebody searches the internet for "test network performance". That it's not the best tool to test high speed interconnects (>=10 GBit/s) and not intended for that purpose (see esnet posts on their iperf3 github page) is rarely mentioned...

 

[jode]

Trying to respond to workload requests...

1. define a set of tasks that simulate workload representing a typical (average) user
2. measure max number of users

• "Home" workload: mix of web browsing a mix of popular sites, online searches (Google search, Duckduck go), Mobile Device (Android/Apple cloud), Social Media (FB, Insta, etc.), AI (mix of providers), spotify audio streams, youtube/netflix/etc video streams, office workloads: MS365/Google/etc., related dns/ntp requests
• "Homelab/small office" workload: video editing, data transfer (SMB/NFS), VM access (Proxmox/etc.), related dns/ntp/AD requests
• "Office" workload: MS365/GoogleOffice, MS Power(BI/Apps/etc.), Cloud (AWS, Azure, etc.), File sharing (SMB), Email (Outlook), office security cams, admin overhead (SNMP, authentication, authorization, etc.)
• "datacenter" workload: ...

 

[Patrick]

All good feedback. @siematos - I think we are going to end up with like a 1-25GbE set, a 50-200GbE set, then a 400GbE+ set. We were going to post on a SMB 400GbE switch today but for the NVIDIA-Intel thing. On the SFP56, that is why we have these (one currently installed but we may end up with two)

@justincormack on the reliability side, we can find things like we did with the Ubiquiti in this post, but real reliability is probably too hard/ expensive for us to do since it requires a LOT of devices, ovens, and so forth.

@i386 that is the challenge with this one. Even the charts come out looking very different from iperf3 where you might see a straight maximum line, but then with applications you often see more ups and downs in the charts depending on the number of users, sessions, and the apps running. The challenge is that we have too much data now. We are going to show off a new free tool soon though.

@jode That "home" we can do easily. Small office is a good one that I think we can mostly do, maybe we would add some mysql tests and such in there. Office, yes. The difference is that I think in our current versions I have YouTube and a few other less business-focused apps, but that you would find people using. Funny thing is that there are also apps like Tinder that we can emulate and even try to block with firewalls if needed.

Lots of different things we can emulate.

All great feedback

 

[kapone]

I’d love to see a “tool” where you “draw” your network topology, down to routes, firewall rules, subnets, gateways, DHCP/Static etc, and it “hooks” into a whole bunch of vendor APIs, and essentially renders your network.

Bonus points for zero touch provisioning on an going basis.

Then run network testing end to end, for real world, on the ground data.

All of this is possible today, except its a hot mess of tools.

 

[Crond]

Wish list

Basic synthetic tests for Ethernet routers for different pkt size 64/512/1518/IMIX across different configuration and modes

1. mode: Bridge | routing | NAT | site-to-site VPN ( OpenVPN | wireguard | ipsec )
2. IP filters: none / 25 / 100 / etc
3. QoS queues: none / 10 / 25 / 100 / etc

Datapoints to collect for tests above

• Max throughput
• Max NDR (non-drop rate)
• Latency histogram at 10/20/30/../100% NDR

---
Stability / security

• Security of the device itself. pen-test from LAN and WAN.
• DoS attacks

---
Use case driven benchmarks to mimic real scenarios depending on type of device (Enterprise / edge / home )
(I.E capture traffic, mix and play back)

1. Home use traffic profiles
   1. Streaming (Netflix / Max / etc)
   2. Video calls (face time / telegram / what's up)
   3. Online gaming ( top 10/100 titles from google and apple stores)
   4. Web download.
   5. etc
2. Enterprise profiles
   1. Outlook / MS office
   2. Webex/Google meet/MS teams
   3. Git / gerrit
   4. Web browsing

---
VPN Home:
Select some solid reference. I.E recent server hardware + ubuntu LTS.
Compare throughtput / latency to major VPN service providers to a device under test.

---
Performance impact of advanced features like DPI

Hope that helps

 

[Kalgen]

At least for my home/smb use, the biggest bandwidth demand is a single workstation copying from a NAS. The current 2.5gb and 10gb switch roundups show a graph of all ports being used at once, which is great for what it is, but they don't (a) include bandwidth as a number so it's pixel peeping to guess what per-port bandwidth in that case is, and (b) I care a lot more about the speed of the single transfer between desktop/NAS because that's what my home network does -- it's not doing simultaneous 10Gbps transfers. And the rest of web browsing/email/streaming/etcetcetc is noise: 0.1 Gbps or less in aggregate. Any switch will handle those fine. (Though maybe not any router.) So please consider including what fraction of link speed it actually achieves in a single transfer?

The other important measures are latency and drops/failures/reliability of packet/flow delivery (not reliability of the hardware). E.g., the sample graphs from your new hotness show 1700 successful connections and a line that could be 0 or could be 50 for failures, but that's indistinguishable. Likewise, latency on the graph is dominated by a single max point in the 100s of ms range, and 99-percentile is "low" somewhere in the 0-10ms range but again the graph scale hides it. Maybe tl/dr: please choose better axes scales and/or include a table with relevant numbers not just graphs?

Reflecting at a high level, those graphs are a data dump, but it's unclear how they can be used to drive real-world decision making around which switch to buy.

 

[Darkmatter501]

Very happy to see more detailed networking coverage!

One thing I would suggest is looking at the tests which vendors find important. Some of this is mostly for dialing in DPDK, but other parts like RFC2544 are staple tests that vendors want since it's a standard setup that can be used to compare hardware easily. In particular, some NICs can have a noticeable perf hit when under heavy load and using SR-IOV or vDPA and having public data about that would be valuable.

Also, just be warned that iPerf will start to run into issues at higher speeds. We probably only have 5-7 more years of being able to push single-flow TCP at line rate before the window size limitation renders it impossible. UDP mode will help with that but you'll need to have tests set up for that and you'll want to be testing that along the way so you know when you've run into network stack limitations.

It would also be neat if you could capture what offloads Linux says the NIC offers as far as Ethernet is concerned. That data isn't really available anywhere and stuff that "just works" with the kernel network stack and any application is very valuable.

A few other tests I want to see:

* High rate small packets vs hardware offloads. Particularly relevant are stateful offloads like flow rate limiting, protocol (TCP, QUIC) offloading/acceleration, stateful firewall offloads, TLS/macSEC/IPSec, etc
* Line rate multicast small ipv6 packets. I have seen many a switch, NIC, and appliance crash and burn under traffic that some databases will actually generate if on an IPv6 network and under heavy load.
* Windows vs Linux. I don't think some people understand just how much slower the Windows network stack is at some tasks. For those of you who remember, running Linux on tiny ARM cores and doing all of the network processing there was faster than doing it on fairly high-end consumer hardware when Barefoot Networks released their proto-DPU. The situation has not improved as much as it should, which is also why Windows leans so heavily on RDMA.
* P2P DMA is not really tested anywhere public outside of microbenchmarks.
* Distributed databases, especially distributed SQL such as CockroachDB or YugabyteDB. Distributed joins are both very high bandwidth and latency sensitive so the features provided by a NIC matter a lot.
* Test whether cryptographic offloads can actually do line rate, same for compression offloads if the NIC has them + storage offloads.
* Testing the latency of getting a packet in and out of the system using DPDK testpmd's io forwarding functionality (for very low overhead) would be good since NICs have wildly varying latency characteristics.
* For anything with tc-flower or OVS offloads, shuffling data between containers/vms on the same host is an important but poorly-covered workload. Some NICs will do far in excess of their total port bandwidth for this (esp. Mellanox/Nvidia with SR-IOV), while others can't even do their total port bandwidth.

 

[Kalgen]

Especially like the comment about high rate small packets, especially for routers or the cheap and cheerful 2.5gb and 10gb switch reviews STH does. Mikrotik's test results with 64 byte packets are always illuminating (e.g. CRS317-1G-16S+RM | MikroTik) to give a concrete example.

 

[( )]

Seems like posting directly on the article isn't working again, it just eats your post and nothing shows up; even without any links.

---

TLDR: I think Teledyne would have been better, and you need a bigger attack / test library than 500 profiles.

If you want to stick with Keysight they have their APS-100/400GE Series Platform, might as well get their best. One thing I noticed is they they make a variety of test equipment, they don't specialize in networking.

Of course, if that appeals to you, there's Teledyne Lecroy's Z800 Freya, like it sounds, it does multiple 800G and goes down to layer 1. The Xena B2400 is a 4U modular chassis that can be equipped with up to 12 Ethernet test modules. Do Automated Testing with ValkyrieBay from Xena Networks.

Somewhere that does specialize (check out their customer list) are places like: Apposite’s Netropy, CyberAttack and their Attack Library, FortiGuard Labs and Fortinet's FortiClient, or SolarWinds WAN Killer network traffic generator (intimidating name).

In the free category there's:
Cisco TRex, Ostinato (stateless only), idsEventGenerator a.k.a Genesids (plays Snort signatures), free makes a great addition to an expensive network testing appliance.

 

[Patrick]

Seems like posting directly on the article isn't working again, it just eats your post and nothing shows up; even without any links.

---

TLDR: I think Teledyne would have been better, and you need a bigger attack / test library than 500 profiles.

If you want to stick with Keysight they have their APS-100/400GE Series Platform, might as well get their best. One thing I noticed is they they make a variety of test equipment, they don't specialize in networking.

Of course, if that appeals to you, there's Teledyne Lecroy's Z800 Freya, like it sounds, it does multiple 800G and goes down to layer 1. The Xena B2400 is a 4U modular chassis that can be equipped with up to 12 Ethernet test modules. Do Automated Testing with ValkyrieBay from Xena Networks.

Somewhere that does specialize (check out their customer list) are places like: Apposite’s Netropy, CyberAttack and their Attack Library, FortiGuard Labs and Fortinet's FortiClient, or SolarWinds WAN Killer network traffic generator (intimidating name).

In the free category there's:
Cisco TRex, Ostinato (stateless only), idsEventGenerator a.k.a Genesids (plays Snort signatures), free makes a great addition to an expensive network testing appliance.

That might have been the case but a few thoughts:

We are already at over 2x the throughput of the Z800. I am hoping in 2026 we are doing 3.2Tbps-4Tbps.

Keysight bought IXIA and more and announced an acquisition for Spirent. On the APS-100/400G, we are using CyPerf but do not have the BreakingPoint/ ixia traffic generator. You are right both of those would be cool, and getting a giant chassis would be neat. I am also looking ahead to next year when we start seeing a lot more 800G. Who knows where this all goes a year or two from now.

I think Apposite says 100Gbps max, WAN Killer I believe was one of the ones that it is more of a packet generator. I wanted something that would generate real flows. CyPerf, as we showed in the accompanying piece, acutally is being detected as real application flows.

Ostinato the founder actually reached out after we were down the CyPerf path. Maybe that is an option to add.

T-Rex we actually even had two of the Napatech 100G FPGAs and expensive validated optics for, but scaling to 1Tbps was basically a no-go. Also, it really is a painful user experience. We had a Spirent appliance that would have been cool for 10G and lower traffic, but it never made it to reviews.

One other, and very important factor to keep in mind is that we test a wide range of devices. We currently have 4x 1/2.5/5/10Gbase-T ports, and 4x SFP56 ports for lower-speed traffic as well. I know that sounds useless, but if you have a 400Gbps port on a test appliance, and have to get down to 2.5GbE NICs, that is probably a different switch. We can just do those directly using onboard ports making 2.5Gbps much easier.

You are right, there is a lot more we can do. On the other hand, the goal was to get one tool that could get us to 1.6Tbps in a single box. Also, I think it is cool to have traffic that looks real to a DUT rather than just shooting random packet size mixes.

@Kalgen - The straight small packet stuff is something I understand the need for. Still working on how we can do that.

@Darkmatter501 The goal of the old Spirent test was to do RFC2544

The way I am looking at this is more of like do we just go and define our own standards for STH. Maybe we need a <25Gbps router/ firewall set of 5-10 traffic mixes. Some may be simple throughput. Some may be 2-3 complex traffic scenarios. One or two might be connection/ user scaling. Then we might have 1-3 where we inject attack profiles.

Then we take those say 5-10 tests, present it as our test suite findings.

The challenge is that many of these will generate thousands (or more) of individual metrics. Realistically, we will get very few folks who want that data rather than a summary view.

 

[( )]

We are already at over 2x the throughput of the Z800. I am hoping in 2026 we are doing 3.2Tbps-4Tbps.

Sorry Patrick, I missed that part.

... Who knows where this all goes a year or two from now.

Hopefully you can use the equipment both for reviews and in-house, but also charge as a service; this recovering some of your money.

... T-Rex we actually even had two of the Napatech 100G FPGAs and expensive validated optics for, but scaling to 1Tbps was basically a no-go. Also, it really is a painful user experience.

Leave it to Cisco to go from leader (decades ago) to the one people complain about the most.

We had a Spirent appliance that would have been cool for 10G and lower traffic, but it never made it to reviews.

One other, and very important factor to keep in mind is that we test a wide range of devices. We currently have 4x 1/2.5/5/10Gbase-T ports, and 4x SFP56 ports for lower-speed traffic as well. I know that sounds useless, but if you have a 400Gbps port on a test appliance, and have to get down to 2.5GbE NICs, that is probably a different switch. We can just do those directly using onboard ports making 2.5Gbps much easier.

You are right, there is a lot more we can do. On the other hand, the goal was to get one tool that could get us to 1.6Tbps in a single box. Also, I think it is cool to have traffic that looks real to a DUT rather than just shooting random packet size mixes.

Agreed, need both the "small stuff" and the "big stuff", as the interconnection cabling (optics) isn't one size fits all.

 

[athurdent]

Hi @Patrick ,
is it possible to elaborate on the 2048 users limit you hit where you had to hard reboot the desktop gateway?
To me this sounds normal for a firewall with a maxed-out state table - it will appear unresponsive as it won't accept new connections, until the state table has freed up timed out states. So, how long did you wait until hard reboot? Did an established SSH session running e.g. top also freeze?
I'd love to recreate this, but it seems the CE edition I could install is limited in that direction, there's no parameter I can find that would specify user count.

 

[Patrick]

Hi @Patrick ,
is it possible to elaborate on the 2048 users limit you hit where you had to hard reboot the desktop gateway?
To me this sounds normal for a firewall with a maxed-out state table - it will appear unresponsive as it won't accept new connections, until the state table has freed up timed out states. So, how long did you wait until hard reboot? Did an established SSH session running e.g. top also freeze?
I'd love to recreate this, but it seems the CE edition I could install is limited in that direction, there's no parameter I can find that would specify user count.

It might have been, the 1536 user stop came back after a few minutes, but in a severely degraded state (from 5-10Gbps to ~1Gbps.) A reboot fixed that as well.

The 2048 user failure we waited 30+ minutes thinking it was the testing engine that failed before we realized it was the UI device not responding to web requests, and we needed to hard power cycle it. Then we tried three more times, where we power cycled after about 5 minutes to confirm, then we set the user limit lower on that test.

 

[athurdent]

It might have been, the 1536 user stop came back after a few minutes, but in a severely degraded state (from 5-10Gbps to ~1Gbps.) A reboot fixed that as well.

The 2048 user failure we waited 30+ minutes thinking it was the testing engine that failed before we realized it was the UI device not responding to web requests, and we needed to hard power cycle it. Then we tried three more times, where we power cycled after about 5 minutes to confirm, then we set the user limit lower on that test.

Thank you for the response.
I'd love to try and create similar tests with a pre-existing SSH session, to see if the gateway just runs out of states or actually locks/crashes.
Can you elaborate on how "a user" is defined in that software? How many concurrent connections does it spin up to simulate a user for example, what else does it do?
I have tested the CE edition now, but it has a pretty limited feature set it seems.
In general, simulating 1536 or 2048 users for a residential desktop gateway seems high, but I like the approach to push a device to its limits and find out its maximum capabilities, while also verifying the tech specs. For the UCG Fiber they state 500+ users, so unsure about the "+" but you could give 501 users a try and see how it behaves.