Different speeds to and from on 10G network.

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.
C

CJRoss

Member
May 31, 2017
91
6
8
This is a new one to me and I'm hoping some of you has an idea as to why it's happening. I'm running an Ubuntu desktop with an Aquantia NIC and connecting to a FreeNAS Xeon D-1540 via a QNAP QSW-1208-8C. This is all 10GBaseT.

When transferring TO the NAS I get 5Gb/s, but when transferring FROM the NAS I get 7Gb/s. This is via iperf3.

Any ideas why it would be so much slower transferring to the NAS than from it?
 
C

CJRoss

Member
May 31, 2017
91
6
8
The one difference I did find is the CPU utilization in the tests.

Code: 
Starting Test: protocol: TCP, 1 streams, 131072 byte blocks, omitting 0 seconds, 10 second test, tos 0
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   846 MBytes  7.09 Gbits/sec    0   3.00 MBytes      
[  5]   1.00-2.00   sec   858 MBytes  7.19 Gbits/sec    0   3.00 MBytes      
[  5]   2.00-3.00   sec   858 MBytes  7.20 Gbits/sec    0   3.00 MBytes      
[  5]   3.00-4.00   sec   880 MBytes  7.39 Gbits/sec    0   3.00 MBytes      
[  5]   4.00-5.00   sec   874 MBytes  7.32 Gbits/sec    0   3.00 MBytes      
[  5]   5.00-6.00   sec   873 MBytes  7.33 Gbits/sec    0   3.00 MBytes      
[  5]   6.00-7.00   sec   887 MBytes  7.44 Gbits/sec    0   3.00 MBytes      
[  5]   7.00-8.00   sec   882 MBytes  7.40 Gbits/sec    0   3.00 MBytes      
[  5]   8.00-9.00   sec   842 MBytes  7.07 Gbits/sec    0   3.00 MBytes      
[  5]   9.00-10.00  sec   842 MBytes  7.06 Gbits/sec    0   3.00 MBytes      
- - - - - - - - - - - - - - - - - - - - - - - - -
Test Complete. Summary Results:
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  8.44 GBytes  7.25 Gbits/sec    0             sender
[  5]   0.00-10.22  sec  8.44 GBytes  7.09 Gbits/sec                  receiver
CPU Utilization: local/sender 99.1% (0.6%u/98.5%s), remote/receiver 16.3% (0.4%u/15.8%s)


Starting Test: protocol: TCP, 1 streams, 131072 byte blocks, omitting 0 seconds, 10 second test, tos 0
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   598 MBytes  5.02 Gbits/sec  3598   1.64 MBytes      
[  5]   1.00-2.00   sec   524 MBytes  4.39 Gbits/sec  4065   1.61 MBytes      
[  5]   2.00-3.00   sec   569 MBytes  4.77 Gbits/sec  4530   1.64 MBytes      
[  5]   3.00-4.00   sec   582 MBytes  4.89 Gbits/sec  4561   1.64 MBytes      
[  5]   4.00-5.00   sec   592 MBytes  4.97 Gbits/sec  3895   1.59 MBytes      
[  5]   5.00-6.00   sec   590 MBytes  4.95 Gbits/sec  3895   1.66 MBytes      
[  5]   6.00-7.00   sec   584 MBytes  4.90 Gbits/sec  3967   1.67 MBytes      
[  5]   7.00-8.00   sec   580 MBytes  4.87 Gbits/sec  3951   1.47 MBytes      
[  5]   8.00-9.00   sec   572 MBytes  4.80 Gbits/sec  4672   1.67 MBytes      
[  5]   9.00-10.00  sec   572 MBytes  4.80 Gbits/sec  4010   1.67 MBytes      
- - - - - - - - - - - - - - - - - - - - - - - - -
Test Complete. Summary Results:
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  5.63 GBytes  4.84 Gbits/sec  41144             sender
[  5]   0.00-10.00  sec  5.63 GBytes  4.83 Gbits/sec                  receiver
CPU Utilization: local/sender 17.2% (0.4%u/16.8%s), remote/receiver 35.7% (1.4%u/34.3%s)
 
C

CJRoss

Member
May 31, 2017
91
6
8
This is what happens when I test with my Xeon E3-1220v3 system.

Code: 
Starting Test: protocol: TCP, 1 streams, 131072 byte blocks, omitting 0 seconds, 10 second test, tos 0
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   542 MBytes  4.55 Gbits/sec  231   46.7 KBytes       
[  5]   1.00-2.00   sec   601 MBytes  5.04 Gbits/sec  215   65.0 KBytes       
[  5]   2.00-3.00   sec   571 MBytes  4.79 Gbits/sec  220   89.1 KBytes       
[  5]   3.00-4.00   sec   578 MBytes  4.84 Gbits/sec  236   32.5 KBytes       
[  5]   4.00-5.00   sec   592 MBytes  4.97 Gbits/sec  221    322 KBytes       
[  5]   5.00-6.00   sec   685 MBytes  5.75 Gbits/sec  249   65.0 KBytes       
[  5]   6.00-7.00   sec   598 MBytes  5.01 Gbits/sec  240   46.7 KBytes       
[  5]   7.00-8.00   sec   621 MBytes  5.21 Gbits/sec  220   65.0 KBytes       
[  5]   8.00-9.00   sec   584 MBytes  4.90 Gbits/sec  211   99.0 KBytes       
[  5]   9.00-10.00  sec   585 MBytes  4.91 Gbits/sec  218   53.7 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
Test Complete. Summary Results:
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  5.82 GBytes  5.00 Gbits/sec  2261             sender
[  5]   0.00-10.00  sec  5.81 GBytes  4.99 Gbits/sec                  receiver
CPU Utilization: local/sender 9.7% (0.2%u/9.5%s), remote/receiver 14.6% (1.6%u/13.0%s)

Starting Test: protocol: TCP, 1 streams, 131072 byte blocks, omitting 0 seconds, 10 second test, tos 0
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  1.09 GBytes  9.39 Gbits/sec    0   3.00 MBytes       
[  5]   1.00-2.00   sec  1.09 GBytes  9.39 Gbits/sec    0   3.00 MBytes       
[  5]   2.00-3.00   sec  1.09 GBytes  9.39 Gbits/sec    0   3.00 MBytes       
[  5]   3.00-4.00   sec  1.09 GBytes  9.39 Gbits/sec    0   3.00 MBytes       
[  5]   4.00-5.00   sec  1.09 GBytes  9.39 Gbits/sec    0   3.00 MBytes       
[  5]   5.00-6.00   sec  1.09 GBytes  9.39 Gbits/sec    0   3.00 MBytes       
[  5]   6.00-7.00   sec  1.09 GBytes  9.39 Gbits/sec    0   3.00 MBytes       
[  5]   7.00-8.00   sec  1.09 GBytes  9.39 Gbits/sec    0   3.00 MBytes       
[  5]   8.00-9.00   sec  1.09 GBytes  9.39 Gbits/sec    0   3.00 MBytes       
[  5]   9.00-10.00  sec  1.09 GBytes  9.39 Gbits/sec    0   3.00 MBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
Test Complete. Summary Results:
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  10.9 GBytes  9.39 Gbits/sec    0             sender
[  5]   0.00-10.12  sec  10.9 GBytes  9.28 Gbits/sec                  receiver
CPU Utilization: local/sender 28.1% (2.4%u/25.7%s), remote/receiver 15.5% (0.3%u/15.2%s)
 
C

CJRoss

Member
May 31, 2017
91
6
8
I can only assume there's an issue with the Aquantia card as I can't believe a Ryzen 5 3600 would be slower than either of the Xeons.
 
D

Decoy

New Member
May 11, 2020
8
3
3
I've got no answer for you (have a similar issue) - but my first thought was the number of retries showing up on your slower iperf3 test. Is there an MTU mismatch or something causing fragmentation or similar?
 
R

Rand__

Well-Known Member
Mar 6, 2014
6,634
1,767
113
No solution either but have a similar issue 10G one way, 4G the other - never resolved it :/
 
B

blinkenlights

Active Member
May 24, 2019
157
65
28
CJRoss said:
I can only assume there's an issue with the Aquantia card as I can't believe a Ryzen 5 3600 would be slower than either of the Xeons.
Click to expand...
I know all of the tech influencers get wobbly in the knees over Aquantia, but I never had much luck with them. Last test was an AQC107 to an ICX7450-48 with 10 GigE modules - the card showed poor performance under CentOS/Fedora and even shut itself off. Only saving grace was an easy return through Amazon.

I admit both FreeNAS and recent Ubuntu releases are unfamiliar to me, but I have found that most *nix distributions are (still) not properly tuned for 10 Gbps out of the box.

Calomel is a good resource for FreeBSD: FreeBSD Network Performance Tuning @ Calomel.org (offline, using archive.org link)

The DoE labs have good resources for Linux: Linux Tuning

Have you tried opening a support ticket with QNAP? No offense intended, but I find something marketed as "My first 10GbE switch" to be highly suspect in terms of performance :rolleyes:
 
C

CJRoss

Member
May 31, 2017
91
6
8
Decoy said:
I've got no answer for you (have a similar issue) - but my first thought was the number of retries showing up on your slower iperf3 test. Is there an MTU mismatch or something causing fragmentation or similar?
Click to expand...
You know, I completely missed that. I'm using all of the defaults for network settings so I'll have to go look.
 
C

CJRoss

Member
May 31, 2017
91
6
8
blinkenlights said:
I know all of the tech influencers get wobbly in the knees over Aquantia, but I never had much luck with them. Last test was an AQC107 to an ICX7450-48 with 10 GigE modules - the card showed poor performance under CentOS/Fedora and even shut itself off. Only saving grace was an easy return through Amazon.
Click to expand...
Agreed. I do have issues where sometimes the Aquantia card appears to forget there's a connection, but that results in a full network drop, not these issues. Additionally, I'm seeing slowness between the two FreeNAS boxes and neither of them have an Aquantia card.

blinkenlights said:
I admit both FreeNAS and recent Ubuntu releases are unfamiliar to me, but I have found that most *nix distributions are (still) not properly tuned for 10 Gbps out of the box.

Calomel is a good resource for FreeBSD: FreeBSD Network Performance Tuning @ Calomel.org (offline, using archive.org link)

The DoE labs have good resources for Linux: Linux Tuning
Click to expand...
I'll check that out. What really puzzles me is the asymetric part of the issue. I would think if they weren't tuned correctly it would manifest on both sides, or iperf would be line speed while actual transfers would be slower.

blinkenlights said:
Have you tried opening a support ticket with QNAP? No offense intended, but I find something marketed as "My first 10GbE switch" to be highly suspect in terms of performance :rolleyes:
Click to expand...
I have not. I did open a ticket and end up upgrading the firmware because some of the ports weren't recognizing some of my cables, but I'm still seeing the same speed issues.
 
C

CJRoss

Member
May 31, 2017
91
6
8
From the E3 to the D.

Code: 
tarting Test: protocol: TCP, 1 streams, 131072 byte blocks, omitting 0 seconds, 10 second test, tos 0
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   614 MBytes  5.15 Gbits/sec   19    892 KBytes       
[  5]   1.00-2.00   sec   680 MBytes  5.70 Gbits/sec   63    958 KBytes       
[  5]   2.00-3.00   sec   669 MBytes  5.62 Gbits/sec    2   1018 KBytes       
[  5]   3.00-4.00   sec   673 MBytes  5.65 Gbits/sec    3   1.04 MBytes       
[  5]   4.00-5.00   sec   672 MBytes  5.64 Gbits/sec    2   1.08 MBytes       
[  5]   5.00-6.00   sec   673 MBytes  5.65 Gbits/sec    3   1.12 MBytes       
[  5]   6.00-7.00   sec   673 MBytes  5.65 Gbits/sec    3   1.17 MBytes       
[  5]   7.00-8.00   sec   674 MBytes  5.65 Gbits/sec    2   1.21 MBytes       
[  5]   8.00-9.00   sec   672 MBytes  5.63 Gbits/sec    2   1.26 MBytes       
[  5]   9.00-10.00  sec   673 MBytes  5.65 Gbits/sec    2   1.29 MBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
Test Complete. Summary Results:
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  6.52 GBytes  5.60 Gbits/sec  101             sender
[  5]   0.00-10.00  sec  6.52 GBytes  5.60 Gbits/sec                  receiver
CPU Utilization: local/sender 77.5% (14.4%u/63.2%s), remote/receiver 72.6% (6.5%u/66.1%s)
snd_tcp_congestion newreno
rcv_tcp_congestion newreno
From the D to the E3

Code: 
Starting Test: protocol: TCP, 1 streams, 131072 byte blocks, omitting 0 seconds, 10 second test, tos 0
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   513 MBytes  4.30 Gbits/sec   24    273 KBytes       
[  5]   1.00-2.01   sec   418 MBytes  3.46 Gbits/sec   18   62.7 KBytes       
[  5]   2.01-3.00   sec   510 MBytes  4.33 Gbits/sec   29   59.9 KBytes       
[  5]   3.00-4.00   sec   521 MBytes  4.37 Gbits/sec   28   77.0 KBytes       
[  5]   4.00-5.00   sec   490 MBytes  4.11 Gbits/sec   32   49.9 KBytes       
[  5]   5.00-6.01   sec   458 MBytes  3.82 Gbits/sec   22   52.7 KBytes       
[  5]   6.01-7.00   sec   519 MBytes  4.37 Gbits/sec   22    328 KBytes       
[  5]   7.00-8.01   sec   440 MBytes  3.65 Gbits/sec   20   38.5 KBytes       
[  5]   8.01-9.00   sec   542 MBytes  4.59 Gbits/sec   15   42.8 KBytes       
[  5]   9.00-10.00  sec   524 MBytes  4.39 Gbits/sec   24   47.0 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
Test Complete. Summary Results:
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  4.82 GBytes  4.14 Gbits/sec  234             sender
[  5]   0.00-10.04  sec  4.82 GBytes  4.12 Gbits/sec                  receiver
CPU Utilization: local/sender 68.0% (0.7%u/67.3%s), remote/receiver 27.8% (4.0%u/23.8%s)
snd_tcp_congestion newreno
rcv_tcp_congestion newreno
I did find that if I have the E3 in the port that was giving me trouble previously that my connection from the desktop drops to this.
Code: 
Starting Test: protocol: TCP, 1 streams, 131072 byte blocks, omitting 0 seconds, 10 second test, tos 0
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   595 KBytes  4.88 Mbits/sec   46   2.83 KBytes       
[  5]   1.00-2.00   sec  2.82 MBytes  23.7 Mbits/sec  244   2.83 KBytes       
[  5]   2.00-3.00   sec   255 KBytes  2.08 Mbits/sec   28   2.83 KBytes       
[  5]   3.00-4.00   sec  1.37 MBytes  11.5 Mbits/sec  120   2.83 KBytes       
[  5]   4.00-5.00   sec   255 KBytes  2.09 Mbits/sec   26   2.83 KBytes       
[  5]   5.00-6.00   sec   509 KBytes  4.17 Mbits/sec   32   2.83 KBytes       
[  5]   6.00-7.00   sec   127 KBytes  1.04 Mbits/sec   12   1.41 KBytes       
[  5]   7.00-8.00   sec   127 KBytes  1.04 Mbits/sec   11   2.83 KBytes       
[  5]   8.00-9.00   sec   127 KBytes  1.04 Mbits/sec   19   1.41 KBytes       
[  5]   9.00-10.00  sec   255 KBytes  2.09 Mbits/sec   22   1.41 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
Test Complete. Summary Results:
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  6.39 MBytes  5.36 Mbits/sec  560             sender
[  5]   0.00-10.22  sec  6.18 MBytes  5.07 Mbits/sec                  receiver
CPU Utilization: local/sender 1.2% (0.2%u/1.0%s), remote/receiver 0.3% (0.1%u/0.2%s)
snd_tcp_congestion cubic
rcv_tcp_congestion newreno
I'm going to reach out to QNAP again as something is still wrong with my switch.
 
sboesch

sboesch

Active Member
Aug 3, 2012
467
95
28
Columbus, OH
CJRoss said:
Can you elaborate? I'm not sure exactly what you mean.
Click to expand...
A good managed switch will show you pauses related to flow control. You can use the ethtool app to turn flow control on and off.

ethtool -A eth0 [autoneg on|off] [rx on|off] [tx on|off]
 
M

madbrain

Active Member
Jan 5, 2019
212
44
28
I can reproduce the asynchronous performance issue.

In the case below, the client is running Win10 with an i7-5820k and X99 chipset.
Server is running Skylake with i5-6600k and Z170.
Both sides are using Aquantia AQN-107 PCIe cards.
They are hooked up through a Trendnet TEG-7080ES switch.
MTU is 9000 (jumbo frames).

D:\Downloads\iperf-3.1.3-win64>type server.bat
@iperf3.exe -c server10g -N -i 10 -t 3600 -P4
D:\Downloads\iperf-3.1.3-win64>iperf3.exe -c server10g -N -i 10 -t 3600
Connecting to host server10g, port 5201
[ 4] local 192.168.1.26 port 3615 connected to 192.168.1.27 port 5201
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-10.00 sec 3.52 GBytes 3.02 Gbits/sec
[ 4] 10.00-20.00 sec 3.61 GBytes 3.10 Gbits/sec
[ 4] 20.00-30.00 sec 4.19 GBytes 3.60 Gbits/sec
[ 4] 30.00-40.00 sec 4.20 GBytes 3.61 Gbits/sec
[ 4] 40.00-50.00 sec 4.19 GBytes 3.60 Gbits/sec
[ 4] 50.00-60.00 sec 4.20 GBytes 3.61 Gbits/sec
[ 4] 60.00-60.28 sec 120 MBytes 3.61 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-60.28 sec 24.0 GBytes 3.42 Gbits/sec sender
[ 4] 0.00-60.28 sec 0.00 Bytes 0.00 bits/sec receiver
iperf3: interrupt - the client has terminated

D:\Downloads\iperf-3.1.3-win64>iperf3.exe -c server10g -N -i 10 -t 3600 -R
Connecting to host server10g, port 5201
Reverse mode, remote host server10g is sending
[ 4] local 192.168.1.26 port 3617 connected to 192.168.1.27 port 5201
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-10.00 sec 7.30 GBytes 6.27 Gbits/sec
[ 4] 10.00-20.00 sec 7.49 GBytes 6.43 Gbits/sec
[ 4] 20.00-30.00 sec 7.50 GBytes 6.44 Gbits/sec
[ 4] 30.00-40.00 sec 7.54 GBytes 6.48 Gbits/sec
[ 4] 40.00-50.00 sec 7.55 GBytes 6.48 Gbits/sec
[ 4] 50.00-60.00 sec 7.47 GBytes 6.41 Gbits/sec
[ 4] 60.00-70.00 sec 7.32 GBytes 6.28 Gbits/sec
[ 4] 70.00-73.24 sec 2.37 GBytes 6.29 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-73.24 sec 0.00 Bytes 0.00 bits/sec sender
[ 4] 0.00-73.24 sec 54.5 GBytes 6.40 Gbits/sec receiver
iperf3: interrupt - the client has terminated

D:\Downloads\iperf-3.1.3-win64>start

Since those machines are in the same room, I tried to connect them with a straight through cable instead.
Sending performance was the same at 3.6 Gbps. Receiving performance was a bit better, at 6.9 Gbps.
So, I just reconnected them to the switch.

It takes 4 streams to get close to the line rate for the sending side.

D:\Downloads\iperf-3.1.3-win64>iperf3.exe -c server10g -N -i 10 -t 3600 -P 4
Connecting to host server10g, port 5201
[ 4] local 192.168.1.26 port 3880 connected to 192.168.1.27 port 5201
[ 6] local 192.168.1.26 port 3881 connected to 192.168.1.27 port 5201
[ 8] local 192.168.1.26 port 3882 connected to 192.168.1.27 port 5201
[ 10] local 192.168.1.26 port 3883 connected to 192.168.1.27 port 5201
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-10.00 sec 2.64 GBytes 2.26 Gbits/sec
[ 6] 0.00-10.00 sec 3.02 GBytes 2.59 Gbits/sec
[ 8] 0.00-10.00 sec 2.99 GBytes 2.57 Gbits/sec
[ 10] 0.00-10.00 sec 2.60 GBytes 2.23 Gbits/sec
[SUM] 0.00-10.00 sec 11.2 GBytes 9.66 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 10.00-20.00 sec 2.65 GBytes 2.27 Gbits/sec
[ 6] 10.00-20.00 sec 3.06 GBytes 2.62 Gbits/sec
[ 8] 10.00-20.00 sec 3.03 GBytes 2.60 Gbits/sec
[ 10] 10.00-20.00 sec 2.62 GBytes 2.25 Gbits/sec
[SUM] 10.00-20.00 sec 11.4 GBytes 9.75 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 20.00-30.00 sec 2.63 GBytes 2.26 Gbits/sec
[ 6] 20.00-30.00 sec 3.07 GBytes 2.64 Gbits/sec
[ 8] 20.00-30.00 sec 3.04 GBytes 2.61 Gbits/sec
[ 10] 20.00-30.00 sec 2.61 GBytes 2.25 Gbits/sec
[SUM] 20.00-30.00 sec 11.4 GBytes 9.76 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 30.00-30.69 sec 190 MBytes 2.31 Gbits/sec
[ 6] 30.00-30.69 sec 215 MBytes 2.60 Gbits/sec
[ 8] 30.00-30.69 sec 212 MBytes 2.57 Gbits/sec
[ 10] 30.00-30.69 sec 188 MBytes 2.28 Gbits/sec
[SUM] 30.00-30.69 sec 804 MBytes 9.76 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-30.69 sec 8.10 GBytes 2.27 Gbits/sec sender
[ 4] 0.00-30.69 sec 0.00 Bytes 0.00 bits/sec receiver
[ 6] 0.00-30.69 sec 9.36 GBytes 2.62 Gbits/sec sender
[ 6] 0.00-30.69 sec 0.00 Bytes 0.00 bits/sec receiver
[ 8] 0.00-30.69 sec 9.27 GBytes 2.59 Gbits/sec sender
[ 8] 0.00-30.69 sec 0.00 Bytes 0.00 bits/sec receiver
[ 10] 0.00-30.69 sec 8.02 GBytes 2.24 Gbits/sec sender
[ 10] 0.00-30.69 sec 0.00 Bytes 0.00 bits/sec receiver
[SUM] 0.00-30.69 sec 34.7 GBytes 9.72 Gbits/sec sender
[SUM] 0.00-30.69 sec 0.00 Bytes 0.00 bits/sec receiver
iperf3: interrupt - the client has terminated

For the receive side, 2 streams are enough to saturate.

D:\Downloads\iperf-3.1.3-win64>iperf3.exe -c server10g -N -i 10 -t 3600 -R -P 2
Connecting to host server10g, port 5201
Reverse mode, remote host server10g is sending
[ 4] local 192.168.1.26 port 3887 connected to 192.168.1.27 port 5201
[ 6] local 192.168.1.26 port 3888 connected to 192.168.1.27 port 5201
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-10.00 sec 5.53 GBytes 4.75 Gbits/sec
[ 6] 0.00-10.00 sec 5.58 GBytes 4.79 Gbits/sec
[SUM] 0.00-10.00 sec 11.1 GBytes 9.54 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 10.00-20.00 sec 5.58 GBytes 4.79 Gbits/sec
[ 6] 10.00-20.00 sec 5.64 GBytes 4.84 Gbits/sec
[SUM] 10.00-20.00 sec 11.2 GBytes 9.64 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 20.00-30.00 sec 5.60 GBytes 4.81 Gbits/sec
[ 6] 20.00-30.00 sec 5.65 GBytes 4.85 Gbits/sec
[SUM] 20.00-30.00 sec 11.2 GBytes 9.66 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 30.00-31.32 sec 756 MBytes 4.80 Gbits/sec
[ 6] 30.00-31.32 sec 765 MBytes 4.85 Gbits/sec
[SUM] 30.00-31.32 sec 1.49 GBytes 9.65 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-31.32 sec 0.00 Bytes 0.00 bits/sec sender
[ 4] 0.00-31.32 sec 17.5 GBytes 4.79 Gbits/sec receiver
[ 6] 0.00-31.32 sec 0.00 Bytes 0.00 bits/sec sender
[ 6] 0.00-31.32 sec 17.6 GBytes 4.83 Gbits/sec receiver
[SUM] 0.00-31.32 sec 0.00 Bytes 0.00 bits/sec sender
[SUM] 0.00-31.32 sec 35.1 GBytes 9.62 Gbits/sec receiver
iperf3: interrupt - the client has terminated
 
M

madbrain

Active Member
Jan 5, 2019
212
44
28
There was an issue on my Linux server (server10g) with an Asus Z170-AR motherboard where the x4 slot was set to run at x2 speed. I fixed that in the BIOS (at the expense of sacrificing two unused SATA ports). That got the speed a little bit higher.

However, on the Windows client, it seems that Aquantia drivers have an issue where the speed isn't actually stable. Here are a few consecutive runs :

D:\Downloads\iperf-3.1.3-win64>iperf3.exe -N -c server10g
Connecting to host server10g, port 5201
[ 4] local 192.168.1.26 port 4187 connected to 192.168.1.27 port 5201
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-1.00 sec 398 MBytes 3.34 Gbits/sec
[ 4] 1.00-2.00 sec 404 MBytes 3.39 Gbits/sec
[ 4] 2.00-3.00 sec 390 MBytes 3.27 Gbits/sec
[ 4] 3.00-4.00 sec 405 MBytes 3.40 Gbits/sec
[ 4] 4.00-5.00 sec 398 MBytes 3.34 Gbits/sec
[ 4] 5.00-6.00 sec 396 MBytes 3.32 Gbits/sec
[ 4] 6.00-7.00 sec 379 MBytes 3.18 Gbits/sec
[ 4] 7.00-8.00 sec 400 MBytes 3.35 Gbits/sec
[ 4] 8.00-9.00 sec 344 MBytes 2.88 Gbits/sec
[ 4] 9.00-10.00 sec 394 MBytes 3.31 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-10.00 sec 3.82 GBytes 3.28 Gbits/sec sender
[ 4] 0.00-10.00 sec 3.82 GBytes 3.28 Gbits/sec receiver

iperf Done.

D:\Downloads\iperf-3.1.3-win64>iperf3.exe -N -c server10g
Connecting to host server10g, port 5201
[ 4] local 192.168.1.26 port 4195 connected to 192.168.1.27 port 5201
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-1.00 sec 427 MBytes 3.58 Gbits/sec
[ 4] 1.00-2.00 sec 392 MBytes 3.29 Gbits/sec
[ 4] 2.00-3.00 sec 426 MBytes 3.58 Gbits/sec
[ 4] 3.00-3.10 sec 42.2 MBytes 3.54 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-3.10 sec 1.26 GBytes 3.48 Gbits/sec sender
[ 4] 0.00-3.10 sec 0.00 Bytes 0.00 bits/sec receiver
iperf3: interrupt - the client has terminated

D:\Downloads\iperf-3.1.3-win64>iperf3.exe -N -c server10g
Connecting to host server10g, port 5201
[ 4] local 192.168.1.26 port 4197 connected to 192.168.1.27 port 5201
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-1.00 sec 849 MBytes 7.12 Gbits/sec
[ 4] 1.00-2.00 sec 867 MBytes 7.28 Gbits/sec
[ 4] 2.00-2.79 sec 640 MBytes 6.81 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-2.79 sec 2.30 GBytes 7.09 Gbits/sec sender
[ 4] 0.00-2.79 sec 0.00 Bytes 0.00 bits/sec receiver
iperf3: interrupt - the client has terminated

D:\Downloads\iperf-3.1.3-win64>iperf3.exe -N -c server10g
Connecting to host server10g, port 5201
[ 4] local 192.168.1.26 port 4199 connected to 192.168.1.27 port 5201
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-1.00 sec 403 MBytes 3.38 Gbits/sec
[ 4] 1.00-2.00 sec 391 MBytes 3.28 Gbits/sec
[ 4] 2.00-2.68 sec 273 MBytes 3.35 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-2.68 sec 1.04 GBytes 3.33 Gbits/sec sender
[ 4] 0.00-2.68 sec 0.00 Bytes 0.00 bits/sec receiver
iperf3: interrupt - the client has terminated

D:\Downloads\iperf-3.1.3-win64>iperf3.exe -N -c server10g
Connecting to host server10g, port 5201
[ 4] local 192.168.1.26 port 4201 connected to 192.168.1.27 port 5201
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-1.00 sec 393 MBytes 3.29 Gbits/sec
[ 4] 1.00-2.00 sec 400 MBytes 3.36 Gbits/sec
[ 4] 2.00-2.11 sec 44.4 MBytes 3.44 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-2.11 sec 838 MBytes 3.33 Gbits/sec sender
[ 4] 0.00-2.11 sec 0.00 Bytes 0.00 bits/sec receiver
iperf3: interrupt - the client has terminated

D:\Downloads\iperf-3.1.3-win64>iperf3.exe -N -c server10g
Connecting to host server10g, port 5201
[ 4] local 192.168.1.26 port 4203 connected to 192.168.1.27 port 5201
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-1.00 sec 853 MBytes 7.15 Gbits/sec
[ 4] 1.00-2.00 sec 861 MBytes 7.22 Gbits/sec
[ 4] 2.00-3.00 sec 834 MBytes 7.00 Gbits/sec
[ 4] 3.00-4.00 sec 855 MBytes 7.17 Gbits/sec
[ 4] 4.00-5.00 sec 845 MBytes 7.09 Gbits/sec
[ 4] 5.00-6.00 sec 830 MBytes 6.96 Gbits/sec
[ 4] 6.00-7.00 sec 828 MBytes 6.94 Gbits/sec
[ 4] 7.00-7.51 sec 403 MBytes 6.66 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-7.51 sec 6.16 GBytes 7.05 Gbits/sec sender
[ 4] 0.00-7.51 sec 0.00 Bytes 0.00 bits/sec receiver
iperf3: interrupt - the client has terminated

All are identical, going against the same server. I just CTRL-C early before 10s are up. As you can see, speed is sometimes in the 3 Gbps range, sometimes 7 Gbps range.

I then turned off "interrupt moderation" in the Windows driver settings for the Aquantia card. Subsequent runs are more stable, in the 7 Gbps range, and never dropping down to 3 Gbps.

D:\Downloads\iperf-3.1.3-win64>iperf3.exe -N -c server10g
Connecting to host server10g, port 5201
[ 4] local 192.168.1.26 port 4377 connected to 192.168.1.27 port 5201
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-1.00 sec 891 MBytes 7.48 Gbits/sec
[ 4] 1.00-2.00 sec 799 MBytes 6.70 Gbits/sec
[ 4] 2.00-3.00 sec 860 MBytes 7.21 Gbits/sec
[ 4] 3.00-4.00 sec 816 MBytes 6.85 Gbits/sec
[ 4] 4.00-5.00 sec 822 MBytes 6.89 Gbits/sec
[ 4] 5.00-6.00 sec 835 MBytes 7.01 Gbits/sec
[ 4] 6.00-6.31 sec 272 MBytes 7.49 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-6.31 sec 5.17 GBytes 7.04 Gbits/sec sender
[ 4] 0.00-6.31 sec 0.00 Bytes 0.00 bits/sec receiver
iperf3: interrupt - the client has terminated

The receive side is also stable, but still faster. Now nearly maxing line rate with a single stream, as it should be.

D:\Downloads\iperf-3.1.3-win64>iperf3.exe -N -c server10g -R
Connecting to host server10g, port 5201
Reverse mode, remote host server10g is sending
[ 4] local 192.168.1.26 port 4380 connected to 192.168.1.27 port 5201
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-1.00 sec 1.13 GBytes 9.73 Gbits/sec
[ 4] 1.00-2.00 sec 1.11 GBytes 9.57 Gbits/sec
[ 4] 2.00-3.00 sec 1.13 GBytes 9.74 Gbits/sec
[ 4] 3.00-4.00 sec 1.11 GBytes 9.54 Gbits/sec
[ 4] 4.00-5.00 sec 1.13 GBytes 9.71 Gbits/sec
[ 4] 5.00-6.00 sec 1.07 GBytes 9.22 Gbits/sec
[ 4] 6.00-7.00 sec 1.12 GBytes 9.62 Gbits/sec
[ 4] 7.00-8.00 sec 1.10 GBytes 9.44 Gbits/sec
[ 4] 8.00-9.00 sec 1.12 GBytes 9.59 Gbits/sec
[ 4] 9.00-10.00 sec 1.13 GBytes 9.71 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth Retr
[ 4] 0.00-10.00 sec 11.2 GBytes 9.59 Gbits/sec 0 sender
[ 4] 0.00-10.00 sec 11.2 GBytes 9.59 Gbits/sec receiver

iperf Done.

I'm going to try to do the same thing on the Linux driver.
 
M

madbrain

Active Member
Jan 5, 2019
212
44
28
Seems like adaptive RX/TX is disabled by default in the Linux 2.2.6 Aquantia driver. The following did seem to help speed slightly, though.

ethtool -C enp3s0 rx-usecs 1 tx-usecs 1
 
  • Like
Reactions: sboesch
You must log in or register to reply here.
Top Bottom