New 10Gbe install, wildly different speeds on similar hardware

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.
G

Gerk

New Member
Oct 24, 2016
26
8
3
Oakville, ON, Canada
Hi everyone

I just installed a Quanta LB6M into my home setup and in testing with iperf I'm seeing some wildly different speeds on different machines and I'm not really sure exactly why or where the bottleneck is. The best speeds I've gotten is around 5.5-6Gbs (which I consistently get going to one particular machine from anywhere else). I'm seeing consistency for the speeds where I run the iperf server (i.e. no matter which machines I'm sending the iperf test from the speed seems to be limited by the machine running the iperf server).

I've swapped around cables and ports to make sure it wasn't a rogue cable or port causing slowdowns, but that didn't make a difference. All of my hardware is very very similar and all of the NICs are all basically the same thing. Here's a list of the machines and NICs.

Dell R710 #1 - ~2.2Gbps (bare metal), ~20Gbps localhost
2x Xeon X5650 (6 cores each @ 2.67Ghz)
96GB ram
Intel X520-DA2 in x8 slot
FreeNAS 9.10

Dell R710 #2 - ~4Gbps (VM), ~20Gbps localhost
2x Xeon L5520 (4 cores each @ 2.27GHz)
96GB ram
Intel X520-DA2 (OEM) in x8 slot
ESXi 6

Dell R610 - ~5.5Gbps (VM), ~40Gbps localhost
2x Xeon X5650 (6 cores each @ 2.67GHz)
72GB ram
Intel X520-DA2 in x8 slot
ESXi 6

Can anyone recommend a good small live boot image to test with? I'd like to make some more "Apples to Apples" comparisons as it were ... just to take the OS and virtualized vs bare metal out of the equation.

The funniest thing is that the performance was exactly backwards to what I expected ... I assumed that the bare metal FN box would outperform the esxi servers.

The text following this is no longer relevant as I have changed slots and the OEM card was in a different machine (that's performing well) but I left it for completeness ...

I don't think that the x4 vs x8 slots account for the difference here but maybe I'm wrong -- in theory the x4 slot should still have sufficient bandwidth (guessing the 20Gbps vs 40Gbps illustrates that difference).

Where am I bottlenecking I wonder ... could it be the X520 clone card that's the biggest slowdown? Am I expecting too much from this gear? I thought I could at least get closer to 10Gbps ...
Click to expand...
 
Last edited:
Tom5051

Tom5051

Active Member
Jan 18, 2017
359
79
28
46
These servers all have gen2 PCI-e slots. The 4x slot is only running 1/2 the lanes the NIC card requires as the Intel specs say the x520 requires pci-e 8x therefore it will only impact throughput. Try putting the 2 in the 4x slots into 8x slots if you can and retest.
 
Last edited:
G

Gerk

New Member
Oct 24, 2016
26
8
3
Oakville, ON, Canada
Also worth reporting here is that I tried different virtual NICs in the esxi guests (both e1000 and vmxnet3) and surprisingly didn't see a change in performance. And I also tried to and from a FN10 nightly VM on the R610 (fastest box) and the performance too and from that was terrible (< 2Gbps consistently).
 
G

Gerk

New Member
Oct 24, 2016
26
8
3
Oakville, ON, Canada
Tom5051 said:
x520 needs 8x gen 2 lanes. This could be your issue. 4x will limit throughput.
Click to expand...
But the strange thing is that one machine gets 5.5Gbps in an x4 and the other gets half of that, and then the other machine that has the same card in x8 gets the same throughput as a previous one with x4. That's the part that's the most puzzling.
 
G

Gerk

New Member
Oct 24, 2016
26
8
3
Oakville, ON, Canada
Tom5051 said:
x520 needs 8x gen 2 lanes. This could be your issue. 4x will limit throughput.
Click to expand...
So to test this theory out I swapped the card into an x8 slot and there is no difference at all in the throughput :( I'm seeing the exact same numbers with all the cards in x8 slots now. Also I'm going to update the first post as the OEM card was in a different machine than I thought ...
 
MiniKnight

MiniKnight

Well-Known Member
Mar 30, 2012
3,073
974
113
NYC
What is the command you are using? Are you pegging threads to CPUs? How many parallel? What are MTU sizes?
 
G

Gerk

New Member
Oct 24, 2016
26
8
3
Oakville, ON, Canada
MTU is 1500
Code: 
ix0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=e400b9<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
	ether 00:1b:21:70:8c:2b
	inet 192.168.10.15 netmask 0xffffff00 broadcast 192.168.10.255 
	nd6 options=9<PERFORMNUD,IFDISABLED>
	media: Ethernet autoselect (10Gbase-Twinax <full-duplex,rxpause,txpause>)
	status: active
Not pegging CPU on a single connection, on the slowest machine (sinestro - FN 9.10 box) I'm seeing about 40-50% usage on iperf process and seeing ~2.2Gbps, if I bump up the number of connections I've gotten it up as high as 5Gbps (8 connections) which pushes the CPU harder (80%-170%), however 4 connections was pretty close to the same 5Gpbs speed.

On other machines single connection doesn't max out iperf at all (25%-40%), multiple connections does peg CPU but doesn't give me anything faster than single connection does.

server:
Code: 
$ iperf -s -i 2
client:
Code: 
$ iperf -n 10G -i 1 -c sinestro -P 1
------------------------------------------------------------
Client connecting to sinestro, TCP port 5001
TCP window size: 85.0 KByte (default)
------------------------------------------------------------
[  3] local 192.168.10.174 port 39682 connected with 192.168.10.15 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0- 1.0 sec   210 MBytes  1.76 Gbits/sec
[  3]  1.0- 2.0 sec   248 MBytes  2.08 Gbits/sec
[  3]  2.0- 3.0 sec   252 MBytes  2.11 Gbits/sec
[  3]  3.0- 4.0 sec   252 MBytes  2.12 Gbits/sec
[  3]  4.0- 5.0 sec   259 MBytes  2.18 Gbits/sec
[  3]  5.0- 6.0 sec   255 MBytes  2.14 Gbits/sec
[  3]  6.0- 7.0 sec   259 MBytes  2.17 Gbits/sec
[  3]  7.0- 8.0 sec   251 MBytes  2.11 Gbits/sec
[  3]  8.0- 9.0 sec   250 MBytes  2.10 Gbits/sec
[  3]  9.0-10.0 sec   253 MBytes  2.12 Gbits/sec
[  3] 10.0-11.0 sec   251 MBytes  2.11 Gbits/sec
Code: 
$ iperf -n 10G -i 1 -c sinestro -P 2
------------------------------------------------------------
Client connecting to sinestro, TCP port 5001
TCP window size: 85.0 KByte (default)
------------------------------------------------------------
[  4] local 192.168.10.174 port 39686 connected with 192.168.10.15 port 5001
[  3] local 192.168.10.174 port 39684 connected with 192.168.10.15 port 5001
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0- 1.0 sec   166 MBytes  1.39 Gbits/sec
[  3]  0.0- 1.0 sec   173 MBytes  1.45 Gbits/sec
[SUM]  0.0- 1.0 sec   339 MBytes  2.84 Gbits/sec
[  4]  1.0- 2.0 sec   177 MBytes  1.49 Gbits/sec
[  3]  1.0- 2.0 sec   178 MBytes  1.49 Gbits/sec
[SUM]  1.0- 2.0 sec   355 MBytes  2.98 Gbits/sec
[  4]  2.0- 3.0 sec   178 MBytes  1.49 Gbits/sec
[  3]  2.0- 3.0 sec   181 MBytes  1.52 Gbits/sec
[SUM]  2.0- 3.0 sec   358 MBytes  3.01 Gbits/sec
[  4]  3.0- 4.0 sec   179 MBytes  1.50 Gbits/sec
[  3]  3.0- 4.0 sec   186 MBytes  1.56 Gbits/sec
[SUM]  3.0- 4.0 sec   365 MBytes  3.06 Gbits/sec
[  4]  4.0- 5.0 sec   178 MBytes  1.49 Gbits/sec
[  3]  4.0- 5.0 sec   187 MBytes  1.57 Gbits/sec
[SUM]  4.0- 5.0 sec   365 MBytes  3.06 Gbits/sec
Code: 
$ iperf -n 10G -i 1 -c sinestro -P 4
------------------------------------------------------------
Client connecting to sinestro, TCP port 5001
TCP window size: 85.0 KByte (default)
------------------------------------------------------------
[  3] local 192.168.10.174 port 39698 connected with 192.168.10.15 port 5001
[  5] local 192.168.10.174 port 39700 connected with 192.168.10.15 port 5001
[  4] local 192.168.10.174 port 39696 connected with 192.168.10.15 port 5001
[  6] local 192.168.10.174 port 39702 connected with 192.168.10.15 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0- 1.0 sec   148 MBytes  1.24 Gbits/sec
[  5]  0.0- 1.0 sec   131 MBytes  1.10 Gbits/sec
[  4]  0.0- 1.0 sec   152 MBytes  1.27 Gbits/sec
[  6]  0.0- 1.0 sec   140 MBytes  1.17 Gbits/sec
[SUM]  0.0- 1.0 sec   570 MBytes  4.78 Gbits/sec
[  3]  1.0- 2.0 sec   151 MBytes  1.26 Gbits/sec
[  5]  1.0- 2.0 sec   140 MBytes  1.17 Gbits/sec
[  4]  1.0- 2.0 sec   151 MBytes  1.27 Gbits/sec
[  6]  1.0- 2.0 sec   136 MBytes  1.14 Gbits/sec
[SUM]  1.0- 2.0 sec   577 MBytes  4.84 Gbits/sec
[  6]  2.0- 3.0 sec   147 MBytes  1.23 Gbits/sec
[  3]  2.0- 3.0 sec   148 MBytes  1.24 Gbits/sec
[  5]  2.0- 3.0 sec   132 MBytes  1.11 Gbits/sec
[  4]  2.0- 3.0 sec   150 MBytes  1.26 Gbits/sec
[SUM]  2.0- 3.0 sec   578 MBytes  4.85 Gbits/sec
[  3]  3.0- 4.0 sec   150 MBytes  1.26 Gbits/sec
[  4]  3.0- 4.0 sec   160 MBytes  1.35 Gbits/sec
[  5]  3.0- 4.0 sec   138 MBytes  1.15 Gbits/sec
[  6]  3.0- 4.0 sec   142 MBytes  1.19 Gbits/sec
[SUM]  3.0- 4.0 sec   590 MBytes  4.95 Gbits/sec
[  5]  4.0- 5.0 sec   143 MBytes  1.20 Gbits/sec
[  4]  4.0- 5.0 sec   161 MBytes  1.35 Gbits/sec
[  3]  4.0- 5.0 sec   150 MBytes  1.25 Gbits/sec
[  6]  4.0- 5.0 sec   144 MBytes  1.20 Gbits/sec
[SUM]  4.0- 5.0 sec   597 MBytes  5.01 Gbits/sec
[  3]  5.0- 6.0 sec   145 MBytes  1.21 Gbits/sec
[  4]  5.0- 6.0 sec   150 MBytes  1.26 Gbits/sec
[  6]  5.0- 6.0 sec   142 MBytes  1.19 Gbits/sec
[  5]  5.0- 6.0 sec   148 MBytes  1.24 Gbits/sec
[SUM]  5.0- 6.0 sec   584 MBytes  4.90 Gbits/sec
[  5]  6.0- 7.0 sec   158 MBytes  1.33 Gbits/sec
[  3]  6.0- 7.0 sec   137 MBytes  1.15 Gbits/sec
[  4]  6.0- 7.0 sec   162 MBytes  1.36 Gbits/sec
[  6]  6.0- 7.0 sec   137 MBytes  1.15 Gbits/sec
[SUM]  6.0- 7.0 sec   595 MBytes  4.99 Gbits/sec
[  3]  7.0- 8.0 sec   146 MBytes  1.22 Gbits/sec
[  5]  7.0- 8.0 sec   148 MBytes  1.24 Gbits/sec
[  6]  7.0- 8.0 sec   143 MBytes  1.20 Gbits/sec
[  4]  7.0- 8.0 sec   149 MBytes  1.25 Gbits/sec
[SUM]  7.0- 8.0 sec   587 MBytes  4.92 Gbits/sec
[  3]  8.0- 9.0 sec   152 MBytes  1.27 Gbits/sec
[  5]  8.0- 9.0 sec   148 MBytes  1.24 Gbits/sec
[  4]  8.0- 9.0 sec   144 MBytes  1.21 Gbits/sec
[  6]  8.0- 9.0 sec   143 MBytes  1.20 Gbits/sec
[SUM]  8.0- 9.0 sec   587 MBytes  4.93 Gbits/sec
[  5]  9.0-10.0 sec   146 MBytes  1.23 Gbits/sec
[  6]  9.0-10.0 sec   144 MBytes  1.20 Gbits/sec
[  3]  9.0-10.0 sec   150 MBytes  1.26 Gbits/sec
[  4]  9.0-10.0 sec   142 MBytes  1.19 Gbits/sec
[SUM]  9.0-10.0 sec   582 MBytes  4.89 Gbits/sec
Code: 
$ iperf -n 10G -i 1 -c sinestro -P 8
------------------------------------------------------------
Client connecting to sinestro, TCP port 5001
TCP window size: 85.0 KByte (default)
------------------------------------------------------------
[  9] local 192.168.10.174 port 39718 connected with 192.168.10.15 port 5001
[  6] local 192.168.10.174 port 39706 connected with 192.168.10.15 port 5001
[  3] local 192.168.10.174 port 39704 connected with 192.168.10.15 port 5001
[  4] local 192.168.10.174 port 39708 connected with 192.168.10.15 port 5001
[  7] local 192.168.10.174 port 39710 connected with 192.168.10.15 port 5001
[  8] local 192.168.10.174 port 39712 connected with 192.168.10.15 port 5001
[  5] local 192.168.10.174 port 39714 connected with 192.168.10.15 port 5001
[ 10] local 192.168.10.174 port 39716 connected with 192.168.10.15 port 5001
[ ID] Interval       Transfer     Bandwidth
[  9]  0.0- 1.0 sec  56.0 MBytes   470 Mbits/sec
[  6]  0.0- 1.0 sec  84.6 MBytes   710 Mbits/sec
[  3]  0.0- 1.0 sec  86.4 MBytes   725 Mbits/sec
[  4]  0.0- 1.0 sec  87.4 MBytes   733 Mbits/sec
[  7]  0.0- 1.0 sec  77.2 MBytes   648 Mbits/sec
[  8]  0.0- 1.0 sec  66.5 MBytes   558 Mbits/sec
[ 10]  0.0- 1.0 sec  74.5 MBytes   625 Mbits/sec
[  5]  0.0- 1.0 sec  56.8 MBytes   476 Mbits/sec
[SUM]  0.0- 1.0 sec   589 MBytes  4.94 Gbits/sec
[  9]  1.0- 2.0 sec  77.1 MBytes   647 Mbits/sec
[  6]  1.0- 2.0 sec  67.4 MBytes   565 Mbits/sec
[  3]  1.0- 2.0 sec  68.6 MBytes   576 Mbits/sec
[  4]  1.0- 2.0 sec  91.1 MBytes   764 Mbits/sec
[  8]  1.0- 2.0 sec  76.5 MBytes   642 Mbits/sec
[  5]  1.0- 2.0 sec  77.2 MBytes   648 Mbits/sec
[ 10]  1.0- 2.0 sec  85.6 MBytes   718 Mbits/sec
[  7]  1.0- 2.0 sec  71.0 MBytes   596 Mbits/sec
[SUM]  1.0- 2.0 sec   615 MBytes  5.16 Gbits/sec
[  9]  2.0- 3.0 sec  73.8 MBytes   619 Mbits/sec
[  3]  2.0- 3.0 sec  85.0 MBytes   713 Mbits/sec
[  8]  2.0- 3.0 sec  66.1 MBytes   555 Mbits/sec
[  6]  2.0- 3.0 sec  78.2 MBytes   656 Mbits/sec
[  4]  2.0- 3.0 sec  68.0 MBytes   570 Mbits/sec
[  7]  2.0- 3.0 sec  62.5 MBytes   524 Mbits/sec
[  5]  2.0- 3.0 sec  70.8 MBytes   593 Mbits/sec
[ 10]  2.0- 3.0 sec  64.8 MBytes   543 Mbits/sec
[SUM]  2.0- 3.0 sec   569 MBytes  4.77 Gbits/sec
[  9]  3.0- 4.0 sec  94.0 MBytes   789 Mbits/sec
[  6]  3.0- 4.0 sec  92.9 MBytes   779 Mbits/sec
[  3]  3.0- 4.0 sec  69.6 MBytes   584 Mbits/sec
[  4]  3.0- 4.0 sec  46.1 MBytes   387 Mbits/sec
[  7]  3.0- 4.0 sec  68.5 MBytes   575 Mbits/sec
[  8]  3.0- 4.0 sec  78.4 MBytes   657 Mbits/sec
[  5]  3.0- 4.0 sec  50.8 MBytes   426 Mbits/sec
[ 10]  3.0- 4.0 sec  73.4 MBytes   616 Mbits/sec
[SUM]  3.0- 4.0 sec   574 MBytes  4.81 Gbits/sec
[  4]  4.0- 5.0 sec  87.1 MBytes   731 Mbits/sec
[  7]  4.0- 5.0 sec  72.5 MBytes   608 Mbits/sec
[  9]  4.0- 5.0 sec  94.1 MBytes   790 Mbits/sec
[  6]  4.0- 5.0 sec  17.5 MBytes   147 Mbits/sec
[  3]  4.0- 5.0 sec  86.0 MBytes   721 Mbits/sec
[  8]  4.0- 5.0 sec  68.0 MBytes   570 Mbits/sec
[  5]  4.0- 5.0 sec  96.8 MBytes   812 Mbits/sec
[ 10]  4.0- 5.0 sec  81.0 MBytes   679 Mbits/sec
[SUM]  4.0- 5.0 sec   603 MBytes  5.06 Gbits/sec
[  9]  5.0- 6.0 sec   102 MBytes   851 Mbits/sec
[  7]  5.0- 6.0 sec  78.2 MBytes   656 Mbits/sec
[ 10]  5.0- 6.0 sec  63.2 MBytes   531 Mbits/sec
[  6]  5.0- 6.0 sec  95.1 MBytes   798 Mbits/sec
[  3]  5.0- 6.0 sec  32.2 MBytes   271 Mbits/sec
[  4]  5.0- 6.0 sec  56.6 MBytes   475 Mbits/sec
[  8]  5.0- 6.0 sec  89.8 MBytes   753 Mbits/sec
[  5]  5.0- 6.0 sec  84.6 MBytes   710 Mbits/sec
[SUM]  5.0- 6.0 sec   601 MBytes  5.04 Gbits/sec
[  6]  6.0- 7.0 sec  80.4 MBytes   674 Mbits/sec
[  3]  6.0- 7.0 sec   108 MBytes   902 Mbits/sec
[  4]  6.0- 7.0 sec  56.6 MBytes   475 Mbits/sec
[  8]  6.0- 7.0 sec  70.5 MBytes   591 Mbits/sec
[ 10]  6.0- 7.0 sec  52.0 MBytes   436 Mbits/sec
[  9]  6.0- 7.0 sec  71.1 MBytes   597 Mbits/sec
[  7]  6.0- 7.0 sec   111 MBytes   932 Mbits/sec
[  5]  6.0- 7.0 sec  67.2 MBytes   564 Mbits/sec
[SUM]  6.0- 7.0 sec   616 MBytes  5.17 Gbits/sec
[  9]  7.0- 8.0 sec  63.2 MBytes   531 Mbits/sec
[  4]  7.0- 8.0 sec  95.8 MBytes   803 Mbits/sec
[  7]  7.0- 8.0 sec  24.6 MBytes   207 Mbits/sec
[ 10]  7.0- 8.0 sec  76.4 MBytes   641 Mbits/sec
[  6]  7.0- 8.0 sec  69.6 MBytes   584 Mbits/sec
[  3]  7.0- 8.0 sec  90.8 MBytes   761 Mbits/sec
[  8]  7.0- 8.0 sec  95.1 MBytes   798 Mbits/sec
[  5]  7.0- 8.0 sec   101 MBytes   847 Mbits/sec
[SUM]  7.0- 8.0 sec   616 MBytes  5.17 Gbits/sec
[  9]  8.0- 9.0 sec  83.0 MBytes   696 Mbits/sec
[  6]  8.0- 9.0 sec  46.5 MBytes   390 Mbits/sec
[  4]  8.0- 9.0 sec  65.1 MBytes   546 Mbits/sec
[  7]  8.0- 9.0 sec  57.1 MBytes   479 Mbits/sec
[  8]  8.0- 9.0 sec   114 MBytes   952 Mbits/sec
[  5]  8.0- 9.0 sec  92.1 MBytes   773 Mbits/sec
[ 10]  8.0- 9.0 sec  71.9 MBytes   603 Mbits/sec
[  3]  8.0- 9.0 sec  73.4 MBytes   616 Mbits/sec
[SUM]  8.0- 9.0 sec   603 MBytes  5.06 Gbits/sec
[  6]  9.0-10.0 sec  73.1 MBytes   613 Mbits/sec
[  3]  9.0-10.0 sec  54.2 MBytes   455 Mbits/sec
[  7]  9.0-10.0 sec  76.2 MBytes   640 Mbits/sec
[  8]  9.0-10.0 sec  44.1 MBytes   370 Mbits/sec
[  5]  9.0-10.0 sec   114 MBytes   953 Mbits/sec
[  9]  9.0-10.0 sec  73.0 MBytes   612 Mbits/sec
[ 10]  9.0-10.0 sec   108 MBytes   908 Mbits/sec
[  4]  9.0-10.0 sec  78.9 MBytes   662 Mbits/sec
[SUM]  9.0-10.0 sec   622 MBytes  5.21 Gbits/sec
 
G

Gerk

New Member
Oct 24, 2016
26
8
3
Oakville, ON, Canada
And going from sinestro (slowest machine, FN 9.10) to other machines:

to 'house' (R610) , ~4.6Gbps single connection, ~5.5Gbps multiple connections
Code: 
[mark@sinestro ~]$ iperf -n 100G -i 1 -c house
------------------------------------------------------------
Client connecting to house, TCP port 5001
TCP window size: 32.5 KByte (default)
------------------------------------------------------------
[  3] local 192.168.10.15 port 27784 connected with 192.168.10.67 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0- 1.0 sec   545 MBytes  4.57 Gbits/sec
[  3]  1.0- 2.0 sec   555 MBytes  4.66 Gbits/sec
[  3]  2.0- 3.0 sec   561 MBytes  4.71 Gbits/sec
[  3]  3.0- 4.0 sec   555 MBytes  4.66 Gbits/sec
[  3]  4.0- 5.0 sec   529 MBytes  4.44 Gbits/sec
[  3]  5.0- 6.0 sec   552 MBytes  4.63 Gbits/sec
^C[  3]  0.0- 6.5 sec  3.48 GBytes  4.61 Gbits/sec
[mark@sinestro ~]$ iperf -n 100G -i 1 -c house -P 2
------------------------------------------------------------
Client connecting to house, TCP port 5001
TCP window size: 32.5 KByte (default)
------------------------------------------------------------
[  3] local 192.168.10.15 port 27793 connected with 192.168.10.67 port 5001
[  4] local 192.168.10.15 port 27792 connected with 192.168.10.67 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0- 1.0 sec   206 MBytes  1.72 Gbits/sec
[  4]  0.0- 1.0 sec   322 MBytes  2.70 Gbits/sec
[SUM]  0.0- 1.0 sec   528 MBytes  4.43 Gbits/sec
[  3]  1.0- 2.0 sec   250 MBytes  2.10 Gbits/sec
[  4]  1.0- 2.0 sec   257 MBytes  2.15 Gbits/sec
[SUM]  1.0- 2.0 sec   507 MBytes  4.25 Gbits/sec
[  3]  2.0- 3.0 sec   293 MBytes  2.45 Gbits/sec
[  4]  2.0- 3.0 sec   293 MBytes  2.46 Gbits/sec
[SUM]  2.0- 3.0 sec   585 MBytes  4.91 Gbits/sec
[  4]  3.0- 4.0 sec   290 MBytes  2.43 Gbits/sec
[  3]  3.0- 4.0 sec   291 MBytes  2.44 Gbits/sec
[SUM]  3.0- 4.0 sec   581 MBytes  4.87 Gbits/sec
[  3]  4.0- 5.0 sec   297 MBytes  2.49 Gbits/sec
[  4]  4.0- 5.0 sec   296 MBytes  2.49 Gbits/sec
[SUM]  4.0- 5.0 sec   593 MBytes  4.98 Gbits/sec
^C[  3]  0.0- 5.5 sec  1.38 GBytes  2.15 Gbits/sec
[  4]  0.0- 5.5 sec  1.50 GBytes  2.34 Gbits/sec
[SUM]  0.0- 5.5 sec  2.89 GBytes  4.49 Gbits/sec
[mark@sinestro ~]$ iperf -n 100G -i 1 -c house -P 4
------------------------------------------------------------
Client connecting to house, TCP port 5001
TCP window size: 32.5 KByte (default)
------------------------------------------------------------
[  6] local 192.168.10.15 port 27804 connected with 192.168.10.67 port 5001
[  3] local 192.168.10.15 port 27803 connected with 192.168.10.67 port 5001
[  5] local 192.168.10.15 port 27802 connected with 192.168.10.67 port 5001
[  4] local 192.168.10.15 port 27801 connected with 192.168.10.67 port 5001
[ ID] Interval       Transfer     Bandwidth
[  6]  0.0- 1.0 sec   173 MBytes  1.45 Gbits/sec
[  3]  0.0- 1.0 sec   165 MBytes  1.38 Gbits/sec
[  4]  0.0- 1.0 sec   165 MBytes  1.38 Gbits/sec
[  5]  0.0- 1.0 sec   155 MBytes  1.30 Gbits/sec
[SUM]  0.0- 1.0 sec   657 MBytes  5.51 Gbits/sec
[  3]  1.0- 2.0 sec   177 MBytes  1.48 Gbits/sec
[  5]  1.0- 2.0 sec   147 MBytes  1.24 Gbits/sec
[  4]  1.0- 2.0 sec   172 MBytes  1.44 Gbits/sec
[  6]  1.0- 2.0 sec   170 MBytes  1.42 Gbits/sec
[SUM]  1.0- 2.0 sec   666 MBytes  5.59 Gbits/sec
[  6]  2.0- 3.0 sec   164 MBytes  1.38 Gbits/sec
[  3]  2.0- 3.0 sec   156 MBytes  1.31 Gbits/sec
[  5]  2.0- 3.0 sec   185 MBytes  1.55 Gbits/sec
[  4]  2.0- 3.0 sec   162 MBytes  1.36 Gbits/sec
[SUM]  2.0- 3.0 sec   668 MBytes  5.60 Gbits/sec
[  5]  3.0- 4.0 sec   198 MBytes  1.66 Gbits/sec
[  4]  3.0- 4.0 sec   174 MBytes  1.46 Gbits/sec
[  6]  3.0- 4.0 sec   120 MBytes  1.00 Gbits/sec
[  3]  3.0- 4.0 sec   173 MBytes  1.45 Gbits/sec
[SUM]  3.0- 4.0 sec   665 MBytes  5.58 Gbits/sec
[  6]  4.0- 5.0 sec   164 MBytes  1.38 Gbits/sec
[  3]  4.0- 5.0 sec   155 MBytes  1.30 Gbits/sec
[  5]  4.0- 5.0 sec   172 MBytes  1.44 Gbits/sec
[  4]  4.0- 5.0 sec   156 MBytes  1.31 Gbits/sec
[SUM]  4.0- 5.0 sec   647 MBytes  5.43 Gbits/sec
[  6]  5.0- 6.0 sec   165 MBytes  1.39 Gbits/sec
[  3]  5.0- 6.0 sec   164 MBytes  1.37 Gbits/sec
[  5]  5.0- 6.0 sec   172 MBytes  1.45 Gbits/sec
[  4]  5.0- 6.0 sec   163 MBytes  1.37 Gbits/sec
[SUM]  5.0- 6.0 sec   664 MBytes  5.57 Gbits/sec
^C[  6]  0.0- 6.4 sec  1018 MBytes  1.34 Gbits/sec
[  3]  0.0- 6.4 sec  1.03 GBytes  1.39 Gbits/sec
[  5]  0.0- 6.4 sec  1.07 GBytes  1.44 Gbits/sec
[  4]  0.0- 6.4 sec  1.03 GBytes  1.39 Gbits/sec
[SUM]  0.0- 6.4 sec  4.13 GBytes  5.55 Gbits/sec
[mark@sinestro ~]$ iperf -n 100G -i 1 -c house -P 8
------------------------------------------------------------
Client connecting to house, TCP port 5001
TCP window size: 32.5 KByte (default)
------------------------------------------------------------
[  9] local 192.168.10.15 port 27818 connected with 192.168.10.67 port 5001
[  5] local 192.168.10.15 port 27816 connected with 192.168.10.67 port 5001
[  8] local 192.168.10.15 port 27817 connected with 192.168.10.67 port 5001
[ 10] local 192.168.10.15 port 27819 connected with 192.168.10.67 port 5001
[  3] local 192.168.10.15 port 27815 connected with 192.168.10.67 port 5001
[  6] local 192.168.10.15 port 27814 connected with 192.168.10.67 port 5001
[  7] local 192.168.10.15 port 27813 connected with 192.168.10.67 port 5001
[  4] local 192.168.10.15 port 27812 connected with 192.168.10.67 port 5001
[ ID] Interval       Transfer     Bandwidth
[  5]  0.0- 1.0 sec  57.8 MBytes   484 Mbits/sec
[  3]  0.0- 1.0 sec   129 MBytes  1.08 Gbits/sec
[  9]  0.0- 1.0 sec  61.6 MBytes   517 Mbits/sec
[ 10]  0.0- 1.0 sec  63.6 MBytes   534 Mbits/sec
[  6]  0.0- 1.0 sec  97.0 MBytes   814 Mbits/sec
[  7]  0.0- 1.0 sec  60.8 MBytes   510 Mbits/sec
[  8]  0.0- 1.0 sec   121 MBytes  1.02 Gbits/sec
[  4]  0.0- 1.0 sec  53.9 MBytes   452 Mbits/sec
[SUM]  0.0- 1.0 sec   645 MBytes  5.41 Gbits/sec
[  9]  1.0- 2.0 sec  83.1 MBytes   697 Mbits/sec
[  5]  1.0- 2.0 sec   121 MBytes  1.01 Gbits/sec
[  8]  1.0- 2.0 sec  48.4 MBytes   406 Mbits/sec
[ 10]  1.0- 2.0 sec   117 MBytes   980 Mbits/sec
[  3]  1.0- 2.0 sec  69.4 MBytes   582 Mbits/sec
[  6]  1.0- 2.0 sec  86.9 MBytes   729 Mbits/sec
[  7]  1.0- 2.0 sec  83.8 MBytes   703 Mbits/sec
[  4]  1.0- 2.0 sec  44.2 MBytes   371 Mbits/sec
[SUM]  1.0- 2.0 sec   654 MBytes  5.48 Gbits/sec
[  9]  2.0- 3.0 sec  79.5 MBytes   667 Mbits/sec
[  5]  2.0- 3.0 sec  83.2 MBytes   698 Mbits/sec
[  8]  2.0- 3.0 sec  86.1 MBytes   722 Mbits/sec
[ 10]  2.0- 3.0 sec  58.5 MBytes   491 Mbits/sec
[  3]  2.0- 3.0 sec  77.9 MBytes   653 Mbits/sec
[  6]  2.0- 3.0 sec  81.9 MBytes   687 Mbits/sec
[  7]  2.0- 3.0 sec  75.5 MBytes   633 Mbits/sec
[  4]  2.0- 3.0 sec  82.0 MBytes   688 Mbits/sec
[SUM]  2.0- 3.0 sec   625 MBytes  5.24 Gbits/sec
[  9]  3.0- 4.0 sec  94.1 MBytes   790 Mbits/sec
[  5]  3.0- 4.0 sec  91.6 MBytes   769 Mbits/sec
[  8]  3.0- 4.0 sec  74.8 MBytes   627 Mbits/sec
[ 10]  3.0- 4.0 sec  87.2 MBytes   732 Mbits/sec
[  3]  3.0- 4.0 sec  80.5 MBytes   675 Mbits/sec
[  6]  3.0- 4.0 sec  70.8 MBytes   593 Mbits/sec
[  7]  3.0- 4.0 sec  79.6 MBytes   668 Mbits/sec
[  4]  3.0- 4.0 sec  89.4 MBytes   750 Mbits/sec
[SUM]  3.0- 4.0 sec   668 MBytes  5.60 Gbits/sec
[  9]  4.0- 5.0 sec  93.1 MBytes   781 Mbits/sec
[  5]  4.0- 5.0 sec  92.9 MBytes   779 Mbits/sec
[  8]  4.0- 5.0 sec  55.8 MBytes   468 Mbits/sec
[ 10]  4.0- 5.0 sec  89.2 MBytes   749 Mbits/sec
[  3]  4.0- 5.0 sec  61.6 MBytes   517 Mbits/sec
[  6]  4.0- 5.0 sec  92.4 MBytes   775 Mbits/sec
[  7]  4.0- 5.0 sec  84.8 MBytes   711 Mbits/sec
[  4]  4.0- 5.0 sec  97.2 MBytes   816 Mbits/sec
[SUM]  4.0- 5.0 sec   667 MBytes  5.60 Gbits/sec
[  9]  5.0- 6.0 sec  80.8 MBytes   677 Mbits/sec
[  5]  5.0- 6.0 sec   103 MBytes   863 Mbits/sec
[  8]  5.0- 6.0 sec  78.0 MBytes   654 Mbits/sec
[ 10]  5.0- 6.0 sec  59.0 MBytes   495 Mbits/sec
[  3]  5.0- 6.0 sec  86.6 MBytes   727 Mbits/sec
[  7]  5.0- 6.0 sec   100 MBytes   842 Mbits/sec
[  4]  5.0- 6.0 sec  95.0 MBytes   797 Mbits/sec
[  6]  5.0- 6.0 sec  62.9 MBytes   527 Mbits/sec
[SUM]  5.0- 6.0 sec   666 MBytes  5.58 Gbits/sec
^C[  9]  0.0- 6.7 sec   557 MBytes   702 Mbits/sec
[  5]  0.0- 6.7 sec   586 MBytes   739 Mbits/sec
[  8]  0.0- 6.7 sec   524 MBytes   660 Mbits/sec
[ 10]  0.0- 6.7 sec   512 MBytes   645 Mbits/sec
[  3]  0.0- 6.7 sec   568 MBytes   717 Mbits/sec
[  6]  0.0- 6.7 sec   532 MBytes   671 Mbits/sec
[  7]  0.0- 6.7 sec   549 MBytes   692 Mbits/sec
[  4]  0.0- 6.7 sec   531 MBytes   669 Mbits/sec
[SUM]  0.0- 6.7 sec  4.26 GBytes  5.50 Gbits/sec
And pretty much the same performance with sinestro as a client to the third server
 
Last edited:
G

Gerk

New Member
Oct 24, 2016
26
8
3
Oakville, ON, Canada
I will if there's nothing else to test as it's a fair bit of work to accomplish that (I've found esxi can be incredibly fussy at times and I don't want to bust things without just cause heh)
 
Z

zhoulander

Active Member
Feb 1, 2016
181
46
28
Try iperf through Ubuntu live CD/USB to rule out the OS. To install iperf you'll need to sudo apt-add-repository universe, apt-get update, then apt-get install iperf. Ubuntu vs Win10 on my X99 workstation goes from 9.4Gbps to 5.4Gbps. Really odd.
 
G

Gerk

New Member
Oct 24, 2016
26
8
3
Oakville, ON, Canada
Ok so a bit more info to report back here ...

I've now tried this all out with live boot CDs, I updated firmware on all cards in question (the problem machine -- sinestro -- was a couple of revisions back in terms of firmware). here's my findings:

Nothing that I've done will get me more than ~5.5Gb/s of incoming bandwidth with the NIC on sinestro (when using it as the iperf server), but when using it as the iperf client to an almost identical machine I was able to get 9.7Gb/s of bandwidth. To rule out the actual NIC I even swapped it with one from another machine and I'm seeing the exact same behaviour with the server, so ... this doesn't seem to be an external issue, or a specific NIC issue, but something specifically to do with this server that's just not letting the full bandwidth happen.

Is there anything else that I've missed out on? Is it possible somehow that the machine is able to send the full bandwidth but not be able to receive the full bandwidth? To me it just doesn't make a lot of sense that one direction is half the speed. BIOS is all fully up to date and the servers both have pretty much identical settings. The only difference between the two is that the X series processor lets me select the QPI to be optimized for either processing or I/O (I've selected I/O, which is the same as my third box which doesn't suffer from bandwidth issues)

Very confusing ...
 
G

Gerk

New Member
Oct 24, 2016
26
8
3
Oakville, ON, Canada
Unfortunately that wasn't the issue, appears to be linking at x8 ok.

Code: 
07:00.0 Ethernet controller: Intel Corporation 82598EB 10-Gigabit AF Dual Port Network Connection (rev 01)
	Subsystem: Intel Corporation 10-Gigabit AF DA Dual Port Server Adapter
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Interrupt: pin A routed to IRQ 40
	Region 0: Memory at df440000 (32-bit, non-prefetchable)
	Region 1: Memory at df480000 (32-bit, non-prefetchable)
	Region 2: I/O ports at dcc0
	Region 3: Memory at df438000 (32-bit, non-prefetchable)
	Expansion ROM at df300000 [disabled]
	Capabilities: [40] Power Management version 3
		Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold-)
		Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=1 PME-
	Capabilities: [50] MSI: Enable- Count=1/1 Maskable- 64bit+
		Address: 0000000000000000  Data: 0000
	Capabilities: [60] MSI-X: Enable+ Count=18 Masked-
		Vector table: BAR=3 offset=00000000
		PBA: BAR=3 offset=00002000
	Capabilities: [a0] Express (v2) Endpoint, MSI 00
		DevCap:	MaxPayload 256 bytes, PhantFunc 0, Latency L0s <512ns, L1 <64us
			ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 0.000W
		DevCtl:	Report errors: Correctable- Non-Fatal+ Fatal+ Unsupported+
			RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
			MaxPayload 256 bytes, MaxReadReq 512 bytes
		DevSta:	CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr- TransPend-
		LnkCap:	Port #2, Speed 2.5GT/s, Width x8, ASPM L0s L1, Exit Latency L0s <4us, L1 <64us
			ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- CommClk+
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 2.5GT/s, Width x8, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
		DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR-, OBFF Not Supported
		DevCtl2: Completion Timeout: 65ms to 210ms, TimeoutDis-, LTR-, OBFF Disabled
		LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
			 EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-

07:00.1 Ethernet controller: Intel Corporation 82598EB 10-Gigabit AF Dual Port Network Connection (rev 01)
	Subsystem: Intel Corporation 10-Gigabit AF DA Dual Port Server Adapter
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Interrupt: pin B routed to IRQ 50
	Region 0: Memory at df460000 (32-bit, non-prefetchable)
	Region 1: Memory at df4c0000 (32-bit, non-prefetchable)
	Region 2: I/O ports at dce0
	Region 3: Memory at df43c000 (32-bit, non-prefetchable)
	Expansion ROM at df300000 [disabled]
	Capabilities: [40] Power Management version 3
		Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold-)
		Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=1 PME-
	Capabilities: [50] MSI: Enable- Count=1/1 Maskable- 64bit+
		Address: 0000000000000000  Data: 0000
	Capabilities: [60] MSI-X: Enable+ Count=18 Masked-
		Vector table: BAR=3 offset=00000000
		PBA: BAR=3 offset=00002000
	Capabilities: [a0] Express (v2) Endpoint, MSI 00
		DevCap:	MaxPayload 256 bytes, PhantFunc 0, Latency L0s <512ns, L1 <64us
			ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 0.000W
		DevCtl:	Report errors: Correctable- Non-Fatal+ Fatal+ Unsupported+
			RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
			MaxPayload 256 bytes, MaxReadReq 512 bytes
		DevSta:	CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr- TransPend-
		LnkCap:	Port #2, Speed 2.5GT/s, Width x8, ASPM L0s L1, Exit Latency L0s <4us, L1 <64us
			ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- CommClk+
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 2.5GT/s, Width x8, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
		DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR-, OBFF Not Supported
		DevCtl2: Completion Timeout: 65ms to 210ms, TimeoutDis-, LTR-, OBFF Disabled
		LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
			 EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
 
C

coolrunnings82

Active Member
Mar 26, 2012
407
92
28
I have this same issue with my X540-T2 cards in my R720's and a Supermicro board with an E5-1620v2.

Sent from my SM-G900V using Tapatalk
 
You must log in or register to reply here.
Top Bottom