[SOLVED]Slow speeds between two Connectx-2 machines

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

arglebargle

H̸̖̅ȩ̸̐l̷̦͋l̴̰̈ỏ̶̱ ̸̢͋W̵͖̌ò̴͚r̴͇̀l̵̼͗d̷͕̈
Jul 15, 2018
657
244
43
I think -p is parallel streams, not # of threads
Ah, yeah, you're right. I assumed -P would spawn one thread per stream but with -P 8 I only see two.
 

William

Well-Known Member
May 7, 2015
789
252
63
66
I updated the Firmware to 2.32.5100, it was 2.11
I see no real improvement :(
argh this is driving me crazy :(
iperf results new#2.JPG
 

saivert

Member
Nov 2, 2015
138
18
18
40
Norway
@arglebargle
Yes you are right as long as you pick Linux or FreeBSD as OS but as long as you go with Windows something fishy is going on. Either issues with receive side scaling handling or some bug in the Windows drivers. Seems to be very reliant on having lots of free CPU cycles. This never impacts TX throughput. Only RX.
Of course this is only my simple conclusions from lots of testing on my old system before I upgraded to Coffee Lake i7 8700k where I can't replicate this issue anymore.

On the server side i have Dual Xeon E5-2670 and I have tried both ConnectX-2 and Intel X520. I'm currently using the X520 card as it was just easier to enable SR-IOV on and also doesn't seem to generate as much heat and has a bigger heatsink too. Just more suitable for my server.
 

BackupProphet

Well-Known Member
Jul 2, 2014
1,083
640
113
Stavanger, Norway
olavgg.com
I did some testing tonight on my Connect-X2. Use iperf2, do not use iperf3. Ubuntu/FreeBSD has iperf2 in their repositories, so that is also what I downloaded for Windows.

The bandwidth numbers from iperf2 on Windows are not correct. Use the task manager to check the actual bandwidth. The numbers are correct on the Linux/FreeBSD side. If you try Windows to Windows you can only rely on the task manager. Good luck.

I managed to fully utilize 10G on both send and receive with Windows.
 

William

Well-Known Member
May 7, 2015
789
252
63
66
Good points BackupProphet.

However, I have used the taskmanager, it shows the highest bandwidth I get at about 2.3G.
ATTO also confirms these numbers.

So, I saw some one mention cables, could this be an issue ?

From reading around through google searchs which many times points back to STH. A large number of people have this issue, but for some everything seems to work fine. I am at a complete loss on my issue.
Maybe its my platform … the ASUS Z10PE-D16 WS.. the BIOS is updated to latest.
I thought for awhile it was my switch D-Link DXS-1210-12TC … but if I direct connect WS to DS1618+ I see about the same numbers... 2.3G
 

William

Well-Known Member
May 7, 2015
789
252
63
66
Here is the Task Manager.
You can see on ATTO that it caps out on reads at 1G, makes it to about 2.3G on writes..

As far as using the ConnectX2, I has these in before and they do the exact same thing.

Task Manager.JPG
 

saivert

Member
Nov 2, 2015
138
18
18
40
Norway
The bandwidth numbers from iperf2 on Windows are not correct.
For what its worth I always had Task manager up when testing as I was also measuring CPU utilization during the iperf2 runs. And iperf2 and Task Manager seemed to agree.
I never had as low as 2.3Gbps though. It was usually either full 9.8Gbps or ~6.5Gbps.

The only reason I got the Mellanox cards was because they were cheap. Certainly not for the quality or ease of use.
 

William

Well-Known Member
May 7, 2015
789
252
63
66
Yup, the first two ConnectX2 cards I got was like $15 each with cables !
The ConnectX3 card I just got was $28.

So all in all it wasn't a huge investment.

But sadly nothing seems to be working to get these cards up to 10G speeds. Very strange.