Slow iperf speed (4gbit) between 2 mellanox connectx 3 vpi cards with 40gps link

[xgiovio]

Hello,

I have 2 mellanox connectx 3 vpi cards. They have been updated to latest firmware and installed one on centos7 and the other on windows 10.



Both cards are on amd threadripper systems with pcie express gen4/3 at 8x or 16x.



On windows I use the latest WinOF drivers.

On linux I use the inbox drives because I can't compile the ofed drivers for the 5.10 kernel.



The link between the two cards is 40gps, tried on infiniband and ethernet mode. The cable is a cisco QSFP-H40G-CU3M.



Problem is, even if the link is rated at 40gbs, real speed is 4gbs.



What could be the problem?



Thank you

 

[i386]

What could be the problem?

Iperf on windows.
Try a linux livecd on the windows host and run iperf from there or boot windows on the centos host and try ntttcp.

 

[xgiovio]

I think iperf is correct because if i try an smb transfer i can’t get more than 600megabyte per second.
However i will try with live images

 

[Roelf Zomerman]

you need to iperf with multiple channels to avoid bottlenecks in cpu cores (not -P as described below)

iperf3 at 40Gbps and above (es.net)

has nothing to do with Windows vs Linux or smb - SMB can get way more than 600 - certainly with MultiChannel SMB - I've had 1.6GB(yte) / s on multiple occasions (using 2x 10G adapters)..

your 600MB/s copy - what is the source / target bus speed - what type of disk is it - how utilized is the disk-queue, how big is the file (larger files have more speed) ?

Can you try with 2x RAMdisks (source and target) : How to Create a RAM Disk on Windows Server? | Windows OS Hub (woshub.com) ?

 

[xgiovio]

On the same machines I also have 10gb ethernet cards. Using iperf with or without -P parameters I get a maximum of 7gbits. With mellanox cards a maximum of 4 gbits.

Accessing the smb folders using the 10gb ethernet cards i reach 1200 megabyte per sec. Using the mellanox cards only 600 megabyte per sec.

So from my point of view, the problem right now is related to mellanox card or the cable even if it seems certified and without any transmission error.

What do you think? I'm open to whatever test.

 

[Roelf Zomerman]

you can check if the cable is indeed showing 40G - my Mellanox - with SFP+DirectCables were still seen as 10Gb only..

what are your CPU utilizations on a copy? perhaps the 10Gb's have offloading so they dont need your CPU's that much

a full guide for tuning can be found: Performance Tuning for Mellanox Adapters

but 600 overall is a bit too low

 

[xgiovio]

the 2 systems are 3960x and 1950x, 24 and 16 cores.
both windows and linux show a link rate of 40gbps.

How can i check if the cable is showing 40gb?

Svr in the bios could create problems? Do you have any personal experience with windows to linux direct cards connection?

 

[xgiovio]

you need to iperf with multiple channels to avoid bottlenecks in cpu cores (not -P as described below)

iperf3 at 40Gbps and above (es.net)

has nothing to do with Windows vs Linux or smb - SMB can get way more than 600 - certainly with MultiChannel SMB - I've had 1.6GB(yte) / s on multiple occasions (using 2x 10G adapters)..

your 600MB/s copy - what is the source / target bus speed - what type of disk is it - how utilized is the disk-queue, how big is the file (larger files have more speed) ?

Can you try with 2x RAMdisks (source and target) : How to Create a RAM Disk on Windows Server? | Windows OS Hub (woshub.com) ?

I did the test and the result is the same. 2 parallel tests, same speed halfed. With 1 iperf3 i get 4 gbit. On 2 iperf3 on 2 different ports, I get 2 gigabit per port!

So what could be the problem? I'm tired!

 

[xgiovio]

I made a test mounting an ubuntu installation on the windows host and now the link is 30gbps in iperf3. 30 seems the maximum right now without any additional enhancement or optimization. It is over infiniband. Running iperf3 with -P or multiple iperf3 instances the result is always 30gb summing all the results. Now we know the problem is related to windows itself.

Do you have suggestions what could be the problem? And do you think 30gps could be improved?
Thank you

 

[xgiovio]

up

 

[TjPjMusic]

I've got the same problem, I've got 2 servers both running ubuntu, and one windows workstation. the servers can talk to each other at about 30gbps with no extra flags (no -P or -l) for iperf3. The workstation is only getting about 300-400mbps with no extra flags but can get to about 6gbps with -P 100 and -l 15000. I've also been able to narrow it down to a windows issue, either drivers or OS.

 

[KaneTW]

I have the same problem. PCIe3 x4, Windows 10 gets like, 4Gbps max. (realistically 2-3) with jumbo frames, <600Mbps otherwise. Boot into Linux, get 20-25Gbps.