Short version: If you're unable to get more than ~7Gbps with a ConnectX-2, ensure your card is running at PCIe x4 and enable jumbo frames.
I wanted to share my experience with Mellanox ConnectX-2 EN MNPA19-XTR cards and Windows 10. Got them off ebay and put one card into my Windows 10 machine running a 6700K on a ASUS Maximus Gene VIII Z170 motherboard. The other card went into a Fedora 27 system and worked without issue (2.9.1000 firmware). They were connected with a Cisco CU3M direct attach cable.
Here is the motherboard slot layout.
Since my video card is in the top slot and covers the middle PCIe slot, I could not put the NIC there so I put it in the open-ended x4 slot even though it is physically a x8 card.
I followed directions from another thread here to update the firmware in the card to 2.10.720. The Windows 2016 WinOF 5.40 drivers from Mellanox appear to work fine (5.4.14004.0).
I was perplexed as to why I was only getting about 3Gbps with a single iperf thread to the Linux box, got up to 5Gbps when I upped the MTU to 9000. Hit about 7Gbps with parallel threads (-P switch in iperf). I then realized the PCIe slot was only running at x2, limiting the maximum theoretical bandwidth to 8Gbps. This was verified with Get-NetAdapterHardwareInfo and the Information tab in the adapter properties.
Code:
PS C:\WINDOWS\system32> Get-NetAdapterHardwareInfo
Name Segment Bus Device Function Slot NumaNode PcieLinkSpeed PcieLinkWidth Version
---- ------- --- ------ -------- ---- -------- ------------- ------------- -------
Ethernet 5 0 2 0 0 5.0 GT/s 2 1.1
I moved the video card to the middle slot and the NIC to the top x16 slot in my motherboard and it ran at x4, so I knew it wasn't some kind of compatibility issue. Benchmarks show there was no issue running the 980Ti at x8, but cooling was suboptimal in this setup so I really wanted the card in the bottom slot.
Eventually, I figured out that there is a BIOS setting that trades off 2 lanes to enable the SATA 5/6/Express slots. Since I was not using those slots, I toggled the BIOS setting (can't remember exactly what it was called) to enable PCIe x4 on the bottom slot. Success! With Jumbo Frames enabled (MTU=9000), I was able to hit 9Gbps with a single iperf thread and up to 9.8Gbps when tweaking various iperf parameters. SSH copies are currently being limited by CPU throughput (1.6Gbps) and disk bottlenecks (300-400 MB/s read/write) but I was able to get up to 6Gbps to/from my Windows SSD raid with a ramdisk on the Linux box. It is currently showing as RDMA capable but I have not tested this yet.
Code:
PS C:\WINDOWS\system32> Get-NetAdapterHardwareInfo
Name Segment Bus Device Function Slot NumaNode PcieLinkSpeed PcieLinkWidth Version
---- ------- --- ------ -------- ---- -------- ------------- ------------- -------
Ethernet 5 0 2 0 0 5.0 GT/s 4 1.1
PS C:\WINDOWS\system32> Get-SmbServerNetworkInterface
Scope Name Interface Index RSS Capable RDMA Capable Speed IpAddress
---------- --------------- ----------- ------------ ----- ---------
* 22 True True 10 Gbps 192.168.2.2
Note: According to
this Get-NetAdapterHardwareInfo won't return anything higher than 1.1 for the version, but since it's 5.0 GT/s it must be PCIe 2.0.
---------------------------------------------------------------------------------------------
Hey,
I just wanted to thank you for your great post and solution.
I had the same issue with MNPA19-XTR connected to Intel X520-2 (2 port SFP+).
I had the MNPA19-XTR installed on my desktop connected to one of the ports of the Intel X520-2 on a server machine.
After a couple of jperf tests I was not getting any speed greater than 7Gbps. Just to confirm your solution I went to my MB BIOS (ASUS MAXIMUS_XI_HERO) and changed the speed of PCIE from X2 to X4 and voila i tested and got the full 10Gbps speeds.
For people with the same issue make sure you follow the steps below:
1-If you are using any PCIE anything other than the first PCIE on you MB make sure you have the full speed enabled.
In my case my GPU was located in PCIE16_1 and the NIC was in PCIE16_3.
Here is how to enable it in you ASUS bios:
2-I installed latest driver 5.50.14688 and it gave me full speed:
3-Under performance make sure to select single port traffic or single stream traffic then hit Run Tuning and restart computer.
4-Under advanced:
Interrupt Moderation---->Disable
Jumbo Packet---->9014 (this could be different but i matched it with the highest value on my server Intel X520-2 NIC)
5-Receive Buffers and Send Buffer---->4096 or to its maximum value.
7-Make sure to assign static IP on each NIC.
8-Add the IP address and name of the target machine to the Host file on windows. Let me know if you have questions on how to do this (this will force windows to use the 10gb and not the 1gb LAN speed).
6-Use iperf or jperf or RAM disk to test the speeds. Below are my results from Jperf:
bin/iperf.exe -s -P 0 -i 1 -p 5001 -f g
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 0.00 GByte (default)
------------------------------------------------------------
OpenSCManager failed - Access is denied. (0x5)
[392] local 192.168.50.3 port 5001 connected with 192.168.50.1 port 53662
[ ID] Interval Transfer Bandwidth
[392] 0.0- 1.0 sec 1.07 GBytes 9.21 Gbits/sec
[392] 1.0- 2.0 sec 1.13 GBytes 9.71 Gbits/sec
[392] 2.0- 3.0 sec 1.15 GBytes 9.86 Gbits/sec
[392] 3.0- 4.0 sec 1.15 GBytes 9.85 Gbits/sec
[392] 4.0- 5.0 sec 1.15 GBytes 9.86 Gbits/sec
[392] 5.0- 6.0 sec 1.15 GBytes 9.87 Gbits/sec
[392] 6.0- 7.0 sec 1.15 GBytes 9.88 Gbits/sec
[392] 7.0- 8.0 sec 1.15 GBytes 9.89 Gbits/sec
[392] 8.0- 9.0 sec 1.15 GBytes 9.86 Gbits/sec
[392] 9.0-10.0 sec 1.13 GBytes 9.72 Gbits/sec
[392] 10.0-11.0 sec 1.14 GBytes 9.75 Gbits/sec
[392] 11.0-12.0 sec 1.15 GBytes 9.86 Gbits/sec
[392] 12.0-13.0 sec 1.14 GBytes 9.80 Gbits/sec
[392] 13.0-14.0 sec 1.14 GBytes 9.76 Gbits/sec
[392] 14.0-15.0 sec 1.14 GBytes 9.80 Gbits/sec
[392] 15.0-16.0 sec 1.14 GBytes 9.83 Gbits/sec
[392] 16.0-17.0 sec 1.15 GBytes 9.89 Gbits/sec
[392] 17.0-18.0 sec 1.15 GBytes 9.90 Gbits/sec
[392] 18.0-19.0 sec 1.15 GBytes 9.90 Gbits/sec
[392] 19.0-20.0 sec 1.14 GBytes 9.75 Gbits/sec
[ ID] Interval Transfer Bandwidth
[392] 0.0-20.0 sec 22.8 GBytes 9.80 Gbits/sec
All the credit goes to mfgmfg for his/her great post and solution!!!!!
Thanks again for this.