Mellanox ConnectX-2 EN and Windows 10?

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

TeleFragger

Active Member
Oct 26, 2016
247
52
28
51
Experiencing the same thing. 1-14-19

Will not connect to my network. Will not acquire an IP nor connect using a static.
Widows 10 V 1809 client
Mellanox driver: 5.50.14643.1
Connectx-2 MNPA19-XSR/XTR SP# 671798-001
FW: 2.9.1000 (Tried to update FW to 2.9.1200 but to no avail.)
Tried deleting Widows 5.50 driver and install an earlier version without success. No matter what 5.50 always in use.
Windows shows card operational, working normally.
I have 4 of these and none will connect including the one from an operational computer which did connect before.
Modules: 850nm 10gb and 8gb all confirmed operational using two routers the using same fiber jumper cable.
Lasers ON on both modules.

Ideas?
Yes.... will reply soon. Let me get to work first.....
 

TeleFragger

Active Member
Oct 26, 2016
247
52
28
51
Experiencing the same thing. 1-14-19

Will not connect to my network. Will not acquire an IP nor connect using a static.
Widows 10 V 1809 client
Mellanox driver: 5.50.14643.1
Connectx-2 MNPA19-XSR/XTR SP# 671798-001
FW: 2.9.1000 (Tried to update FW to 2.9.1200 but to no avail.)
Tried deleting Widows 5.50 driver and install an earlier version without success. No matter what 5.50 always in use.
Windows shows card operational, working normally.
I have 4 of these and none will connect including the one from an operational computer which did connect before.
Modules: 850nm 10gb and 8gb all confirmed operational using two routers the using same fiber jumper cable.
Lasers ON on both modules.

Ideas?

so do this... I have a ConnectX-1 card but hopefully it will help as drivers are drivers.
go to mellanox and DL the 5.50 drivers from them.
go into device manager on win10 and go under system devices and look for the mellanox card.
go in and click drivers
update, let me specify, have drivers, browse...
point to c:\program files\mellanox\hw and use that driver for it
once done, you should be able to go back in and get the mellanox add on tabs...
that is what I had to do and I had to take it a step further.
for me I had to choose ETH instead of IB ( was checked to use default).
Then I had to go up under network adapters and do the same for driver but point to mellanox\eth for its drivers.

so I think Microsoft has default drivers that were provided by mellanox but it doesn't give you the added features and doing this should.

hope that helps...
 

shrike0064

New Member
Jan 14, 2019
12
3
3
so do this... I have a ConnectX-1 card but hopefully it will help as drivers are drivers.
go to mellanox and DL the 5.50 drivers from them.
go into device manager on win10 and go under system devices and look for the mellanox card.
go in and click drivers
update, let me specify, have drivers, browse...
point to c:\program files\mellanox\hw and use that driver for it
once done, you should be able to go back in and get the mellanox add on tabs...
that is what I had to do and I had to take it a step further.
for me I had to choose ETH instead of IB ( was checked to use default).
Then I had to go up under network adapters and do the same for driver but point to mellanox\eth for its drivers.

so I think Microsoft has default drivers that were provided by mellanox but it doesn't give you the added features and doing this should.

hope that helps...
Sir,
Thank you for your reply as this is driving me up the wall.
Unfortunately nothing worked. I downloaded the 5.50 driver.
I found the ETH that you mentioned, but it is already selected and greyed out so I cannot change it to auto.
When I redirect the driver it comes up with best driver currently installed.
ARRRRG!
 
Last edited:

TeleFragger

Active Member
Oct 26, 2016
247
52
28
51
Sir,
Thank you for your reply as this is driving me up the wall.
Unfortunately nothing worked. I downloaded the 5.50 driver.
I found the ETH that you mentioned, but it is already selected and greyed out so I cannot change it to auto.
When I redirect the driver it comes up with best driver currently installed.
ARRRRG!
no I had that too... you have to do it a different way...

you have to start with the item under System Devices....


upload_2019-1-18_21-0-30.png

upload_2019-1-18_21-0-47.png


upload_2019-1-18_21-1-2.png

then click browse....
you must use the one not in ETH... you must use this one..
upload_2019-1-18_21-1-39.png

for me I have this one...
upload_2019-1-18_21-2-49.png



and I use it... it looks like the others that were in the list....

then I close out all the windows and go back in … now I have the new tab...
upload_2019-1-18_21-3-19.png



then I choose ETH...

then I repeat same steps for items under network cards but then I choose ETH folder....

then I get all the new tabs... only had a few now I got 2 rows...
upload_2019-1-18_21-4-6.png
 
  • Like
Reactions: abq and nikalai

shrike0064

New Member
Jan 14, 2019
12
3
3
Yes sir. Did that.
I have those tabs.
Those options remain greyed out.
Has the IP: 169.254.44.65/16
However my network is 192.168.1. _ _ _
That's what I don't understand.
If I set a static I get: 169.254.44.65/16, 192.168.1.55/24
But still will not connect.
Gateway: 192.168.1.1
DNS: 192.168.1.1
 
Last edited:
  • Like
Reactions: TeleFragger

shrike0064

New Member
Jan 14, 2019
12
3
3
Apologies about the delay as I had to go into work. Over time even though not getting paid due to shutdown.
Thanks for the snippet tool. Forgot about that. What other screen shots? Capture1.JPG Capture2.JPG
 

TeleFragger

Active Member
Oct 26, 2016
247
52
28
51
Thats weird... firmware update? I know i had a wacked firmware and only saw 1 port... flashed to newer then saw other...
 

shrike0064

New Member
Jan 14, 2019
12
3
3
Tried. Kept saying failed on all 4 cards.
Now wondering if it's that my PCI-e3 x4 slot somehow isn't compatible. But then why is the card seen and drivers loaded?
Computers, such fun.

I found a couple 10gb connectx-3 single ports on eBay for $28 each.
Bought 2. Hoping these will be supported.
Found somewhere that the connectx-2 are no longer supported on Windows 10 Client. They may or may not work.
 
  • Like
Reactions: TeleFragger

TeleFragger

Active Member
Oct 26, 2016
247
52
28
51
Tried. Kept saying failed on all 4 cards.
Now wondering if it's that my PCI-e3 x4 slot somehow isn't compatible. But then why is the card seen and drivers loaded?
Computers, such fun.

I found a couple 10gb connectx-3 single ports on eBay for $28 each.
Bought 2. Hoping these will be supported.
Found somewhere that the connectx-2 are no longer supported on Windows 10 Client. They may or may not work.

Well ive got connectx-1 cx4 style ports working in esxi..freenas 11 and win10
 
  • Like
Reactions: shrike0064

shrike0064

New Member
Jan 14, 2019
12
3
3
Well, crap.
I'm going to try these and see what happens. I know all 16 4k IP cameras will be fine on the gigabit LAN but it's the principle of the thing.
I bought a mini ATX with two slots just for the fiber network I have.
 
  • Like
Reactions: TeleFragger

Maxim

New Member
May 17, 2019
3
0
1
Hi guys!

I will tell my small story. I have just registered account here to help community.

So I have 2 end points with Mellanox Connect X2 cards.
One is Fedora 29 and one Windows Server 2008 R2 (I guess everybody knows that it is same thing as Windows 7 x64 but Server).
I have downloaded latest drivers available for that platform for X2.
It is older version that drivers for Windows 10 (which itself is terrible piece of shit).

When I tested link with iperf3 I have experienced speed issues.
Trandfer speed was limited as described in this thread.

So my network configuration is the following: Mellanox X2 <-> MikroTik 10 Gbps switch (level 3) <-> Mellanox X2
I have MikroTik and Mellanox cables but all that does not matter (both works fine).

In worst case iperf 3 said me 3-4 Gbps in best case up to 7.2 Gbps (depending CPU load - from 100% all cores to idle).
But even if CPU is idle it is still not expected 10Gbps. So I worried about that and came to this thread (and multiple others).

I started experiments and finally got full speed. So what was needed to achieve that in my case?

1) On Windows machine - in Mellanox settings set 9000 in Jumbo Packet field.
2) Send buffers make like Receive buffers - 4096 (it is maximum).
3) Interrupt moderation set to Disabled.

4) On Linux machine set MTU to 9000. There is a lot of docs about that.

5) And now final and crucial thing - check your router if it is not direct connection.
In my case it was needed to change MTU/L2MTU from 1500 to 9000 for each SFP+ interface in Mikrotik Router OS settings.
You will need to find appropriate settings in your router.

After that I got 9.3-9.5 Gbps in iperf3!

Have a nice day guys!
 

tommybackeast

Active Member
Jun 10, 2018
286
105
43
I can confirm that these ConnectX-2 cards work great on Windows 10, though definitely don't use the default drivers included with Windows. They perform terribly. WinOF v4.95 installs perfectly fine on Windows 10 if you use the 8.1 x64 version. The performance is much, much better -- full 10Gb instead of about 6Gb or less in my testing.

WinOF can be found here: http://www.mellanox.com/page/products_dyn?product_family=32
Don't install WinOF-2 (It's for ConnectX-3 and above cards), and don't install the latest version of WinOF (5.10) as it is only for Win7. Click "Archived Versions" and install the 4.95 Win8.1 version.

(I just posted this in the ConnectX-2 great deals thread as well)
Re-visiting a 3yr post - seem Mellanox changed their pages and many old URLs no longer work, any chance you have the current URL for ConnectX-2 Win7 drivers? thanks
 

tommybackeast

Active Member
Jun 10, 2018
286
105
43
  • Like
Reactions: nikalai

Parham

New Member
Jan 8, 2020
1
2
3
Short version: If you're unable to get more than ~7Gbps with a ConnectX-2, ensure your card is running at PCIe x4 and enable jumbo frames.

I wanted to share my experience with Mellanox ConnectX-2 EN MNPA19-XTR cards and Windows 10. Got them off ebay and put one card into my Windows 10 machine running a 6700K on a ASUS Maximus Gene VIII Z170 motherboard. The other card went into a Fedora 27 system and worked without issue (2.9.1000 firmware). They were connected with a Cisco CU3M direct attach cable.

Here is the motherboard slot layout.

Since my video card is in the top slot and covers the middle PCIe slot, I could not put the NIC there so I put it in the open-ended x4 slot even though it is physically a x8 card.

I followed directions from another thread here to update the firmware in the card to 2.10.720. The Windows 2016 WinOF 5.40 drivers from Mellanox appear to work fine (5.4.14004.0).

I was perplexed as to why I was only getting about 3Gbps with a single iperf thread to the Linux box, got up to 5Gbps when I upped the MTU to 9000. Hit about 7Gbps with parallel threads (-P switch in iperf). I then realized the PCIe slot was only running at x2, limiting the maximum theoretical bandwidth to 8Gbps. This was verified with Get-NetAdapterHardwareInfo and the Information tab in the adapter properties.

Code:
PS C:\WINDOWS\system32> Get-NetAdapterHardwareInfo

Name                           Segment Bus Device Function Slot NumaNode PcieLinkSpeed PcieLinkWidth Version
----                           ------- --- ------ -------- ---- -------- ------------- ------------- -------
Ethernet 5                           0   2      0        0                    5.0 GT/s             2 1.1
I moved the video card to the middle slot and the NIC to the top x16 slot in my motherboard and it ran at x4, so I knew it wasn't some kind of compatibility issue. Benchmarks show there was no issue running the 980Ti at x8, but cooling was suboptimal in this setup so I really wanted the card in the bottom slot.

Eventually, I figured out that there is a BIOS setting that trades off 2 lanes to enable the SATA 5/6/Express slots. Since I was not using those slots, I toggled the BIOS setting (can't remember exactly what it was called) to enable PCIe x4 on the bottom slot. Success! With Jumbo Frames enabled (MTU=9000), I was able to hit 9Gbps with a single iperf thread and up to 9.8Gbps when tweaking various iperf parameters. SSH copies are currently being limited by CPU throughput (1.6Gbps) and disk bottlenecks (300-400 MB/s read/write) but I was able to get up to 6Gbps to/from my Windows SSD raid with a ramdisk on the Linux box. It is currently showing as RDMA capable but I have not tested this yet.

Code:
PS C:\WINDOWS\system32> Get-NetAdapterHardwareInfo

Name                           Segment Bus Device Function Slot NumaNode PcieLinkSpeed PcieLinkWidth Version
----                           ------- --- ------ -------- ---- -------- ------------- ------------- -------
Ethernet 5                           0   2      0        0                    5.0 GT/s             4 1.1

PS C:\WINDOWS\system32> Get-SmbServerNetworkInterface

Scope Name Interface Index RSS Capable RDMA Capable Speed   IpAddress
---------- --------------- ----------- ------------ -----   ---------
*          22              True        True         10 Gbps 192.168.2.2
Note: According to this Get-NetAdapterHardwareInfo won't return anything higher than 1.1 for the version, but since it's 5.0 GT/s it must be PCIe 2.0.

---------------------------------------------------------------------------------------------
Hey,
I just wanted to thank you for your great post and solution.
I had the same issue with MNPA19-XTR connected to Intel X520-2 (2 port SFP+).

I had the MNPA19-XTR installed on my desktop connected to one of the ports of the Intel X520-2 on a server machine.

After a couple of jperf tests I was not getting any speed greater than 7Gbps. Just to confirm your solution I went to my MB BIOS (ASUS MAXIMUS_XI_HERO) and changed the speed of PCIE from X2 to X4 and voila i tested and got the full 10Gbps speeds.

For people with the same issue make sure you follow the steps below:
1-If you are using any PCIE anything other than the first PCIE on you MB make sure you have the full speed enabled.
In my case my GPU was located in PCIE16_1 and the NIC was in PCIE16_3.

upload_2020-1-9_10-46-59.png
Here is how to enable it in you ASUS bios:

20200109_083102 (Large).jpg


2-I installed latest driver 5.50.14688 and it gave me full speed:
upload_2020-1-9_10-49-10.png

3-Under performance make sure to select single port traffic or single stream traffic then hit Run Tuning and restart computer.

4-Under advanced:
Interrupt Moderation---->Disable
Jumbo Packet---->9014 (this could be different but i matched it with the highest value on my server Intel X520-2 NIC)

5-Receive Buffers and Send Buffer---->4096 or to its maximum value.

7-Make sure to assign static IP on each NIC.

8-Add the IP address and name of the target machine to the Host file on windows. Let me know if you have questions on how to do this (this will force windows to use the 10gb and not the 1gb LAN speed).

6-Use iperf or jperf or RAM disk to test the speeds. Below are my results from Jperf:

bin/iperf.exe -s -P 0 -i 1 -p 5001 -f g
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 0.00 GByte (default)
------------------------------------------------------------
OpenSCManager failed - Access is denied. (0x5)
[392] local 192.168.50.3 port 5001 connected with 192.168.50.1 port 53662
[ ID] Interval Transfer Bandwidth
[392] 0.0- 1.0 sec 1.07 GBytes 9.21 Gbits/sec
[392] 1.0- 2.0 sec 1.13 GBytes 9.71 Gbits/sec
[392] 2.0- 3.0 sec 1.15 GBytes 9.86 Gbits/sec
[392] 3.0- 4.0 sec 1.15 GBytes 9.85 Gbits/sec
[392] 4.0- 5.0 sec 1.15 GBytes 9.86 Gbits/sec
[392] 5.0- 6.0 sec 1.15 GBytes 9.87 Gbits/sec
[392] 6.0- 7.0 sec 1.15 GBytes 9.88 Gbits/sec
[392] 7.0- 8.0 sec 1.15 GBytes 9.89 Gbits/sec
[392] 8.0- 9.0 sec 1.15 GBytes 9.86 Gbits/sec
[392] 9.0-10.0 sec 1.13 GBytes 9.72 Gbits/sec
[392] 10.0-11.0 sec 1.14 GBytes 9.75 Gbits/sec
[392] 11.0-12.0 sec 1.15 GBytes 9.86 Gbits/sec
[392] 12.0-13.0 sec 1.14 GBytes 9.80 Gbits/sec
[392] 13.0-14.0 sec 1.14 GBytes 9.76 Gbits/sec
[392] 14.0-15.0 sec 1.14 GBytes 9.80 Gbits/sec
[392] 15.0-16.0 sec 1.14 GBytes 9.83 Gbits/sec
[392] 16.0-17.0 sec 1.15 GBytes 9.89 Gbits/sec
[392] 17.0-18.0 sec 1.15 GBytes 9.90 Gbits/sec
[392] 18.0-19.0 sec 1.15 GBytes 9.90 Gbits/sec
[392] 19.0-20.0 sec 1.14 GBytes 9.75 Gbits/sec
[ ID] Interval Transfer Bandwidth
[392] 0.0-20.0 sec 22.8 GBytes 9.80 Gbits/sec

All the credit goes to mfgmfg for his/her great post and solution!!!!!
Thanks again for this.
 

klui

Well-Known Member
Feb 3, 2019
824
453
63
I learned afterwards that while ConnectX-2 was not mentioned, that driver still works for it.

Simply was confusing language from mellanox
You can get the info you need by going to Mellanox's site and select Support & Education > Ethernet Drivers (or Infiniband/VPI). Then in the navigation pane on the left, click Windows Drivers. In the header portion, click the matrix of "WinOF/WinOF-2 drivers versions vs. supported hardware and firmware." Scroll all the way down until you see CX2 references for "supported" versions. Most likely newer versions will support the part under W10. Just choose WinOF.
 
  • Like
Reactions: tommybackeast