Connectivity Issues on Windows 10 with Mellanox ConnectX-2 and Driver 5.50

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

darkarn

New Member
Aug 31, 2017
17
3
3
33
I fixed it I hope! Best of all, I can stay on 5.50 with 2.10.0720 and not need to reformat Windows 10!

You see, I took notes of what the error messages are saying about the DNS Client Service, which is similarly to what was mentioned here: Get rid of yellow mark on network icon on windows server. I tried going to the Microsoft website mentioned on that page (http://www.msftncsi.com/ncsi.txt) and noticed that it is unresponsive at times, even though it is just a very simple web page, indicating that something was wrong with the way Internet goes in and out of my computers.

Given that the page was important since Windows uses it as a way of finding out the quality of Internet connection and the Techspot blog (https://blogs.technet.microsoft.com/networking/2012/12/20/the-network-connection-status-icon/) mentioned that this specifically test the quality of DNS responses, I then decided to check all my network adapters. I found out that I was setting DNS servers even for my Mellanox adapters even though they are connected only to other computers (I was setting two computers and a NAS to connect to each other directly; I wanted to transfer files between them via 10Gbe while they are still connected to the Internet with Gigabit) instead of a switch.

I tried to do a "Network Reset" to clear all settings and set only the IP address, Netmask and Default Gateway for each Mellanox adapter while leaving the DNS servers blank. This idea worked to my surprise as I was expecting Windows to give me error messages for not setting a DNS server for those 10 Gbe connections. I repeated all of these on the other computer (which was using Windows inbox drivers instead of 5.50) and the NAS (which was using Linux inbox drivers) and have similar successes.

So far, they are as snappy as before, which is very noticeable since I am on 1Gbps up/down connection. Youtube videos can load at FHD quality right away instead of being stuck at 360p. All of these are happening even on Incognito mode of the browsers (I tested using this all along to bypass the browser cache to ensure that the connection is fast not due to the cached files)

All in all, this shows that:
1. You do not need to set a DNS server for peer to peer connection; this should have been obvious to me on hindsight, I'll chalk this up to experience then
2. A 10 Gbe switch would have prevented this in the first place since in that case, I can just use the 10 Gbe adapters for both data transfer and Internet connection at the same time, making configuration and troubleshooting a lot easier. I can see why these switches cost a lot now...
3. Maybe I should have just went with ConnectX-3 in the first place? I am still getting error messages about the ConnectX-2 being too old etc, it is making me feel bad about my purchases! (Paid about 40 USD per card including a DAC per card and S&H)
 
  • Like
Reactions: Tha_14

Z3R0

New Member
Sep 5, 2018
6
1
3
36
Florida, USA
Try flashing to 2.9.8350. I had a few really strange issues with one of my cards (also dual port, MHQH29C) on 2.10.0720 that were completely resolved by switching firmwares. My MPNA19* single port cards never demonstrated the same problem.

Connect-X2 Firmware - Google Drive

Thank you very much for your work and the link. I'd tried a few months back to update the firmware with no luck. Thanks to you, I've finally got mine on 2.10.0720. Very much appreciated.
 

anomaly

Active Member
Jan 8, 2018
235
48
28
Ran into this thread now... having issues with this card on both Linux and Win10:

Code:
Image type:            FS2
FW Version:            2.9.1200
Rom Info:              type=PXE version=3.3.400 proto=VPI
Device ID:             26448
Description:      Port1            Port2
MACs:                      xx xx
VSD:
PSID:                  MT_0F60110010
The speeds are laughable and I replicated the issue with both Twinax and fiber+optics:


Code:
[ ID] Interval           Transfer     Bandwidth
[  5]   0.00-1.00   sec  5.70 KBytes  46.7 Kbits/sec
[  5]   1.00-2.00   sec  7.13 KBytes  58.4 Kbits/sec
[  5]   2.00-3.00   sec  7.13 KBytes  58.3 Kbits/sec
[  5]   2.00-3.00   sec  7.13 KBytes  58.3 Kbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth
[  5]   0.00-3.00   sec  0.00 Bytes  0.00 bits/sec                  sender
[  5]   0.00-3.00   sec  21.4 KBytes  58.4 Kbits/sec                  receiver
iperf3: the client has terminated
-----------------------------------------------------------
I'm beginning to think the card is defective, but this could be a FW issue too. Anyone has a clue? And what is the process to flash these with the HP firmware that is more recent? SFP+ and ethernet mode is what I use, no IB here.
 

andy007

New Member
Sep 19, 2018
7
0
1
Ran into this thread now... having issues with this card on both Linux and Win10:

Code:
Image type:            FS2
FW Version:            2.9.1200
Rom Info:              type=PXE version=3.3.400 proto=VPI
Device ID:             26448
Description:      Port1            Port2
MACs:                      xx xx
VSD:
PSID:                  MT_0F60110010
The speeds are laughable and I replicated the issue with both Twinax and fiber+optics:


Code:
[ ID] Interval           Transfer     Bandwidth
[  5]   0.00-1.00   sec  5.70 KBytes  46.7 Kbits/sec
[  5]   1.00-2.00   sec  7.13 KBytes  58.4 Kbits/sec
[  5]   2.00-3.00   sec  7.13 KBytes  58.3 Kbits/sec
[  5]   2.00-3.00   sec  7.13 KBytes  58.3 Kbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth
[  5]   0.00-3.00   sec  0.00 Bytes  0.00 bits/sec                  sender
[  5]   0.00-3.00   sec  21.4 KBytes  58.4 Kbits/sec                  receiver
iperf3: the client has terminated
-----------------------------------------------------------
I'm beginning to think the card is defective, but this could be a FW issue too. Anyone has a clue? And what is the process to flash these with the HP firmware that is more recent? SFP+ and ethernet mode is what I use, no IB here.
Me too. I have the same card (same PSID, single port) and my speed is a little bit better at around 3-4Gbit/s (but nowhere near expected). I tried all the settings I could find in many guide/tutorials but nothing changes. I am on Win 10 Pro.
Not sure if this is true or not but I saw most people with Mellanox X2 success stories are using Dual Port X2 cards, including this video:
. Could anyone who has successfully configured their Single Port Mellanox X2 share their configuration?
 

AiC

New Member
Oct 28, 2018
29
15
3
Me too. I have the same card (same PSID, single port) and my speed is a little bit better at around 3-4Gbit/s (but nowhere near expected). I tried all the settings I could find in many guide/tutorials but nothing changes. I am on Win 10 Pro.
Not sure if this is true or not but I saw most people with Mellanox X2 success stories are using Dual Port X2 cards, including this video:
. Could anyone who has successfully configured their Single Port Mellanox X2 share their configuration?
I have an X2 on my Win10 and my Freenas box. On my last iPerf test I got 9.5Gbs I think. I'll rerun the test when I get home. What config info do you want?
 

andy007

New Member
Sep 19, 2018
7
0
1
I have an X2 on my Win10 and my Freenas box. On my last iPerf test I got 9.5Gbs I think. I'll rerun the test when I get home. What config info do you want?
1.iperf result

2.What kind of cable/connection are you using? Cable Length?

3.Do you use any form of virtual machine? If yes, what is it

4.A screenshot of 'Information' tab.
(For this, I need Driver Version, Firmware Version, Bus Type, Part Number, Device ID)

5.Output of 'flint -d /dev/mst/mt4099_pci_cr0 query'

6.Under Advanced tab
a.Interupt Moderation
b.Jumbo Packet
c.Maximum number of RSS Processors
d.Receive Buffers
e.Send Buffers
f.Anything else under this tab that you made changes?

7.Something crucial configuration that I may not know of? please share

Really appreciate it
 

AiC

New Member
Oct 28, 2018
29
15
3
1.iperf result

2.What kind of cable/connection are you using? Cable Length?

3.Do you use any form of virtual machine? If yes, what is it

4.A screenshot of 'Information' tab.
(For this, I need Driver Version, Firmware Version, Bus Type, Part Number, Device ID)

5.Output of 'flint -d /dev/mst/mt4099_pci_cr0 query'

6.Under Advanced tab
a.Interupt Moderation
b.Jumbo Packet
c.Maximum number of RSS Processors
d.Receive Buffers
e.Send Buffers
f.Anything else under this tab that you made changes?

7.Something crucial configuration that I may not know of? please share

Really appreciate it
I'll post up the info when I get home.
 

AiC

New Member
Oct 28, 2018
29
15
3
1.iperf result

2.What kind of cable/connection are you using? Cable Length?

3.Do you use any form of virtual machine? If yes, what is it

4.A screenshot of 'Information' tab.
(For this, I need Driver Version, Firmware Version, Bus Type, Part Number, Device ID)

5.Output of 'flint -d /dev/mst/mt4099_pci_cr0 query'

6.Under Advanced tab
a.Interupt Moderation
b.Jumbo Packet
c.Maximum number of RSS Processors
d.Receive Buffers
e.Send Buffers
f.Anything else under this tab that you made changes?

7.Something crucial configuration that I may not know of? please share

Really appreciate it
1. Iperf3 9.64Gb/s both ways

2. Currently using 15m fiber. Same result with 5m DAC

3. No virtual machine

5. flint -d /dev/mst/mt4099_pci_cr0 query
-E- mst service failed: Could not open the SC Manager
-E- mst service failed: Could not open the SC Manager
-E- Cannot open Device: /dev/mst/mt4099_pci_cr0. No such file or directory. MFE_CR_ERROR

6. a.enabled
b.9000
c.8
d.4096
e.2048
f. Nothing special

7. Nothing (Edit: This is peer to peer)
 

Attachments

  • Like
Reactions: anomaly

anomaly

Active Member
Jan 8, 2018
235
48
28
Can you show us screenshots of the iperf output and settings?

I'm running the same driver version with an upgraded card. Also output of System Information (feel free to remove personal details!). This is weird.
 

AiC

New Member
Oct 28, 2018
29
15
3
Can you show us screenshots of the iperf output and settings?

I'm running the same driver version with an upgraded card. Also output of System Information (feel free to remove personal details!). This is weird.
Do you want the entire system information output file?
 

AiC

New Member
Oct 28, 2018
29
15
3
I forgot that I added 2 lines to the hosts file.
C:\Windows\System32\drivers\etc\hosts

I added the IP of each card and the system name associated with that IP

10.10.10.1 DESKTOP
10.10.10.2 FreeNAS
 
Last edited:

anomaly

Active Member
Jan 8, 2018
235
48
28
Do you want the entire system information output file?
Just the first page is fine, what I'm mostly after is the information on logical CPUs, architecture, etc. I'm using a Threadripper CPU here. Definitely not seeing those numbers. I do in Linux though.
 

andy007

New Member
Sep 19, 2018
7
0
1
1. Iperf3 9.64Gb/s both ways

2. Currently using 15m fiber. Same result with 5m DAC

3. No virtual machine

5. flint -d /dev/mst/mt4099_pci_cr0 query
-E- mst service failed: Could not open the SC Manager
-E- mst service failed: Could not open the SC Manager
-E- Cannot open Device: /dev/mst/mt4099_pci_cr0. No such file or directory. MFE_CR_ERROR

6. a.enabled
b.9000
c.8
d.4096
e.2048
f. Nothing special

7. Nothing (Edit: This is peer to peer)
Thank you so much for the config.
Other than using VM in ESXI, and iperf 2, my config is pretty much the same. If passing-thru directly, I got about 6-7gbit/s. Other than that, I don't know what to do now.
I think ESXI is the limiting factor here. Anyone using mellanox x2 in ESXI can chime in maybe? I may just gather some old parts to build another desktop for testing purpose.
Another idea: does anyone know if there is a 10gb nic that workss easily in ESXI and win 10? I might just switch to Intel or something if there is such card.
 

andy007

New Member
Sep 19, 2018
7
0
1
@AiC, what type of cable do you use? DAC or fiber?
I read somewhere that using fiber will give you less trouble. I am grabbing some from Amazon to try out. Will report back with result.
 

anomaly

Active Member
Jan 8, 2018
235
48
28
@AiC, what type of cable do you use? DAC or fiber?
I read somewhere that using fiber will give you less trouble. I am grabbing some from Amazon to try out. Will report back with result.
DAC and fiber should not make a difference at all, fiber introduces more variables in the mix too. Clean contact, no kinks in the fiber run, etc.
If your DAC is not damaged, and it is rare to see one damaged unless you abused it, it will give you the highest speed your NIC can achieve.

The issue here seems to be the software stack. Not the connections...
 

andy007

New Member
Sep 19, 2018
7
0
1
Update: I figured that when I do 2 parallel streams in iperf, I was able to get the SUM of about 9.5Gbit/s. With one stream, maximum is about 8gbit/s. Looking @AiC 's result with single stream, I think I'm still missing something. Oh, and I did make sure to disable firewall/internet security on both ends of the connection.
 

AiC

New Member
Oct 28, 2018
29
15
3
@AiC, what type of cable do you use? DAC or fiber?
I read somewhere that using fiber will give you less trouble. I am grabbing some from Amazon to try out. Will report back with result.
Sorry about not getting back to this, I've used both a DAC and fiber. Same results with both.