Connectivity Issues on Windows 10 with Mellanox ConnectX-2 and Driver 5.50

[darkarn]

Hi, I am unsure how or why but I noticed strange slowdowns to my Internet connection (e.g. initial calls to pages taking longer to load, Youtube defaulting to 360p instead of the usual 1080p 60fps) after using driver 5.50 with my Mellanox ConnectX-2 cards even though they are not being used for connecting to the Internet (am using Intel NICs for this). I suspect the problem has something to do with DNS since in my Event Viewer I noticed an unusually high number of errors related to the DNS Client Service.

I noticed this problem on both Windows 10 computers that I have on build 1709, and can confirm that the problem does not happen if using inbox drivers or on 4.80.

I do not foresee any more patches or fixes to this card so the best I can do for now is to put this out here. In the meantime, has anyone else noticed this problem too on other builds of Windows 10?

 

[i386]

I have a cx-2 in a workstation with windows 10 (everything from the first rtm version up to 1803), 2+ years so far without problems.

 

[arglebargle]

I ran a CX-2 in my win 10 workstation for about a year and had zero issues with it. I don't recall exactly which OFED version I used at the time, 5.35 maybe?

 

[darkarn]

Thanks guys, I don't recall this problem either when I started using the cards since last year, it started only after I did a fresh install of Windows 10 two weeks ago.

Strangely, if I were to do another fresh install now, the problem would go away (had tested and confirmed via writing current config to a disk image using Clonzilla, reinstalling Windows, and then restoring back after testing) but for how long I am unsure, which is why I am trying to trace the problem first. Alternatively I can just bite the bullet and reinstall again but this means another round of reinstallation of stuff, which I am trying to avoid.

These cards are HP branded dual port cards running on 2.10.0720 firmware by the way

 

[arglebargle]

Try flashing to 2.9.8350. I had a few really strange issues with one of my cards (also dual port, MHQH29C) on 2.10.0720 that were completely resolved by switching firmwares. My MPNA19* single port cards never demonstrated the same problem.

Connect-X2 Firmware - Google Drive

 

[darkarn]

Thanks, let me try that and report back accordingly! A pity that I may lose RDMA support though; I followed the guide here for it and therefore got the 2.10.0720 firmware

 

[arglebargle]

You won't, 2.9.8350 onward support RDMA and SR-IOV.

 

[i386]

You won't, 2.9.8350 onward support RDMA and SR-IOV.

The last firmware version publicly available on mellanox website is 2.9.1200 for the ethernet nics.

 

[arglebargle]

Yes, I know. Those aren't the latest available firmwares out in the wild though. I've been collecting and mirroring later firmware packages as I find them: Connect-X2 Firmware - Google Drive

2.9.1000 doesn't support RoCE, 2.9.8350 and on do. I think they added SR-IOV beta support in 2.9.1000.

 

[darkarn]

You won't, 2.9.8350 onward support RDMA and SR-IOV.

Thanks but why would I still get messages in Event Viewer that SR-IOV is not supported?

 

[i386]

Maybe it's not enabled in the mainboards bios?

 

[arglebargle]

You have to manually enable SR-IOV in the board's firmware on the CX-2 boards, if you're just flashing with the stock .ini file in the firmware package or with a prebuilt .bin it's possible that SR-IOV isn't enabled. Dump the configuration from the board and look under the [HCA] section and make sure that the following are set:

num_pfs = 1
total_vfs = 64
sriov_en = true

I typically set total_vfs to something like 12, 16 or 24.

If those aren't set then SR-IOV *isn't* active on the board.

Full instructions here: https://forums.servethehome.com/index.php?threads/sr-iov-for-mellanox-connectx-2.12693/#post-121167

 

[darkarn]

Maybe it's not enabled in the mainboards bios?

I can't find any option to set this on the motherboards I am using. They are consumer grade stuff though

You have to manually enable SR-IOV in the board's firmware on the CX-2 boards, if you're just flashing with the stock .ini file in the firmware package or with a prebuilt .bin it's possible that SR-IOV isn't enabled. Dump the configuration from the board and look under the [HCA] section and make sure that the following are set:

num_pfs = 1
total_vfs = 64
sriov_en = true

I typically set total_vfs to something like 12, 16 or 24.

If those aren't set then SR-IOV *isn't* active on the board.

Full instructions here: https://forums.servethehome.com/index.php?threads/sr-iov-for-mellanox-connectx-2.12693/#post-121167

Thanks, let me try that along with flashing the firmware too!

 

[arglebargle]

Quick question: can you actually make use of SR-IOV or RDMA on your version of Windows? I'm not sure what versions of Windows support SR-IOV (maybe it works with optional Hyper V features on Win10?) I do know what RDMA isn't available unless you're using Windows 10 with a "for Workstations" license which is an additional purchase on top of Win 10 Pro.

Just wanted to throw that out there and possibly save you some time in case you're trying to get things that aren't supported working.

 

[compuguy]

Hi, I am unsure how or why but I noticed strange slowdowns to my Internet connection (e.g. initial calls to pages taking longer to load, Youtube defaulting to 360p instead of the usual 1080p 60fps) after using driver 5.50 with my Mellanox ConnectX-2 cards even though they are not being used for connecting to the Internet (am using Intel NICs for this). I suspect the problem has something to do with DNS since in my Event Viewer I noticed an unusually high number of errors related to the DNS Client Service.

I noticed this problem on both Windows 10 computers that I have on build 1709, and can confirm that the problem does not happen if using inbox drivers or on 4.80.

I do not foresee any more patches or fixes to this card so the best I can do for now is to put this out here. In the meantime, has anyone else noticed this problem too on other builds of Windows 10?

Everytime I try to install the 5.50 drivers for my connectx 2 on my desktop, the entire computer freezes. It works fine with the windows stock drivers. I have a feeling the Asrock motherboard is the culprit.

 

[arglebargle]

Everytime I try to install the 5.50 drivers for my connectx 2 on my desktop, the entire computer freezes. It works fine with the windows stock drivers. I have a feeling the Asrock motherboard is the culprit.

It could be, but try older drivers before you give up entirely. I think I recall using 5.35.xxxx a year or two ago when I installed a CX-2 in my Win 10 desktop.

Also, when the machine freezes what's actually causing the crash? Try something like "WhoCrashed" to parse any available dump.

 

[darkarn]

Quick question: can you actually make use of SR-IOV or RDMA on your version of Windows? I'm not sure what versions of Windows support SR-IOV (maybe it works with optional Hyper V features on Win10?) I do know what RDMA isn't available unless you're using Windows 10 with a "for Workstations" license which is an additional purchase on top of Win 10 Pro.

Just wanted to throw that out there and possibly save you some time in case you're trying to get things that aren't supported working.

I am on Windows 10 Pro. This is the first time I heard about this "For Workstations" license though, thanks for the heads up! Maybe I should try Enterprise instead, I heard it pretty much gives you full control of the OS.

And no, I haven't done the flashing of firmware yet; been busy. Will start shortly

Everytime I try to install the 5.50 drivers for my connectx 2 on my desktop, the entire computer freezes. It works fine with the windows stock drivers. I have a feeling the Asrock motherboard is the culprit.

Try 4.80, it is the last documented driver to be working for these cards

 

[arglebargle]

I am on Windows 10 Pro. This is the first time I heard about this "For Workstations" license though, thanks for the heads up! Maybe I should try Enterprise instead, I heard it pretty much gives you full control of the OS.

Yeah, you won't have any luck getting RDMA working without the upsell to a "Windows 10 Pro for Workstations" license, I think they're an additional $100-120 after a Win10 Pro license. RDMA on client OSes is feature locked to an optional package that's only available with that license or another enterprise SKU.

Try 4.80, it is the last documented driver to be working for these cards

I can confirm that 5.35.xxxx works. Hell, 5.50.xxxx worked for me the last time I tried it too so I suspect the problem isn't the driver (but it never hurts to check.)

 

[compuguy]

It could be, but try older drivers before you give up entirely. I think I recall using 5.35.xxxx a year or two ago when I installed a CX-2 in my Win 10 desktop.

Also, when the machine freezes what's actually causing the crash? Try something like "WhoCrashed" to parse any available dump.

Its worth a shot. I'm not 100% sure if it made a dump or not (the whole machine freezes, no BSOD).

I am on Windows 10 Pro. This is the first time I heard about this "For Workstations" license though, thanks for the heads up! Maybe I should try Enterprise instead, I heard it pretty much gives you full control of the OS.

And no, I haven't done the flashing of firmware yet; been busy. Will start shortly



Try 4.80, it is the last documented driver to be working for these cards

I think I did try the 4.80 drivers a year ago. I'll try that again when I get a chance.

 

[darkarn]

Yeah, you won't have any luck getting RDMA working without the upsell to a "Windows 10 Pro for Workstations" license, I think they're an additional $100-120 after a Win10 Pro license. RDMA on client OSes is feature locked to an optional package that's only available with that license or another enterprise SKU.

I can confirm that 5.35.xxxx works. Hell, 5.50.xxxx worked for me the last time I tried it too so I suspect the problem isn't the driver (but it never hurts to check.)

Ah I see... Yeah come to think of it, 5.50 was working perfectly all these while; this issue was pretty recent and ironically happened only after this new fresh install of Windows 10. This is puzzling to say the least.

I'll try to reinstall the NIC's driver along with 5.50 and see how this goes. If not I'll just bite the bullet and do another reinstall of Windows since I have confirmed that would resolve the problem