Mellanox WinOF powershell cmdlets not working in Windows 10 (ConnectX-3 SMB Direct/RDMA)

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

marv

Active Member
Apr 2, 2015
155
34
28
38
Hello,

I got two MCX314 cards, and I am trying to make SMB direct in windows 10 working. I enabled DCB and SMB Direct in windows features, updated firmware to 2.42 and installed WinOF 5.35

In Mellanox guide for Windows 2012, It is specified that for WinOF 4.9 or higher, RoCE version has to be specifically enabled via powershell cmdlet.

HowTo Configure SMB Direct (RoCE) over PFC on W... | Mellanox Interconnect Community
HowTo Configure RoCE in Windows Environment (Gl... | Mellanox Interconnect Community

now the issue. None of the cmdlets are working for me. When I open new PS window, It "knows" the cmdlets, because clicking tab finishes the names, but trying anything just renders error

Code:
PS C:\WINDOWS\system32> Get-MlnxDriverCoreSetting
Get-MlnxDriverCoreSetting : The term 'Get-MlnxDriverCoreSetting' is not recognized as the name of a cmdlet, function, s
cript file, or operable program. Check the spelling of the name, or if a path was included, verify that the path is cor
rect and try again.
At line:1 char:1
+ Get-MlnxDriverCoreSetting
+ ~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : ObjectNotFound: (Get-MlnxDriverCoreSetting:String) [], CommandNotFoundException
    + FullyQualifiedErrorId : CommandNotFoundException
Importing module doesnt do anything
Code:
PS C:\WINDOWS\system32> Import-Module -name "C:\Program Files\Mellanox\MLNX_CIMProvider\WMI\Modules\MLNXProvider" -verbose
PS C:\WINDOWS\system32>
So, does anyone know how to set RoCE version without powershell or how to make the cmdlets working? I am using Windows 10 Pro for Workstations, which should have RDMA enabled. I also tried WinOF 5.25 and same behavior.
 
  • Like
Reactions: Frank173

cesmith9999

Well-Known Member
Mar 26, 2013
1,417
468
83
Rdma is only enabled in 2 places

Server 2012+
Windows 10 pro for workstations

Regular pro and home sku will not have smb direct enabled.

Chris
 

i386

Well-Known Member
Mar 18, 2016
4,220
1,540
113
34
Germany
Did you install the SMB direct feature on your windows system? By default it's not installed.

Control panel > programms & features > windows features, scroll down, check "SMB Direct" and restart your system.
 

marv

Active Member
Apr 2, 2015
155
34
28
38
@cesmith9999 yeah, I use windows 10 pro for workstations, wrote that at the end of my post.

@i386 I enabled SMB Direct and DCB in Windows Features, its at beginning of my post :)

Originally I was using W10 to W10 connection, but today I installed Win 2016 on one host. cmdlets are working fine on there and RoCE v2 is set by default.

netstat -xan on server actually says its listening on 445, on W10 its not listening. This probably means that SMB Direct in W10 is only available in client mode.

doing test with diskspd when accessing share on server, RDMA counters in performance monitor are showing some numbers, so seems like its working fine.
 

i386

Well-Known Member
Mar 18, 2016
4,220
1,540
113
34
Germany

William

Well-Known Member
May 7, 2015
789
252
63
66
Thanks for your guide Necr.
I followed it and did get the workstation upgrade for Windows 10 Pro. My version is 1809.
Everything shows RDMA enabled with the powershell commands.
I am using a dual port ConnextX-3 Pro card.
enabled SMB Direct and DCB in Windows Features. They had been actually from past attempts.

I do not see the RDMA information like you showed in the Performance Monitor.
I see a graph.

Still network speeds are very low, like 2G.

I have looked at this issue maybe 3-4 times, spent days each time but no go. I thought the Workstation Upgrade would do the trick but so far its not happening :(
 

zkrr01

Member
Jun 28, 2018
106
6
18
I was having the same problems until I updated my workstations to "Windows 10 Pro for Workstations". Then when I ran :
PS C:\work> Get-SmbClientNetworkInterface it showed my 40GbE NIC returned it was RDMA Capable it started working!

It looks like you have upgraded your workstations, does Get-SmbClientNetworkInterface show your NIC is RDMA Capable?
 

William

Well-Known Member
May 7, 2015
789
252
63
66
Yes it does.
I have a dual port ConnectX-3 Pro. Only one port is connected and that is Interface 21.RDMA.JPG
 

William

Well-Known Member
May 7, 2015
789
252
63
66
Much of that I have done and all report RDMA is working, but there are extra steps in that link I will do.

Thank you !
 

zkrr01

Member
Jun 28, 2018
106
6
18
I just noticed your interface 21 showed 10 GbE I would have thought your NIC would do 40GbE
 

zkrr01

Member
Jun 28, 2018
106
6
18
I just noticed ConnectX-3 Pro and it stuck in my mind. I recently brought 40GbE cards on the recommendation of Mellanox and now I wish I did more research. Mellanox also had 40/50 and 40.56 cards for about the same price. I should have brought the used ConnectX-3 card and also got 40/56. Now I really need 56GbE and am stuck at 40GbE.
 

William

Well-Known Member
May 7, 2015
789
252
63
66
So that guide was written for a Server A to Server B direct connect setup.
I use a switch, DLink DXS-1210-12TC or NETGEAR ProSafe XS708T. Have tried both with no change.
The only server I have is a Synology DS1618+ with ConnectX-3 Pro card inside it.

I could try a direct connect but that defeats the purpose of a NAS shared across my network. Not everything is 10GbE but need to share to several machines.

I saw no change after completing the guide.
 

zkrr01

Member
Jun 28, 2018
106
6
18
That guide had nothing to so with workstations or servers or direct connect or switches. It's just was not possible to use that guide with workstations prior to "Windows 10 Pro for Workstations" version. Prior to that the client side could be a workstation but the other side had to be a server. Since the guide is not working for you, I suspect other factors are coming into play.
 

William

Well-Known Member
May 7, 2015
789
252
63
66
Sadly I think so :(

With my test bench based on the ASUS WS C612 SAGE I boot with Server 2012 R2, ConnectX-2 cards and do not have issues. Its just my main workstation that I would love to get 10G working, but sadly its been nothing but trouble. Its an older ASUS Z10PE-D16 WS with 2x E5-2699v3's running Windows 10 Pro Workstation that gives me issues.
 

necr

Active Member
Dec 27, 2017
151
47
28
124
Hey William, I think I had an issue like that before and it was solved by unloading and then loading NetworkDirect module. Can't remember the command now though - could be Set-NetOffloadGlobalSetting -NetworkDirect Disabled

What does Get-SmbMultichannelConnection show? As noted above, share has to be on the Windows Server!
Another possibility (less likely) is that your switches don't support RoCEv1 (fallback mode) which uses specific frame ethertype 0x8915. Are both sides in RoCEv2 mode?
 
Last edited: