Mellanox WinOF powershell cmdlets not working in Windows 10 (ConnectX-3 SMB Direct/RDMA)

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.
W

William

Well-Known Member
May 7, 2015
789
252
63
66
First command errors. I will have to check the correct command spelling.
Get-SmbMultichannelConnection shows nothing.

Ok so from what I can tell 10G cards with RDMA enabled will only work on Windows 10 Pro Workstation if it is connecting to a Windows Server ?

Is that true ?

Than what are people doing if they connect to something else, like a Synology or QNAP box ?
I cannot believe that you cannot get 10G any other way besides a direct connect ?
 
I

i386

Well-Known Member
Mar 18, 2016
4,240
1,546
113
34
Germany
William said:
Is that true ?
Click to expand...
Nope, I have server with server 2019 with shares and different windows 10 clients (enterprise, pro & pro for workstations) at work and home.
None of the windows 10 clients can use rdma features.
The power shell scripts all show that the nics are rdma capable, but the "Get-SmbMultichannelConnection" command shows nothing.

Server 2019 <-> server 2019 (and 2012 r2, 2016) works with rdma.
 
N

necr

Active Member
Dec 27, 2017
156
48
28
124
William said:
Get-SmbMultichannelConnection shows nothing.
Click to expand...
Please try to access shared folder (on your Windows Server) from W10 machine, that should initiate it.

William said:
Ok so from what I can tell 10G cards with RDMA enabled will only work on Windows 10 Pro Workstation if it is connecting to a Windows Server ?
Click to expand...
Yes, Windows 10 Pro Workstation can only access Windows Server share (upload and download data really fast without utilizing CPU). Otherwise, it's Windows Server to Windows Server.

William said:
I cannot believe that you cannot get 10G any other way besides a direct connect ?
Click to expand...
Of course connecting through switch should work. I'd start with direct connect first and then just plug cables to switch.
 
W

William

Well-Known Member
May 7, 2015
789
252
63
66
As I have explained, I am trying to connect to a Synology or QNAP box, I do not have a windows server running atm.
Windows 10 Pro Workstation 1809 <-> Synology/QNAP
I will try a direct connect in a bit.
 
N

necr

Active Member
Dec 27, 2017
156
48
28
124
William said:
As I have explained, I am trying to connect to a Synology or QNAP box,
Click to expand...
That was new to me. Your previous post mentions 2012R2.
William said:
With my test bench based on the ASUS WS C612 SAGE I boot with Server 2012 R2
Click to expand...
Upd: Ah, see it
William said:
The only server I have is a Synology DS1618+ with ConnectX-3 Pro card inside it.
Click to expand...
No need to try direct connect then. You will first need somehow to configure SMB3+ on that. Otherwise, it's going to be just a 10G network share.
 
Last edited:
W

William

Well-Known Member
May 7, 2015
789
252
63
66
Just tried direct connect to a Synology 1618+ (has ConnectX-3 Pro inside). Still low speed :(
 
N

necr

Active Member
Dec 27, 2017
156
48
28
124
William said:
Just tried direct connect to a Synology 1618+ (has ConnectX-3 Pro inside). Still low speed :(
Click to expand...
William said:
Still network speeds are very low, like 2G.
Click to expand...
To sum it up: you did everything right on W10 side, but similar stuff has to be done on NAS side. You'll need to get your hands dirty in samba configuration to get SMB3+ and Multichannel working - that's target #1. Getting RDMA to work on that Synology box is a separate task but it also involves getting to shell and checking all mlx4 related stuff - that doesn't guarantee problem-free Windows - Linux (Synology) operation (never supported in the first place!). Last, 2G isn't that bad - average disk performance is 250MB/s which is exactly 2000Mbit/s (250*8), and if you don't have some kind of mirroring or you have slow drives, or you have just 1 drive there - that's the speed you're going to get.

Synology DS1618+ Performance Tweaking -- Enabling SMB Multichannel
 
F

Frank173

Member
Feb 14, 2018
75
9
8
marv,

did you ever get the Mellanox cmdlets to work in Windows 10 Pro for Workstation? I am facing similar issues right now and nowhere does it mention that roce or rdma only works by sharing from Windows Server, it should work equally well in Windows 10 Pro for Workstation, but I cannot even configure roce because the cmdlets are not recognized. This is what I currently get on my Windows 10 Pro for Workstation machines, so clearly rdma is working but I can't configure further using the MLNX cmdlets.

Capture.PNG
Capture10.JPG

but when copying files between the two machines it shows that "client is not RDMA capable"

Capture5.JPG

marv said:
Hello,

I got two MCX314 cards, and I am trying to make SMB direct in windows 10 working. I enabled DCB and SMB Direct in windows features, updated firmware to 2.42 and installed WinOF 5.35

In Mellanox guide for Windows 2012, It is specified that for WinOF 4.9 or higher, RoCE version has to be specifically enabled via powershell cmdlet.

HowTo Configure SMB Direct (RoCE) over PFC on W... | Mellanox Interconnect Community
HowTo Configure RoCE in Windows Environment (Gl... | Mellanox Interconnect Community

now the issue. None of the cmdlets are working for me. When I open new PS window, It "knows" the cmdlets, because clicking tab finishes the names, but trying anything just renders error

Code: 
PS C:\WINDOWS\system32> Get-MlnxDriverCoreSetting
Get-MlnxDriverCoreSetting : The term 'Get-MlnxDriverCoreSetting' is not recognized as the name of a cmdlet, function, s
cript file, or operable program. Check the spelling of the name, or if a path was included, verify that the path is cor
rect and try again.
At line:1 char:1
+ Get-MlnxDriverCoreSetting
+ ~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : ObjectNotFound: (Get-MlnxDriverCoreSetting:String) [], CommandNotFoundException
    + FullyQualifiedErrorId : CommandNotFoundException
Importing module doesnt do anything
Code: 
PS C:\WINDOWS\system32> Import-Module -name "C:\Program Files\Mellanox\MLNX_CIMProvider\WMI\Modules\MLNXProvider" -verbose
PS C:\WINDOWS\system32>
So, does anyone know how to set RoCE version without powershell or how to make the cmdlets working? I am using Windows 10 Pro for Workstations, which should have RDMA enabled. I also tried WinOF 5.25 and same behavior.
Click to expand...
 
Last edited:
tjk

tjk

Active Member
Mar 3, 2013
481
199
43
I think MLNX pulled the powershell cmdlets in the latest WinOF, at least I don't have them/can't find them when I install the latest WinOF.

Anyone else?
 
F

Frank173

Member
Feb 14, 2018
75
9
8
They are not part of WinOF2 but are part of WinOF. I just used them today they are definitely part of WinOF

tjk said:
I think MLNX pulled the powershell cmdlets in the latest WinOF, at least I don't have them/can't find them when I install the latest WinOF.

Anyone else?
Click to expand...
 
F

Frank173

Member
Feb 14, 2018
75
9
8
I gave up on smb direct, rdma, and roce. I did not witness any performance increases for file servers. Windows file manager operates single threaded and I max out bandwidth at 1GB/sec no matter what. I have file access to a striped array at over 12 GB/sec but can't transfer that performance over to clients who have access over a 100 Gb/sec network. Mellanox performance tools show a 95 Gb/sec transfer speed and I can load data off the array at around 7 GB/s single threaded but as soon as the windows file manager is involved performance drops off to 1 GB/s levels. Rmda might be useful for server clusters and latency sensitive applications but I find no benefit regarding serving data off a file server.

I am happy to worship and call anyone a hero who can explain a way to force Window's file manager to use multiple threads by default for file copies. Not robocopy or other tools but the default file manager that Windows 10 and Windows Server comes with.
 
  • Like
Reactions: William
I

i386

Well-Known Member
Mar 18, 2016
4,240
1,546
113
34
Germany
Explorer is the wrong tool.

Use something that supports multithreading, like robocopy.
 
F

Frank173

Member
Feb 14, 2018
75
9
8
Even robocopy does not split up a single file copy into segments that can be transferred concurrently. And anyway, it remains a total mystery why Windows limits single threaded file transfers to around 1GB/sec when the network can handle a multiple, when the file storage can handle a multiple and when the client can handle a multiple of that. I have in many years not come across a single source that provided a reasonable explanation for this limitation other than poor implementation. The kernel is the only bottleneck it seems.

i386 said:
Explorer is the wrong tool.

Use something that supports multithreading, like robocopy.
Click to expand...
 
  • Like
Reactions: William
You must log in or register to reply here.
Top Bottom