Mellanox WinOF powershell cmdlets not working in Windows 10 (ConnectX-3 SMB Direct/RDMA)

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

cesmith9999

Well-Known Member
Mar 26, 2013
1,417
468
83
This is from the help

Thread Description
Directory search:
Specify the number of thread to be used for directory search. Increasing number of thread for directory search is effective when directory structure is rather complex, but it does not contribute to the performance so much when directory structure is simple. - make this some small number greater than 1

Directory copy:
Specify the number of thread to be used to copy files. Thread number actually identical to the number of copy process executed simultaneously. When 10 is specified, and then 10 files are copied simultaneously. - this is like the /MT for robocopy

File copy:
Specified the number of thread to be used to copy a file. This is effective to copy small number of huge file copy. It is not recommended to use more than 1 thread for a file copy as it could consume large size of system resource. (Multiple number of directory copy thread by number of file copy thread. ) - this is the multiple threads per file

it is recommended (this is from memory) that you have file copy or Directory copy set to 1. and not have both set to values set to values greater than 1 at the same time.

Chris
 

Frank173

Member
Feb 14, 2018
75
9
8
Thanks, I will try it out and report back.

This is from the help

Thread Description
Directory search:
Specify the number of thread to be used for directory search. Increasing number of thread for directory search is effective when directory structure is rather complex, but it does not contribute to the performance so much when directory structure is simple. - make this some small number greater than 1

Directory copy:
Specify the number of thread to be used to copy files. Thread number actually identical to the number of copy process executed simultaneously. When 10 is specified, and then 10 files are copied simultaneously. - this is like the /MT for robocopy

File copy:
Specified the number of thread to be used to copy a file. This is effective to copy small number of huge file copy. It is not recommended to use more than 1 thread for a file copy as it could consume large size of system resource. (Multiple number of directory copy thread by number of file copy thread. ) - this is the multiple threads per file

it is recommended (this is from memory) that you have file copy or Directory copy set to 1. and not have both set to values set to values greater than 1 at the same time.

Chris
 

eskiuzmi

New Member
Dec 10, 2022
12
0
1
Necroing this great thread, thank you everyone for your help.
First, a reply to someone who wasn't seeing the RDMA stats:
I do not see the RDMA information like you showed in the Performance Monitor.
click on this button twice 1671877687805.png


I followed both tutos but still not working, I might need your sets of eyes to see what I am missing
window guarden's tests on both machines:
1671878294716.png
1671878302242.png
1671878379391.png
1671878385123.png
1671878449736.png
1671878489992.png
1671878507315.png
1671878533593.png
1671878540321.png
1671878584146.png
1671878589992.png

and yet another infiniband tuto:
1671878695662.png
1671878715817.png
1671878732252.png1671878737307.png
1671878765290.png



Try that and see if any of the commands fail. It looks like you are almost there!
Thanks for the link, I went through all that and Install-WindowsFeature as well as diskspd.exe fail in "pro workstation", I think this is server feature. Samba Direct and Data Center Binding are enabled in windows feature so maybe that's the same thing.

Set-NetOffloadGlobalSetting -NetworkDirect Disabled
Unfortunately
1671878091035.png
Get-SmbMultichannelConnection
1671878225380.png
on one machine
and
1671878133221.png
on the other
 

Attachments

i386

Well-Known Member
Mar 18, 2016
4,221
1,540
113
34
Germany
to see what I am missing
Sadly you miss nothing. Rdma with client versions of windows seems to be broken.
With every (big) windows and mellanox driver update I reran the same commands you used (and that are on the microsoft and nvidia/mellanox site) and still didn't get any rdma counters in perfmon. Tried the same commands with a server installation and everything work fine...
 

eskiuzmi

New Member
Dec 10, 2022
12
0
1
Sadly you miss nothing. Rdma with client versions of windows seems to be broken.
With every (big) windows and mellanox driver update I reran the same commands you used (and that are on the microsoft and nvidia/mellanox site) and still didn't get any rdma counters in perfmon. Tried the same commands with a server installation and everything work fine...
Thanks for telling me.
I'll abandon that.
The amusing thing is when I activated the workstation license the performance halved, went from copy at 1.6GB/s and direct transfer at 6GB/s to 500MB/s
I'll re install windows on both machines and call it a day.

note: when I disabled then enabled SMB direct I got this
1671879767262.png
1671879779124.png
and on heavy copy
1671879855802.png
2 failed connection...
could be close...
or just taunting us
 
Last edited:

Martin Kuras

New Member
Feb 20, 2020
1
0
1
Berlin
www.youtube.com
Heho,

i found the culprit!
The DataCenterBridging-Feature on the Windows 10 Machine causes the glitch. Since i removed it, Client-RDMA works just fine.


1680550983900.png

Reboot & voila!
 
Last edited: