Windows 10 Pro for Workstations - RDMA

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.
Z

zkrr01

Member
Jun 28, 2018
106
6
18
I am trying to get RDMA working, however no success. I got the Diskspd version for RDMA working and followed the instructions from several online reports on enabling RDMA such as:

My little Windows Garden

Everything on my two workstations is setup as the above article states. But the RDMA counters are zero and I can see the Diskspd network activity on the task manager performance ethernet graph. If RDMA was working, the task manager should show little or no network activity on the 40GbE NIC's.

Has anyone been able to get RDMA working with Windows 10 Pro for Workstations?
 
cesmith9999

cesmith9999

Well-Known Member
Mar 26, 2013
1,420
470
83
are you using Mellanox (RoCEv2)? or iWARP adapters?

are you direct connect or are you using a Switch with DCB and PFC?



Chris
 
Z

zkrr01

Member
Jun 28, 2018
106
6
18
cesmith9999 said:
are you using Mellanox (RoCEv2)? or iWARP adapters?

are you direct connect or are you using a Switch with DCB and PFC?
I am direct connect and using a Mellanox 40GbE NIC



Chris
Click to expand...
I checked my list with your list and it appears I have done all of them. However in the performance monitor, "RDMA Failed Connection Attempts" has a value of 2.000 and all the other counters are 0. So there is something I have not done correctly, but what?
 
cesmith9999

cesmith9999

Well-Known Member
Mar 26, 2013
1,420
470
83
You still did not answer my questions.

What is the switch are you using? If you are using mellanox, you also have switch configurations to do.

Chris
 
Z

zkrr01

Member
Jun 28, 2018
106
6
18
I am using Mellanox NIC's and I am not using a switch, the two workstations are direct connected.
 
Z

zkrr01

Member
Jun 28, 2018
106
6
18
I think I see the problem but I don't know to make the below share "Client RDMA Capable"?
For that matter, how do you make any share ""Client RDMA Capable"? I never heard that term used for Windows shares.

PS C:\work> Get-SmbMultichannelConnection

Server Name Selected Client IP Server IP Client RSS Capable Client RDMA Capable

----------- -------- --------- --------- ---------------------- -------------------
Eagle True 10.0.1.2 10.0.1.1 True False
 
cesmith9999

cesmith9999

Well-Known Member
Mar 26, 2013
1,420
470
83
A share is SMB. There is no distinction as if it is RDMA or not. That is handled at a lower level in the stack.

Really you should look at getting a switch and tagging the vlan so you have PFC.

I have never tried to do RDMA in a direct connection configuration. MS certainly does not support that. I know, I talked to the dev's one night about it.


Chris
 
M

marv

Active Member
Apr 2, 2015
155
34
28
38
Vlan tagging works with or without switch when its done on PC... and RDMA works with direct connection. Just if you have switch, you need one with DCB and PFC support.

But @zkrr01, are you connecting W10 -> WS2016? With W10 -> W10 or WS2016 -> W10, RDMA will not work because W10 will not listen.

check it with

Code: 
C:\Windows\System32>netstat -xan

Active NetworkDirect Connections, Listeners, SharedEndpoints

  Mode   IfIndex Type           Local Address          Foreign Address        PID

  Kernel      21 Listener       [fe80::c022:5c42:4256:caea%21]:445  NA           0
  Kernel      21 Listener       10.9.0.9:445        NA                     0
output will be empty on W10.
 
  • Like
Reactions: zkrr01
Z

zkrr01

Member
Jun 28, 2018
106
6
18
@marv do you know why W10 is not listening? Is there any articles on this? Is it only Windows 10 Pro for Workstations since its so new?
 
Last edited:
cesmith9999

cesmith9999

Well-Known Member
Mar 26, 2013
1,420
470
83
Only W10 Pro for workstations and Server has RDMA enabled. if you are doing W10 Pro <-> W10 Pro 4W then it will not work.

Chris
 
Z

zkrr01

Member
Jun 28, 2018
106
6
18
I have W10 Pro for Workstations <-> W10 Pro for Workstations and it does NOT work. When W10 for Workstations was released, none of the articles or Microsoft said anything about this. The only clue I had was that netstat -xln did not show any ports listening.
 
P

PigLover

Moderator
Jan 26, 2011
3,185
1,545
113
Microsoft only supports the server side SMB RDMA connection on Windows Server SKUs. It is disabled on all desktop and workstation licenses. Very annoying.
 
M

marv

Active Member
Apr 2, 2015
155
34
28
38
zkrr01 said:
@marv do you know why W10 is not listening? Is there any articles on this? Is it only Windows 10 Pro for Workstations since its so new?
Click to expand...
I believe its not a bug, but business decision. Win10 pro for workstations normally listens on port 445, accessing shares works fine. Its just RDMA server side thats not working on Windows 10 Pro for Workstations as PigLover says.

If you want to use RDMA, share has to always reside on Windows Server. W10 Pro for Workstations <-> W10 Pro for Workstations will not utilize RDMA. At least not in current versions (1709,1803). And I am not aware of any mod that would make it work.
 
K

kapone

Well-Known Member
May 23, 2015
1,095
642
113
marv said:
I believe its not a bug, but business decision. just a money grab, since they can sell server licenses for more money. Win10 pro for workstations normally listens on port 445, accessing shares works fine. Its just RDMA server side thats not working on Windows 10 Pro for Workstations as PigLover says.

If you want to use RDMA, share has to always reside on Windows Server. W10 Pro for Workstations <-> W10 Pro for Workstations will not utilize RDMA. At least not in current versions (1709,1803). And I am not aware of any mod that would make it work.
Click to expand...
There, I fixed it for you. :)
 
Z

zkrr01

Member
Jun 28, 2018
106
6
18
I went back and read the following article from Microsoft that I thought said otherwise, but now I see the one word that I missed.

https://www.microsoft.com/en-us/mic...10-pro-workstations-power-advanced-workloads/

"You’ll also save valuable time on file transfers between Server and PC with SMB Direct2, a Microsoft solution for network adapters with Remote Direct Memory Access (RDMA) capability. This technology uses RDMA capable network adapters to utilize the full throughput of high-speed networks, transferring data at line speeds. It also provides very low-latency network requests, making remote storage perform like local storage. Finally, it significantly reduces CPU utilization, so you have more computing power available to run other applications such as Autodesk AutoCAD or Adobe Creative Suite."
 
Z

zkrr01

Member
Jun 28, 2018
106
6
18
In one of my Windows 10 Pro for Workstations, I have Hyper-V running. If I create a new virtual machine with Fedora 28 and attach a virtual NIC with the Mellanox ConnectX-4 Ethernet Adapter, then could I test RDMA with that? That would be W10 <-> Fedora with Fedora as the server and W10 the client. Has anyone tried something like this?
 
Last edited:
You must log in or register to reply here.
Top Bottom