Windows 10 Pro for Workstations - RDMA

Discussion in 'Networking' started by zkrr01, Aug 7, 2018.

  1. zkrr01

    zkrr01 Member

    Joined:
    Jun 28, 2018
    Messages:
    34
    Likes Received:
    2
    I am trying to get RDMA working, however no success. I got the Diskspd version for RDMA working and followed the instructions from several online reports on enabling RDMA such as:

    My little Windows Garden

    Everything on my two workstations is setup as the above article states. But the RDMA counters are zero and I can see the Diskspd network activity on the task manager performance ethernet graph. If RDMA was working, the task manager should show little or no network activity on the 40GbE NIC's.

    Has anyone been able to get RDMA working with Windows 10 Pro for Workstations?
     
    #1
  2. cesmith9999

    cesmith9999 Well-Known Member

    Joined:
    Mar 26, 2013
    Messages:
    972
    Likes Received:
    298
    are you using Mellanox (RoCEv2)? or iWARP adapters?

    are you direct connect or are you using a Switch with DCB and PFC?



    Chris
     
    #2
  3. marv

    marv Member

    Joined:
    Apr 2, 2015
    Messages:
    85
    Likes Received:
    19
    #3
  4. zkrr01

    zkrr01 Member

    Joined:
    Jun 28, 2018
    Messages:
    34
    Likes Received:
    2
    I checked my list with your list and it appears I have done all of them. However in the performance monitor, "RDMA Failed Connection Attempts" has a value of 2.000 and all the other counters are 0. So there is something I have not done correctly, but what?
     
    #4
  5. cesmith9999

    cesmith9999 Well-Known Member

    Joined:
    Mar 26, 2013
    Messages:
    972
    Likes Received:
    298
    You still did not answer my questions.

    What is the switch are you using? If you are using mellanox, you also have switch configurations to do.

    Chris
     
    #5
  6. zkrr01

    zkrr01 Member

    Joined:
    Jun 28, 2018
    Messages:
    34
    Likes Received:
    2
    I am using Mellanox NIC's and I am not using a switch, the two workstations are direct connected.
     
    #6
  7. zkrr01

    zkrr01 Member

    Joined:
    Jun 28, 2018
    Messages:
    34
    Likes Received:
    2
    I think I see the problem but I don't know to make the below share "Client RDMA Capable"?
    For that matter, how do you make any share ""Client RDMA Capable"? I never heard that term used for Windows shares.

    PS C:\work> Get-SmbMultichannelConnection

    Server Name Selected Client IP Server IP Client RSS Capable Client RDMA Capable

    ----------- -------- --------- --------- ---------------------- -------------------
    Eagle True 10.0.1.2 10.0.1.1 True False
     
    #7
  8. cesmith9999

    cesmith9999 Well-Known Member

    Joined:
    Mar 26, 2013
    Messages:
    972
    Likes Received:
    298
    A share is SMB. There is no distinction as if it is RDMA or not. That is handled at a lower level in the stack.

    Really you should look at getting a switch and tagging the vlan so you have PFC.

    I have never tried to do RDMA in a direct connection configuration. MS certainly does not support that. I know, I talked to the dev's one night about it.


    Chris
     
    #8
  9. marv

    marv Member

    Joined:
    Apr 2, 2015
    Messages:
    85
    Likes Received:
    19
    Vlan tagging works with or without switch when its done on PC... and RDMA works with direct connection. Just if you have switch, you need one with DCB and PFC support.

    But @zkrr01, are you connecting W10 -> WS2016? With W10 -> W10 or WS2016 -> W10, RDMA will not work because W10 will not listen.

    check it with

    Code:
    C:\Windows\System32>netstat -xan
    
    Active NetworkDirect Connections, Listeners, SharedEndpoints
    
      Mode   IfIndex Type           Local Address          Foreign Address        PID
    
      Kernel      21 Listener       [fe80::c022:5c42:4256:caea%21]:445  NA           0
      Kernel      21 Listener       10.9.0.9:445        NA                     0
    
    output will be empty on W10.
     
    #9
    zkrr01 likes this.
  10. zkrr01

    zkrr01 Member

    Joined:
    Jun 28, 2018
    Messages:
    34
    Likes Received:
    2
    #10
  11. zkrr01

    zkrr01 Member

    Joined:
    Jun 28, 2018
    Messages:
    34
    Likes Received:
    2
    @marv do you know why W10 is not listening? Is there any articles on this? Is it only Windows 10 Pro for Workstations since its so new?
     
    #11
    Last edited: Aug 10, 2018
  12. cesmith9999

    cesmith9999 Well-Known Member

    Joined:
    Mar 26, 2013
    Messages:
    972
    Likes Received:
    298
    Only W10 Pro for workstations and Server has RDMA enabled. if you are doing W10 Pro <-> W10 Pro 4W then it will not work.

    Chris
     
    #12
  13. matthelm

    matthelm New Member

    Joined:
    Sep 19, 2013
    Messages:
    4
    Likes Received:
    0
    Will W10 Pro for Workstations <-> W10 Pro for Workstations work? (This was my plan for my setup.)
     
    #13
  14. zkrr01

    zkrr01 Member

    Joined:
    Jun 28, 2018
    Messages:
    34
    Likes Received:
    2
    I have W10 Pro for Workstations <-> W10 Pro for Workstations and it does NOT work. When W10 for Workstations was released, none of the articles or Microsoft said anything about this. The only clue I had was that netstat -xln did not show any ports listening.
     
    #14
  15. PigLover

    PigLover Moderator

    Joined:
    Jan 26, 2011
    Messages:
    2,655
    Likes Received:
    1,040
    Microsoft only supports the server side SMB RDMA connection on Windows Server SKUs. It is disabled on all desktop and workstation licenses. Very annoying.
     
    #15
  16. marv

    marv Member

    Joined:
    Apr 2, 2015
    Messages:
    85
    Likes Received:
    19
    I believe its not a bug, but business decision. Win10 pro for workstations normally listens on port 445, accessing shares works fine. Its just RDMA server side thats not working on Windows 10 Pro for Workstations as PigLover says.

    If you want to use RDMA, share has to always reside on Windows Server. W10 Pro for Workstations <-> W10 Pro for Workstations will not utilize RDMA. At least not in current versions (1709,1803). And I am not aware of any mod that would make it work.
     
    #16
  17. kapone

    kapone Active Member

    Joined:
    May 23, 2015
    Messages:
    251
    Likes Received:
    64
    There, I fixed it for you. :)
     
    #17
  18. zkrr01

    zkrr01 Member

    Joined:
    Jun 28, 2018
    Messages:
    34
    Likes Received:
    2
    I went back and read the following article from Microsoft that I thought said otherwise, but now I see the one word that I missed.

    https://www.microsoft.com/en-us/mic...10-pro-workstations-power-advanced-workloads/

    "You’ll also save valuable time on file transfers between Server and PC with SMB Direct2, a Microsoft solution for network adapters with Remote Direct Memory Access (RDMA) capability. This technology uses RDMA capable network adapters to utilize the full throughput of high-speed networks, transferring data at line speeds. It also provides very low-latency network requests, making remote storage perform like local storage. Finally, it significantly reduces CPU utilization, so you have more computing power available to run other applications such as Autodesk AutoCAD or Adobe Creative Suite."
     
    #18
  19. zkrr01

    zkrr01 Member

    Joined:
    Jun 28, 2018
    Messages:
    34
    Likes Received:
    2
    In one of my Windows 10 Pro for Workstations, I have Hyper-V running. If I create a new virtual machine with Fedora 28 and attach a virtual NIC with the Mellanox ConnectX-4 Ethernet Adapter, then could I test RDMA with that? That would be W10 <-> Fedora with Fedora as the server and W10 the client. Has anyone tried something like this?
     
    #19
    Last edited: Aug 11, 2018
  20. zkrr01

    zkrr01 Member

    Joined:
    Jun 28, 2018
    Messages:
    34
    Likes Received:
    2
    #20
Similar Threads: Windows Workstations
Forum Title Date
Networking 40gbe, 4x10gbe, Freenas, Windows 10, server 2016, and Switch Jul 26, 2018
Networking Intel X710-DA2 + windows 10 Jul 24, 2018
Networking Windows 10 40GbE Network Jul 5, 2018
Networking Windows 10 with Mallenox MNPA19-XTR May 29, 2018
Networking Chelsio S320 windows blue screen! Mar 4, 2018

Share This Page