Mellanox WinOF powershell cmdlets not working in Windows 10 (ConnectX-3 SMB Direct/RDMA)

Discussion in 'Windows Server, Hyper-V Virtualization' started by marv, Nov 25, 2017.

  1. William

    William Active Member

    Joined:
    May 7, 2015
    Messages:
    747
    Likes Received:
    228
    First command errors. I will have to check the correct command spelling.
    Get-SmbMultichannelConnection shows nothing.

    Ok so from what I can tell 10G cards with RDMA enabled will only work on Windows 10 Pro Workstation if it is connecting to a Windows Server ?

    Is that true ?

    Than what are people doing if they connect to something else, like a Synology or QNAP box ?
    I cannot believe that you cannot get 10G any other way besides a direct connect ?
     
    #21
  2. i386

    i386 Well-Known Member

    Joined:
    Mar 18, 2016
    Messages:
    1,659
    Likes Received:
    400
    Nope, I have server with server 2019 with shares and different windows 10 clients (enterprise, pro & pro for workstations) at work and home.
    None of the windows 10 clients can use rdma features.
    The power shell scripts all show that the nics are rdma capable, but the "Get-SmbMultichannelConnection" command shows nothing.

    Server 2019 <-> server 2019 (and 2012 r2, 2016) works with rdma.
     
    #22
  3. necr

    necr New Member

    Joined:
    Dec 27, 2017
    Messages:
    20
    Likes Received:
    11
    Please try to access shared folder (on your Windows Server) from W10 machine, that should initiate it.

    Yes, Windows 10 Pro Workstation can only access Windows Server share (upload and download data really fast without utilizing CPU). Otherwise, it's Windows Server to Windows Server.

    Of course connecting through switch should work. I'd start with direct connect first and then just plug cables to switch.
     
    #23
  4. William

    William Active Member

    Joined:
    May 7, 2015
    Messages:
    747
    Likes Received:
    228
    As I have explained, I am trying to connect to a Synology or QNAP box, I do not have a windows server running atm.
    Windows 10 Pro Workstation 1809 <-> Synology/QNAP
    I will try a direct connect in a bit.
     
    #24
  5. necr

    necr New Member

    Joined:
    Dec 27, 2017
    Messages:
    20
    Likes Received:
    11
    That was new to me. Your previous post mentions 2012R2.
    Upd: Ah, see it
    No need to try direct connect then. You will first need somehow to configure SMB3+ on that. Otherwise, it's going to be just a 10G network share.
     
    #25
    Last edited: Mar 29, 2019
  6. William

    William Active Member

    Joined:
    May 7, 2015
    Messages:
    747
    Likes Received:
    228
    Thanks necr, sorry for the confusion.
     
    #26
  7. William

    William Active Member

    Joined:
    May 7, 2015
    Messages:
    747
    Likes Received:
    228
    Just tried direct connect to a Synology 1618+ (has ConnectX-3 Pro inside). Still low speed :(
     
    #27
  8. necr

    necr New Member

    Joined:
    Dec 27, 2017
    Messages:
    20
    Likes Received:
    11
    To sum it up: you did everything right on W10 side, but similar stuff has to be done on NAS side. You'll need to get your hands dirty in samba configuration to get SMB3+ and Multichannel working - that's target #1. Getting RDMA to work on that Synology box is a separate task but it also involves getting to shell and checking all mlx4 related stuff - that doesn't guarantee problem-free Windows - Linux (Synology) operation (never supported in the first place!). Last, 2G isn't that bad - average disk performance is 250MB/s which is exactly 2000Mbit/s (250*8), and if you don't have some kind of mirroring or you have slow drives, or you have just 1 drive there - that's the speed you're going to get.

    Synology DS1618+ Performance Tweaking -- Enabling SMB Multichannel
     
    #28
  9. William

    William Active Member

    Joined:
    May 7, 2015
    Messages:
    747
    Likes Received:
    228
    Thanks necr !
    I will test this out.
     
    #29
  10. Frank173

    Frank173 Member

    Joined:
    Feb 14, 2018
    Messages:
    75
    Likes Received:
    9
    marv,

    did you ever get the Mellanox cmdlets to work in Windows 10 Pro for Workstation? I am facing similar issues right now and nowhere does it mention that roce or rdma only works by sharing from Windows Server, it should work equally well in Windows 10 Pro for Workstation, but I cannot even configure roce because the cmdlets are not recognized. This is what I currently get on my Windows 10 Pro for Workstation machines, so clearly rdma is working but I can't configure further using the MLNX cmdlets.

    Capture.PNG
    Capture10.JPG

    but when copying files between the two machines it shows that "client is not RDMA capable"

    Capture5.JPG

     
    #30
    Last edited: Apr 23, 2019
  11. i386

    i386 Well-Known Member

    Joined:
    Mar 18, 2016
    Messages:
    1,659
    Likes Received:
    400
    Try this in powershell with admin rights:
    Code:
    Set-ExecutionPolicy -ExecutionPolicy AllSigned -Scope LocalMachine
    and then try to import the mellanox modules
     
    #31
    kiteboarder and Frank173 like this.
  12. Frank173

    Frank173 Member

    Joined:
    Feb 14, 2018
    Messages:
    75
    Likes Received:
    9
    that did the trick, thanks a lot!

     
    #32
  13. tjk

    tjk Active Member

    Joined:
    Mar 3, 2013
    Messages:
    236
    Likes Received:
    32
    I think MLNX pulled the powershell cmdlets in the latest WinOF, at least I don't have them/can't find them when I install the latest WinOF.

    Anyone else?
     
    #33
  14. Frank173

    Frank173 Member

    Joined:
    Feb 14, 2018
    Messages:
    75
    Likes Received:
    9
    They are not part of WinOF2 but are part of WinOF. I just used them today they are definitely part of WinOF

     
    #34
  15. tjk

    tjk Active Member

    Joined:
    Mar 3, 2013
    Messages:
    236
    Likes Received:
    32
    #35
  16. Frank173

    Frank173 Member

    Joined:
    Feb 14, 2018
    Messages:
    75
    Likes Received:
    9
    #36
  17. Frank173

    Frank173 Member

    Joined:
    Feb 14, 2018
    Messages:
    75
    Likes Received:
    9
    I gave up on smb direct, rdma, and roce. I did not witness any performance increases for file servers. Windows file manager operates single threaded and I max out bandwidth at 1GB/sec no matter what. I have file access to a striped array at over 12 GB/sec but can't transfer that performance over to clients who have access over a 100 Gb/sec network. Mellanox performance tools show a 95 Gb/sec transfer speed and I can load data off the array at around 7 GB/s single threaded but as soon as the windows file manager is involved performance drops off to 1 GB/s levels. Rmda might be useful for server clusters and latency sensitive applications but I find no benefit regarding serving data off a file server.

    I am happy to worship and call anyone a hero who can explain a way to force Window's file manager to use multiple threads by default for file copies. Not robocopy or other tools but the default file manager that Windows 10 and Windows Server comes with.
     
    #37
    William likes this.
  18. i386

    i386 Well-Known Member

    Joined:
    Mar 18, 2016
    Messages:
    1,659
    Likes Received:
    400
    Explorer is the wrong tool.

    Use something that supports multithreading, like robocopy.
     
    #38
  19. Frank173

    Frank173 Member

    Joined:
    Feb 14, 2018
    Messages:
    75
    Likes Received:
    9
    Even robocopy does not split up a single file copy into segments that can be transferred concurrently. And anyway, it remains a total mystery why Windows limits single threaded file transfers to around 1GB/sec when the network can handle a multiple, when the file storage can handle a multiple and when the client can handle a multiple of that. I have in many years not come across a single source that provided a reasonable explanation for this limitation other than poor implementation. The kernel is the only bottleneck it seems.

     
    #39
    William likes this.
  20. cesmith9999

    cesmith9999 Well-Known Member

    Joined:
    Mar 26, 2013
    Messages:
    1,095
    Likes Received:
    333
    #40

Share This Page