RDMA slow in one direction.

Discussion in 'Windows Server, Hyper-V Virtualization' started by mb300sd, Jun 18, 2019.

  1. mb300sd

    mb300sd Active Member

    Joined:
    Aug 1, 2016
    Messages:
    100
    Likes Received:
    36
    I'm having a strange issue with SMB over RDMA. The file transfer is very slow on one direction only, and only when initiated by the client. Connection is between Windows 10 and Server 2016, 40GbE with Mellanox ConnectX-3 cards.

    If I do the file copy from the Windows 10 computer, it's very slow copying from server to client, but fast from client to server. If I log into the server and do the transfer from there, it's fast in both directions.

    I set up a ramdisk on both systems for testing, so it's not disk limited.

    2016->W10, initiated from W10: 100-200MB/s
    W10->2016, initiated from W10: >2GB/s
    2016->W10, initiated from 2016: >2GB/s
    W10->2016, initiated from 2016: >2GB/s

    Anyone experience something like this before?
     
    #1
  2. zxv

    zxv The more I C, the less I see.

    Joined:
    Sep 10, 2017
    Messages:
    152
    Likes Received:
    46
    One option to troubleshoot this kind of issue is to use the mellanox bandwidth
    utilities to test increasing message sizes, to test whether its' a network issue.

    A lack of flow control will show up as a drop in bandwidth with increasing message size.
     
    #2
  3. red_vette

    red_vette New Member

    Joined:
    Jul 13, 2019
    Messages:
    3
    Likes Received:
    1
    I've had a lot of issues with Windows 2019 and Windows 10 machines past the initial release build. I found that disabling Large Send Offload, Receive Side Scaling and Recv Segment Coalescing on the adapters resolved the issue. Might be worth digging into and seeing if that helps.
     
    #3
Similar Threads: RDMA slow
Forum Title Date
Windows Server, Hyper-V Virtualization Mellanox WinOF powershell cmdlets not working in Windows 10 (ConnectX-3 SMB Direct/RDMA) Nov 25, 2017
Windows Server, Hyper-V Virtualization WS2012r2, Hyper-V, SMB 3.0 and RDMA Sep 25, 2015
Windows Server, Hyper-V Virtualization SMBdirect (RDMA) and SMBmultichannel Oct 13, 2013
Windows Server, Hyper-V Virtualization Slow diskperfomance in virtual guest Feb 22, 2019
Windows Server, Hyper-V Virtualization WS 2016 Slow VPN May 23, 2018

Share This Page