RDMA slow in one direction.

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

mb300sd

Active Member
Aug 1, 2016
204
80
28
34
I'm having a strange issue with SMB over RDMA. The file transfer is very slow on one direction only, and only when initiated by the client. Connection is between Windows 10 and Server 2016, 40GbE with Mellanox ConnectX-3 cards.

If I do the file copy from the Windows 10 computer, it's very slow copying from server to client, but fast from client to server. If I log into the server and do the transfer from there, it's fast in both directions.

I set up a ramdisk on both systems for testing, so it's not disk limited.

2016->W10, initiated from W10: 100-200MB/s
W10->2016, initiated from W10: >2GB/s
2016->W10, initiated from 2016: >2GB/s
W10->2016, initiated from 2016: >2GB/s

Anyone experience something like this before?
 

zxv

The more I C, the less I see.
Sep 10, 2017
156
57
28
One option to troubleshoot this kind of issue is to use the mellanox bandwidth
utilities to test increasing message sizes, to test whether its' a network issue.

A lack of flow control will show up as a drop in bandwidth with increasing message size.
 

red_vette

New Member
Jul 13, 2019
8
1
3
I've had a lot of issues with Windows 2019 and Windows 10 machines past the initial release build. I found that disabling Large Send Offload, Receive Side Scaling and Recv Segment Coalescing on the adapters resolved the issue. Might be worth digging into and seeing if that helps.