MNPA19-XTR Connectx-2 / Windows 10 - Slow LAN download speed

[Saints09]

I'm new to the Servethehome community and using 10gb network equipment in general, however, I'm experiencing some transfer speed issues using two MNPA19-XTR Connectx-2 10gb cards and I'm hoping someone can help.

System #1

Windows 10 Pro Build 1903
Intel i7-4790k
16gb Ram
Using a 3gb Ramdisk for testing

System #2
Windows Server 2016 Standard Build 1607
Xeon E5-2630 v2
16gb Ram
Using a 3gb Ramdisk for testing

Network Equipment:
Mikrotik CRS305-1G-4S+IN SFP+ 10gb Switch
Fiber connected Finisar FTLX8571D3BCL transceivers or Cisco SFP-H10GB-CU3M SFP+ DAC's
2x Mellanox MNPA19-XTR Connectx-2 10gb SFP+ network cards.

The specifics of the issue I'm experiencing is that downloading a 2.5gb test file shared on the Windows server 2016 Ramdisk to the Windows 10 Ramdisk only yields an average transfer speed of 45MB/s. The same file, uploaded from my Windows 10 Ramdisk back to the Windows server 2016 Ramdisk yields a transfer speed of 632MB/s.

I've tried various matching driver versions, MTU settings, Large Send offload on or off, Send / Receive buffer sizes, fiber or copper cabling, different switch ports but the download speed problem persists on Windows 10. I would like to share that I've tested the same Ramdisk file transfers using Windows 7 (network card driver version 4.80) and the 2016 server, the download / upload speed are exactly the same and as expected, an average of 630MB/s either way.

For the life of me, I can't figure out what's causing the downloads from my server to be so slow using Windows 10.

I appreciate any help you all can offer me.

Thanks for your time.

 

[nikey22]

I use the same cards and similar 2 computer setups and Fiber connected. I'm having similar issues with it peaking at 7Gb/s.
I did a few things below and got it to 9.5Gb/s, but it hasn't survived a few reboots, so I'm trying to troubleshoot that.

Mellanox Drivers. Get 5.50 drivers, March/2019. Don't use windows drivers.
Firmware up-to-date?
After installing the Mellanox Drivers, there should be a tab in the configure section of the adapter that says "Performance", choose either single stream or single port mode. Then press "Run Tuning", it will optimize some internal settings on the card and the windows registry. You only need to do this on ONE of the cards.

Try a peer-to-peer connection and bypass the switch for now for testing.
Run Iperf between them and see what you get. use a TCP window size of 84.
on the adapters:
I keep Flow Control on 'Rx/Tx Enabled'
I set my MTU to 9610 (yes it made a bit of a difference compared to 9014 or 9000)
buffer sizes to 4096

Make sure the PCIe slot is registering 5GT/s @x8. The link width should be x8 as well, just confirm that.

I'm starting to wonder if its a Windows network stack or SMB that is causing this slow down.

Try this using powershell:
Get-SmbClientConfiguration

There is an entry there called bandwidth throttling, I wonder if turning it off helps. I'm gonna try.
Set-SmbClientConfiguration -EnableBandwidthThrottling -0

 

[saivert]

Here is my anecdotal take:

This is highly dependent on your setup. I suffered this with my older Intel Core i7 2600k Asus P8Z68V motherboard PC but not with my newer 8700k setup.

Basically I noticed that when I had MPC-HC (a video player) up the network throughput was cut in half. It wasn't even playing video. It was in the stopped state. But just by having that application running it affected the network throughput. I did all this while testing speeds with iperf to a Linux server with the same network card.

Apparently MMCSS can wreak havoc on your network throughput.

In the Windows registry under:

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Multimedia\SystemProfile

NetworkThrottlingIndex=ffffffff
to Disable

YMMV as always.

Also make sure your network card is sufficiently cooled. It needs airflow over the passive heatsink. I had to mount an additional small fan blowing directly onto the card in my setup as there is still air in that area of the case otherwise.