MNPA19-XTR Connectx-2 / Windows 10 - Slow LAN download speed

Discussion in 'Networking' started by Saints09, Sep 10, 2019.

  1. Saints09

    Saints09 New Member

    Joined:
    Sep 10, 2019
    Messages:
    1
    Likes Received:
    0
    I'm new to the Servethehome community and using 10gb network equipment in general, however, I'm experiencing some transfer speed issues using two MNPA19-XTR Connectx-2 10gb cards and I'm hoping someone can help.

    System #1

    Windows 10 Pro Build 1903
    Intel i7-4790k
    16gb Ram
    Using a 3gb Ramdisk for testing

    System #2
    Windows Server 2016 Standard Build 1607
    Xeon E5-2630 v2
    16gb Ram
    Using a 3gb Ramdisk for testing

    Network Equipment:
    Mikrotik CRS305-1G-4S+IN SFP+ 10gb Switch
    Fiber connected Finisar FTLX8571D3BCL transceivers or Cisco SFP-H10GB-CU3M SFP+ DAC's
    2x Mellanox MNPA19-XTR Connectx-2 10gb SFP+ network cards.

    The specifics of the issue I'm experiencing is that downloading a 2.5gb test file shared on the Windows server 2016 Ramdisk to the Windows 10 Ramdisk only yields an average transfer speed of 45MB/s. The same file, uploaded from my Windows 10 Ramdisk back to the Windows server 2016 Ramdisk yields a transfer speed of 632MB/s.

    I've tried various matching driver versions, MTU settings, Large Send offload on or off, Send / Receive buffer sizes, fiber or copper cabling, different switch ports but the download speed problem persists on Windows 10. I would like to share that I've tested the same Ramdisk file transfers using Windows 7 (network card driver version 4.80) and the 2016 server, the download / upload speed are exactly the same and as expected, an average of 630MB/s either way.

    For the life of me, I can't figure out what's causing the downloads from my server to be so slow using Windows 10.

    I appreciate any help you all can offer me.

    Thanks for your time.
     
    #1
  2. nikey22

    nikey22 New Member

    Joined:
    Feb 19, 2018
    Messages:
    14
    Likes Received:
    0
    I use the same cards and similar 2 computer setups and Fiber connected. I'm having similar issues with it peaking at 7Gb/s.
    I did a few things below and got it to 9.5Gb/s, but it hasn't survived a few reboots, so I'm trying to troubleshoot that.

    Mellanox Drivers. Get 5.50 drivers, March/2019. Don't use windows drivers.
    Firmware up-to-date?
    After installing the Mellanox Drivers, there should be a tab in the configure section of the adapter that says "Performance", choose either single stream or single port mode. Then press "Run Tuning", it will optimize some internal settings on the card and the windows registry. You only need to do this on ONE of the cards.

    Try a peer-to-peer connection and bypass the switch for now for testing.
    Run Iperf between them and see what you get. use a TCP window size of 84.
    on the adapters:
    I keep Flow Control on 'Rx/Tx Enabled'
    I set my MTU to 9610 (yes it made a bit of a difference compared to 9014 or 9000)
    buffer sizes to 4096

    Make sure the PCIe slot is registering 5GT/s @x8. The link width should be x8 as well, just confirm that.

    I'm starting to wonder if its a Windows network stack or SMB that is causing this slow down.

    Try this using powershell:
    Get-SmbClientConfiguration

    There is an entry there called bandwidth throttling, I wonder if turning it off helps. I'm gonna try.
    Set-SmbClientConfiguration -EnableBandwidthThrottling -0
     
    #2
    Last edited: Sep 10, 2019
  3. saivert

    saivert Member

    Joined:
    Nov 2, 2015
    Messages:
    118
    Likes Received:
    12
    Here is my anecdotal take:

    This is highly dependent on your setup. I suffered this with my older Intel Core i7 2600k Asus P8Z68V motherboard PC but not with my newer 8700k setup.

    Basically I noticed that when I had MPC-HC (a video player) up the network throughput was cut in half. It wasn't even playing video. It was in the stopped state. But just by having that application running it affected the network throughput. I did all this while testing speeds with iperf to a Linux server with the same network card.

    Apparently MMCSS can wreak havoc on your network throughput.

    In the Windows registry under:

    HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Multimedia\SystemProfile

    NetworkThrottlingIndex=ffffffff
    to Disable

    YMMV as always.

    Also make sure your network card is sufficiently cooled. It needs airflow over the passive heatsink. I had to mount an additional small fan blowing directly onto the card in my setup as there is still air in that area of the case otherwise.
     
    #3
Similar Threads: MNPA19-XTR Connectx-2
Forum Title Date
Networking How do I flash the MAC address on a Mellanox MNPA19-XTR (connectX-2) ? Nov 20, 2015
Networking Question about Mellanox ConnectX-2 MNPA19-XTR Drivers Sep 28, 2014
Networking Windows 10 with Mallenox MNPA19-XTR May 29, 2018
Networking Dell MNPA19-XTR and LB4M Nov 27, 2016
Networking MNPA19-XTR Windows 10 speed/iperf issues Sep 10, 2016

Share This Page