Windows 10 40GbE Network

Discussion in 'Networking' started by zkrr01, Jul 5, 2018.

  1. zkrr01

    zkrr01 New Member

    Joined:
    Jun 28, 2018
    Messages:
    9
    Likes Received:
    0
    I finally got a 40GbE network connection between two Windows 10 8920 Dell Workstations where I have some Samsung 970 PRO SSD's. On initial tests I am able to transfer 500 GB's of data at 2120 MB/s using the standard Microsoft drag/drop interface. Since the SSD's spec's show 3500 MB/s read and 2700 MB/s write, the initial tests are a bit low since the overall transfer will be restricted by the write speed of 2700 MB/s.

    I am using a Mellanox MCX4131A-BCAT ConnectX-4 Lx EN Network Interface Card 40GbE Single-Port QSFP28 PCIe3.0 x8 ROHS R6 NIC in each Dell 8920 Workstation. I have jumbo packets enabled.

    Any recommendations on how to get the transfer speed from 2120 MB/s closer to the 2700 MB/s?
     
    #1
  2. cactus

    cactus Moderator

    Joined:
    Jan 25, 2011
    Messages:
    775
    Likes Received:
    52
    Map a drive and run something like Crystal Disk Mark(CDM) on the mapped drive. Also run CDM locally.
    I dont trust what Explorer is reporting and you have nothing to compare it to to say that you should be getting a full 2700MB/s.
     
    #2
  3. MiniKnight

    MiniKnight Well-Known Member

    Joined:
    Mar 30, 2012
    Messages:
    2,679
    Likes Received:
    745
    That's good if you're getting that without doing anything more.
     
    #3
  4. zkrr01

    zkrr01 New Member

    Joined:
    Jun 28, 2018
    Messages:
    9
    Likes Received:
    0
    The numbers I provided was from CrystalDiskMark 6.0.0 x64. The exact numbers were 3496.9 MB/s for read and 2725.8 MB/s for write.
     
    #4
  5. cesmith9999

    cesmith9999 Well-Known Member

    Joined:
    Mar 26, 2013
    Messages:
    947
    Likes Received:
    294
  6. i386

    i386 Well-Known Member

    Joined:
    Mar 18, 2016
    Messages:
    1,252
    Likes Received:
    283
    Explorer is limited by a single thread, try a tool that can copy multiple files simultaneously like robocopy. That should get you the max performance.
     
    #6
    MiniKnight likes this.
  7. cesmith9999

    cesmith9999 Well-Known Member

    Joined:
    Mar 26, 2013
    Messages:
    947
    Likes Received:
    294
    with robocopy there are a few tricks to make it copy faster
    1) /mt:xx - multi threaded copy - I usually maxed out to 2x the logical proc count...
    2) /ndl /nfl to suppress output to the console
    3) /log in place of /ndl /nfl logs output to a file.

    surprisingly, in tests I helped with a long time ago, #3 was the best option to speed up robocopy. as displaying the output was the biggest thing slowing robocopy down.

    Chris
     
    #7
    ecosse likes this.
  8. zkrr01

    zkrr01 New Member

    Joined:
    Jun 28, 2018
    Messages:
    9
    Likes Received:
    0
    I use the Windows 10 Hyper-V Manager to export each virtual machine to a directory on my primary machine and use the Microsoft drag/drop to copy this directory over to a backup machine. What is the recommended way to do this using robocopy? Examples would be useful.
     
    #8
  9. zkrr01

    zkrr01 New Member

    Joined:
    Jun 28, 2018
    Messages:
    9
    Likes Received:
    0
    I tried what you suggested and boy were you correct! I used the following:

    ROBOCOPY /MT /MIR /R:0 /W:0 /LOG:G:\files.log /NP /NDL \\Robin\g\HyperV-Exported-Systems \\Eagle\g\HyperV-Exported-Systems

    and is went a lot faster and is a much better way of doing it. Lots of experts on this forum! Thanks!
     
    #9
  10. zkrr01

    zkrr01 New Member

    Joined:
    Jun 28, 2018
    Messages:
    9
    Likes Received:
    0
    Upon additional tests I determined Robocopy performed best when /MT:2 was used. The average speed I observed was 2500MB/s , the maximum possible was 2700MB/s, which is the rated Samsung 970 pro write speed.
     
    #10
  11. oddball

    oddball Member

    Joined:
    May 18, 2018
    Messages:
    53
    Likes Received:
    9
    Have you been able to hit line rates on 40Gbe in Windows?

    We have a 40Gbe backbone and Windows has some issues. I've been able to hit mid-30s Gbe with iPerf3 between Windows and Linux, but from Windows to Windows we're in the mid-20s.

    Any help would be appreciated. Windows just seems to have a slower network stack.
     
    #11
  12. i386

    i386 Well-Known Member

    Joined:
    Mar 18, 2016
    Messages:
    1,252
    Likes Received:
    283
    Try NTttcp, it supports multithreading.
     
    #12
  13. oddball

    oddball Member

    Joined:
    May 18, 2018
    Messages:
    53
    Likes Received:
    9
    Wow, big improvement, 20Gb out of the box with default params. I need to mess with this, but a big improvement.
     
    #13
  14. zkrr01

    zkrr01 New Member

    Joined:
    Jun 28, 2018
    Messages:
    9
    Likes Received:
    0
    I also use NTttcp and with 4 threads, the throughput I was getting was 3456.496MB/s
    That was Windows 10 to Windows 10 with Dell 8920 workstations
     
    #14
  15. oddball

    oddball Member

    Joined:
    May 18, 2018
    Messages:
    53
    Likes Received:
    9
    Do you have a switch in between the computers?

    I'm doing Windows Server 2016 to Windows Server 2016. Both servers have a network team of 2x40Gbps through a mlag of Arista 7050qx-32s switches.

    The lag on Windows is switch independent, which I think is an issue. We're working to convert to LACP.
     
    #15
  16. zkrr01

    zkrr01 New Member

    Joined:
    Jun 28, 2018
    Messages:
    9
    Likes Received:
    0
    No, we use a direct connect using a QSFP passive copper cable.
     
    #16
  17. zkrr01

    zkrr01 New Member

    Joined:
    Jun 28, 2018
    Messages:
    9
    Likes Received:
    0
    I changed the Jumbo Packet size to 9614 and I am now able to achieve 2700MB/s writing to the Samsung Pro SSD.
    I was using the following robocopy setup:
    ROBOCOPY /MT:2 /MIR /R:0 /W:0 /LOG:G:\files.log /NP /NDL
     
    #17
Similar Threads: Windows 40GbE
Forum Title Date
Networking Windows 10 with Mallenox MNPA19-XTR May 29, 2018
Networking Chelsio S320 windows blue screen! Mar 4, 2018
Networking Infiniband HBA ports show as "cable unplugged" in Windows 2016? Jan 3, 2018
Networking Intel XL720 - allow unsupported qsfp-modules on Windows Nov 22, 2017
Networking Finisar FTLX8571D3BCL sfp+ & x520-da2 in Windows 10 Nov 6, 2017

Share This Page