MCX3131A-BCBT Getting only ~6Gb/s

[RaymondLC92]

So I'm bamboozled here, I've been debugging this stuff for several hours now and still don't know if I'm doing something outrageously wrong or if there's something wrong with my cards/setup.

I have two PCs, an Ubuntu linux machine with:
CPU: AMD Ryzen 3400G
RAM: 16GB DDR4 2933Mhz
DISK: 1TB NVMe Samsung 970 Evo

And a Windows 10 Enterprise machine with:
CPU: AMD Ryzen Threadripper 1950x
RAM: 16GB DDR4 2933MHz
Disk: 2TB NVMe Samsung 970 Evo
Network Share: 1TB NVMe Samsung 970 Evo

The cards are attached directly to each other via a MC2207310-010 cable, there is no switch/router in the current setup. In Ubuntu the port is shown as type "eth" when using the Mellanox config tool. Both NICs were assigned an IP manually.

I've also updated the firmware of both cards to their current version. Both cards are in a PCI-E 3.0 x8 slot in their respective motherboards (technically the linux machine is PCI-E 4.0 but it's limited by the CPU which is Zen+ rather than Zen 2),

I've been attempting to debug the performance of these cards for quite some time, the cards themselves are from eBay so knowing that they can actually perform as claimed is kind of a necessity.

During the following tests I completely detached the outgoing network connections for these two machines, they shared only the 40Gbe connection.

IPERF v2:
I've tried iperf v2 with -P set to variations of 1,2,4 and 8 and -w values of 1M,2M,4M, and 8M.

I only seem to get 40Gb/s speeds when the window size (-w) aligns with (-n) and with multiple threads (-P)

It doesn't seem to make any difference if the client is the linux machine or if it is reversed.

I tried with various MTU values (1500, 9000, 9014) and have also used the mlnx_tune program ("HIGH_THROUGHPUT" profile) as well as tuning the card in the device properties on windows for Single Port).

I followed all of the instructions in the "Performance Tuning Guidelines for Mellanox Network Adapters" for both the linux and Windows machines. No virtual machines or virtualization is in use, both are baremetal.

ntttcp + ntttcp-for-linux:
When using ntttcp on Windows and ntttcp-for-linux on Linux, I used 8 cores on both the receiver and sender, otherwise I left the options as they were in the performance guide under "Performance Testing" I did have to add the -ns flag to windows and the -N flag to the linux command in order to avoid an interop issue.

Rate: ~6Gb/s

I'm seeing 5-7 errors and 20-22 retransmits when performing this test, CPU usage is between 3-5% on the Windows size and between 12-15% on the Ubuntu size. The test sent 11GB of data and ran for 15 seconds, this seems to be about normal for this test.

dd file copy benchmark to windows NVMe network share:
The dd file copy test which used the following command:

dd if=/dev/zero of=~/nvme_share/test_file bs=10G count=1 oflag=dsync

I also tried with direct rather than dsync and I tried specifying the SMB version to 3.0 in /etc/fstab/ - neither had any impact.

Rate: ~100MB/s

File copy from ramdisk to ramdisk:
I created a Linux RAM Disk and a Windows RAM Disk (for windows I used AMDs utility, for linux I used what was baked into the OS), and then I copied a 4GB file between the two.

Rate was ~1.35GB/s which is about 10.8Gb/s.

Screenshots of the adapters in their respective operating systems:



I've been reading posts on this forum for the last couple of weeks leading up to the delivery, and it's been a huge help, but now I'm at the end of my rope as I have no idea what my next step should be.

Any help would be hugely appreciated!

 

[Rand__]

Is the performance ok if you do local iperf?
i.e. client and server on same host, either on an individual port or on one if you got a single port card.

Are both cards similar in their performance? If not does the performance stay the same if you switch the cards around?

Edit - ah i see its a single port card
Mellanox MCX313A-BCCT ConnectX-3 Pro EN

You sure the cable is good?

 

[RaymondLC92]

Here's a local iperf, it's not nearly as high as I expected it to be but I don't see what this really shows?


Client:

iperf -c 127.0.0.1 -P 16 -w 1M -n 1024M

Server:

iperf -s -w 1M

How would I be able to tell if the cards are similar in performance if I only have the two? Everything else I have is much slower than 40Gb/s.

Edit: I have more than 1 cable, and both cables yield the same results.

 

[Rand__]

well the goal of this test was to
a.) identify whether one of the cards has a HW issue by running both cards in both boxes comparing performance to see whether one is significantly worse than the other
b.) see what speeds the card can do in a best case situation (no network)

If you only get 8Gbit locally why would you expect to get 40 GBit over another cable? so that clearly shows there is an issue on that box that you need to solve - be it card/hw, card/sw (firmware), OS/driver or configuration...

 

[RaymondLC92]

well the goal of this test was to
a.) identify whether one of the cards has a HW issue by running both cards in both boxes comparing performance to see whether one is significantly worse than the other
b.) see what speeds the card can do in a best case situation (no network)

If you only get 8Gbit locally why would you expect to get 40 GBit over another cable? so that clearly shows there is an issue on that box that you need to solve - be it card/hw, card/sw (firmware), OS/driver or configuration...

That's a very good point, I suppose I hadn't thought about it, performing the same test on the Ubuntu machine yields a rate of about 100Gb/s, so there's definitely something wrong here with my Windows machine.

I can blow away the OS pretty easily, or boot into a fresh Ubuntu on this machine and test, but my only other guess as to issues would be that I don't have every RAM channel populated and that my RAM is relatively slow on this PC.

Do you think those issues could cause this?

 

[RaymondLC92]

Not sure if this is a Windows/Linux issue or if my current install of Windows is somehow bunked, but I'll do a fresh install of Windows and see if that helps.

Edit: After rebooting into an Ubuntu live CD and installing iperf 2 I got about 80Gb/s local.

Edit 2: After reinstalling Windows, I get ~40Gb/s on the localhost iPerf which is still half of the Ubuntu Live CD performance, any reason that this may be the case?

Edit 3: I played around in the Ubuntu live CD for a while and noticed that the card was only running at x4 width which is weird as the motherboard doesn't even allow me to change the lane allotment on that slot! After taking the card out and putting it back in, I now get x8 width in lspci.

With that I am now getting reasonable benchmark scores:

~1.5GB/s on my NVMe Share a nice 150x speedup over the terrible 100MB/s I was getting before.
I'm also seeing ~35-37Gb/s on iperf when I used multiple cores.

Thanks for the help, I never would've though that it'd be a mix of a jacked up Windows installation and an improperly inserted PCI-E card.

Now I can go to bed and get back to other more important tasks.

 

[saivert]

There is not a lot of information on network tuning for Windows. I have seen so many ideas over the past months since I got into 10gigabit networking. Some suspect it is about how Windows handles Receive Side Scaling. Several threads about it on the forum and I also had the issue until I moved on from Sandy Bridge to a Coffee Lake CPU for my desktop PC. And I had ruled out any PCIe link width issue and such as I dual booted with Linux and had zero bandwidth issues on Linux but when booting into Windows they were back.

One thing other forums users that come by may try is open a powershell admin prompt and type:

Set-NetTCPSetting -SettingName Internet -AutoTuningLevelLocal Experimental

or:

Set-NetTCPSetting -SettingName Internet -AutoTuningLevelLocal Disabled


RSS should be enabled by default for these adapters. But in case it is not type:

Set-NetAdapterRss -Name <name of adapter> -Enabled $true

(You may have to specify max processor number using -MaxProcessorNumber)

You can see it with Get-NetAdapterRss powershell cmdlet.

I wish Microsoft stopped fooling around and fixed this once and for all.