Getting very odd results with Connectx-3 and ConnectX-4 cards.

[calmasacow]

I'm having rather disappointing and confusing results with my transfer rates.
here is a little info about the setup.

I have a Unraid file server it has a Mellanox ConnectX-3 dual port card in it and is connected via fiber to my Mellanox-SX6036 that has been flashed with ETH license This side checks out and seems great.
I can connect via a 10GB thunderbolt adapter with my MacBook and I get the expected transfer rates.


The Windows 11 problem. Isn't it always windows?

The windows machine is as follows:

i9-13900K
64GB Memory
Windows 11 Pro 24H2

I can connect a Asus Q107 based 10GBE card and I get expected transfer rates.

I can connect another Connectx-3 card that is identical to the one in the Unraid server and I get iPerf3 of :


[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.00 sec 3.32 GBytes 2.85 Gbits/sec
[ 5] 0.00-10.00 sec 3.32 GBytes 2.85 Gbits/sec


But when I install a ConnectX-4 and run it I get a even lower score between 1.2 and 1.9Gbit Sec
it is a MCX455A-FCAT

Connections are using Mellanox MC2207411-SR4L QSFP+ modules rated at 56Gbit and using MPO fiber runs that are 10M cables.
Firmware and drivers are upto date

 

[i386]

Don't use iperf3 on Windows.

 

[nasbdh9]

disable recv segment coalescing in mlx5 of windows (both ipv4 and ipv6) and change receive buffers to 4096.
iperf 3.16 or higher does not have any serious performance issues on Windows.
Please make sure your iperf version is at least 3.16.

 

[nasbdh9]

It is also important that you have enough memory bandwidth.
My customers often complain that their iperf3 test speed cannot reach 10Gbps, and then find out that their PC is only configured with single channel DDR4.