problem with mixed port speeds on Mellanox MCX4121A-ACAT

[XiMMiX]

I have 2 Mellanox MCX4121A-ACAT 25Gbit NICs. One for my homeserver (Proxmox) and one for my desktop (Win11).
I have a 25Gbit passive DAC cable and if I connect port2 on each NIC with the other it works great, both machines report 25Gbit link speed and I see actual speeds above 10Gbit.

Connecting a 10Gbit passive DAC cable to port 1 of my server creates a problem. The speed on port2 is then also reduced to 10Gbit.
It doesn't matter whether the other end of the DAC is connected or not.
This problem doesn't exist on my Win11 desktop, I can connect port1 using the 10Gbit dac to a switch without the speed on port 2 being affected.


Things I've tried:
Swap cards between server and desktop. This doesn't help, the problem is only on the server.
Try a different brand 10Gbit DAC (cisco instead of 10Gtek)
Upgrade both cards to the latest firmware

Has anyone ever had this problem and maybe know a solution?

I'm at a loss. Is this
a. normal behaviour?
b. cheaping out on the cards too much? They're refurbished cards from aliexpress
c. cheaping out on the DACs too much? They're from 10Gtek
d. a driver setting that defaults to the correct setting on Win11 but the wrong one on Linux?
e. a bug in the linux driver?

thx for any help.

 

[Chunks]

I actually was troubleshooting something similar yesterday. In my case, I have a windows VM on Unraid that shows 10gb link (latest virtio drivers) but using ntttcp I only get 250MB/s (2.5gbit performance not 10gbit). So what I did to test, was to create a Windows 11 "Windows to Go" disk on a spare NVME (using Rufus) I had in a USB enclosure, boot off the USB NVME and then use the temporary Windows 11 install to download and run ntttcp test directly, windows to windows.

My speeds went from 250MB to 900ish, so in my case at least, I believe the problem lies somewhere withing Linux/LinuxKVM/Unraid defaults/VM Settings.

I didn't solve it yet, but I confirmed the hardware is all mostly fine at least. Maybe you can at least rule out some sources of problems?

 

[pimposh]

Did you play with mlx5cmd or ethtool to set-off auto neg ?

 

[SlowmoDK]

I had the same issue when using a 25Gbit and 10Gbit DAC both ran at 10Gbit

But if i use a 25Gbit fiber connection and a 10Gbit dac both run at correct speed

I haven't tested 25Gbit DAC with a 10Gbit fiber

But running 2 DAC cables seem to cause this issue on linux (TrueNAS Scale in my instance)

 

[XiMMiX]

I actually was troubleshooting something similar yesterday. In my case, I have a windows VM on Unraid that shows 10gb link (latest virtio drivers) but using ntttcp I only get 250MB/s (2.5gbit performance not 10gbit). So what I did to test, was to create a Windows 11 "Windows to Go" disk on a spare NVME (using Rufus) I had in a USB enclosure, boot off the USB NVME and then use the temporary Windows 11 install to download and run ntttcp test directly, windows to windows.

My speeds went from 250MB to 900ish, so in my case at least, I believe the problem lies somewhere withing Linux/LinuxKVM/Unraid defaults/VM Settings.

I didn't solve it yet, but I confirmed the hardware is all mostly fine at least. Maybe you can at least rule out some sources of problems?

I didn't use nttcp, but iperf which is a similar tool afaik. I know that in practice I probably won't reach the 25gbit using filetransfers either because the underlying blockdevice can't reach that speed or because of limitations of windows filetransfers.

Did you play with mlx5cmd or ethtool to set-off auto neg ?

I used ethtool on the server, but not mlx5cmd. Mlx5cmd is windows only afaik and since I didn't have a problem on the windows side I never tried. Once I have some time I'll try forcing 25gbit on the windows end as well, that I haven't tried yet.

I had the same issue when using a 25Gbit and 10Gbit DAC both ran at 10Gbit

But if i use a 25Gbit fiber connection and a 10Gbit dac both run at correct speed

I haven't tested 25Gbit DAC with a 10Gbit fiber

But running 2 DAC cables seem to cause this issue on linux (TrueNAS Scale in my instance)

thx, that sounds like a solution, even if it's a bit more expensive than I hoped for.

 

[Chunks]

I realized after writing the following that it's a pretty big tangent. Anyway, regarding windows + iperf, Microsoft wrote a blog article about why you'd want to use ntttcp vs iperf3 on windows:

Three Reasons Why You Should Not Use iPerf3 on Windows | Microsoft Community Hub

This article will explain why you should not use iPerf3 on Windows for synthetic network benchmarking and testing. Followed by a brief explanation of...

techcommunity.microsoft.com

Here's a quote I found from an article on CNX summarizing the above blog post:

Microsoft explains iperf3 should not be used in Windows for three main reasons:

1. The maintainer of iperf – ESnet (Energy Sciences Network) – says “iperf3 is not officially supported on Windows, but iperf2 is. We recommend you use iperf2. Some people are using Cygwin to run iperf3 in Windows, but not all options will work”
2. iPerf3 is Emulated on Windows – iPerf3 does not make Windows native API calls as it only knows how to make Linux/POSIX calls, and this may impact performance.
3. I usually download iperf3 3.1.3 for Windows released in 2016, and Microsoft notes the one offered by ESnet (version 3.16) is more recent but still 15 versions behind, so users are not running the latest version of the utilities.

Lastly, here's an article on how to use it (I'm really starting to feel like an MS Shill )

Test VM network throughput by using NTTTCP

Use the NTTTCP tool to test network bandwidth and throughput performance for Windows and Linux VMs on a virtual network.

learn.microsoft.com

Sorry if this is just spam, I'm pretty new to digging into this stuff.

 

[bugacha]

I realized after writing the following that it's a pretty big tangent. Anyway, regarding windows + iperf, Microsoft wrote a blog article about why you'd want to use ntttcp vs iperf3 on windows:

Three Reasons Why You Should Not Use iPerf3 on Windows | Microsoft Community Hub

This article will explain why you should not use iPerf3 on Windows for synthetic network benchmarking and testing. Followed by a brief explanation of...

techcommunity.microsoft.com

Here's a quote I found from an article on CNX summarizing the above blog post:


Lastly, here's an article on how to use it (I'm really starting to feel like an MS Shill )

Test VM network throughput by using NTTTCP

Use the NTTTCP tool to test network bandwidth and throughput performance for Windows and Linux VMs on a virtual network.

learn.microsoft.com

Sorry if this is just spam, I'm pretty new to digging into this stuff.

thanks for posting this, I didnt realize iperf3 can't be used on windows