Mellanox Connectx-6 Dx adapter slow only 6gbps Throughput

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.
Y

ycp

Member
Jun 22, 2014
251
19
18
I have 2 servers specifically Lenovo SR635 both with Mellanox Connectx-6 Dx OCP 100G network cards.
One can transfer data speed at high throughputs and one is stuck at 6.5gbps. It wont go any higher than 6.5gbps.
The cpus and memory and os configurations are the same.
I can't figure out why its stuck at such a speed.
 
Y

ycp

Member
Jun 22, 2014
251
19
18
I am doing large file transfers from older servers to this new one. I am using windows server. Around 80TB
 
M

mattventura

Well-Known Member
Nov 9, 2022
591
305
63
If you swap the cards, does the issue follow the card, or does it stick with the host server?
 
N

necr

Active Member
Dec 27, 2017
196
65
28
125
Could be many reasons for file transfer, usual suspects are MTU size and NIC buffers. I'd recommend to test with ib_send_bw (part of WinOF2) and nttcp (not iperf3 on Windows!) to exclude any disk issues.
 
aosudh

aosudh

Member
Jan 25, 2023
64
15
8
How exactly do you transfer files? Do you simply drag the files using the Windows File Explorer on the two servers?
 
aosudh

aosudh

Member
Jan 25, 2023
64
15
8
In addition, if it is not a bandwidth-specialized fs, the most appropriate way to transfer files is to directly move the drive to the new machine for copying.
 
I

i386

Well-Known Member
Mar 18, 2016
4,588
1,737
113
35
Germany
Are the cx6 dx from israel and have "prototype", "es" or o stickers at all on them?
 
G

gb00s

Well-Known Member
Jul 25, 2018
1,309
702
113
Poland
Check if you run the card x8 or x16.

EDIT: mlxconfig -d <deviceID> q | grep LINK_TYPE or lspci -s <deviceID> -vvv | grep -i width
EDIT: 6.5Gbps (Gbit/s) or 6.5GBps (GB/s). Because 6.5GB/s sounds pretty much x8.
 
Last edited:
Y

ycp

Member
Jun 22, 2014
251
19
18
@aosudh I transfer files using Windows explorer.
@bugacha The MTU is 1500. I have not configured anything for SMB multichannel or RDMA
@i386 - The cards are not prototype or es and are actually Dell and Facebook branded
@gb00s - The cards are running at x16.
@mattventura - I Swapped the cards the issue is not the cards but probably tuning the network and MTU settings.

I tried running iperf3 on windows and getting over 80Gbps in the tests. Its the file copying that doesnt show the same performance not even close.
Also the the servers have multiple nvme ssds in RAID configurations so the storage is capable of pushing close to 100gbps.
 
aosudh

aosudh

Member
Jan 25, 2023
64
15
8
i386 said:
Are the cx6 dx from israel and have "prototype", "es" or o stickers at all on them?
Click to expand...
Generally speaking, es and prototype are unlikely to have a significant impact on the transmission performance.
 
You must log in or register to reply here.
Top Bottom