Humbling.

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

pepperspray

New Member
Jan 8, 2021
6
0
1
I thought I knew what I was doing but I'm calling no joy.

3 desktop Windows 10 computers.

1 Intel x520-2 NIC
2 Mellanox Connect X-2
Microtik 5 port switch

Netgear Orbi Mesh

I have the internet line coming into the Orbi.
One of the ports from the Orbi into the Microtik switch.
All 3 desktops connected to the Microtik with sfp+ copper.

Internet is great 800 up/down on all the machines.
Transfer times are horrible. 30-70MB/s
Transfers are all to folders on m.2 drives.

I've updated firmware on all the cards. I've tried different tweaks on both the switch and cards. If anything, I've likely made it worse.
It's currently set to default everything.

Please help.
 

pepperspray

New Member
Jan 8, 2021
6
0
1
C:\WINDOWS\system32>/iperf/iPerf3.exe -c 192.168.1.125
Connecting to host 192.168.1.125, port 5201
[ 4] local 192.168.1.121 port 52162 connected to 192.168.1.125 port 5201
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-1.00 sec 184 MBytes 1.54 Gbits/sec
[ 4] 1.00-2.00 sec 184 MBytes 1.54 Gbits/sec
[ 4] 2.00-3.00 sec 184 MBytes 1.54 Gbits/sec
[ 4] 3.00-4.00 sec 184 MBytes 1.54 Gbits/sec
[ 4] 4.00-5.00 sec 180 MBytes 1.51 Gbits/sec
[ 4] 5.00-6.00 sec 171 MBytes 1.43 Gbits/sec
[ 4] 6.00-7.00 sec 182 MBytes 1.53 Gbits/sec
[ 4] 7.00-8.00 sec 172 MBytes 1.44 Gbits/sec
[ 4] 8.00-9.00 sec 182 MBytes 1.53 Gbits/sec
[ 4] 9.00-10.00 sec 180 MBytes 1.51 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-10.00 sec 1.76 GBytes 1.51 Gbits/sec sender
[ 4] 0.00-10.00 sec 1.76 GBytes 1.51 Gbits/sec receiver

iperf Done.

Anything jump out that I should change?
 

Falloutboy

Member
Oct 23, 2011
221
23
18
I thought I knew what I was doing but I'm calling no joy.

3 desktop Windows 10 computers.

1 Intel x520-2 NIC
2 Mellanox Connect X-2
Microtik 5 port switch

Netgear Orbi Mesh

I have the internet line coming into the Orbi.
One of the ports from the Orbi into the Microtik switch.
All 3 desktops connected to the Microtik with sfp+ copper.

Internet is great 800 up/down on all the machines.
Transfer times are horrible. 30-70MB/s
Transfers are all to folders on m.2 drives.

I've updated firmware on all the cards. I've tried different tweaks on both the switch and cards. If anything, I've likely made it worse.
It's currently set to default everything.

Please help.
If your mellanox cards which i presume are running at 10gb have a default gateway to your switch DNS they will be bottlenecked at the switch speed... note I don't know this equipment but this is my rampant guess, those GW requests are at 1Gb p/s i have no idea what an orbi is.
 

pepperspray

New Member
Jan 8, 2021
6
0
1
If your mellanox cards which i presume are running at 10gb have a default gateway to your switch DNS they will be bottlenecked at the switch speed... note I don't know this equipment but this is my rampant guess, those GW requests are at 1Gb p/s i have no idea what an orbi is.
Orbi is just a Netgear Mesh router. I appreciate the reply. At least it points me in a direction.
 

Falloutboy

Member
Oct 23, 2011
221
23
18
Orbi is just a Netgear Mesh router. I appreciate the reply. At least it points me in a direction.
No problems, try removing the GW on the mellanox cards. you will have no internet at that point on those nics but see if it resolves your lan speed issue.
if it does might i suggest 1 lan of mellanox with no gateway for file xfers between pcs on one ip address with no gw and a sepeeate gigabit lan using the orbi with gigabit nics and a gw.
 

pepperspray

New Member
Jan 8, 2021
6
0
1
No problems, try removing the GW on the mellanox cards. you will have no internet at that point on those nics but see if it resolves your lan speed issue.
if it does might i suggest 1 lan of mellanox with no gateway for file xfers between pcs on one ip address with no gw and a sepeeate gigabit lan using the orbi with gigabit nics and a gw.
I'm doing this all remote whilst slacking at work. When I get home I'll try it out. Thank you again!
 

BoredSysadmin

Not affiliated with Maxell
Mar 2, 2019
1,050
437
83
also try this:
 

Rand__

Well-Known Member
Mar 6, 2014
6,626
1,767
113
C:\WINDOWS\system32>/iperf/iPerf3.exe -c 192.168.1.125
Iperf3 is single threaded, depending on your cpu this might cause issues. But thats so slow that most cpu's should be able to top this, this is barely above 1Gbps

Thats why I said iperf2 or ntttp :)
 

pepperspray

New Member
Jan 8, 2021
6
0
1
Iperf3 is single threaded, depending on your cpu this might cause issues. But thats so slow that most cpu's should be able to top this, this is barely above 1Gbps

Thats why I said iperf2 or ntttp :)
Ooops. You did. Let me try it the right way.
 

Rand__

Well-Known Member
Mar 6, 2014
6,626
1,767
113
iperf2 has multiple threads option (-P or -p, one is port) to run at 4 or 8 processes in parallel to ensure total bandwith is not limited by a single thread
 

pepperspray

New Member
Jan 8, 2021
6
0
1
Iperf3 is single threaded, depending on your cpu this might cause issues. But thats so slow that most cpu's should be able to top this, this is barely above 1Gbps

Thats why I said iperf2 or ntttp :)
C:\WINDOWS\system32>/iperf/iPerf2.exe -c 192.168.1.125
------------------------------------------------------------
Client connecting to 192.168.1.125, TCP port 5001
TCP window size: 64.0 KByte (default)
------------------------------------------------------------
[ 1] local 192.168.1.121 port 52891 connected with 192.168.1.125 port 5001
[ ID] Interval Transfer Bandwidth
[ 1] 0.00-10.00 sec 1.79 GBytes 1.54 Gbits/sec

Looks about the same.
 

Rand__

Well-Known Member
Mar 6, 2014
6,626
1,767
113
Well, play with window size and multiple threads/processes a bit
 

RageBone

Active Member
Jul 11, 2017
617
159
43
iperf3 has the same -P x argument.
i had other bandwith issues on mellanox cx3s 40GbE and thought it were hardware related, turned out that 1500 MTU was limiting and upping that to 9000 made a jump from about 7Gbit/s to between 17 and 27 Gbit/s, direction matters.

-R to reverse iperf test direction on the client.
 

Rand__

Well-Known Member
Mar 6, 2014
6,626
1,767
113
Ah I see,
-P, --parallel n number of parallel client streams to run

finally.
Have not looked at it in years to be honest since it didnt have that option when I started with this (in 1853 or so:p)
 

j_h_o

Active Member
Apr 21, 2015
644
179
43
California, US
SMB file transfer performance is almost certainly related to RSS/RSC and/or the congestion provider changes.

For my Windows 10/Server 2019 machines, just the CTCP congestion provider changes back to Server 2016 values restored SMB file transfer performance; I kept the RSS/RSC stuff enabled on Mellanox NICs since it ~generally worked OK for me.

So if you run (Powershell) on each client:

Code:
Set-NetTCPSetting -SettingName "InternetCustom" -CongestionProvider CTCP
Set-NetTCPSetting -SettingName "InternetCustom" -DelayedAckTimeoutMs 50
Set-NetTCPSetting -SettingName "InternetCustom" -ForceWS Disabled

Set-NetTCPSetting -SettingName "DatacenterCustom" -CongestionProvider DCTCP
Set-NetTCPSetting -SettingName "DatacenterCustom" -CwndRestart True
Set-NetTCPSetting -SettingName "DatacenterCustom" -ForceWS Disabled

Set-NetTCPSetting -SettingName "Compat" -ForceWS Disabled

Set-NetTCPSetting -SettingName "Datacenter" -CongestionProvider DCTCP
Set-NetTCPSetting -SettingName "Datacenter" -CwndRestart True
Set-NetTCPSetting -SettingName "Datacenter" -ForceWS Disabled

Set-NetTCPSetting -SettingName "Internet" -CongestionProvider CTCP
Set-NetTCPSetting -SettingName "Internet" -DelayedAckTimeoutMs 50
Set-NetTCPSetting -SettingName "Internet" -ForceWS Disabled
you should find the performance is vastly improved.

 
  • Like
Reactions: itronin and Rand__
Jul 19, 2020
51
19
8
iperf3 vs iperf2: prefer iperf3. I just did some testing with this a couple of days ago. iperf2 is multithreaded, specifying -P starts up multiple threads in parallel that can run on multiple CPUs. iperf3 is single-threaded, specifying -P only opens multiple TCP connections that are served by a single thread running on a single CPU. However, iperf2 uses a LOT more CPU than iperf3 for the same transfer rate. In general, one iperf3 process should be able to hit north of 20 Gbps on a modern high-end CPU, and possibly a lot more with an MTU of 9 KB. If you run multiple iperf3 connections in parallel, you can get close to 100 Gbps. However, iperf2 will max out the CPU long before hitting 100 Gbps.
 
Last edited:

Rand__

Well-Known Member
Mar 6, 2014
6,626
1,767
113
iperf2 is multithreaded, specifying -P starts up multiple threads in parallel that can run on multiple CPUs. iperf2 is single-threaded, specifying -P only opens multiple TCP connections that are served by a single thread running on a single CPU.
Might want to clear that up :)