Poor performance 10GBE network.. why??

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

craig5571

Member
May 31, 2020
60
6
8
I got a new 10GBE switch from Santa.. ( ICX 7250P) in hopes it would speed up my VM performance (ESXi 7) and general large file copying..
well golly gee, was I shocked,, when I saw on how slow things were... I'm usually very smart... great with Wi-Fi ...

I am at a loss here.. as to why I have such poor Iperf performance..
maybe I can't see the forest because of the trees..
both the iperf client and server are on 10gb sfp+ ports

Iperf results
C:\stuff\iperf>iperf3 -c 192.168.1.254
Connecting to host 192.168.1.254, port 5201
[ 4] local 192.168.1.205 port 49951 connected to 192.168.1.254 port 5201
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-1.00 sec 57.5 MBytes 482 Mbits/sec
[ 4] 1.00-2.00 sec 36.1 MBytes 303 Mbits/sec
[ 4] 2.00-3.00 sec 48.2 MBytes 405 Mbits/sec
[ 4] 3.00-4.00 sec 38.1 MBytes 320 Mbits/sec
[ 4] 4.00-5.00 sec 56.1 MBytes 471 Mbits/sec
[ 4] 5.00-6.00 sec 38.9 MBytes 326 Mbits/sec
[ 4] 6.00-7.00 sec 49.1 MBytes 412 Mbits/sec
[ 4] 7.00-8.00 sec 78.9 MBytes 662 Mbits/sec
[ 4] 8.00-9.00 sec 39.1 MBytes 328 Mbits/sec
[ 4] 9.00-10.00 sec 38.5 MBytes 323 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-10.00 sec 481 MBytes 403 Mbits/sec sender
[ 4] 0.00-10.00 sec 480 MBytes 403 Mbits/sec receiver

iperf Done.


My PC has Intel Xeon 14 Core Processor E5-2683v3 2.0ghz 35mb Smart Cache
(4x16gb sticks) for a total 64 gb ddr4 ram
samsung 970 m.2 512gb
Asrock x99 extreme 4

OS
Microsoft Windows [Version 10.0.19042.746]
Edition Windows 10 Pro
Version 20H2
Installed on ‎6/‎29/‎2020
OS build 19042.746
Experience Windows Feature Experience Pack 120.2212.551.0


Mellanox ConnectX-2 Ethernet Adapter installed in slot 1 which is a pci-e 3.0 x16 slot
driver provider microsoft
driver date 6/19/2019
driver version 5.50.14695.0
packet size from advanced tab of driver 1514

client iperf3 software on windows
iPerf 3.1.3

iperf server on synology 918+ NAS
DSM 6.2.3-25426 Update 3
4 x 8tb drives with a intel 3700 800gb SSD as read cache
NAS NIC Intel X520-DA1 INTEL 10GB PCIE SFP+
running docker image


connected with finisar 850nm ftlx8571d3bcl sfp+
optical fiber cable this is on the cable its blue "mm 50 /125um om4 lszh 01086m"

MY Switch
Brocade ICX 7250p

SSH@brocade#sh ver
Copyright (c) Ruckus Networks, Inc. All rights reserved.
UNIT 1: compiled on Nov 4 2019 at 10:30:28 labeled as SPR08092
(32121708 bytes) from Primary SPR08092.bin (UFI)
SW: Version 08.0.92T213
Compressed Primary Boot Code size = 786944, Version:10.1.17T215 (spz10117)
Compiled on Wed Oct 9 04:08:27 2019

HW: Stackable ICX7250-24-HPOE
==========================================================================
UNIT 1: SL 1: ICX7250-24P POE 24-port Management Module
Serial #:D
Software Package: ICX7250_L3_SOFT_PACKAGE (LID: fwkINHMnFJe)
Current License: l3-prem-8X10G
P-ASIC 0: type B344, rev 01 Chip BCM56344_A0
==========================================================================
UNIT 1: SL 2: ICX7250-SFP-Plus 8-port 80G Module
==========================================================================
1000 MHz ARM processor ARMv7 88 MHz bus
8192 KB boot flash memory
2048 MB code flash memory
2048 MB DRAM
STACKID 1 system uptime is 1 day(s) 14 hour(s) 8 minute(s) 6 second(s)
The system started at 22:33:14 GMT-05 Mon Jan 18 2021

The system : started=cold star

Happy to provide any other details if necessary
I know very little about Iperf, so happy to find out that its user error.. (lol)
 
Last edited:

i386

Well-Known Member
Mar 18, 2016
4,221
1,540
113
34
Germany
I am at a loss here.. as to why I have such poor Iperf performance..
There are multiple reasons:
- Iperf is a linux application that runs with the help of cygwin on windows (Iperf3 shows extremely low throughput but wget shows throughput differently · Issue #463 · esnet/iperf)
- iperf binaries are distributed by third parties and shipped by with older versions of cygwin1.dll (Iperf3 shows extremely low throughput but wget shows throughput differently · Issue #463 · esnet/iperf)
- iperf3 is not using multiple cores (iperf3 at 40Gbps and above) (esnet is the organization behind iperf)
 
  • Like
Reactions: craig5571

craig5571

Member
May 31, 2020
60
6
8
There are multiple reasons:
- Iperf is a linux application that runs with the help of cygwin on windows (Iperf3 shows extremely low throughput but wget shows throughput differently · Issue #463 · esnet/iperf)
- iperf binaries are distributed by third parties and shipped by with older versions of cygwin1.dll (Iperf3 shows extremely low throughput but wget shows throughput differently · Issue #463 · esnet/iperf)
- iperf3 is not using multiple cores (iperf3 at 40Gbps and above) (esnet is the organization behind iperf)
thanks for the reply , also when I try to copy to an smb share on my nas it copies at 86mb .. so it's really not the iperf i'm concerned about I want to optimize my network and i spent a fair amount of time and $$$ and this poor performance feels like a punch in the gut.

any other good tools to benchmark network performance.

would it help if I booted a linux machine or mac.. to try the iperf??

this could all be in my head.. but I like to research an verify .. do every project like you were going to support it .. i find that breeds quality
 

BoredSysadmin

Not affiliated with Maxell
Mar 2, 2019
1,050
437
83
try "C:\stuff\iperf>iperf3 -P 20 -c 192.168.1.254"
This test will run 20 simultaneous connections against the Iperf listening server
It's also possible that the relatively weak cpu on NAS (J3455) isn't fast enough to saturate 10gig with 1500mtu. I'd definitely bump the MTU on both server and client to 9k. That should take much of the load off the NAS cpu.

more reading:
 
  • Like
Reactions: craig5571

azev

Well-Known Member
Jan 18, 2013
768
251
63
iperf.JPG

Not sure what the issue is, but I have no issues with getting good iperf3 performance on windows... above result is from win2k16 server running iperf -s and a win10 client. For me to get similar result I have to enable -P and run the test with 4 thread.

iperf2.JPG

Server hardware is similar to OP with oneconnect oce11102 nic and the win10 desktop is running Ryzen processor and intel x550.
 
  • Like
Reactions: craig5571

i386

Well-Known Member
Mar 18, 2016
4,221
1,540
113
34
Germany
does windows really suck that bad..
I think that depends on your expectations.
When I ran into low numbers with iperf i started to look for other tools and found ntttcp from microsoft. With ntttcp I had no problems with vanilla windows settings (no jumbo frames etc) and out of the box drivers to saturate a 40GBE link on a dual core 2.2GHz cpu:Unbenannt.JPG
 
  • Like
Reactions: craig5571

craig5571

Member
May 31, 2020
60
6
8
another thing copying large files to my nas is slow as well.. (86mb and less the gig) both have 10gb nics in them (details of entire config in first post). I think its windows issue.. i just use copy and paste.. is there a 3rd party copy utility that is multi-threaded ( not sure if that is the right word) . what I am looking to do is squeeze performance out of my window 10 box. I can't switch OS due to the nature of my jobs tools being windows only. just looking for ideas. thank you
 

i386

Well-Known Member
Mar 18, 2016
4,221
1,540
113
34
Germany
Robocopy

Edit: 86MByte/s is low, even for 1GBE. What 8TB hdds do you use in your nas? Raid 10?
 
Last edited:

craig5571

Member
May 31, 2020
60
6
8
i have a couple of different NAS they all copy at the same speed ironically ..
qnap ts 509 3gbps sata 5 2tb drives in raid 6 only has gig nics

synology 918+ with 4 HGST/Sun 8TB 7.2K 3.5" SAS HUH728080AL5200
in it mode raid 5. has intel 10gb 520 dat1.

i think it has something to do with windows...

i sometimes get to 150mb when copying files in windows but normally its around 70 -120 with small bursts upto 300 for a few seconds..

ok, i tried a program called extremcopy

and i move a 22 gig file from my c: drive ( samsumg 970 nvme) to my 918+ nas

it was way faster....
i found this article.. i can not vouch for any of the programs here.. but it seems interesting...
 
Last edited:

azev

Well-Known Member
Jan 18, 2013
768
251
63
what protocols are you using to move files between your windows machine and the nas ?? is it CIFS/SMB ?
Here's the transfer result of me pulling files from the server win2016 (ssd on my desktop, and raid6 of 16x 4TB drives with adaptec adapter)
transfer.JPG
I am also experiencing some weird issue when going the opposite direction, here's what I get when sending the same file to the server

transfer1.JPG

This seems to be a common issue, I found tons of people complaining the transfer speed wave, however mine seems to be very extreme from maxing out a 10Gb nics, to barely 1Gbps in the valley of the graph.

I don't have this problem when transferring or pulling data from a ZFS NAS, although the peak transfer speed is much less than the windows servers.
 
  • Like
Reactions: craig5571

itronin

Well-Known Member
Nov 24, 2018
1,233
793
113
Denver, Colorado
@azev
Sure looks like in the direction from your desktop to the server is 1Gbe constrained?
Do you have both a 10Gbe and 1Gbe connection plugged into your desktop?
 

azev

Well-Known Member
Jan 18, 2013
768
251
63
both servers and client only have 10Gb active nics, If you see the transfer speed to the server actually hits 1.11GB/s, that is about 10Gbps.
If I pay attention during the transfer, I can see that thansfer speed would drop to less than 100Mbps in the valley, before it would pick back up to 10Gbps, and rinse and repeat as the graph shows.
 

craig5571

Member
May 31, 2020
60
6
8
i have a couple of different NAS they all copy at the same speed ironically ..
qnap ts 509 3gbps sata 5 2tb drives in raid 6 only has gig nics

synology 918+ with 4 HGST/Sun 8TB 7.2K 3.5" SAS HUH728080AL5200
in it mode raid 5. has intel 10gb 520 dat1.

i think it has something to do with windows...

i sometimes get to 150mb when copying files in windows but normally its around 70 -120 with small bursts upto 300 for a few seconds..

ok, i tried a program called extremcopy

and i move a 22 gig file from my c: drive ( samsumg 970 nvme) to my 918+ nas

it was way faster....
i found this article.. i can not vouch for any of the programs here.. but it seems interesting...
for my share i was created an SMB share on the synology and used regular windows map a drive to connect to it
 

itronin

Well-Known Member
Nov 24, 2018
1,233
793
113
Denver, Colorado
both servers and client only have 10Gb active nics, If you see the transfer speed to the server actually hits 1.11GB/s, that is about 10Gbps.
If I pay attention during the transfer, I can see that thansfer speed would drop to less than 100Mbps in the valley, before it would pick back up to 10Gbps, and rinse and repeat as the graph shows.
nevermind and sorry my bad I miss read that as 1Gb in the graph. Good thing I have new glasses on the way...
 

craig5571

Member
May 31, 2020
60
6
8
It shouldn't. This is a Windows 7 x64 box talking to one of my homebuilt RAIDzilla servers running FreeBSD with Samba, single 10Gig card:
from what i have read "they" ( microsoft ) made changes in windows 10.. I don't know this to be true.. I just read it on several blogs. I would be happy to be wrong
 

Terry Kennedy

Well-Known Member
Jun 25, 2015
1,140
594
113
New York City
www.glaver.org
from what i have read "they" ( microsoft ) made changes in windows 10.. I don't know this to be true.. I just read it on several blogs. I would be happy to be wrong
Well, I just tried it for you on a Windows 10 box and got:



Both the file server and this PC were doing other things at the same time, which explains the varying transfer speed (the file server is 16 HGST He8 drives and the PC is a Class 6 NVMe drive).
 
  • Like
Reactions: craig5571

craig5571

Member
May 31, 2020
60
6
8
Well, I just tried it for you on a Windows 10 box and got:



Both the file server and this PC were doing other things at the same time, which explains the varying transfer speed (the file server is 16 HGST He8 drives and the PC is a Class 6 NVMe drive).
thanks my NAS only has 4 HGST He8 (8tb) drives.. I really appreciate the help, it convinces me I'm not crazy and it is doable...