2.5GbE network not running as expected

[PlannerOne]

Hi, I'm a new forum member. Probably like most forum members, I joined because I'm having an issue that I cannot figure out. I am a home user that set up a 2.5GbE network that isn't working correctly. I'll apologize in advance if this forum is geared more towards a business/IT setting with professional level equipment, and would take suggestions on where to properly point my issue.
If I am in the right place, then I will present my case. I'm at a loss, because it seems that what I have done so far should have given me the result I expected, but has not.
I was running a gigabit LAN that was performing as expected for each respective device.
I purchased an Asustor AS5304T NAS, and populated with 3x Toshiba N300 drives in RAID 5.
I wanted to utilize the faster option of the 2.5GbE port on the NAS, so I purchased the required equipment to set up part of my LAN as 2.5GbE.
I purchased a TP Link SG105-M2 unmanaged switch, and an Asustor AS-U2.5G2.
Components were installed, and I am using CAT6 cable from a computer to the switch, and a CAT6 cable from the NAS to the switch.
I did make sure that on the Asustor USB adapter, that the arrow aligns with the USB SS logo, as stated in the manual, to get 2.5GbE speed when using the type C to type A adapter.
I also made sure that the Asustor USB adapter was plugged into the correct type of USB port on the motherboard. (USB 3.2 Gen2x1)
The latest Realtek USB drivers were installed.
So, thinking everything is in place, I start running some tests between the connected PC to the NAS, thru the 2.5GbE switch, and I'm seeing no improvement in read and write speeds to my NAS. Three different test methods provided results no better than what I had before, anywhere from 90-110 MB/s. (720-880 Mb/s)
I did notice that using a USB to ethernet adapter did create some pauses with my other USB devices when I was testing, so as the adapter was a temporary idea just to get the 2.5GbE network off the ground, I purchased a NIC. The NIC is based on Realtek 8125B chipset.
The latest Realtek 2.5GbE PCIe drivers were installed.
I swapped over from the USB adapter to the NIC, rebooted all equipment, and started running some transfer tests.
I am still getting the same general read and write speeds to the NAS.
I did also try changing the adapter settings from auto negotiate to 2.5GbE full duplex, there was no difference in the results.
The Asustor NAS shows a 2.5GbE LAN connection.
The computer connected to the 2.5GbE switch shows a 2.5Gb connection to the switch.
In the Windows network connection information for the adapter, also shows a 2500/2500 (Mb/s) connection.
In all my internet research before putting up this post, I have all the proper equipment, and see all the proper indications, to be getting better than gigabit numbers in read/write tests to my NAS.
To provide more data, I took the Asustor USB adapter, installed it on my laptop thru the correct type USB port, and still got the same level of read and write speeds to the NAS, so I eliminated a configuration issue with the computer being used before.
All my tests were run with the source data on a NVMe drive so as to not bottleneck transfer rates.
One test I have not run yet is to remove the NAS from the equation, and check transfer rates between two computers connected to the 2.5GbE switch.
What have I missed? Everything I see so far on the internet indicates I should be seeing faster speeds from this part of my LAN.

 

[Mushishi]

Do you by any chance still have connection over the gigabit side from the computer, that can access the nas as windows can be tricky if you run multiple lan connections in the same network to make it understand you want some over one connection and some traffic over another connection.

So the way i would set it up to be sure you are using the 2.5G and not sending traffic over the 1G connection is like this:

Router/Modem -> 2.5G Switch -> one cable to nas and one to computer.

 

[aero]

As Mushishi already mentioned, the max rate you achieved in your testing is suspicious because it never exceeds 1Gb/s, so definitely confirm you don't have a 1G link in the mix still.

Proceed with the next test, which is to eliminate storage from the testing, and focus on network only, between two computers. I recommend the iperf utility.

 

[PlannerOne]

Last night I eliminated the NAS from the mix. I have my laptop with 2.5Gb adapter going to 2.5GbE switch. I have my desktop with 2.5Gb network card going to 2.5Gb switch. Both laptop and desktop are running NVMe drives. This eliminates storage medium speed as a slowing factor. I copied a large file from laptop to desktop, then from desktop back to laptop. Read and write speeds peaked 113MB/s (904 Mb/s). By all accounts, this setup should have provided a direct patch from point A to point B, at 2.5Gb speeds.
It is a relatively simple home network. All the 2.5Gb equipment is connected, just like all of the research I did before getting the equipment. That's the part I cannot figure out, if all 2.5Gb equipment has a path, why am I only seeing gigabit speeds?

 

[BlueFox]

Is everything actually negotiating at 2.5gbit? Start eliminating things from the equation to see what might be the culprit, e.g. run a cable directly between the two computers. You may also wish to use iperf for testing instead of just copying files back and forth.

 

[PlannerOne]

I did check to assure any network adapter was set to 2.5Gb speed full duplex, not auto negotiate. All indicators say link speed is at 2500 (Mb/s), what it is actually doing seems to be something different.

I did stop using the NAS as the target device for speed testing. I saw no difference.

I don't know how to use iperf but will search and learn.

What does connecting a network cable directly from one computer to another allow me to do? How does (this work?) one pc identify the other pc without going through a router or switch (assigning an IP address)?

 

[BlueFox]

You can set a static IP address on both ends when directly connecting. Don't need a router or switch for that.

 

[aero]

I thought I'd harp a bit on troubleshooting. Many people seem to want to skip steps and make poor assumptions, which leads to inadequate troubleshooting.

Every system is comprised of various subsystems, with those subsystems comprised of individual components. When a system is not working as expected, one must seek to test at the lowest level of those systems to find the fault. That is why in this thread we have suggested eliminating storage of any kind (the NAS, and also local disk), and focus on raw networking first. Directly connecting two hosts eliminates the switch from being the culprit.

Storage subsystem is comprised of the NAS itself, individual disks, cpu, interrupts, protocols, and more. The storage subsystem depends on the underlying network. The network is comprised of adapters on the hosts, the switch, the cabling, protocols, etc.

Making incorrect assumptions about any of those things will just cause you to chase your tail.

 

[PlannerOne]

I will connect the two computers, each with a 2.5Gb NIC/adapter, and run iperf, per all the best practices mentioned above. I will try to get to this no later than this weekend and report results back.
Thank you for all the responses so far.

 

[PlannerOne]

I have iperf test results after connecting the two computers directly to one another. The top results are with both adapters speed/duplex set to 2.5Gb, the bottom results are both adapters on auto negotiate.

 

[prdtabim]

I have iperf test results after connecting the two computers directly to one another. The top results are with both adapters speed/duplex set to 2.5Gb, the bottom results are both adapters on auto negotiate.

View attachment 24163

Try iperf3 with the additinal key "-P 4" to invoke 4 parallel thread transfering data.
The rates in that point to point test must be 2.38Gb/s using MTU 1500 and 2.47Gb/s using MTU 9000.

Consider deactivate antivirus or other software ( virus ) used by Banks. They can limit the rate in the interface.

 

[PlannerOne]

I will run more tests with the additional parameter, as well as deactivate AV software, and report results back.
Thanks

 

[PlannerOne]

I have iperf test results. AV was deactivated, as well as using the "-P 4" parameter. The results between the two adapters did not perform as they should. Any thoughts on how to proceed are appreciated.

 

[i386]

I have iperf test results. AV was deactivated, as well as using the "-P 4" parameter. The results between the two adapters did not perform as they should. Any thoughts on how to proceed are appreciated.

View attachment 24206

The windows version of iperf sucks, try a linux live cd and run iperf from there.

 

[aero]

I'd generally say that USB NICs are not very good. However, you also had speed issues with your desktop Realtek NIC. What is the exact card model? And what are the specs of the desktop?

 

[PlannerOne]

It is a generic card based on Realtek RTL8125B chipset, running drivers version 10.60.

https://www.amazon.com/dp/B09HGRK5XB?psc=1&ref=ppx_yo2ov_dt_b_product_details

What do you need to know about the desktop? It's an Intel i7-8700k on an Asus Prime Z390-P, 16GB, Win 10 Pro.

I got the USB NIC first just for ease of installation, but then when I didn't see any improvement in network performance, I got the PCIe NIC.

 

[PlannerOne]

Well I have some additional data, but I can't make sense of it. Just to see if it made any difference, I turned on jumbo frames on both adapters (9014) and ran iperf. These test results are now exactly what is expected. I connected both computers back to the 2.5Gb switch and ran the test again, and again got numbers close to what is expected. I changed the jumbo frames to off, and the test results dropped back down to gigabit speeds. My research leans towards that jumbo frame enabled is not necessary with modern cards, nor is recommended due to risk of fragmentation if all devices not using same MTU.

So I don't know what to do, because it seems everything should work if jumbo frames is off, but does not.

NIC to NIC



NICs to switch

 

[prdtabim]

Well I have some additional data, but I can't make sense of it. Just to see if it made any difference, I turned on jumbo frames on both adapters (9014) and ran iperf. These test results are now exactly what is expected. I connected both computers back to the 2.5Gb switch and ran the test again, and again got numbers close to what is expected. I changed the jumbo frames to off, and the test results dropped back down to gigabit speeds. My research leans towards that jumbo frame enabled is not necessary with modern cards, nor is recommended due to risk of fragmentation if all devices not using same MTU.

So I don't know what to do, because it seems everything should work if jumbo frames is off, but does not.

NIC to NIC

View attachment 24271

NICs to switch

View attachment 24272

Looks like a bottleneck in packet rate. Since mtu was increased the packet rate falls at the point to achieve 2.47Gb/s.
The fact that using the switch in between lower the rate is very odd. This Tplink switch is new and have switching capability more than enough.
Did you try with mtu 9000 or other value ? its important that all devices involved uses the same mtu.

 

[PlannerOne]

For both adapters, in the settings, the MTU drop down list is 9014. It does not give the options to enter a number, only select from a list, and 9000 is not in that list. The NAS with the 2.5Gb port (and the reason for the upgrade to 2.5Gb equipment) has MTU of 9000. I did not try any other speed in the drop down list, since I was expecting everything to work out of the box at MTU 1500, and certainly not expecting to change the MTU to anything other than 1500, so everything would be the same. So there is the concern of all the 2.5Gb devices not having the same MTU, as well as why does the tests at MTU 1500 not run as they should. How can I check and see what are the bottlenecks at MTU 1500?

 

[prdtabim]

For both adapters, in the settings, the MTU drop down list is 9014. It does not give the options to enter a number, only select from a list, and 9000 is not in that list. The NAS with the 2.5Gb port (and the reason for the upgrade to 2.5Gb equipment) has MTU of 9000. I did not try any other speed in the drop down list, since I was expecting everything to work out of the box at MTU 1500, and certainly not expecting to change the MTU to anything other than 1500, so everything would be the same. So there is the concern of all the 2.5Gb devices not having the same MTU, as well as why does the tests at MTU 1500 not run as they should. How can I check and see what are the bottlenecks at MTU 1500?

You could test if a device is operating with mtu larger than 1500 using ping with no fragment key ( -f in windows ) and "-l size " to set the payload.
MTU 1500 implies in payload max size of 1472. MTU 9000 implies in payload max size of 8972 .

ping -f -l <size>