10GB read speeds, not so great write speeds

[madbrain]

For those who care, I have almost 1000 iperf3 data points between 4 machines on my network running 10 Gbe in a spreadsheet at

10 GBASE-T Ethernet performance in my home network

Tests Client,Server,Mode,Streams,Gbps,Gbps,Gbps,Gbps,Gbps,Gbps,Gbps,Gbps,Gbps,Gbps,Slowest,Fastest EEE,off,off,off,off,on,on,off,off,on,on Jumbo frames,off,off,4KB,4KB,4KB,4KB,9KB,9KB,off,9KB Notes,Run #1,Run #2,Run #1,Run #2,Run #1,Run #2,Run #1,Run #2 HIGGS,BUMBLEBEE,Receiving,1,7.16,7.53,7.16...

docs.google.com

All 4 boxes have Aquantia AQN-107 x4 NICs.

I test every host combination from 1 to 4 TCP streams, without jumbo frames, with 4KB jumbo frames, and 9KB jumbo frames. Also some tests measuring the impact for Energy-efficient-ethernet.

Tests are between 4 machines of various generations, which are described in the second sheet (click at the bottom).
One is as old as 2011 - AMD FX-8120 with a PCIe 2.0 bus. All others are PCIe 3.0 . The most recent CPU is a Ryzen 2700 from 2800.

I haven't fully analyzed them all the results - there are a hell of a lot of data points. Some could be erroneous due to manual entry. Each column takes about 20 minutes to start and run, and a few more to enter the data.

So far, I can see that :
1) the two switches, one consumer, one pro, don't appear to be slowing anything down, as most tests with recent machines and multiple streams get close to line rate
2) jumbo frames aren't helping with single stream tests, which can never saturate the network. Actually, jumbos seem to be hurting throughput in many single stream tests, which I thought odd. Jumbos do help with multiple streams .
3) None of the tests appear to be strictly bound on the PCIe bus. Even the year 2011 PCIe 2.0 box, BUMBLEBEE, achieves 9.79 Gbps in a few of the tests.

Data that isn't in the spreadsheet : CPU usage. I didn't try to record that. I can say that the NICs behave wildly differently between these 4 machines, that have 4 different CPUs and chipsets. Sender always uses less CPU due to way LSO works and some of the TCP decoding cannot be fully offloaded. I read a paper about it over the weekend while waiting for some tests to complete.

Things I still want to test :
- the various "interrupt moderation" settings. It is left at the default in all these tests. On Windows, that's "Enabled: Adaptive". On Linux, I believe it is "Disabled".
- disk writes. I do have one real drive that performs beyond 10 Gb/s. It is a 5 x 1TB Samsung 860 SSD EVO. Crystaldiskmark local test attached. Userbenchmark always tells me it detected a cached drive, but that's not the case ... Remote test not done yet.

I guess this drives justifies 25 Gbe, right ?

 

[madbrain]

I do have one real drive that performs beyond 10 Gb/s. It is a 5 x 1TB Samsung 860 SSD EVO. Crystaldiskmark local test attached. Userbenchmark always tells me it detected a cached drive, but that's not the case ... Remote test not done yet.

So I did the remote test.


This is without jumbo frames. Also going through 2 switches. And at least 100 ft of cables, between my home office and home theater.
The client in the test above is a Ryzen 2700 in the home theater.
Server is an i7-5820k in the home office. That's the box with the 5 x SATA 1 TB SSD.
Interesting how the writes are a better than the reads in my case for lines 3-4 .
And how big the difference is vs the local test above. 1/3 the speed for SEQ Q1T1.

 

[n00bftw]

So I did the remote test.

View attachment 15113
This is without jumbo frames. Also going through 2 switches. And at least 100 ft of cables, between my home office and home theater.
The client in the test above is a Ryzen 2700 in the home theater.
Server is an i7-5820k in the home office. That's the box with the 5 x SATA 1 TB SSD.
Interesting how the writes are a better than the reads in my case for lines 3-4 .
And how big the difference is vs the local test above. 1/3 the speed for SEQ Q1T1.

SSD's will do that haha. Now where can i buy 100TB for a minimal price. Knowhere haha.

 

[i386]

So I did the remote test.

View attachment 15113
This is without jumbo frames. Also going through 2 switches. And at least 100 ft of cables, between my home office and home theater.
The client in the test above is a Ryzen 2700 in the home theater.
Server is an i7-5820k in the home office. That's the box with the 5 x SATA 1 TB SSD.
Interesting how the writes are a better than the reads in my case for lines 3-4 .
And how big the difference is vs the local test above. 1/3 the speed for SEQ Q1T1.

Your raid consists of 860 evos, I rember that many samsung sata ssds throttle at some point.
What numbers do you get when you restart the storage "server" between the benchmarks?

 

[madbrain]

Your raid consists of 860 evos, I rember that many samsung sata ssds throttle at some point.
What numbers do you get when you restart the storage "server" between the benchmarks?

What do you mean "restart" ? Reboot the Win10 machine that has those 5 x 1TB 860 EVOs ?
I haven't noticed any difference in performance from doing that.
The volume is over 80% full. I think it was a bit faster locally when empty, but not much.

 

[n00bftw]

Got it sorted now, the problem was that I had to add 'tso lro' into the options field of the NIC. Beautiful. I also have it running in ESXi . Runs perfect.

 

[acquacow]

Dang, I can hit the 1M 10gigE speeds with my FreeNAS box just fine, but I can't seem to replicate your 4k numbers.

My 4k reads/writes with those settings only seem to hit ~10MB/sec, but I'm sync + compression. I even tested with my SLOG-backed pool and I get about the same.

I'll have to dig into this.

 

[lowfat]

Tried to check to see if change large offload on my ConnectX-3 in Ubuntu Server 19.10. But unfortunately it doesn't look like I can.

brendon@server:~$ sudo ethtool -k enp1s0d1 | grep large-receive-offload
large-receive-offload: off [fixed]
brendon@server:~$ sudo ethtool -K enp1s0d1 lro on
Cannot change large-receive-offload
Could not change any device features

I know for sure I can change the values with the same adapter in Windows.