Super Low 10G Performance

[coolrunnings82]

I recently got a Dell R720 server for my lab and outfitted it with a 10G X540/i350 quad port mezzanine card. I'm having difficulty with performance of the network. Here's my setup:

Machine 1: Dell R720.
2x 2670v1 CPUs
384gb DDR3 RAM
Mezzanine Intel x540 10G NIC w/latest drivers and firmware
OS: Server 2012 R2

Machine 2:
Whitebox
E3 1240v2 CPU on Supermicro board
16gb DDR3 1600 RAM
PCI-E Intel X540T2 w/latest drivers and firmware
OS: Windows 10 Pro x64

Both machines are connected to a Netgear XS708E
Jumbo frames are not enabled
Both are connected with brand new CAT6 cables from Monoprice

I tried copying files over the network and was capping out around 1.8 gigabit going solid state to solid state. If I did just a single file copy it was around 1.2 gigabit and maxed out at 1.8 with multiple simultaneous file transfers. Not impressed!

Next I grabbed jperf 2.02 and installed it on both machines. With a single stream, I max out at 1.54 gigabit. Ugh! No wonder the file transfers were bad. If I bump it to 20 streams I can get it to 5 gigabit max. I logged into the gui on the switch and I'm not seeing any CRC errors. The cables I'm running are only 7' long. I'm at a loss for what else I can do to improve this performance. Any ideas?

 

[T_Minus]

To clear up some basics

1. Cables have done faster before ?
2. Switch has "worked" and done faster before ?
   
3. PCIE NIC has done > performance before ? (for you not prev. owner)
   
4. Mezz NIC has done > performance before ? (for you not prev. owner)

 

[bds1904]

Copying files over a network is not a good way to test initial perfoance. Windows can/does have some flaws in regards to file sharing. You should be using iperf to test initial performance and tweaking from there. I have also had issues with jperf, that's why I suggest using iperf.

Also try disabling QOS (in windows)on the interface to see if that makes a difference.

 

[coolrunnings82]

@bds1904
I'm pretty familiar with the issues associated with windows file transfers. However, the jperf results mirror exactly the same problem that I'm experiencing with Windows file transfers. The whole purpose of my 10G network is large Windows file transfers so that's as real-world as it gets. No point to it for me if I can't improve that metric. Will try disabling QOS and see if that helps! Thanks!

@T_Minus
Regarding cables and switch: yes, previous hosts on same cables could do at least 5-6 gigabit
Regarding PCIE NIC - Yep, better performance before
Mezz NIC > That's what's new. Hope it's not that part or if it is that it's not that model of part... LOL!

 

[coolrunnings82]

Update: I had enabled Hyper-V and forgot that the NIC was shared with the host OS. I removed the v-switch and reran the tests. This improved things a little bit. Here are the results from that test:
[ ID] Interval Transfer Bandwidth
[292] 0.0- 1.0 sec 242545 KBytes 1986928 Kbits/sec
[292] 1.0- 2.0 sec 254560 KBytes 2085353 Kbits/sec
[292] 2.0- 3.0 sec 254992 KBytes 2088894 Kbits/sec
[292] 3.0- 4.0 sec 255288 KBytes 2091323 Kbits/sec
[292] 4.0- 5.0 sec 265616 KBytes 2175923 Kbits/sec
[292] 5.0- 6.0 sec 287384 KBytes 2354247 Kbits/sec
[292] 6.0- 7.0 sec 266577 KBytes 2183800 Kbits/sec
[292] 7.0- 8.0 sec 260127 KBytes 2130962 Kbits/sec
[292] 8.0- 9.0 sec 261544 KBytes 2142565 Kbits/sec
[292] 9.0-10.0 sec 261342 KBytes 2140915 Kbits/sec
[292] 0.0-10.0 sec 2610992 KBytes 2138545 Kbits/sec

I bypassed the switch and used a patch cable going between the two NICs and I get the exact same results. This was also a new cable. Tried turning off QOS and get the same results. Ideas???

 

[dwright1542]

No jumbo frames is going to be an issue. Even with jumbo frames it's tough to get wirespeed. Have you tried it?

 

[coolrunnings82]

I did try enabling jumbo frames. Here's the result:
[ ID] Interval Transfer Bandwidth
[212] 0.0- 1.0 sec 197152 KBytes 1615069 Kbits/sec
[212] 1.0- 2.0 sec 251608 KBytes 2061173 Kbits/sec
[212] 2.0- 3.0 sec 276800 KBytes 2267546 Kbits/sec
[212] 3.0- 4.0 sec 262288 KBytes 2148663 Kbits/sec
[212] 4.0- 5.0 sec 197832 KBytes 1620640 Kbits/sec
[212] 5.0- 6.0 sec 259496 KBytes 2125791 Kbits/sec
[212] 6.0- 7.0 sec 226136 KBytes 1852506 Kbits/sec
[212] 7.0- 8.0 sec 197432 KBytes 1617363 Kbits/sec
[212] 8.0- 9.0 sec 197584 KBytes 1618608 Kbits/sec
[212] 9.0-10.0 sec 202912 KBytes 1662255 Kbits/sec
[212] 0.0-10.0 sec 2269248 KBytes 1858961 Kbits/sec
Done.

Still no improvement.

 

[coolrunnings82]

I tried 10 connections at once and I get the following results:
bin/iperf.exe -c 10.0.0.50 -P 10 -i 1 -p 5001 -f g -t 10 -T 1
------------------------------------------------------------
Client connecting to 10.0.0.50, TCP port 5001
TCP window size: 0.00 GByte (default)
------------------------------------------------------------
[216] local 10.0.0.126 port 50518 connected with 10.0.0.50 port 5001
[280] local 10.0.0.126 port 50526 connected with 10.0.0.50 port 5001
[224] local 10.0.0.126 port 50519 connected with 10.0.0.50 port 5001
[212] local 10.0.0.126 port 50517 connected with 10.0.0.50 port 5001
[272] local 10.0.0.126 port 50525 connected with 10.0.0.50 port 5001
[240] local 10.0.0.126 port 50521 connected with 10.0.0.50 port 5001
[248] local 10.0.0.126 port 50522 connected with 10.0.0.50 port 5001
[256] local 10.0.0.126 port 50523 connected with 10.0.0.50 port 5001
[264] local 10.0.0.126 port 50524 connected with 10.0.0.50 port 5001
[232] local 10.0.0.126 port 50520 connected with 10.0.0.50 port 5001
[ ID] Interval Transfer Bandwidth
[212] 0.0- 1.0 sec 0.08 GBytes 0.73 Gbits/sec
[248] 0.0- 1.0 sec 0.16 GBytes 1.35 Gbits/sec
[224] 0.0- 1.0 sec 0.09 GBytes 0.74 Gbits/sec
[264] 0.0- 1.0 sec 0.09 GBytes 0.75 Gbits/sec
[280] 0.0- 1.0 sec 0.09 GBytes 0.76 Gbits/sec
[256] 0.0- 1.0 sec 0.17 GBytes 1.42 Gbits/sec
[216] 0.0- 1.0 sec 0.09 GBytes 0.74 Gbits/sec
[272] 0.0- 1.0 sec 0.09 GBytes 0.74 Gbits/sec
[240] 0.0- 1.0 sec 0.16 GBytes 1.37 Gbits/sec
[232] 0.0- 1.0 sec 0.17 GBytes 1.45 Gbits/sec
[SUM] 0.0- 1.0 sec 1.17 GBytes 10.0 Gbits/sec
[264] 1.0- 2.0 sec 0.09 GBytes 0.77 Gbits/sec
[212] 1.0- 2.0 sec 0.09 GBytes 0.75 Gbits/sec
[248] 1.0- 2.0 sec 0.15 GBytes 1.32 Gbits/sec
[256] 1.0- 2.0 sec 0.15 GBytes 1.31 Gbits/sec
[216] 1.0- 2.0 sec 0.09 GBytes 0.77 Gbits/sec
[272] 1.0- 2.0 sec 0.09 GBytes 0.76 Gbits/sec
[232] 1.0- 2.0 sec 0.17 GBytes 1.44 Gbits/sec
[240] 1.0- 2.0 sec 0.15 GBytes 1.26 Gbits/sec
[224] 1.0- 2.0 sec 0.09 GBytes 0.76 Gbits/sec
[ ID] Interval Transfer Bandwidth
[280] 1.0- 2.0 sec 0.09 GBytes 0.78 Gbits/sec
[SUM] 1.0- 2.0 sec 1.15 GBytes 9.91 Gbits/sec
[256] 2.0- 3.0 sec 0.14 GBytes 1.24 Gbits/sec
[216] 2.0- 3.0 sec 0.09 GBytes 0.80 Gbits/sec
[248] 2.0- 3.0 sec 0.15 GBytes 1.32 Gbits/sec
[240] 2.0- 3.0 sec 0.14 GBytes 1.19 Gbits/sec
[232] 2.0- 3.0 sec 0.16 GBytes 1.39 Gbits/sec
[212] 2.0- 3.0 sec 0.09 GBytes 0.78 Gbits/sec
[280] 2.0- 3.0 sec 0.09 GBytes 0.80 Gbits/sec
[264] 2.0- 3.0 sec 0.09 GBytes 0.79 Gbits/sec
[272] 2.0- 3.0 sec 0.09 GBytes 0.79 Gbits/sec
[224] 2.0- 3.0 sec 0.09 GBytes 0.80 Gbits/sec
[SUM] 2.0- 3.0 sec 1.15 GBytes 9.91 Gbits/sec
[216] 3.0- 4.0 sec 0.09 GBytes 0.77 Gbits/sec
[272] 3.0- 4.0 sec 0.09 GBytes 0.75 Gbits/sec
[232] 3.0- 4.0 sec 0.15 GBytes 1.33 Gbits/sec
[256] 3.0- 4.0 sec 0.15 GBytes 1.28 Gbits/sec
[212] 3.0- 4.0 sec 0.09 GBytes 0.75 Gbits/sec
[240] 3.0- 4.0 sec 0.17 GBytes 1.42 Gbits/sec
[280] 3.0- 4.0 sec 0.09 GBytes 0.78 Gbits/sec
[ ID] Interval Transfer Bandwidth
[264] 3.0- 4.0 sec 0.09 GBytes 0.76 Gbits/sec
[248] 3.0- 4.0 sec 0.15 GBytes 1.31 Gbits/sec
[224] 3.0- 4.0 sec 0.09 GBytes 0.77 Gbits/sec
[SUM] 3.0- 4.0 sec 1.15 GBytes 9.91 Gbits/sec
[216] 4.0- 5.0 sec 0.09 GBytes 0.74 Gbits/sec
[232] 4.0- 5.0 sec 0.15 GBytes 1.26 Gbits/sec
[272] 4.0- 5.0 sec 0.09 GBytes 0.74 Gbits/sec
[256] 4.0- 5.0 sec 0.15 GBytes 1.31 Gbits/sec
[280] 4.0- 5.0 sec 0.09 GBytes 0.75 Gbits/sec
[240] 4.0- 5.0 sec 0.17 GBytes 1.42 Gbits/sec
[264] 4.0- 5.0 sec 0.09 GBytes 0.74 Gbits/sec
[224] 4.0- 5.0 sec 0.09 GBytes 0.74 Gbits/sec
[248] 4.0- 5.0 sec 0.15 GBytes 1.32 Gbits/sec
[212] 4.0- 5.0 sec 0.09 GBytes 0.73 Gbits/sec
[SUM] 4.0- 5.0 sec 1.14 GBytes 9.76 Gbits/sec
[232] 5.0- 6.0 sec 0.14 GBytes 1.22 Gbits/sec
[240] 5.0- 6.0 sec 0.16 GBytes 1.37 Gbits/sec
[280] 5.0- 6.0 sec 0.09 GBytes 0.78 Gbits/sec
[212] 5.0- 6.0 sec 0.09 GBytes 0.77 Gbits/sec
[248] 5.0- 6.0 sec 0.16 GBytes 1.37 Gbits/sec
[ ID] Interval Transfer Bandwidth
[224] 5.0- 6.0 sec 0.09 GBytes 0.78 Gbits/sec
[256] 5.0- 6.0 sec 0.15 GBytes 1.28 Gbits/sec
[264] 5.0- 6.0 sec 0.09 GBytes 0.78 Gbits/sec
[272] 5.0- 6.0 sec 0.09 GBytes 0.77 Gbits/sec
[216] 5.0- 6.0 sec 0.09 GBytes 0.78 Gbits/sec
[SUM] 5.0- 6.0 sec 1.15 GBytes 9.91 Gbits/sec
[256] 6.0- 7.0 sec 0.14 GBytes 1.21 Gbits/sec
[280] 6.0- 7.0 sec 0.09 GBytes 0.79 Gbits/sec
[224] 6.0- 7.0 sec 0.09 GBytes 0.77 Gbits/sec
[232] 6.0- 7.0 sec 0.17 GBytes 1.45 Gbits/sec
[240] 6.0- 7.0 sec 0.15 GBytes 1.30 Gbits/sec
[212] 6.0- 7.0 sec 0.09 GBytes 0.77 Gbits/sec
[248] 6.0- 7.0 sec 0.15 GBytes 1.28 Gbits/sec
[216] 6.0- 7.0 sec 0.09 GBytes 0.77 Gbits/sec
[264] 6.0- 7.0 sec 0.09 GBytes 0.79 Gbits/sec
[272] 6.0- 7.0 sec 0.09 GBytes 0.78 Gbits/sec
[SUM] 6.0- 7.0 sec 1.15 GBytes 9.91 Gbits/sec
[232] 7.0- 8.0 sec 0.17 GBytes 1.43 Gbits/sec
[256] 7.0- 8.0 sec 0.14 GBytes 1.22 Gbits/sec
[216] 7.0- 8.0 sec 0.09 GBytes 0.77 Gbits/sec
[ ID] Interval Transfer Bandwidth
[272] 7.0- 8.0 sec 0.09 GBytes 0.77 Gbits/sec
[248] 7.0- 8.0 sec 0.16 GBytes 1.37 Gbits/sec
[280] 7.0- 8.0 sec 0.09 GBytes 0.78 Gbits/sec
[264] 7.0- 8.0 sec 0.09 GBytes 0.78 Gbits/sec
[240] 7.0- 8.0 sec 0.14 GBytes 1.24 Gbits/sec
[224] 7.0- 8.0 sec 0.09 GBytes 0.77 Gbits/sec
[212] 7.0- 8.0 sec 0.09 GBytes 0.76 Gbits/sec
[SUM] 7.0- 8.0 sec 1.15 GBytes 9.92 Gbits/sec
[280] 8.0- 9.0 sec 0.09 GBytes 0.79 Gbits/sec
[256] 8.0- 9.0 sec 0.16 GBytes 1.40 Gbits/sec
[240] 8.0- 9.0 sec 0.14 GBytes 1.17 Gbits/sec
[216] 8.0- 9.0 sec 0.09 GBytes 0.78 Gbits/sec
[272] 8.0- 9.0 sec 0.09 GBytes 0.77 Gbits/sec
[248] 8.0- 9.0 sec 0.16 GBytes 1.34 Gbits/sec
[232] 8.0- 9.0 sec 0.16 GBytes 1.36 Gbits/sec
[224] 8.0- 9.0 sec 0.09 GBytes 0.76 Gbits/sec
[212] 8.0- 9.0 sec 0.09 GBytes 0.76 Gbits/sec
[264] 8.0- 9.0 sec 0.09 GBytes 0.78 Gbits/sec
[SUM] 8.0- 9.0 sec 1.15 GBytes 9.91 Gbits/sec
[272] 9.0-10.0 sec 0.09 GBytes 0.74 Gbits/sec
[ ID] Interval Transfer Bandwidth
[248] 9.0-10.0 sec 0.15 GBytes 1.28 Gbits/sec
[212] 9.0-10.0 sec 0.09 GBytes 0.73 Gbits/sec
[232] 9.0-10.0 sec 0.15 GBytes 1.31 Gbits/sec
[224] 9.0-10.0 sec 0.09 GBytes 0.74 Gbits/sec
[280] 9.0-10.0 sec 0.09 GBytes 0.77 Gbits/sec
[240] 9.0-10.0 sec 0.17 GBytes 1.48 Gbits/sec
[264] 9.0-10.0 sec 0.09 GBytes 0.75 Gbits/sec
[256] 9.0-10.0 sec 0.16 GBytes 1.35 Gbits/sec
[216] 9.0-10.0 sec 0.09 GBytes 0.75 Gbits/sec
[SUM] 9.0-10.0 sec 1.15 GBytes 9.91 Gbits/sec
[248] 0.0-10.0 sec 1.55 GBytes 1.33 Gbits/sec
[240] 0.0-10.0 sec 1.54 GBytes 1.32 Gbits/sec
[256] 0.0-10.0 sec 1.52 GBytes 1.30 Gbits/sec
[232] 0.0-10.0 sec 1.59 GBytes 1.36 Gbits/sec
[216] 0.0-10.0 sec 0.89 GBytes 0.77 Gbits/sec
[272] 0.0-10.0 sec 0.89 GBytes 0.76 Gbits/sec
[280] 0.0-10.0 sec 0.90 GBytes 0.78 Gbits/sec
[224] 0.0-10.0 sec 0.89 GBytes 0.76 Gbits/sec
[212] 0.0-10.0 sec 0.88 GBytes 0.75 Gbits/sec
[264] 0.0-10.0 sec 0.90 GBytes 0.77 Gbits/sec
[ ID] Interval Transfer Bandwidth
[SUM] 0.0-10.0 sec 11.5 GBytes 9.91 Gbits/sec
Done.

So that's nearly 10G throughput (close as I could expect, anyway!) but why does it take 10x streams to equal that? I was under the impression that I would be able to get full throughput on a single connection.

 

[azev]

It has something to do with this

 

[coolrunnings82]

Booted off a Linux LiveCD called the Network Security Toolkit to test with iPerf there.... It appears it's a Windows or Windows Driver related issue. Getting 9+ Gbits/sec in Linux consistently.

@azev - Sorry I'm a bit confused how to interpret that link you sent. Any way you could explain?

 

[cesmith9999]

What are your adapter settings on the net cards?

Get-NetAdapterAdvancedProperty | fl

where we work we have a specific set of network card changes that we do
1) max receive/transmit buffers
2) increase the RSS queues

the rest is dependent on the specific card in the server.

Chris

 

[coolrunnings82]

@cesmith9999 I just tried 1 & 2 there with no measurable increase in speed. Here's the results of that command

 

[coolrunnings82]

I tried monkeying with most of the settings I found on the advanced tab in the nic properties in device manager. Here are my results:
Diabling Interrupt moderation got me above 3 gigabit
Changing Maximum Number of RSS Processes to 1 instead of the default of 16 and changing Maximum Number of RSS Queues to 1 instead of 8 got me to 3.78-4.4 gigabit.
Changing Send and Receive Buffers from 512 to 4096 didn't do much.
Turning off Virtual machine queues dropped performance substantially below 3 gigabit
Turning off RSS kept me above 3 gigabit even with interrupt moderation enabled. If I turned off interrupt moderation and RSS I got 3.6+ gigabit.

 

[cesmith9999]

what are your power settings with the server? what happens if you are set to MAX performance? and in the BIOS set to max performance?

Chris

 

[coolrunnings82]

Power settings in Windows were set to balanced. I set them to high performance and reran the test but got no improvement. I went into the BIOS and checked and the profile was set to PerformancePerWatt. I just set it to max performance and reran the test with the same results.

 

[cesmith9999]

when you are running these tests. are there any cores that are maxed out?

Chris

 

[coolrunnings82]

Nope, no maxed out cores. Downloaded the latest Intel drivers directly from Intel instead of Dell and set it for low latency profile in Device Manager and now I'm getting upwards of 4 gig! Still not the 10 I expected based on the Linux speed test though and isn't going to help me with the Windows 10 side of things since they disable all options in the driver there... Grrr.

 

[dwright1542]

I've got something of the same issue. We are evaluating VSAN (Starwind, SvSAN, etc) and both tech support teams are killing themselves why we can't get better performance. X520 adapters, C2100's and DL385's. Max I can just eek out is 4G.

 

[coolrunnings82]

Any other ideas?

 

[cesmith9999]

there is a settings issue some where.

just today I am in the middle of backing up 200 TB to a single server (cold storage) and I am getting 9.5 GB/sec transfers.

we have a slew of optimizations for the file server. and a few for the networking. I will see if I can share our config script.

Chris