Slow Network speed between identical Intel 10Gb cards

[RyuHimora]

I have a newly built server (Gigabyte X99 UD4P, Xeon E5 2650 v4, 64GB Samsung ECC RAM) and am trying to set it up as a NAS for my workstation (Gigabyte Aorus Elite AM4, R7 5700G, 32GB RAM). The NAS solution is a TrueNAS Core VM running under Hyper-V, but I've experienced the same problems even when TrueNAS was installed on bare metal. Both systems have the Intel 10gb AF DA 2 Port Server Adapter which took some convincing on both ends to get drivers installed, but they both see each other and report a 10gb network link. They are connected by a copper Direct Attach Cable, no ethernet involved as far as I can tell.

However, the maximum network speed is 2.6-2.8Gb. File transfers onto the NAS top out at 1.4-1.6Gb while iPerf3 at 4 parallel connections tops out at 2.8Gb and goes down with more connections. I've enabled Jumbo Frames at 9000 to no effect. Intel's troubleshooting guide seems to have a complete blindspot for the issue. What troubleshoots steps should I take to track down the problem?

 

[DavidWJohnston]

To help pinpoint the issue, you might want to try:

• Try running iperf3 in both directions (The -R flag on the client side will reverse it) and see if it's different.
• Have a look at your CPU utilization on both sides. Are you maxing out any core(s) when iperf3 is running?
• Is your workstation running Windows? Have a look and see if a security product (ex. AV) has high utilization during high I/O
• Check jumbos are active - ping 192.168.1.2 -f -l 8900 and see if you get a successful ping out and back. (use your IP)
• Did you also enable other features like Receive Side Scaling and checksum offloading? Sometimes these can help, but I know from past experience enabling LRO (large receive offload) actually kills the performance instead of improving it. Sometimes drivers have strange bugs.

Testing in my lab on a server slower than yours (X79 vintage), I get about 5Gbps on a single iperf3 stream when I max out a core. I would expect you to be able to beat that number.

 

[i386]

Did you run iperf on a windows machine (hyper v host, workstation)?
Then try to run iperf from a linux live cd or use another tool like ntttcp for testing the network

 

[fohdeesha]

I have a newly built server (Gigabyte X99 UD4P, Xeon E5 2650 v4, 64GB Samsung ECC RAM) and am trying to set it up as a NAS for my workstation (Gigabyte Aorus Elite AM4, R7 5700G, 32GB RAM). The NAS solution is a TrueNAS Core VM running under Hyper-V, but I've experienced the same problems even when TrueNAS was installed on bare metal. Both systems have the Intel 10gb AF DA 2 Port Server Adapter which took some convincing on both ends to get drivers installed, but they both see each other and report a 10gb network link. They are connected by a copper Direct Attach Cable, no ethernet involved as far as I can tell.

However, the maximum network speed is 2.6-2.8Gb. File transfers onto the NAS top out at 1.4-1.6Gb while iPerf3 at 4 parallel connections tops out at 2.8Gb and goes down with more connections. I've enabled Jumbo Frames at 9000 to no effect. Intel's troubleshooting guide seems to have a complete blindspot for the issue. What troubleshoots steps should I take to track down the problem?

where'd you get the cards? I've had similar results when I ended up with intel clones/knockoffs off ebay

 

[RyuHimora]

To help pinpoint the issue, you might want to try:

Running iperf from server to workstation actually yielded a surprising improvement, it went up to 5 Gbps. Consistent across several tests.

No anomalous CPU utilization on either side. Workstation increases by about 4%, Server by about 12%.

I only have the default Windows AV enabled.

So, pinging with jumbo frames is showing very strange results. Trying from workstation to server using ping my.ip.address -f -l 8900 doesn't work at all, it just times out. However, from server to workstation, it appears to work correctly however there is 4-8% packet loss. I'm assuming that's a root cause of the issue, but I have limited resources to troubleshoot. I tried swapping out the DAC, no change.

Receive Side Scaling and Checksum offloading were on by default. LRO doesn't seem to exist on either the workstation or server.

 

[DavidWJohnston]

When you're getting a ping timeout, that means the outbound frame was potentially successful, but the inbound (response coming back in) frame was not. If it was being dropped at the sender, you'd get an error "Packet needs fragmentation but DF bit not set" not a timeout. In your case it can't be a switch because you don't have one.

The max allowed outbound frame size is controlled by the OS's MTU setting. The max allowed inbound frame is controlled by the NIC hardware setting. BOTH of these on both sides need to be set to jumbo.

By your description, it sounds like the MTU on your Windows machine is not set to 9000, but your NIC advanced setting is. Usually this is done automatically, but who knows, check it anyway. On the server, you may need to set the MTU via the TrueNAS GUI.

On windows: How To: Change and Check Windows MTU Size
To see a path's MTU in FreeBSD, I think this command will work: route get 192.168.1.2

As far as the packet drops, I'm not so sure. Try with normal size pings and see if the same thing happens. See if you can catch it in a packet trace at either end. There might be some useful ICMP messages present.

Edit: Actually, I think I got the direction wrong - It's probably the TrueNAS not being set to MTU 9000. Your Windows box might be OK. Here's where I set it on mine: