40 GbE on Win10 - Only getting ~10Gb/s

[jro]

I just put an Intel XL710-QDA2 NIC in my Windows 10 machine (TR 2990WX + 128GB RAM) and have it directly connected via QSFP+ DAC to a Chelsio T580-LP-CR in my FreeNAS system. Running iperf3, I only get ~10Gbit/s between the two machines (regardless of which is the host).

Interestingly enough, if I run iperf on the loopback address on the Windows machine, I only get ~11.5Gbit/s... I booted the machine into Ubuntu and got spikes up t0 30Gb/s in iperf, but lots of retransmits and things settled around 15Gb/s. iperf to the 127.0.0.1 under Ubuntu gave me the full 65Gb/s I'd expect.

I've got all the network tuning on the FreeNAS side in place per https://calomel.org/freebsd_network_tuning.html

I also ran all the tunings in Windows suggested here: https://forums.servethehome.com/index.php?threads/super-low-10g-performance.8394/page-2#post-76577

All the buffers/rss queues on the Windows driver are maxed, running v24 of the driver software. I've confirmed in Intel PROSet that the NIC is running at full PCIe x8.

Any ideas? I'm losing my mind trying to get this thing going... I've got another DAC cable on order thinking I may have kinked it too tightly while running it through the rack?

edit: MTU 9000 on both ends too... Also, if I do iperf with -P 4, I get like ~15-20Gbit/s. Going beyond -P 4 doesn't seem to help things.

 

[jro]

I wanted to provide an update on this after some tinkering on my end. If I run multiple iperf instances, I can get ~30Gb/s aggregate. I think SMB-Direct (SMB over RDMA) would be a good solution, but it's not supported in Samba yet, so I'm out of luck there.

SMB multichannel is supported, in Samba, so I'm going to try getting QSFP+ to 4x SFP+ breakout cables for both my Win10 machine and the FreeNAS system, run them all through my SFP+ switch and see if multichannel lets me get faster transfers. Samba's SMB multichannel implementation requires each interface to run on a seperate network, so I'll just have to create a bunch of /30 subnets. I'm hoping that when it's running over multichannel, SMB is multi-threaded and I can get faster speeds.

 

[mb300sd]

QSFP breakouts don't work with NICs, only with switches. You'd have to put 2x 2 port NICs in to get multichannel at 40G. Maybe you could set up 4 ip addresses on the 40G interface and see if that works. Or try 4 VLAN interfaces so it shows up as 4 interfaces to most software.

 

[jro]

Well crap... I guess you saved me $100 in cables. I'm not sure how to set up multiple IPs on the 40G interface, but I'll look into that as well. I appreciate the input.

EDIT: Okay, even more tinkering. I got 4 IPs, each on their own subnet, on both the Windows side and the FreeNAS side and enabled SMB multichannel. Windows forms the multichannel connection, but it reports the NICs as not being RSS capable in Get-SmbMultichannelConnection. However, in Get-SmbClientNetworkInterface and Get-NetAdapterrss, it DOES say they're RSS capable... I'm going to dig into this a bit more. All symptoms point to RSS not working properly, so hopefully this will lead to a resolution.

 

[kiteboarder]

Sorry if I'm stating something you already know, but you've got to have the most expensive Windows 10 in order to get all of the RDMA stuff to work: Windows 10 Pro for Workstations

Either that or Windows Server...