10Gbe performance issue in FreeNas 11

[Mattias]

I should probably explain my setup before explaining my performance issue. I have a server with FreeNas11 on it and a Mellanox ConnectX-2 card and another server with two Mellanox ConnectX-2 cards bridged via vyOS in Hyper-V on Server 2016 and my Client is a normal gaming rig with a Mellanox ConnectX-2 card in it.

When i check with iperf i get 9Gbit from my FreeNas to my Client which is just fine but when i have iperf as server on the FreeNas box and check speed from my client to the FreeNas box i only get around 3Gbit which bottlenecks my transfers to the storage. All cards are setup with jumbo frames 9000, if i don't change it from the defaults on the FreeNas box i only get 1.5Gbit on average.

I figured maybe it was a problem with Mellanox cards on freenas so i put in my Chelsio CC2-S320E-SR card in there but it has the same 3Gbit max to the server and from the server i get around 8Gbit.

I installed windows 10 on another disk on the freenas server and when i use the latest mellanox drivers there i get 9Gbit on both ways so it doesn't seem to be a hardware issue since it works in windows but not on FreeNas.

Both the FreeNas server and my Bridge server has the S2600CP motherboard, freenas has a E5-2620 v1 and bridge a E5-2670 v2.

EDIT: I updated to the latest stable FreeNas 11 build as of today and it did not make a difference.

 

[Rand__]

what about the gaming client ? Can you boot a linux live cd and test?

 

[Mattias]

not sure what u want me to test, if i get degraded performance on my client if using linux?

 

[Rand__]

Yes - maybe you need some windows tweaking to get the best performance and it might be better in linux ootb

 

[Mattias]

alright i think i have it solved, got the tip from freenas forum to add some tunables for the network part.

kern.ipc.maxsockbuf=16777216
net.inet.ip.intr_queue_maxlen=2048
net.inet.tcp.recvbuf_max=16777216
net.inet.tcp.recvspace=4194304
net.inet.tcp.recvbuf_inc=524288
net.inet.tcp.sendbuf_max=16777216
net.inet.tcp.sendspace=2097152
net.inet.tcp.sendbuf_inc=32768
net.route.netisr_maxqlen=2048

now im getting around 8Gbit speed on the receiving side and 9 on send so im happy but will play around and see if i can get send speed higher still.

for anyone intrested this is the topic settings are taken from: 10 Gig Networking Primer

 

[Rand__]

Lol, thats actually an old post of mine

Still it wont hurt to double check windows tuning as well - similar TCP values (eg buffers et al) buried in adapter settings. Especially queues & usable cpu cores-