Super Low 10G Performance

[PGlover]

s

there is a settings issue some where.

just today I am in the middle of backing up 200 TB to a single server (cold storage) and I am getting 9.5 GB/sec transfers.

we have a slew of optimizations for the file server. and a few for the networking. I will see if I can share our config script.

Chris

Looking forward to the config script. I have the intel x520-da2 NIC card in my WIN 2012 Server R2 and only able to get 3.5 - 4 Gbps based on using the settings from the other forum member (coolrunnings82).

 

[cesmith9999]

#these are our file server settings in general - taken from the performance tuning guides on technet.

new-itemproperty hklm:system\currentcontrolset\services\lanmanserver\parameters -name SrvMaxThreadsPerQueue -value 100
set-itemproperty hklm:system\currentcontrolset\services\lanmanserver\parameters -name SrvMaxThreadsPerQueue -value 100
new-itemproperty hklm:system\currentcontrolset\services\lanmanserver\parameters -name MaxThreadsPerQueue -value 100
set-itemproperty hklm:system\currentcontrolset\services\lanmanserver\parameters -name MaxThreadsPerQueue -value 100
new-itemproperty hklm:system\currentcontrolset\services\lanmanserver\parameters -name TreatHostAsStableStorage -value 1
set-itemproperty hklm:system\currentcontrolset\services\lanmanserver\parameters -name TreatHostAsStableStorage -value 1


new-itemproperty hklm:system\currentcontrolset\control\filesystem -name NtfsMftZoneReservation -value 3
set-itemproperty hklm:system\currentcontrolset\control\filesystem -name NtfsMftZoneReservation -value 3
new-itemproperty hklm:system\currentcontrolset\control\filesystem -name NtfsDisableLastAccessUpdate -value 0
set-itemproperty hklm:system\currentcontrolset\control\filesystem -name NtfsDisableLastAccessUpdate -value 0
new-itemproperty hklm:system\currentcontrolset\control\filesystem -name MaximumTunnelEntries -value 50
set-itemproperty hklm:system\currentcontrolset\control\filesystem -name MaximumTunnelEntries -value 50

new-itemproperty 'hklm:system\currentcontrolset\control\Session Manager\Memory Management' -name DisablePagingExecutive -value 1
set-itemproperty 'hklm:system\currentcontrolset\control\Session Manager\Memory Management' -name DisablePagingExecutive -value 1

new-itemproperty 'hklm:system\currentcontrolset\control\Session Manager\Executive' -name AdditionalCriticalWorkerThreads -value 100
set-itemproperty 'hklm:system\currentcontrolset\control\Session Manager\Executive' -name AdditionalCriticalWorkerThreads -value 100

new-itemproperty hklm:system\currentcontrolset\Services\TCPIP\parameters -name TcpNumConnections -value 16777214
set-itemproperty hklm:system\currentcontrolset\Services\TCPIP\parameters -name TcpNumConnections -value 16777214

#NIC Settings

netsh int tcp set heuristics disabled
netsh int tcp set global autotuninglevel = restricted

$NICName = (get-netadapter -Name * -Physical | where {$_.Status -eq "Up" -and $_.LinkSpeed -eq "10 Gbps"}).Name

Set-NetAdapterAdvancedProperty -Name $NICName -DisplayName "Flow Control" -DisplayValue "Disabled"
Set-NetAdapterAdvancedProperty -Name $NICName -DisplayName "Interrupt Moderation Rate" -DisplayValue "Adaptive"
Set-NetAdapterAdvancedProperty -Name $NICName -DisplayName "Receive Buffers" -DisplayValue "4096"
Set-NetAdapterAdvancedProperty -Name $NICName -DisplayName "Transmit Buffers" -DisplayValue "16384"
Set-NetAdapterAdvancedProperty -Name $NICName -DisplayName "Starting RSS CPU" -DisplayValue "0"
Set-NetAdapterAdvancedProperty -Name $NICName -DisplayName "Maximum Number of RSS Processors" -DisplayValue "16"
Set-NetAdapterAdvancedProperty -Name $NICName -DisplayName "Maximum Number of RSS Queues" -DisplayValue "16 Queues"

Get-NetAdapterAdvancedProperty -Name $NICName

#Chris

 

[PGlover]

Anyone else tried these configuration settings yet?

 

[PGlover]

#these are our file server settings in general - taken from the performance tuning guides on technet.

new-itemproperty hklm:system\currentcontrolset\services\lanmanserver\parameters -name SrvMaxThreadsPerQueue -value 100
set-itemproperty hklm:system\currentcontrolset\services\lanmanserver\parameters -name SrvMaxThreadsPerQueue -value 100
new-itemproperty hklm:system\currentcontrolset\services\lanmanserver\parameters -name MaxThreadsPerQueue -value 100
set-itemproperty hklm:system\currentcontrolset\services\lanmanserver\parameters -name MaxThreadsPerQueue -value 100
new-itemproperty hklm:system\currentcontrolset\services\lanmanserver\parameters -name TreatHostAsStableStorage -value 1
set-itemproperty hklm:system\currentcontrolset\services\lanmanserver\parameters -name TreatHostAsStableStorage -value 1


new-itemproperty hklm:system\currentcontrolset\control\filesystem -name NtfsMftZoneReservation -value 3
set-itemproperty hklm:system\currentcontrolset\control\filesystem -name NtfsMftZoneReservation -value 3
new-itemproperty hklm:system\currentcontrolset\control\filesystem -name NtfsDisableLastAccessUpdate -value 0
set-itemproperty hklm:system\currentcontrolset\control\filesystem -name NtfsDisableLastAccessUpdate -value 0
new-itemproperty hklm:system\currentcontrolset\control\filesystem -name MaximumTunnelEntries -value 50
set-itemproperty hklm:system\currentcontrolset\control\filesystem -name MaximumTunnelEntries -value 50

new-itemproperty 'hklm:system\currentcontrolset\control\Session Manager\Memory Management' -name DisablePagingExecutive -value 1
set-itemproperty 'hklm:system\currentcontrolset\control\Session Manager\Memory Management' -name DisablePagingExecutive -value 1

new-itemproperty 'hklm:system\currentcontrolset\control\Session Manager\Executive' -name AdditionalCriticalWorkerThreads -value 100
set-itemproperty 'hklm:system\currentcontrolset\control\Session Manager\Executive' -name AdditionalCriticalWorkerThreads -value 100

new-itemproperty hklm:system\currentcontrolset\Services\TCPIP\parameters -name TcpNumConnections -value 16777214
set-itemproperty hklm:system\currentcontrolset\Services\TCPIP\parameters -name TcpNumConnections -value 16777214

#NIC Settings

netsh int tcp set heuristics disabled
netsh int tcp set global autotuninglevel = restricted

$NICName = (get-netadapter -Name * -Physical | where {$_.Status -eq "Up" -and $_.LinkSpeed -eq "10 Gbps"}).Name

Set-NetAdapterAdvancedProperty -Name $NICName -DisplayName "Flow Control" -DisplayValue "Disabled"
Set-NetAdapterAdvancedProperty -Name $NICName -DisplayName "Interrupt Moderation Rate" -DisplayValue "Adaptive"
Set-NetAdapterAdvancedProperty -Name $NICName -DisplayName "Receive Buffers" -DisplayValue "4096"
Set-NetAdapterAdvancedProperty -Name $NICName -DisplayName "Transmit Buffers" -DisplayValue "16384"
Set-NetAdapterAdvancedProperty -Name $NICName -DisplayName "Starting RSS CPU" -DisplayValue "0"
Set-NetAdapterAdvancedProperty -Name $NICName -DisplayName "Maximum Number of RSS Processors" -DisplayValue "16"
Set-NetAdapterAdvancedProperty -Name $NICName -DisplayName "Maximum Number of RSS Queues" -DisplayValue "16 Queues"

Get-NetAdapterAdvancedProperty -Name $NICName

#Chris

Chris.. Do you have Jumbo Frames turned on? If yes, what is the size?

 

[cesmith9999]

We do not have jumbo frames turned on.

I ask occasionally. With over 500 switches that we would have to modify, it is not a simple request to ask.

Chris

 

[Chuntzu]

Anyone else tried these configuration settings yet?

I have and they work great. I have also added jumbo frames and that helps as well. Especially at 10gb/40gb speeds. Thank you for posting these commands as well, really speeds up applying them to many hosts.

 

[coolrunnings82]

Wow, those settings are interesting and helpful. Thanks for sharing them. They didn't fix my bottleneck issue but they sure did make everything else way more responsive on the network! It's interesting: after setting those settings up and rebooting, I can saturate the 10G connection with 3 simultaneous streams, just not full 10G with 1 stream. Before it took 5x. The streams are much more consistent as well. I'm super curious what this 2-3gb/s limiting is caused by! I wonder if Intel's support would be of any help...

 

[coolrunnings82]

So I checked with the command netstat -nt and every connection says InHost. From what I read, that means none of the connections are being offloaded. netsh int tcp show global shows Chimney Offload State enabled. Would that be why the connections seem to be limited to 2ish gigabit? What else can I do to get the connections offloaded? Could someone who has 10G working and is getting proper bandwidth throughput on a single stream file-copy process run netstat-nt while doing a large file copy and see if they are getting offloading working properly according to netstat -nt?

UPDATE 1: I just found that if Hyper-V is enabled, TCP Offloading gets disabled. Removing Hyper-V and checking status here... More info on this here:
https://support.microsoft.com/en-us/kb/951037

UPDATE 2: Removing the Hyper-V role did nothing regarding offloading or performance. Back to the drawing board.

 

[cesmith9999]

Do you have a DC role on either server?

Chris

 

[coolrunnings82]

No DC.

 

[dwright1542]

Both of my possible storage vendors still are banging their heads on this one. FusionIO card can push 120,000 IOPS, when it goes over iSCSI, it's 50-60k IOPS. SATA SSD array can do 45k, over the 10G network, 20k.

It's nearly half speed.

C2100 ESXi as clients to a T320 with the FusionIO Duo, X520 cards, with or without switch doesnt matter. The whole thing is just odd.