Ran through the network tuning and still the same.
I also check out ZIL usage and I do see the IO hit the ZIL when I have sync=always but it drops to 35MB/s? That makes zero sense too me.
You can see in the graph (I only inluded one spindle to show what i'm talking about da5 but the I/O looked the same on all spindles during this write) and the two Mirrored SSD's for ZIL, da6 and da7
You can clearly see when sync is enabled that the I/O hits the ZIL and drops on the spindles, but drops the actual throughput to 35MB/s from 160MB/s when sync=always.
I turn sync=standard or disabled and you can clearly see that ZIL is not used and the I/O increases on the spindles. In the middle of the transfer I enabled sync=always then disabled after a short bit.
So in conclusion, I don't believe it's SMB protocol or network that's the issue at all.
_alex, to answer your question, the testing i've seen done and output were with the same 6 x 4TB RAIDZ2 VDEV with no ZIL and they were getting 400MB/s Writes and over 1GB/s Reads.