I finally made the jump to 10GB on my nappit AIO boxes and thought I had followed all of the recommendations, but I am still getting gigabit speed ZFS sends using NC. The weird thing is iperf gives me 8.9 Gbits/sec throughput so I'm not sure if any of the tuneables will help me here. The pool is a stripe of 8 mirrors of 4TB HGST ultrastars.
This is going from one physical AIO box to another AIO box over the 10gb link. Everything along the way is configured with 9000 MTU and the switch is a cisco SG500XG-8F-8T
Additional hardware info below:
intel X520-da2 nics
intel/finisar SFP modules
2x E5-2690 v2
768 GB ECC ram
nappit VM has 16 cores and 64 GB ram
LSI 2116 HBA
Let me know if you have any suggestions for what to try. My iozone 1gb benchmarks are looking decent too so I dont think its pool speed:
This is going from one physical AIO box to another AIO box over the 10gb link. Everything along the way is configured with 9000 MTU and the switch is a cisco SG500XG-8F-8T
Code:
Iperf:
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-1.00 sec 876 MBytes 7.34 Gbits/sec
[ 4] 1.00-2.00 sec 1.03 GBytes 8.87 Gbits/sec
[ 4] 2.00-3.00 sec 1.03 GBytes 8.85 Gbits/sec
[ 4] 3.00-4.00 sec 1.04 GBytes 8.94 Gbits/sec
[ 4] 4.00-5.00 sec 1.05 GBytes 8.97 Gbits/sec
[ 4] 5.00-6.00 sec 1.04 GBytes 8.95 Gbits/sec
[ 4] 6.00-7.00 sec 1.04 GBytes 8.93 Gbits/sec
[ 4] 7.00-8.00 sec 1.04 GBytes 8.92 Gbits/sec
[ 4] 8.00-9.00 sec 1.04 GBytes 8.93 Gbits/sec
[ 4] 9.00-10.00 sec 1.04 GBytes 8.92 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-10.00 sec 10.2 GBytes 8.76 Gbits/sec sender
[ 4] 0.00-10.00 sec 10.2 GBytes 8.76 Gbits/sec receiver
Zfs send:
zfs send pool01-esx/datastore09a-hdd-backup@weekly-1517848527_2018.09.14.23.00.14 | pv | nc -w 20 10.10.10.10 9090
10.2GiB 0:02:21 [74.0MiB/s]
intel X520-da2 nics
intel/finisar SFP modules
2x E5-2690 v2
768 GB ECC ram
nappit VM has 16 cores and 64 GB ram
LSI 2116 HBA
Let me know if you have any suggestions for what to try. My iozone 1gb benchmarks are looking decent too so I dont think its pool speed:
Last edited: