STHers - I'm really struggling with this one and could really use your help. I've been stuck on this for 2 weeks and have an urgent need to finish this system build.
Summary: I need to set up a FreeNAS box as an ESXi NFS datastore. For data integrity, I want to use sync=enabled. To "win back" at least some of the performance, I want to use striped SLOGs and vdevs. However, no matter how many SSDs I throw at the problem, I can't get ESXi NFS writes higher than 250 MB/s. There appears to be a bottleneck when I have sync writes with the smaller block sizes that NFS tends to use.
Box #1
VMware ESXi 6.0 U2
Supermicro X9SRE-F, E5-1650V2, 16GB ECC memory
Intel X710-DA4 10GbE
1 local SSD with 1 test Win 8.1 VM VMDK
Box #2
FreeNAS 9.10.2 U2
Supermicro X9SRE-F, E5-1650V2, 64GB ECC memory
LSI 9305-16i
Intel X710-DA4 10GbE
1 pool:
- 3 x Intel DC S3710 200GB striped SLOG (overprovisioned so only 16 GB is usable)
- 3 x Samsung SSD 850 Pro 1TB striped
- atime=off, sync=always
Tests:
- NFS test: VM runs Iometer 2MB sequential write test on drive that's on the NFS datastore
- Local disk test: FreeNAS box runs "dd" test copying 1GB of data from RAM disk to pool
Observations from NFS test:
In the sync=always cases, it's especially strange watching the 1 SLOG + 1 pool drive push 250 MB/s each drive, then adding drives and watching the 250 MB/s spread evenly across the drives instead of the new drives accepting extra writes.
Any ideas what I'm doing wrong?
Summary: I need to set up a FreeNAS box as an ESXi NFS datastore. For data integrity, I want to use sync=enabled. To "win back" at least some of the performance, I want to use striped SLOGs and vdevs. However, no matter how many SSDs I throw at the problem, I can't get ESXi NFS writes higher than 250 MB/s. There appears to be a bottleneck when I have sync writes with the smaller block sizes that NFS tends to use.
Box #1
VMware ESXi 6.0 U2
Supermicro X9SRE-F, E5-1650V2, 16GB ECC memory
Intel X710-DA4 10GbE
1 local SSD with 1 test Win 8.1 VM VMDK
Box #2
FreeNAS 9.10.2 U2
Supermicro X9SRE-F, E5-1650V2, 64GB ECC memory
LSI 9305-16i
Intel X710-DA4 10GbE
1 pool:
- 3 x Intel DC S3710 200GB striped SLOG (overprovisioned so only 16 GB is usable)
- 3 x Samsung SSD 850 Pro 1TB striped
- atime=off, sync=always
Tests:
- NFS test: VM runs Iometer 2MB sequential write test on drive that's on the NFS datastore
- Local disk test: FreeNAS box runs "dd" test copying 1GB of data from RAM disk to pool
Observations from NFS test:
- If sync=always, the test maxes out at roughly 250MB/s.
- I get roughly the same results with the following sets of drives:
- 1 Intel SSD SLOG + 1 Samsung SSD pool
- 2 Intel SSDs in a striped SLOG + 2 Samsung SSDs in a striped pool
- 3 Intel SSDs in a striped SLOG + 3 Samsung SSDs in a striped pool
- 1 Intel SSD SLOG + 1 Samsung SSD pool
- Modifying the Iometer test to increase QD from 1 to 4 or even 32 barely increases the throughput
- If sync=disabled, the 3/3 combo can write 1050 MB/s.
- If sync=enabled, the output blocksize of the dd test has a significant effect on the throughput and using extra SSDs only helps at the larger block sizes (see below)
- Here are the results with 1 Intel SSD SLOG + 1 Samsung SSD pool:
- obs=1024k: 304 MB/s
- obs=512k: 293 MB/s
- obs=256k: 281 MB/s
- obs=128k: 258 MB/s
- obs=64k: 191 MB/s
- obs=32k: 128 MB/s
- obs=16k: 78 MB/s
- obs=1024k: 304 MB/s
- Here are the results with 3 Intel SSDs in a striped SLOG + 3 Samsung SSDs in a striped pool:
- obs=1024k: 690 MB/s
- obs=512k: 535 MB/s
- obs=256k: 417 MB/s
- obs=128k: 247 MB/s
- obs=64k: 189 MB/s
- obs=32k: 127 MB/s
- obs=16k: 77 MB/s
- obs=1024k: 690 MB/s
- Here are the results with 1 Intel SSD SLOG + 1 Samsung SSD pool:
- If sync=disabled, the 3/3 combo can write 1500 MB/s, even at obs=16k.
In the sync=always cases, it's especially strange watching the 1 SLOG + 1 pool drive push 250 MB/s each drive, then adding drives and watching the 250 MB/s spread evenly across the drives instead of the new drives accepting extra writes.
Any ideas what I'm doing wrong?