Interesting results.
Would be curious to get a specific characterization of the risks associated with async write. Given the significant performance benefit there is significant pressure to use it. i know - data loss - but what, exactly? Is it just the integrity of a single file? Or, like in the old days of ext2/ext3 filesystems is the filesystem metadata itself at risk?
Also, for your iSCSI tests, were you using file-mode or block mode targets? I did some comparatives a couple of years ago on Solaris 10. Less structured than yours but clearly indicating that the Solaris that suggested Sun/Oracle's SMB implementation is just aweful over 10Gbe links (applies to OI/etc, since they are fundamentally the same). I saw file-mode iSCSI results similar to yours, but block mode iSCSI blew SMB away for both reads and writes.
I have used volumebased LUs for my benchmarks but have added to a comparison to filebased LUs.
I prefer them due to much easier copy/clone/restore from snap handling.
The difference is quite small with the volumebased ones slightly faster.
I have also added a comparison of Raid-Z1 with 4 datadisks vs 4 x vdev of basic disks
(same number of datadisks, same pool size).
about risk of async write:
Regarding the ZFS filesystem, there is no risk. Due to Copy on Write, a write is done correctly or not.
The problem is your application or VM on ESXi. If you do multiple transaktions in a database one by one or if you have an OS in ESXi that do some filesystem optimizations in RAM it can happen in case of a power failure, that some transaktions are on disk, others not (inconsistent database) or that your VM filesystem is corrupt.
From ZFS view, everything is ok, Your ZFS filesystem is valid but may contain garbage.
The critical timeframe is about 5s. These 5s of last writes may be lost.
I have a simple rule:
Whenever I can accept last 5s of writes or my VM is quite uncritical against a sudden power-loss (most are), I disable sync and do backups. For some services, sync-write is a must.
