Here's a table of safety for powerloss by datastore type for ZFS backed NFS and iSCSI storage, and also guest VM connected/mounted storage.
Why you should care:
Operating Systems filesystems, and databases will perform sensitive writes in "write through" mode in a synchronous manner, which will wait for the write to succeed and be written in stable storage before returning to the calling code. These "write through" mode writes are tagged as such to bypass disk write caches and save the data directly to disk.
If important filesystem data is written and the "write through" (sync) request isn't honored by the disks or raid controller then guest filesystem corruption or fatal filesystem damage is possible (and probable) with a loss of power. Traditional true "hardware" Raid controllers will usually lie to the requesting system, but instead contain a battery backed cache to preserve important data until power is restored. Once power is restored the cache will write out its contents to stable storage disks preserving filesystem integrity.
Datastore connections from ESXi to ZFS storage are used to save and access virtual machine disk files (VMDK files) which contain the filesystems and data for each virtual machine. Virtual Machines disk files (VMDKs) in ESXi are often written to in a datastore that is mounted over NFS or iSCSI network protocols.
When guest writes are to a VMDK stored on a ZFS filesystem that is set to sync=disabled for the ESXi NFS datastore:
- the Virtual machine OS writes some sensitive filesystem data that ESXi sends
- It's received via NFS or iSCSI on Solaris, FreeBSD, ZoL
- Sensitive write data ends up in volatile memory in the asynchronous data queue on the receiving end in ZFS (or in the iSCSI target's volatile write cache).
- Everything is fine, until power is lost and it's not.
- The ZFS filesystem is fine that the VMDK files are stored on.
- But the VMDK file's guest OS filesystem could have missed vital data being written and could now be corrupted.
Disclaimers:
1) I define "Corrupted" as the filesystem becoming corrupted within the VMDK from a guest's perspective.
2) Async writes 'lost' are those that have not been commited to stable storage. (eg writes in the writeback cache pending flush or flushed async lazy writes)
If you have updates or corrections please post below and I'll updated the image.
Guest connected iSCSI with write cache enabled disks (and zfs sync=standard on the iSCSI lu) are more similar to a non-virtualized OS connected to a single physical disk with a non-battery backed cache. All file writes to these disks are written asynchronously, and unless written in a "write-through" mode by the OS or database/program, any in-flight data will be lost if the power is lost.
Note: If using VM guest iSCSI for data disks you must ensure that the iSCSI write cache settings in the guest are setup correctly and are supported. Otherwise "write-through" sensitive writes may also be lost to power loss causing potential data loss/and or filesystem corruption.
Note 2: ESXi VM snapshots will not cover guest iSCSI filesystems as they are external to the VMDK file of the VM and a ZFS snapshot must also be coordinated.
Note 3. Quiescence of guest mounted iSCSI disks is required for snapshots. Otherwise snapshots of a powered on guest mounted iSCSI disk from ZFS may result in inconsistent data (similar to corrupted filesystem). Powering down the VM which has mounted the iSCSI disk is required or some other method of guaranteeing no-writes will be made. (ZFS filesystem snap will be fine, but upon mounting the snap the filesystem may be in an inconsistent state when iSCSI disk snap is mounted from client OS).
Why you should care:
Operating Systems filesystems, and databases will perform sensitive writes in "write through" mode in a synchronous manner, which will wait for the write to succeed and be written in stable storage before returning to the calling code. These "write through" mode writes are tagged as such to bypass disk write caches and save the data directly to disk.
If important filesystem data is written and the "write through" (sync) request isn't honored by the disks or raid controller then guest filesystem corruption or fatal filesystem damage is possible (and probable) with a loss of power. Traditional true "hardware" Raid controllers will usually lie to the requesting system, but instead contain a battery backed cache to preserve important data until power is restored. Once power is restored the cache will write out its contents to stable storage disks preserving filesystem integrity.
Datastore connections from ESXi to ZFS storage are used to save and access virtual machine disk files (VMDK files) which contain the filesystems and data for each virtual machine. Virtual Machines disk files (VMDKs) in ESXi are often written to in a datastore that is mounted over NFS or iSCSI network protocols.
When guest writes are to a VMDK stored on a ZFS filesystem that is set to sync=disabled for the ESXi NFS datastore:
- the Virtual machine OS writes some sensitive filesystem data that ESXi sends
- It's received via NFS or iSCSI on Solaris, FreeBSD, ZoL
- Sensitive write data ends up in volatile memory in the asynchronous data queue on the receiving end in ZFS (or in the iSCSI target's volatile write cache).
- Everything is fine, until power is lost and it's not.
- The ZFS filesystem is fine that the VMDK files are stored on.
- But the VMDK file's guest OS filesystem could have missed vital data being written and could now be corrupted.
Disclaimers:
1) I define "Corrupted" as the filesystem becoming corrupted within the VMDK from a guest's perspective.
2) Async writes 'lost' are those that have not been commited to stable storage. (eg writes in the writeback cache pending flush or flushed async lazy writes)
If you have updates or corrections please post below and I'll updated the image.
Guest connected iSCSI with write cache enabled disks (and zfs sync=standard on the iSCSI lu) are more similar to a non-virtualized OS connected to a single physical disk with a non-battery backed cache. All file writes to these disks are written asynchronously, and unless written in a "write-through" mode by the OS or database/program, any in-flight data will be lost if the power is lost.
Note: If using VM guest iSCSI for data disks you must ensure that the iSCSI write cache settings in the guest are setup correctly and are supported. Otherwise "write-through" sensitive writes may also be lost to power loss causing potential data loss/and or filesystem corruption.
Note 2: ESXi VM snapshots will not cover guest iSCSI filesystems as they are external to the VMDK file of the VM and a ZFS snapshot must also be coordinated.
Note 3. Quiescence of guest mounted iSCSI disks is required for snapshots. Otherwise snapshots of a powered on guest mounted iSCSI disk from ZFS may result in inconsistent data (similar to corrupted filesystem). Powering down the VM which has mounted the iSCSI disk is required or some other method of guaranteeing no-writes will be made. (ZFS filesystem snap will be fine, but upon mounting the snap the filesystem may be in an inconsistent state when iSCSI disk snap is mounted from client OS).
Last edited: