Each VM on it's own iSCSI target - issue with backup

[katit]

I have 5 VMs running on single server. Used windows backup and all was well (as far as backup goes)

Now I built new FreeNAS server, connected it to HyperV machine with 40Gb card. And now all my VMs live on it's own iSCSI targets.

Performance great, I get ZFS snapshots and all seem to be working as expected. Except for backup.

I am not sure how "dirty" will be ZFS snapshot when it takes it on running VM, so just in case I want to have windows backup. But now those backups fail.
Because of "file being locked by another process". The only difference is that files now in iSCSI drive.

Did anybody have this issue? Any suggstion?

Error in backup of C:\HyperV-iSCSI\build-agent-1\Virtual Machines\5FA6A330-6603-4CC1-B97D-5D0C7C825D74.VMRS during enumerate: Error [0x80070020] The process cannot access the file because it is being used by another process.
Error in backup of C:\HyperV-iSCSI\svn\Virtual Machines\6858BF64-26FB-4592-95BC-37E291DA05AF.VMRS during enumerate: Error [0x80070020] The process cannot access the file because it is being used by another process.
Error in backup of C:\HyperV-iSCSI\dev-1\Virtual Machines\766CFF05-7074-42F4-9CD0-B24BEEE852FC.VMRS during enumerate: Error [0x80070020] The process cannot access the file because it is being used by another process.
Error in backup of C:\HyperV-iSCSI\jira\Virtual Hard Disks\JIRA-X_3C936EE4-8609-44B6-8BE3-A5026D8CAC3E.avhdx during enumerate: Error [0x80070020] The process cannot access the file because it is being used by another process.
Application backup
Writer Id: {66841CD4-6DED-4F4B-8F17-FD23F8DDC3DE}
Component: 5FA6A330-6603-4CC1-B97D-5D0C7C825D74
Caption : Online\BUILD-AGENT-1
Logical Path:
Error : 8078010D
Error Message : Enumeration of the files failed.

Detailed Error : 80070020
Detailed Error Message : (null)

 

[gea]

You cannot copy/backup open files so ZFS snapshots is a workaround.

ZFS snapshots are in a filesystem state like a sudden powerloss. While the ZFS filesystem itself is always safe due CopyonWrite, a guest VM filesystem is not. A corrupt guest filesystem can happen unless you shut down the VM prior a snap or set the VM to a backup safe state.

 

[unwind-protect]

Yes, to safely backup the raw device under a filesystem that filesystem has to be unmounted or mounted read-only. Otherwise you run a risk of reversing carefully sequenced disk transactions.

 

[alaricljs]

I work in a shop with several hundred Windows VMs stored as files on ZFS over NFS (vmware). Same basic issue around filesystem safety and snapshots. We use ZFS snapshots in preference to vmware and haven't had an issue with corruption after a restore or rollback. Not even once.

 

[gea]

With VMware ESXi you can go around rare but possible VM corruptions in ZFS snapshots by creating an ESXi snapshot (hot memory or quiesce) prior the ZFS snap. This ESXi snap can be destroyed when the ZFS snap is created.

For a safe rollback, first restore the ZFS snap, then the included ESXi snap. This can happen under scriptcontrol or embedded in ZFS autosnap scripts (I have done such in my napp-it web-gui)