TIL: Snapshots. Don't let them get too big!

[sd11]

Long story short. I forgot I had taken a snapshot on a Win Server 2016 VM in esxi. I removed the snapshot forgetting that there is a 6TB disk attached. The VM is relatively low when it comes to IO changes. Just a small file server. It's still is taking forever to remove. 6% in the past 20 hours. It's now at 65%. The first 50% was slow but tolerable. It must have been consolidating the disk on the NVME drive. The 6TB is on a LSI 9266 R1 array. Every VM is shutdown.

In hindsight, I should have done a V2V.

By my estimate, it should be done by Friday and I can boot it back up.

Anyone else run into this problem?

 

[Haitch]

Had a customer forget about a snapshot - the LUN it was on ran out of space and the snapshot became corrupted and unrecoverable, even by VMware support. They had to revert to the pre-snap shot vmdk. 18 months of data lost.

 

[gea]

ESXi snaps are growing delta files with all modifications.
This is why this is a shortterm snap where you should not create more than one or a few.

This is different to ZFS snaps. They are a freeze of a former file state due Copy and Write
and do not affect current data. No problem with endless keep or 10000 snaps.

Use NFS on ZFS to store your VMs and you do not have a problem with snaps.
You can create or delete ZFS snaps nearly without a delay and go back either via rollback
or Windows "previous versions" when you enable SMB on the NFS share. You can also
use these ZFS snaps for backup/copy/clone either via zfs send or "previous versions" that
gives a direct SMB file access to the former state of a VM folder.

To use ZFS on ESXi, you can either use a barebone ZFS SAN or you can virtualize
a ZFS storage appliance and use an NFS datastore internally, what I do for 10 years now.
https://napp-it.org/doc/downloads/napp-in-one.pdf

There are also options to include ESXi hot memory snaps in ZFS snaps when you create
an ESXi hot snap, then a ZFS snap with this included and then delete the ESXi snap.

A ZFS rollback give you then the ability to go back to the last ESXi hot memory state.

 

[Spartacus]

VMware's maximum recommended time frame for a snapshot is 24 hours for a reason.

 

[saedrin]

Our monitoring tool is configured to alert on any snapshot older than 24 hours.. Took a few 'situations' to develop before we decided to put that in place.