TIL: Snapshots. Don't let them get too big!

Discussion in 'VMware, VirtualBox, Citrix' started by sd11, Sep 8, 2019.

  1. sd11

    sd11 New Member

    Joined:
    Jun 2, 2016
    Messages:
    19
    Likes Received:
    0
    Long story short. I forgot I had taken a snapshot on a Win Server 2016 VM in esxi. I removed the snapshot forgetting that there is a 6TB disk attached. The VM is relatively low when it comes to IO changes. Just a small file server. It's still is taking forever to remove. 6% in the past 20 hours. It's now at 65%. The first 50% was slow but tolerable. It must have been consolidating the disk on the NVME drive. The 6TB is on a LSI 9266 R1 array. Every VM is shutdown.

    In hindsight, I should have done a V2V.

    By my estimate, it should be done by Friday and I can boot it back up.

    Anyone else run into this problem?
     
    #1
  2. Haitch

    Haitch Member

    Joined:
    Apr 18, 2011
    Messages:
    112
    Likes Received:
    13
    Had a customer forget about a snapshot - the LUN it was on ran out of space and the snapshot became corrupted and unrecoverable, even by VMware support. They had to revert to the pre-snap shot vmdk. 18 months of data lost.
     
    #2
  3. gea

    gea Well-Known Member

    Joined:
    Dec 31, 2010
    Messages:
    2,184
    Likes Received:
    708
    ESXi snaps are growing delta files with all modifications.
    This is why this is a shortterm snap where you should not create more than one or a few.

    This is different to ZFS snaps. They are a freeze of a former file state due Copy and Write
    and do not affect current data. No problem with endless keep or 10000 snaps.

    Use NFS on ZFS to store your VMs and you do not have a problem with snaps.
    You can create or delete ZFS snaps nearly without a delay and go back either via rollback
    or Windows "previous versions" when you enable SMB on the NFS share. You can also
    use these ZFS snaps for backup/copy/clone either via zfs send or "previous versions" that
    gives a direct SMB file access to the former state of a VM folder.

    To use ZFS on ESXi, you can either use a barebone ZFS SAN or you can virtualize
    a ZFS storage appliance and use an NFS datastore internally, what I do for 10 years now.
    https://napp-it.org/doc/downloads/napp-in-one.pdf

    There are also options to include ESXi hot memory snaps in ZFS snaps when you create
    an ESXi hot snap, then a ZFS snap with this included and then delete the ESXi snap.

    A ZFS rollback give you then the ability to go back to the last ESXi hot memory state.
     
    #3
  4. Spartacus

    Spartacus Active Member

    Joined:
    May 27, 2019
    Messages:
    183
    Likes Received:
    39
    VMware's maximum recommended time frame for a snapshot is 24 hours for a reason.
     
    #4
  5. saedrin

    saedrin New Member

    Joined:
    May 13, 2019
    Messages:
    1
    Likes Received:
    0
    Our monitoring tool is configured to alert on any snapshot older than 24 hours.. Took a few 'situations' to develop before we decided to put that in place.
     
    #5
Similar Threads: Snapshots Don't
Forum Title Date
VMware, VirtualBox, Citrix ESXi 6.7 can't change settings, don't survive reboot (passthrough issues, too!) May 1, 2018

Share This Page