ZFS - lost space from snapshots

[Stril]

Hi!

I am using a ZFS-system without deduplication but with LZ4-compression.
The system is running fine, but there is one VERY strange thing:

The size of my volume WITH snapshots is:

NAME USED AVAIL REFER MOUNTPOINT
zpool1/volume 8.07T 138T 1.33T /zpool1/volume

--> 1.33T without snapshots
--> 8.11T with snapshots
---> size of all snapshots should be 6.78TB

#zfs get usedbysnapshots zpool1/volume
zpool1/volume usedbysnapshots 6.78T


The strange thing: If I add the size of the snapshots, the sum is about 3.08TB
Where are the lost terrabytes?

Can you give me a hint?

Thank you for your help!
Stril

 

[Stril]

Hi!

That seems to be a quite common problem. What I found is:
mafm/zfs-snapshot-disk-usage-matrix

--> The script determines the size of snapshots by executing a "dry-run-destroy" for all snapshots between two.

Can anybody here give me a tip, what happens there, that leads to different sizes - although snapshots are strictly "serialized"?

Thank you and best wishes,
Stril

 

[gea]

Only a few remarks
First you should not mix TB and T (ZFS counts T=Tebibyte).
They are different at around 10% as a thumb rule.

Next start from a usable capacity ex 10T on a new pool.
If you copy 4T onto , then your remaining capacity is 6T. Then take a snap. Nothing changes as a snap is just a freeze of the current occupied datablocks. Then delete the 4T. Nothing changes as the 4T are still blocked from further use. You must delete the snap to regain its capacity.

To make it more complicated, any ZFS filesystem can use as much as is available from a pool. Unlike partitions that have a fixed size, ZFS filesystem can access capacity dynamically. You can restrict with quotas and guarantee a capacity with reservations. Such a reservation reduces the available capacity of all other filesystems. Napp-it for exampe sets a 10% reservation to a pool what means all filesystems can see only 90% of pool size. ZFS itself sets an internal small reservation of a few percents to avoid a crash on a pool full.

Then you can create zvols. Their size can be based on used Bytes (thin provisioned) or otherwise block their nominal size. If you compress data, data size and size on storage is also different.

 

[Stril]

Hi gea!

But here, there is a factor of nearly 2 between the sizes.

Do you have any idea about that?

I am aware of the snapshot feature.
My only problem are the different sizes between the sum of the snapshot sizes and the uses size, reported by the ZFS command of the volume.

Thank you for your help

 

[gea]

I can only asume
used=data+reservation+ZFA datablocks blocked by snaps

The size of a snap is the sum of all modified ZFS datablocks.This means that snaps only counts Bytes that are modified on the array.

Data counts filesize and ignores compress.

 

[gea]

Maybe this video from one of the ZFS creators can give some more background infos

How ZFS snapshots really work And why they perform well (usually) by Matt Ahrens