Recursive encrypted zfs send / recv with snapshots

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

deviantintegral

New Member
Dec 29, 2020
6
0
1
I currently have datasets for lxd and kvm VMs on one zfs pool that I'd like to migrate to another. Both the source and destination pools use zfs's encryption, though they have different keys. My end goal is to:

  1. Replicate all existing snapshots, mostly created with zfs-auto-snapshot.
  2. Preserve any cloned datasets (lxc starts new containers with a clone from an "images" dataset)
  3. Be able to do incremental sends until I'm ready to cut lxc and kvm over to the new pool.
  4. Preserve encryption - while it would be nice to re-encrypt the data at the destination so I'm only dealing with one set of keys, I can always work around that later if I have to send the raw data stream.
I've had trouble getting any sort of send / recv to work at all. I expected I could do this in one or two zfs commands. I'm currently running openzfs 2.0, but will be needing to load the pool on a zfs 0.8 system later.

First, I create a dataset to host the backups on the destination pool rpool: $ zfs create rpool/backup

It looks like for the first send, I should use --replicate to send all descendent filesystems, snapshots and so on. I'd need to use --raw because the source is encrypted.

Here is attempting to send the oldest snapshot on the lxc dataset:

Code:
# zfs send --replicate --raw spool/lxc@syncoid_spool_2019-09-24:20:36:30 | zfs recv rpool/backup
cannot send spool/lxc@syncoid_spool_2019-09-24:20:36:30 recursively: snapshot spool/lxc/custom@syncoid_spool_2019-09-24:20:36:30 does not exist
warning: cannot send 'spool/lxc@syncoid_spool_2019-09-24:20:36:30': backup failed
cannot receive: failed to read from stream
It looks like --replicate is expecting that identical snapshot names exist on each child dataset, which isn't the case. Perhaps -I will fix it?

Code:
# zfs send --replicate --raw -I spool/lxc@syncoid_spool_2019-09-24:20:36:30 spool/lxc@zfs-auto-snap_frequent-2020-12-29-1845 | zfs recv rpool/backup
cannot receive incremental stream: most recent snapshot of rpool/backup does not
match incremental source
While I haven't tried it yet, it does seem likely I could create a brand new recursive snapshot and send it. But, that would omit all of my current snapshots.

Suggestions? I tried with syncoid (as it's what I used to move the dataset to this pool), but it also had errors.
 

gea

Well-Known Member
Dec 31, 2010
3,141
1,182
113
DE
You must first create recursive source snaps. Only then a recursive send is possible as send wants to transfer filesystem by filesystem based on their snaps.

Beside the --raw option to send encrypted data that is Open-ZFS only, you can use the quite detailled Oracle manuals like Sending and Receiving ZFS Data - Oracle Solaris ZFS Administration Guide

If you use a replication script this should care about initial and incremental replications and the needed snaps on both sides then.
 

deviantintegral

New Member
Dec 29, 2020
6
0
1
Thank you for confirming that! I created a new recursive snapshot, but it's still a little tricky. It seems like it's not ordering the stream so any clone origins are sent first, causing sends to error out. For example, lxc containers are cloned from an image dataset, and I'm having to manually send the child datasets at least for the first round. We'll see when this is done if at least catch-up sends are straight forward.