Proxmox + zfs + encryption guide (work in progress)

zdude

What is a Computer?
Aug 29, 2017
58
8
8
40
I have been playing around with encryption for a system for a while, 4th post contains my notes on what I have so far. If anyone has more experience with crypttab and knows how to allow booting with a missing device, input would be appreciated.

Presently I am attempting to set up a Proxmox node with complete disk encryption. The end goal is to have to enter two passwords for the encrypted zfs mirror Proxmox is booting from and have data drives be decrypted by keys stored on the boot drives.

The current state of this project is as follows. A Proxmox VM has been configured with two disks in a zfs mirror. The first step I am trying to accomplish is to encrypt a single disk and have the system boot and zfs recognize both boot disks at boot time.

I have tried following a variety of guides, and while data is kept on the drives and appears to work until reboot on reboot zfs does not recognize the encrypted drive.

zfs on lvm on luks attempt (following guides to the letter)

zfs on crypt directly attempt before reboot

After reboot lukszfs1 is not recognized even though it is mounted by grub and initramfs.


Has anybody accomplished zfs on luks whole disk encryption and/or have a guide on how to implement it?

EDIT: added second config to current status spoiler
 
Last edited:

zdude

What is a Computer?
Aug 29, 2017
58
8
8
40
The future (nearly ready) for all Open-ZFS platforms is real ZFS encryption (encryption as a ZFS filesystem property), similar to what is implemented in Oracle Solaris 11.

8727 Native data and metadata encryption for zfs by lundman · Pull Request #489 · openzfs/openzfs
https://zfs.datto.com/slides/caputi.pdf
While I would not be opposed to rolling a newer version of ZFS into Proxmox, I would greatly prefer to not use a release candidate in what will become a production system within the next 6 months. And looking at the progress towards release I do not believe that ZFS 0.8 will be out by then. Otherwise the native encryption would be the ideal solution.
 

zdude

What is a Computer?
Aug 29, 2017
58
8
8
40
So I figured out the issues I was having, as per usual it was just a single config I was not setting correctly. Below is a guide that I wrote mainly so I can follow it when migrating my production systems. It is being posted here in case somebody else wants to move an existing Proxmox 5.3 ZFS install to whole disk encryption (including boot partition).

After setup is complete, each boot disk must be unlocked twice. Once for grub and once in initramfs. While annoying, if you are rebooting Proxmox a lot something has probably already gone horribly wrong.

IMPORTANT NOTE:
As this guide stands if one of the boot disks fails, the system will not boot without repair. That repair involves commenting out the failed disk from crypttab and updating initramfs. I will update this post if I find a way to allow booting with a device listed in crypttab missing.

Note:
I am doing all of my testing in a virtual machine, as such device names such as /dev/sda are acceptable. On a physical machine it is recommended to use the /dev/disk/by-id/ device.

Requirements:
Proxmox 5.3 on ZFS boot
ZFS MUST have at least one disk redundancy. (note only tested on 2 way mirror, no raidz testing was done).


Pre-reqs:
install cryptsetup
Code:
apt-get install cryptsetup

Disk One setup:
Remove the disk from the boot pool
Code:
zpool detach rpool sda3

Format the partition ZFS was using for luks.
Code:
cryptsetup luksFormat /dev/sda3

Mount the LUKS formatted partition
Code:
cryptsetup luksOpen /dev/sda3 lukszfs1

At this point disk layout should look like this

Attach the encrypted partition to the pool
Code:
zpool attach rpool sdb3 lukszfs1

Add the lukszfs1 mount to /etc/crypttab
IMPORTANT: the initramfs option forces the luks partion to be unlocked before ZFS tries importing the pool without this option the system will not boot.
Code:
lukszfs1 /dev/sda3 none luks,discard,initramfs

Update initramfs to apply the correct mounting order.
Code:
update-initramfs -u

Once ZFS is done resilvering disk layout and pool should look something like this

At this point it is theoretically possible to reboot successfully. However there is still an un-encrypted disk in the rpool.


Before encrypting the next disk, add support to grub to boot with an encrypted /boot partition. To do so add the following line to /etc/default/grub
Code:
GRUB_ENABLE_CRYPTODISK=y

Update and re-install grub:
Code:
update-grub
grub-install /dev/sda
grub-install /dev/sdb

Detach the unencrypted disk from the rpool
Code:
zpool detach rpool sdb3


Format, mount and re-add the disk to rpool as done with the first disk.
Code:
cryptsetup luksFormat /dev/sdb3
cryptsetup luksOpen /dev/sdb3 lukszfs2
zpool attach rpool lukszfs1 lukszfs2

Add the second disk to /etc/crypttab
Code:
lukszfs2 /dev/sdb3 none luks,discard,initramfs

Update initramfs
Code:
update-initramfs -u


At this point both disks are encrypted and bootable.
 

noemis

New Member
Jan 21, 2019
1
0
1
Regarding booting without a missing device; there is the
Code:
nofail
option in the crypttab. See crypttab for reference.
 

zdude

What is a Computer?
Aug 29, 2017
58
8
8
40
Regarding booting without a missing device; there is the
Code:
nofail
option in the crypttab. See crypttab for reference.
The version of crypttab that ships with proxmox does not have the nofail option. I haven't had time the past couple of weeks to see about running a newer version than is in repos... hypothetically it should work.
 

janosch1337

New Member
Feb 27, 2019
3
0
1
hey zdude,

i followed your guide. i install proxmox 5.3 with zfs raid 1 (2x 500gb for testing hdd). after the last step when i'm rebooting my system it ask for the encryption key and then goes to the rescue panel because hostdisk//dev/sdb,gpt3 not found
 

zdude

What is a Computer?
Aug 29, 2017
58
8
8
40
hey zdude,

i followed your guide. i install proxmox 5.3 with zfs raid 1 (2x 500gb for testing hdd). after the last step when i'm rebooting my system it ask for the encryption key and then goes to the rescue panel because hostdisk//dev/sdb,gpt3 not found
I ended up having a similar issue when I migrated my production system. I don't remember exactly how I solved it (panic because a production system is down) but it involved a process similar to the following...

  1. Boot into live envoronment (I used ubuntu for pre-compiled zfs)
  2. Unlock and mount drives
  3. import zfs pool
  4. chroot into zfs root of proxmox
  5. regenerate intiramfs (probably not the issue)
  6. regenerate grub config
  7. reinstall grub


My grub config has the following to boot correctly.

Code:
GRUB_DEFAULT=0
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="Proxmox Virtual Environment"
GRUB_CMDLINE_LINUX_DEFAULT="quiet"
GRUB_CMDLINE_LINUX="root=ZFS=rpool/ROOT/pve-1 boot=zfs"
GRUB_ENABLE_CRYPTODISK=y
To get the chroot running I followed the guide here:
zfsonlinux/zfs
 

janosch1337

New Member
Feb 27, 2019
3
0
1
ok now i got it working without booting live cd. but now i have to enter the password 4 times. is it possible to enter the passwords without physical access to the server (dropbear)?
 

zdude

What is a Computer?
Aug 29, 2017
58
8
8
40
This encryption method requires either physical access or IPMI access to the system. It encrypts everything including initramfs.
 

janosch1337

New Member
Feb 27, 2019
3
0
1
Ok is there a possibility to install the server without the password enter in grub?

I have only rescue access (hetzner) and no ipmi Access. For my home server it’s working good because I have ipmi or I go to the server and type the password in it.
 

zdude

What is a Computer?
Aug 29, 2017
58
8
8
40
Ok is there a possibility to install the server without the password enter in grub?

I have only rescue access (hetzner) and no ipmi Access. For my home server it’s working good because I have ipmi or I go to the server and type the password in it.

This is not the encryption for that then. Even if grub and intiramfs is moved to another partition you still need a preboot console access to enter the initramfs password to mount the overall system.