Leveraging LUKS to Add (Almost) Native Encryption Key Slots to ZFS

Have you ever wanted to encrypt a ZFS pool, and have the option to use multiple keys, including per-user passphrases to unlock it? For those that have used LUKS in the past, this seems like a trivial feature: simply have each user add a password to a keyslot, and add a backup key for if you forget the password or want friction-free setup when a device is connected to a trusted system, and you're off to the races. For an advanced filesystem like OpenZFS, you'd assume it'd be easy too, ...right?

Unfortunately, after a bit of documentation hunting and Googling, you'll realise that this isn't yet a feature in ZFS. In the rest of this (quite short, and more braindump than anything) post, I'll show you how to setup LUKS-like keyslots for your ZFS pools, which doesn't address the root issue but is definitely much easier than filing a PR in the openzfs project, and means you no longer have to re-enter keys for your pools after a reboot.

How do keyslots work?

Warning

There are instructions in this blog post that interact with the filesystem and can potentially destroy data. As with best practices, test on a test system first, then ensure you have a working backup in place before deploying to a live system. Ensure you have working offline backups of encryption keys. You have been warned!

In a LUKS-encrypted volume, you can easily add and remove keys and change passphrases to unlock the volumes. Currently LUKS2 limits you to 32 keyslots, as defined in the header file. Each keyslot contains an encrypted version of the master key, with empty slots not containing the master key at all. For a user to decrypt the master key, they must either provide the keys to decrypt the master key or use a key derivation function (KDF) to derive a key from a user supplied password and a fixed salt.

This KDF generates the key to decrypt the master key in that specific slot. Therefore, for a user with access to a LUKS-encrypted device simply needs to provide their passphrase or a new keyfile to decrypt their copy of the master key and mount the device.

It is also due to this that changing the passphrase does not incur any penalties, as all encrypted data can continue to be read and written with the master key. You can (and should) rotate the master key if access to the system changes and a previously authorised user has their key removed from a keyslot, as it is trivial to dump the master key in LUKS. To dump the key, the user simply has to run:

cryptsetup luksDump -q --dump-master-key /dev/disk-identifier

Luckily, it's just as simple to re-encrypt the volume, simply issue a

cryptsetup-reencrypt /dev/mapper/mapped-volume

Consult the man page on this for further details.

How ZFS Manages Keys

ZFS itself supports native encryption without the user having to wrap their entire device in dm-crypt and adding the potential performance penalties such as whole disk encryption versus per dataset encryption, and whether to use LUKS as the internal or external filesystem.

We want to be able to continue to use ZFS with native encryption where possible, so it's in our best interests to work with the current technologies and see how we can provide keys in a way that ZFS can work with.

Currently, ZFS needs to have either a keyformat=raw or keyformat=passphrase set as the encryption parameter. The passphrase option generates an interactive prompt, whilst the raw option requires the user to specify a keyfile, which can then be used to encrypt and decrypt.

Solution

We're going to create a LUKS-encrypted ext4 filesystem on a ZVOL, which we can then automount at boot, unlock (via a file on boot drive, or passphrase) and provide a keyfile for ZFS to load-key with, which will finally allow us to mount the filesystems. We need to be a bit careful with the order here, as we can't mount ZFS filesystems until we are sure that the keyfile mount exists.

Luckily, systemd has a handy RequiresMountsFor key, which we can use to help. Here's a quick flowchart so we can see what's going on at boot:

flowchart TD
    A((Start)) --> B[System Boot] --> root[Mount Root FS] --> crypt[Start creating dm-crypt devices] --> C{Is key available on root FS?}
    C -->|Yes| D[Decrypt ZVOL with Keyfile]
    C -->|No| E[Wait for user to enter password to mount ZVOL]
    D --> F[Finish creating dm-crypt devices]
    E --> F
    F --> G[Mount fstab filesystems, including ZFS encryption key volume]
    G --> H[Use ZFS encryption key at mount to unlock ZFS datasets]

Simple flowchart of unlock procedure

Instructions

Now comes the quick and dirty instructions set, I've replaced the name of my pool with $POOL_NAME where I can, but as always, don't blindly run these commands without checking. My pool is rl-pool and my datasets are under rl-pool/data, for reference.

# Template Variables
POOL_NAME="rl-pool"
ROOT_DATASET_NAME="data"

# Create Zvol
sudo zfs create -V 100M $POOL_NAME/$POOL_NAME-encryption-key

# Create LUKS container, specify initial password here
sudo cryptsetup luksFormat /dev/zvol/$POOL_NAME/$POOL_NAME-encryption-key

# Open the outer container
sudo cryptsetup open /dev/zvol/$POOL_NAME/$POOL_NAME-encryption-key $POOL_NAME-encryption-key

# Create inner filesystem
sudo mkfs.ext4 /dev/mapper/$POOL_NAME-encryption-key

# Mount inner filesystem
sudo mount /dev/mapper/$POOL_NAME-encryption-key /mnt/$POOL_NAME-encryption-key

# Create random inner key
sudo dd if=/dev/urandom of=/mnt/$POOL_NAME-encryption-key/inner-key

# Load current key
sudo zfs load-key $POOL_NAME-pool/data

# Change current key
sudo zfs change-key -o keylocation=file:///mnt/$POOL_NAME-encryption-key/inner-key -o keyformat=raw $POOL_NAME/$POOL_DATASET_NAME

# Create a keyfile on trusted filesystem to unlock LUKS outer filesystem
sudo dd bs=512 count=4 if=/dev/urandom of=/root/$POOL_NAME-luks-key iflag=fullblock

# Add outer keyfile to LUKS volume
sudo cryptsetup luksAddKey /dev/zvol/$POOL_NAME/$POOL_NAME-encryption-key /root/$POOL_NAME-luks-key

# Back up this file and the inner key securely, this will differ depending on where you're backing up to
sudo rsync somehost:/mnt/$POOL_NAME-encryption-key/inner-key /mnt/offline-backup
sudo rsync somehost:/root/$POOL_NAME-luks-key /mnt/offline-backup

# Set up /etc/crypttab
cat <<EOF | sudo tee -a /etc/crypttab
$POOL_NAME-encryption-key   /dev/zvol/$POOL_NAME/$POOL_NAME-encryption-key  /root/$POOL_NAME-luks-key   nofail
EOF

# Set up /etc/fstab
cat <<EOF | sudo tee -a /etc/fstab
/dev/mapper/$POOL_NAME-encryption-key   /mnt/$POOL_NAME-encryption-key  ext4    auto,ro,nofail  0   2
EOF

# Set ZFS automount to depend on /etc/crypttab and /etc/fstab working correctly
# Stolen from https://github.com/openzfs/zfs/issues/8750#issuecomment-497500144
cat <<EOF > /etc/systemd/system/zfs-load-key@.service
[Unit]
Description=Load ZFS keys
DefaultDependencies=no
Before=zfs-mount.service
After=zfs-import.target
Requires=zfs-import.target
Requires=local-fs.target
RequiresMountsFor=/mnt/$POOL_NAME-encryption-key
[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/sbin/zfs load-key %I
[Install]
WantedBy=zfs-mount.service
EOF

# Create and enable the service, being careful to escape my dashes
# E.g., for a pool rl-pool/data, this gets escaped to rl\\x2dpool-data
# Slashes are substituted with dashes, dashes need to be escaped as \x2d
# Single backslash is escaped by CLI, so add an extra
systemctl enable --now zfs-load-key@$POOL_NAME # systemd name encoded

Testing

Once all setup, reboot your device, and it should all magically come to life, datasets decrypted and all! You can simulate what would happen if the drive was taken elsewhere or plugged into another computer by moving the LUKS file at /root/$POOL_NAME-luks-key elsewhere, then test to check you are still able to open the filesystem using the passphrase, and it mounts the underlying ZFS system.

Further Work

This system provides a basic proof-of-concept. Further work to streamline and ensure security could be done in the following areas:

Storage of keys in hardware TPM
Cryptographic assurances of key length and randomness
Filesystem permissions and access to the ZFS key, besides restricting permissions to root

Conclusion

Whilst a bit tedious to setup, the solution in this post shows how you can leverage both the benefits of LUKS and ZFS to generate a secure but convenient encrypted pool that can be unlocked at-boot when connected to an encrypted system, or (with some initial faff) unlocked when connected to a different computer, ensuring all data is stored encrypted at-rest, and readily available following a reboot.