In this post, I’m going to show you the required steps and downfalls on running a LUKS encrypted Ubuntu Server setup and how it can be extended to allow remote unlocking.

Prerequisites

1. A server to install on
2. Static public IP address
3. The so called Alternative Ubuntu Server installer¹
4. Some patience 🙂

Installing and Setting up encrypted LVM

It is assumed that you already know your way around ISO files and how to boot them on your server.

We will also use the simplest possible setup: A server with a single disk

We are going to use LVM inside the LUKS container, it is not only supported, but the recommended way as we could also make use of advanced LVM functionalities later on.

Follow the installation until you reach the disk partitioning section:

1. The disk might need to be unmounted first:

2. choose Guided - use entire disk and setup encrypted LVM option:

3. In the next window, take extreme care to select the right HDD in case you have multiple ones:

4. Confirm changing the partition scheme to LVM if it was asked for:

5. Setup a strong passphrase for LUKS and confirm it:

6. Decide on how much of the disk space you want to dedicate to the root partition. For advanced setups, you can use only a percentage of the available space and create more partitions later on (And all of them would be automatically encrypted). The default is to make a single partition the size of the HDD:

7. Confirm the partition setup and continue with the installation:

8. Since There is no other OS on this system, it’s safe to install GRUB boot loader:

9. You will be prompted for the LUKS password after reboot to unlock the disk:

Our overall setup would be something like this:

NAME                     FSTYPE
sda
├─sda1                   ext4
├─sda2
└─sda5                   crypto_LUKS
  └─sda5_crypt           LVM2_member
    ├─ubuntu--vg-root    ext4
    └─ubuntu--vg-swap_1  swap

• sda1 is our boot partition. It is NOT ENCRYPTED²
• sda2 marks the start of the logical partitions
• sda5 is our encrypted LUKS partition
• sda5_crypt is the virtual crypt partition after unlocking (which uses LVM)
• ubuntu--vg-root is our root partition
• ubuntu--vg-swap_1 is the swap partition

Remote unlocking overview

The process behind this fairly simple. The kernel loads initramfs image, inside this image are the required files/modules/scripts for decrypting/mounting root.

Now if we could somehow run a SSH server in initramfs and make it accessible via network, one could remotely connect to it to unlock root partition.

As initramfs runs in memory, we are somewhat limited in the size and complexity of the running programs. This is the main reason why Dropbear is being used as the SSH server combined with BusyBox to provide the shell and basic utilities.

Update packages

Make sure your system (specially cryptsetup package) is up-to-date:
apt-get update && apt-get --assume-yes upgrade

Install Dropbear package for initramfs

As i said earlier, Ubuntu uses special Dropbear package to provide SSH server functionalities in initramfs environment with all the required hooks and scripts. Install it by issuing:
apt-get --assume-yes install dropbear-initramfs

If you get this warning while installing it, just ignore it for now, we will fix it soon enough:

dropbear: WARNING: Invalid authorized_keys file, remote unlocking of cryptroot via SSH won’t work!

Dropbear SSH keys

When you install the package for the first time, it also generates dss,rsa and ecdsa host keys³ placed in /etc/dropbear-initramfs/.

Although possible, It is not wise to share your real OpenSSH host keys with the dropbear-initramfs ones. This is because for the keys to be accessible by the SSH server, they must not be encrypted (The same also applies to the initramfs itself as the kernel needs to be able to load it).

This means that even on a fully encrypted root system, physical access would be enough to retrieve the dropbear-initramfs private keys (unless boot partition is also encrypted. That unfortunately however, would also render our remote unlocking approach useless)

The downside of using a different private key for the Dropbear server, is that it will likely result in the client getting a scary warning about the possibility of a man-in-the-middle attack. This is because the server keys would be different before/after unlocking the root partition. The simplest (and possibly the best) way to avoid this issue, is to run the Dropbear SSH instance on another port. We will cover this shortly.

Dropbear config file

Dropbear’s config file for the special dropbear-initramfs package, are placed in /etc/dropbear-initramfs/config

Changing default port

For the reason discussed above, we’re better off using a custom port to listen on. This also would have the advantage of reducing attacks on the server, as no firewall is running in initramfs environment.

To make it listen on port 4748, edit the said config file and add the following line:
DROPBEAR_OPTIONS="-p 4748"

Alternativ method:

A user reported that for some reason, the above method did not work for him and he added the same line to /etc/initramfs-tools/initramfs.conf instead.

Please note that this would result in these messages when rebuilding initramfs later on:

dropbear: WARNING: Setting DROPBEAR_* or PKGOPTION_dropbear_* in /etc/initramfs-tools/initramfs.conf is deprecated and will be ignored in a future release dropbear: WARNING: Use /etc/dropbear-initramfs/config instead

Further adjustments

I have also added -s -j -k -I 60 to DROPBEAR_OPTIONS just for the peace of mind. See man dropbear for details.

SSH Authentication

Password login has been disabled for dropbear-initramfs and only publickey authentication is allowed. Public keys should be placed in /etc/dropbear-initramfs/authorized_keys, one entry at a line. rsa based authentication is advised over ecdsa and dss.

You can also limit shell access to unlocking encrypted root partition only, by adding further per-user limitations in authorized_keys file like this:
no-port-forwarding,no-agent-forwarding,no-x11-forwarding,command="/bin/cryptroot-unlock" ssh-rsa ...

After changing Dropbear’s settings, do not forget to regenerate initramfs:
update-initramfs -u
This time, no error should appear.

Kernel IP parameters

Connecting remotely to the SSH server, would require the kernel to be able to setup network interfaces properly.

This would require that the kernel to first recognize the network interface (which is usually the case⁴), and also be able to setup IP parameters correctly.

Setting up static IP

The default kernel’s behavior is getting the IP address via dhcp (ip=dhcp). If your network lacks a DHCP server, special kernel boot IP parameter is needed. This would usually be in the form of:
ip=<client-ip>::<gw-ip>:<netmask>⁵

Append that to the GRUB_CMDLINE_LINUX_DEFAULT parameter in /etc/default/grub and regenerate GRUB config file:
update-grub

Alternativ method:

A user reported that for some reason, the above method did not work for him and instead he added this line to /etc/initramfs-tools/initramfs.conf:
IP="<client-ip>::<gw-ip>:<netmask>::eth0:off"

The above line, uses the same structure for most part. Except that its location is different and it uses the capital IP variable.

And then of course rebuild initramfs: update-initramfs -u

Fixing dorpbear-initramfs

This is a long story, but basically, there is an inconsistency between the dropbear-initramfs and busybox-initramfs packages.

After a complete setup, this problem manifests itself in these forms:

• Remote SSH sessions might not get closed automatically after a successful remote LUKS unlocking.
• You will get a couple of ps errors in your terminal right after LUKS unlocking.
• You might get this scary message in your terminal after boot up:

  Aiee, segfault! You should probably report this as a bug to the developer

• Your network interfaces might refuse to automatically come up after a successful boot.

PS: I have yet to confirm whether the last issue in this list, does in fact have something to do with this bug. So if you believe that after applying the workaround, it fixes your specific issue, please let me know in the comments section below.

To overcome this, I have written a small script to address the issue in the most non-aggressive way I could think of. More information on how to download and apply it, could be found here: dropbear-init-fix

You are advised to apply this workaround, but you may as well skip it and settle with the buggy version of the dropbear-initramfs package.

The result

At this point if you have set up everything correctly, after a restart and right after the kernel loads initramfs, network’s IP settings would be applied. Dropbear would start shortly after, listening for new connections:

And now we could connect to the Dropbear server remotely:
ssh -o "HostKeyAlgorithms ssh-rsa" -p 4748 [email protected]

Which brings us to the BusyBox built-in shell:

To unlock root partition, and maybe others like swap, run \`cryptroot-unlock\`


BusyBox v1.27.2 (Ubuntu 1:1.27.2-2ubuntu3) built-in shell (ash)
Enter 'help' for a list of built-in commands.

# 

cryptroot-unlock command is all that stands now between you and booting up your server!