[solved] install to existing encrypted Btrfs subvolumes

waynedpj · 2020-01-11 16:01:32

ahoy all,

first, thanks for the freedom options, init and otherwise. i have been looking to get away from systemd and friends, find something more simple and minimal, so excited to finally give Devuan a go.

i am trying to install Devuan to a HDD drive with an existing partition scheme based on Btrfs and subvolumes:

/dev/sda1    GRUB BIOS boot partition (for booting GPT disk with BIOS)
/dev/sda2    LUKS1 dm encrypted Btrfs partition (remainder of disk)

within the sda2 Btrfs partition there are 2 subvolumes that i would like to use for the Devuan root / and /home mounts:

/      ->    /systems/Devuan/root/
/home/ ->    /shares/home

thus in the installer i do not need to partition anything as i prepare it all ahead of time with parted, all subvolumes empty.

and while it seems that the Debian/Devuan installers do not directly support installing to Btrfs subvolumes https://aykevl.nl/2015/11/debian-btrfs-subvolume, https://wiki.debian.org/Btrfs#Status, i have found a post in this forum that basically tricks the installer into using the subvolumes after playing with the /target mount created after partitioning but before installation begins: https://dev1galaxy.org/viewtopic.php?pid=14462#p14462.

however using the method described in the above post with expert mode netinst i cannot get past the partitioning step since the installer wants a root partition: after the installer starts in a separate shell i unlock the LUKS1 encrypted partition and mount the Btrfs subvolumes for / and /home. but in the installer partitioning step i cannot find a way to say "hey, use this partition for /" without it seemingly ready to reformat sda2, overwriting the LUKS1 header, Btrfs, etc. thus the /target mount is not created.

there most likely is a step i am missing but after a good deal of research i cannot figure it out, thus any help appreciated. or perhaps someone else has another updated method for this type of installation with Devuan and Btrfs subvolumes?

thanks, w

Last edited by waynedpj (2020-01-20 18:16:03)

waynedpj · 2020-01-13 03:50:03

adding some more info here after discussion on IRC regarding the existing Btrfs subvolumes to which i am trying to install Devuan:

the btrfs mount commands:

mount -o subvol=/shares/home /dev/mapper/storage /mnt/home/
mount -o subvol=/systems/Devuan/root /dev/mapper/storage /mnt/Devuan/root/

output of mount with Btrfs subvolumes mounted:

root@devuan:~# mount
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
udev on /dev type devtmpfs (rw,nosuid,relatime,size=1978076k,nr_inodes=494519,mode=755)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,noexec,relatime,size=399652k,mode=755)
/dev/shm on /lib/live/mount/medium type tmpfs (rw,relatime,size=1084164k)
/dev/loop0 on /lib/live/mount/rootfs/filesystem.squashfs type squashfs (ro,noatime)
tmpfs on /lib/live/mount/overlay type tmpfs (rw,relatime)
overlay on / type overlay (rw,noatime,lowerdir=//filesystem.squashfs/,upperdir=/live/overlay//rw,workdir=/live/overlay//work)
tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k)
pstore on /sys/fs/pstore type pstore (rw,relatime)
tmpfs on /run/shm type tmpfs (rw,nosuid,nodev,noexec,relatime,size=799300k)
tmpfs on /tmp type tmpfs (rw,nosuid,nodev,relatime)
rpc_pipefs on /run/rpc_pipefs type rpc_pipefs (rw,relatime)
tmpfs on /sys/fs/cgroup type tmpfs (rw,nosuid,nodev,noexec,mode=755)
cgroup on /sys/fs/cgroup/elogind type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/elogind/elogind-cgroups-agent,name=elogind)
tmpfs on /run/user/1000 type tmpfs (rw,nosuid,nodev,relatime,size=399648k,mode=700,uid=1000,gid=1000)
/dev/mapper/storage on /mnt/home type btrfs (rw,relatime,space_cache,subvolid=257,subvol=/shares/home)
/dev/mapper/storage on /mnt/Devuan/root type btrfs (rw,relatime,space_cache,subvolid=259,subvol=/systems/Devuan/root)

fsmithred · 2020-01-13 11:11:02

When you get to the partitioning phase in the installer, if your subvolumes are listed, you should be able to highlight one, press enter, and get to a screen that shows options for that parititon - use as (filesystem type), format, mountpoint, etc. If you highlight a line and press enter, you can edit that line. For the format line, pressing enter will toggle yes/no.

When you're at the screen that lists your partitions, there may be an upper-case F or K on each line. F means the partition will be formatted, K means keep the current filesystem (don't format).

Meanwhile, I will try to reproduce your configuration and see if I can do it with the live installer.

fsmithred · 2020-01-13 13:12:04

When you get to the partitioning phase in the installer, if your subvolumes are listed, you should be able to highlight one, press enter, and get to a screen that shows options for that parititon - use as (filesystem type), format, mountpoint, etc. If you highlight a line and press enter, you can edit that line. For the format line, pressing enter will toggle yes/no.

When you're at the screen that lists your partitions, there may be an upper-case F or K on each line. F means the partition will be formatted, K means keep the current filesystem (don't format).

Meanwhile, I will try to reproduce your configuration and see if I can do it with the live installer.

OK, I tried a couple of times. I think it's possible with the live installer, but I haven't succeeded yet - I'm booting to a grub_rescue prompt.

I had to hard-code the subvolid into the mount commands in the install script. If/when I try it again, I will make a separate /boot partition. Oh yeah, I just remembered. In beowulf, you have to use luks format type1 for full-disk encryption to work. That's probably why my grub is lost. (I did add the cryptodisk line to /etc/default/grub.)

fsmithred · 2020-01-13 14:11:13

I got it to install with the live installer if I do a lot of manual fiddling with it. But I can't get it to boot, even with a separate /boot partition.

See if you can get the regular installer to work without formatting as I described above. If that doesn't work, we can come back to a more manual approach.

Edit/Update: I can't get it to work with the installer isos, either. I tried ascii-2.1. Expert install, check cyptsetup at the extra installer components list, drop to a shell when it reaches partitioning, try to open the encrypted volume with cryptsetup, and the command is not found. (the debs are on the media, but they haven't been installed at this point.)

If I try to select the btrfs partition to use as btrfs, it tells me that no root has been selected. Looks like you have to do it the way it was described in the other thread (install to a temp dir and then rsync it all into place.)

waynedpj · 2020-01-13 17:14:51

fsmithred wrote:

When you get to the partitioning phase in the installer, if your subvolumes are listed, you should be able to highlight one, press enter, and get to a screen that shows options for that parititon - use as (filesystem type), format, mountpoint, etc. If you highlight a line and press enter, you can edit that line. For the format line, pressing enter will toggle yes/no.
When you're at the screen that lists your partitions, there may be an upper-case F or K on each line. F means the partition will be formatted, K means keep the current filesystem (don't format).

i assume that you mean netinst by "installer" and "regular installer"? i tried what you said with netinst expert mode:

in the "Partition disks" step at first i only saw the sda3 LUKS1 encrypted partition (the Btrfs partition is inside the encrypted container)
i was able to open the encrypted /sda3 in a separate shell using cryptsetup (after doing something (maybe selecting "Configure encrypted volumes"?) that made the cryptsetup command available in the shell, as it initially was not?)
at that point after exiting and re-entering the "Partition disks" step i now see the Btrfs partition under a new heading "Encrypted volume". and while i can select it for installation without formatting, unfortunately i still cannot see/select the subvolumes as installation targets. in addition i could not add the mount options for selecting the subvolume

this is an improvement as maybe i can just set the root subvolume as the default subvolume for this Btrfs partition which would install the system at least to a single subvolume. then copy over home afterwards? however, it still would be nice to have the installer set up correctly the fstab, cryptab, and GRUB paths.

just in case i also tried with the desktop-live installer i.e. refractainstaller but could not find a way to even see unlocked Btrfs partition (again in a separate shell via cryptsetup) as above. maybe because you most likely meant the netinst in the first place

fsmithred wrote:

I got it to install with the live installer if I do a lot of manual fiddling with it. But I can't get it to boot, even with a separate /boot partition.

thanks for testing this out. i have set up full disk encryption (i.e. with encrypted /boot) with a single encrypted partition containing Btrfs and subvolumes with other distros using GRUB (LUKS1 only). one thing that always got me was that GRUB does not understand Btrfs subvolumes so you need to use the full paths to each subvolume in the config. maybe that is tripping you up?

OK, let me know if you have any other ideas and thanks again!

peace, w

fsmithred · 2020-01-13 18:04:41

Yup. netinst = installer iso (one of them, anyway) = regular installer = debian-installer (the actual package name)

refractainstaller run in a root terminal (the live installer) won't show you the btrfs volumes. I manually opened the encrypted volume, and when the installer asked for the root partition and the home partition, I entered the /dev/mapper path for the opened luks volume.

You'll get a warning that the partition doesn't end with a digit (unless you named it that way) but you can still proceed. I had to edit the mount commands for / and /home inside refractainstaller to add the subvolid option, and I also had to manually edit fstab on /target. (subvolume path should work here instead of subvolid.)

Full path missing in grub is probably why it won't boot.

Yes, if you can get the installer (on the netinst iso) to install to the root subvolume, you should be able to copy /home/* to the second subvolume. I've done the equivalent with encrypted or non-encrypted ext partitions many times.

waynedpj · 2020-01-14 19:48:32

fsmithred wrote:

Yup. netinst = installer iso (one of them, anyway) = regular installer = debian-installer (the actual package name)

thanks for clearing that up!

fsmithred wrote:

refractainstaller run in a root terminal (the live installer) won't show you the btrfs volumes. I manually opened the encrypted volume, and when the installer asked for the root partition and the home partition, I entered the /dev/mapper path for the opened luks volume.

OK so basically you set $install_dev and $home_dev to the same /dev/mapper/storage device that you opened with cryptsetup?

fsmithred wrote:

You'll get a warning that the partition doesn't end with a digit (unless you named it that way) but you can still proceed. I had to edit the mount commands for / and /home inside refractainstaller to add the subvolid option,

do you mean the mount commands for $install_part and $home_part? was this the only change that you had to make in the script? would these changes make the GRUB installation not work?

fsmithred wrote:

and I also had to manually edit fstab on /target. (subvolume path should work here instead of subvolid.)

can you explain what and when exactly you did this fstab edit? after installation but before rebooting?

fsmithred wrote:

Yes, if you can get the installer (on the netinst iso) to install to the root subvolume, you should be able to copy /home/* to the second subvolume. I've done the equivalent with encrypted or non-encrypted ext partitions many times.

i prefer the command line and having the source right there so the refractainstaller seems like the way to go for me. however given that you are having problems getting a booting setup perhaps it is better if i use the debian-installer and just do the default subvolume trick, as hopefully netinst will get the GRUB config correct. any advice?

regardless thanks again.

peace, w

fsmithred · 2020-01-14 22:42:53

waynedpj wrote:

fsmithred wrote:
refractainstaller run in a root terminal (the live installer) won't show you the btrfs volumes. I manually opened the encrypted volume, and when the installer asked for the root partition and the home partition, I entered the /dev/mapper path for the opened luks volume.
OK so basically you set $install_dev and $home_dev to the same /dev/mapper/storage device that you opened with cryptsetup?

Yes.

fsmithred wrote:

You'll get a warning that the partition doesn't end with a digit (unless you named it that way) but you can still proceed. I had to edit the mount commands for / and /home inside refractainstaller to add the subvolid option,

waynedpj wrote:

do you mean the mount commands for $install_part and $home_part? was this the only change that you had to make in the script? would these changes make the GRUB installation not work?

Yes, I added a subvolume option. This should not affect grub. I used subvolid, but you could also use the path to the subvolume.

To get the subvolid, run btrfs subvolume list <path> where path is the path to the mounted luks volume.

# btrfs subvolume list /mnt 
ID 257 gen 37 top level 5 path devuan/rootfs.subvol
ID 258 gen 20 top level 5 path devuan/homefs.subvol

On or around line 1137 of /usr/bin/refractainstaller:
mount -o subvolid=257 "$install_part" /target ; check_exit

Line 1163:
mount -o subvolid=258 "$home_part" /target ; check_exit

waynedpj wrote:

fsmithred wrote:
and I also had to manually edit fstab on /target. (subvolume path should work here instead of subvolid.)
can you explain what and when exactly you did this fstab edit? after installation but before rebooting?

I did it in the middle of the installation. There's a pause when it's ready to install the bootloader, so that you can go to another terminal and modify files on the target or even chroot the target. Then go back and tell the installer to do the bootloader. All I did was add the subvol option. You might use more or different options.

/dev/mapper/sda3_crypt	/	btrfs	defaults,noatime,subvolid=257	0	1
/dev/mapper/sda3_crypt	/home	btrfs	defaults,noatime,subvolid=258	0	2
/dev/sda2	/boot	ext2	defaults,noatime	0	1
/swapfile	none	swap	sw	0	0

fsmithred · 2020-01-14 23:29:58

HA! I got it to boot. It was dropping me to initramfs prompt and complaining about a missing UUID. That was after about a hundred lines of looking for the floppy and looking for a raid array (neither of which exist).

So, at the boot menu, I edited the entry to remove that uuid and changed it to
root=/dev/mapper/sda3_crypt
And it still dropped me to the initramfs prompt. So I manually opened the volume with cryptsetup, was asked for the password, then I ran exec /sbin/init and it booted.

waynedpj · 2020-01-15 02:40:07

fsmithred wrote:

HA! I got it to boot. It was dropping me to initramfs prompt and complaining about a missing UUID. That was after about a hundred lines of looking for the floppy and looking for a raid array (neither of which exist).

way to go! unfortunately after doing the same edits as you described above with the live refractainstaller i find myself in a similar position: i enter the encryption password to GRUB (my /boot is in the encrypted Btrfs partition) and can start booting. however after the line "loading the initial ramdisk" i get tons of the following messages mdadm: No arrays found in config file or automatically then Gave up waiting for root file system device before finally dropping me to an initramfs prompt like you, complaining about the UUID not existing.

strangely enough the first time that i booted the fresh install and this occurred, in the initramfs shell i was able to use cryptsetup to open the LUKS partition, then exit and the shell and continue booting. however on every consecutive reboot since then the cryptsetup command is no longer available in the initramfs shell thus i cannot get any further?! the one time that i was able to use cryptsetup and continue booting, while i could not log in via slim i did run an apt upgrade in a terminal, perhaps that removed cryptsetup somehow? or maybe this points to the problem: the crypt stuff is not getting loaded during boot and thus the initramfs cannot be found in the unopened encrypted partition. however i see the crypt insmod in the GRUB config.

fsmithred wrote:

So, at the boot menu, I edited the entry to remove that uuid and changed it to
root=/dev/mapper/sda3_crypt
And it still dropped me to the initramfs prompt. So I manually opened the volume with cryptsetup, was asked for the password, then I ran exec /sbin/init and it booted.

i tried your fix by changing the root= kernel command line argument to use the /dev/mapper/storage value but ended up with the same thing i.e. stuck in the initramfs shell with no cryptsetup.

at this point i feel like i have hit a dead end. i may try the netinst method to see if at least that gives me a booting system, though the refractainstaller is refreshingly simple and straightforward.

thanks as always, let me know if you make any progress.

peace, w

fsmithred · 2020-01-15 17:24:48

I missed you in IRC by a few minutes. There's a 5.3 kernel in beowulf-backports. And there's a refracta live-iso I made with that kernel for testing purposes. It's here: https://get.refracta.org/files/experime … 5_0440.iso

To install a backports kernel:
Add this line to /etc/apt/sources.list, apt-update, find the kernel you want and install it.
deb http://deb.devuan.org/merged beowulf-backports main

Run:

apt -t beowulf-backports install linux-image-5.whatever

reboot. The highest version kernel will be the first choice in the boot menu.

fsmithred · 2020-01-17 01:31:57

I think there are two problems.

1. cryptsetup was not in the initramfs, even though I had CRYPTSETUP=y in /etc/cryptsetup-initramfs/conf-hook. I added dm_crypt and dm_mod to /etc/initramfs-tools/modules and ran update-initramfs -u. That fixed the initramfs, but it still dropped me to initramfs prompt. It was still looking for the non-existent UUID, which is the UUID of /dev/mapper/sda3_crypt.

2. GRUB can't deal with luks2 format. That seems to be OK with a separate boot partition, but it might be complicated by the btrfs. That's a guess. If I'm right, creating the luks volume with cryptsetup luksFormat --type luks1 <device> might be the answer.

Note: on one reboot, I created /run/cryptsetup at the initramfs prompt before I opened the luks volume, and that got rid of the error message about the missing locking directory. But it didn't help with booting.

https://www.debian.org/releases/buster/ … etup-luks2
https://cryptsetup-team.pages.debian.ne … -boot.html

waynedpj · 2020-01-20 18:14:00

for visitors from the future:

i was able to solve this with the help of fsmithred (thanks!) and a good amount of trial & error. the details are in the above posts but to summarize there are 2 ways that we tried to install Devuan ASCII 2.1 to existing Btrfs subvolumes. for both cases in another shell/vt the encrypted partition must be opened first with cryptsetup and the Btrfs subvolume(s) then mounted before starting the installation.

with the netinst/Debian installer: set the Btrfs default subvolume to be the desired one on which to install /, perform the installation to the Btrfs partition, then copy /home, etc. to their desired subvolumes after installation
with the live/refractainstaller from the command line: edit the refractainstaller script to add the correct --options subvol= to the mount commands for the / and /home partitions, which are given as the /dev/mapper/XXXX

in both cases after the installation is completed some additional steps need to be performed in a chroot, preferably before rebooting:

apt update && apt upgrade
edit /etc/fstab and /etc/crypttab appropriately for the subvolumes
set CRYPTSETUP=y in /etc/cryptsetup-initramfs/conf-hook
if using encryption and the installer did not take care of it, set GRUB_ENABLE_CRYPTODISK=y and GRUB_CMDLINE_LINUX_DEFAULT= in /etc/default/grub
update-grub
updateinitramfs

there may be some small steps that i missed but the details are in the above threads. i ended using the refractainstaller as the steps were all laid out in the script that is well documented and easy to follow, making it simple to perform the above mentioned edits. good luck!

Last edited by waynedpj (2020-01-20 18:31:27)

The officially official Devuan Forum!

#1 2020-01-11 16:01:32

[solved] install to existing encrypted Btrfs subvolumes

#2 2020-01-13 03:50:03

Re: [solved] install to existing encrypted Btrfs subvolumes

#3 2020-01-13 11:11:02

Re: [solved] install to existing encrypted Btrfs subvolumes

#4 2020-01-13 13:12:04

Re: [solved] install to existing encrypted Btrfs subvolumes

#5 2020-01-13 14:11:13

Re: [solved] install to existing encrypted Btrfs subvolumes

#6 2020-01-13 17:14:51

Re: [solved] install to existing encrypted Btrfs subvolumes

#7 2020-01-13 18:04:41

Re: [solved] install to existing encrypted Btrfs subvolumes

#8 2020-01-14 19:48:32

Re: [solved] install to existing encrypted Btrfs subvolumes

#9 2020-01-14 22:42:53

Re: [solved] install to existing encrypted Btrfs subvolumes

#10 2020-01-14 23:29:58

Re: [solved] install to existing encrypted Btrfs subvolumes

#11 2020-01-15 02:40:07

Re: [solved] install to existing encrypted Btrfs subvolumes

#12 2020-01-15 17:24:48

Re: [solved] install to existing encrypted Btrfs subvolumes

#13 2020-01-17 01:31:57

Re: [solved] install to existing encrypted Btrfs subvolumes

#14 2020-01-20 18:14:00

Re: [solved] install to existing encrypted Btrfs subvolumes

Board footer