EFI Booting a Virtual Machine with BTRFS Root

Symptom:

You have a virtual machine. (Mine is on KVM.) It has a BTRFS filesystem providing its root directory, a subvolume. It also needs to boot using EFI (Extensible Firmware Interface) rather than legacy BIOS.

I'm running OpenSuSE Tumbleweed, and the layout of the BTRFS goes like this:

• There is a subvolume called '@' (this is common but some distros call it 'root'), which is mounted on the '/' mountpoint,
• There are various other subvolumes such as 'var', which are mounted on the same-named mount points in the root.
• 'var' and friends, when being mounted, are referred to as '@/var', so they are directly under '@' in the subvolume hierarchy, not at the top level.
• The EFI and swap partitions are outside of BTRFS; they're real partitions.

I run the SuSE installer, and the resulting disc can be booted. I install and remove other packages and do other setup and customization steps. I sneeze. Now it won't boot. Grub claims it can't read font and theme files, nor can it read the kernel and initrd; for the latter it gave this error message:

Loading Linux 6.1.4-1-default
Error: ../../grub-core/fs/btrfs.c:2156: file '/@/boot/vmlinuz-6.1.4-1-default' not found

How do I recover, or, what could I have done to avoid breaking the bootloader.

What's happening:

Compare these mount commands.

losetup --find #prints /dev/loop0
losetup --partscan /dev/loop0 orion-disc1.raw
mount /dev/disk/by-label/ROOT-DRAGON /mnt #Wrong
mount /@/dev/disk/by-label/ROOT-DRAGON /mnt #Wronger
mount -o subvol=@ /dev/disk/by-label/ROOT-DRAGON /mnt #Correct

The one tagged wrong looks like it got the root partition, no error messages. /mnt/boot contains grub.cfg and the font and theme files that it ought to. /mnt/boot/vmlinuz points to vmlinuz-6.1.3-1-default which points to ../usr/lib/modules/6.1.3-1-default/vmlinuz which is a large, kernel-sized plain file. Looks good so far, but why won't the disc boot?

The one tagged wronger is seen at boot time; see below for how to fix this.

The one tagged correct mounts the root subvolume, which is where the content really is. Now we can reinstall the bootloader.

How to fix:

To un-sneeze, i.e. to make sure this problem is banished, look at /etc/default/grub in the mounted filesystem. It needs these control parameters set:

• GRUB_USE_LINUXEFI="true" #If you want it to use EFI to boot
• SUSE_BTRFS_SNAPSHOT_BOOTING="true" #To mount BTRFS the right way

On the second item, I figured that I had not enabled snapshots, so I had better not turn that one on. Bad move! Basically, grub2-mkconfig runs all (or most?) filenames through a function make_system_path_relative_to_its_root which chops off the mount point of the partition containing the file. But on btrfs filesystems only, it prepends the subvolume path (rooted at /@ on SuSE). Grub at boot time cannot read such a path, with the result that after you rebuild /boot/grub2/grub.cfg (e.g. after getting a new kernel), the machine cannot boot. Except, if /etc/default/grub contains SUSE_BTRFS_SNAPSHOT_BOOTING="true", the -r option is added to the call to grub2-mkrelpath (from btrfsprogs package), which makes it behave like on other filesystem types, i.e. no prepended subvolume path.

Now, to reinstall the bootloader, go through these steps. Other distros actually have this scripted under the name arch-chroot, ubuntu-chroot, etc. Actually I do this often enough that it's worth it to write my own script; here it is. Beware, it has these idiosyncracies which definitely differ from stock SuSE as well as other distros:

• All partitions have labels and mounting is always by label.
• It's assumed that the target will be mounted on /mnt . In some distros, miscellaneous devices are mounted on subdirs of /mnt and covering these mount points with your VM's partition would be unfriendly.
• I have a command hostgroup which consults a database of group memberships of hosts. For this script I use btrfs (hosts that have the root on BTRFS) and grub_efi (those which use EFI).

The steps go like this:

• Pick a loopback device to use.
  losetup -f # Prints /dev/loop0, or another one if occupied
• Attach the VM's disc to that device. --partscan tells it to read the disc file's partition table and create the by-label and by-uuid symlinks, which I use. Give your own name for the disc's filename.
  losetup --partscan /dev/loop0 ./dragon-disc1.raw
• At least once, it took 2 sec for the labels to appear; 1 sec was not enough.
  sleep 2
• Check that the partition scan worked.
  ls /dev/disk/by-label/
• Mount the root partition on /mnt (or whatever mountpoint). Use your own disc label, UUID, or device path.
  mount /dev/disk/by-label/ROOT-DRAGON /mnt
• Mount the EFI partition in its proper place.
  mount /dev/disk/by-label/EFI-DRAGON /mnt/boot/efi
• Bind-mount system partitions. Probably you don't have to bind-mount, but it seems to me that you're less likely to wake sleeping dragons if you stick with just one instance of each one.
  for f in dev sys proc ; do
  mount -o bind /$f /mnt/$f
  done
• Change root into the VM's disc.
  chroot /mnt
• If you got a new kernel, or if other issues invalidated the initrd like getting a disc label or UUID wrong, rebuild initrds for all available kernel versions. (This is the modern replacement for mkinitrd.)
  dracut -vf
• If your /etc/default/grub is messed up, edit it, and then run:
  grub2-mkconfig -o /boot/grub2/grub.cfg
• If there is a question whether your bootloader is installed correctly, run this. It relies on GRUB_USE_LINUXEFI. To force the use of EFI, squeeze in --target x86_64-efi.
  grub2-install /dev/loop0
• Exit from the chroot jail; just…
  exit
• Unmount the VM's disc and all sub-mounts, and disconnect the disc from the loop device:
  umount -v -R /mnt
  losetup -d /dev/loop0