SPDX-License-Identifier
CC-BY-SA-4.0

New Virtual Machine on KVM-qemu

I have a project ( Home Automation Upgrade (2022)) that requires a virtual machine with booting by EFI (Extensible Firmware Interface), and as part of this campaign I needed to upgrade my procedures for creating virtual machines in general. In particular I wanted to burst into the new millennium with a btrfs (binary tree filesystem) for the root, which has been the default for OpenSuSE for several years and which has a lot of technical advantages.

Of course I knew that all this would be a big learning experience, so instead of delaying the upgrade by making the btrfs root a prerequisite for running Home Assistant, I used the Debian x86_64 image for Home Assistant. It's working well and has given no trouble. But the OS and kernel are not going to get updated, just the Home Assistant software, and we don't get root access to the OS, similar to the situation with commercial home automation appliances. [Update: the OS and kernel do get updated, removing my biggest objection to using the H-A OS image.] All my other machines run OpenSuSE Tumbleweed and my system administration software is configured for that distro, so I continued the sub-project to put OpenSuSE's package of Home Assistant Supervised on a second VM running Tumbleweed.

Dramatis personae: this subprojet involves five hosts:

• Orion is the virtual machine that is going to get Home Assistant Supervised. It is my first machine with a BTRFS filesystem, and is the third with EFI booting (the first VM with EFI). It is currently operational and is awaiting installation of Home Assistant Supervised.

  [Update: I discovered something nasty while putting together all the pieces of the puzzle. H-A Core lives in a Docker instance, for which they want the overlay2 union filesystem. But the Docker docs warn that overlay2 is bad mojo on any underlying filesystem that has copy-on-write, specifically btrfs; it is typically used over ext4. Docker does have a storage driver specifically for btrfs, but this isn't the storage driver that H-A instructions specify. I see a big can of big worms opening, discussed more at the end of this document.]

• Dragon currently is running the Home Assistant OS image, but when ready, Orion will be renamed to Dragon. Dragon has EFI, but I actually don't know what it's using as a root filesystem.

• Ocelot is a disused VM for a project that didn't work out. But I saved its identity, IP addresses, and host keys. I'm temporarily resurrecting it for this demo.

• Jacinth is the host of Dragon and Orion. It's an Intel NUC6CAYH with a Celeron J3455 CPU. Its main jobs are the main router and the directory service master. It runs 24/7, and the low power of the Celeron is appreciated. And it's not that slow, though others of my machines are faster. My IoT devices use Z-Wave to communicate, via a proprietary USB dongle (radio) called a Z-Stick, which needs to be passed through to Dragon. But to avoid taking it away from Dragon while I try to set up Orion, I provided a flash memory stick that can be passed through to Orion, on which I deposited my SSH public key, which the Tumbleweed installer will offer to copy to become /root/.ssh/authorized_keys, helpful for post-installation setup.

• Iris is the host of Ocelot. It's identical to Jacinth, effectively a hot spare, but its main jobs are backup storage, distro installation master, and audio-visual. As on Jacinth, I provided and passed through a flash memory stick with my SSH public key. Fakeout: it had an old rescue disc on it, and when Iris woke from Linux hibernation, it rebooted into the rescue disc. Wipe and reformat the media if needed.

Here's what I did to get OpenSuSE Tumbleweed (with jimc package selection, admin scripts, etc) onto a KVM-qemu virtual machine with EFI booting and a btrfs root filesystem. The reader should keep in mind that, while documenting this installation, my goal is not only to share commonly applicable points with the general community, but also to aid my memory in reproducing my idiosyncratic preferred features on future virtual machines. This to-do list is the union of quite a number of false starts and repair interventions. It was done first on Orion, and again on Ocelot to verify that the procedure was working reliably and/or to shake out remaining bugs. The procedure is described in terms of Ocelot.

Create a New Disc

This is all happening on the VM's host, Jacinth hosting Orion, or Iris hosting Ocelot. On my machines that host VMs I have a partition with a lot of free space, called /s1, a directory /s1/kvm, and within that, a home directory for each guest.

• The new machine will be called ocelot. The VM with the Home Assistant image is called dragon, and in a very few places this name appears on Ocelot, to imitate the procedure on Orion, where I wanted to avoid forgetting to rename things when Orion is eventually renamed to Dragon.
• Make the homedir: mkdir -p /s1/kvm/ocelot
• Change to that directory; most examples will use paths relative to it.
• This will be the small size of VM: 16 GiB (2³⁴ bytes). Rather than copying this much out of /dev/zero, the easiest way to create a new disc is:
  fallocate --length 16GiB ocelot-disc1.raw
  The allocated range contains synthetic zeroes due to metadata magic, and happens almost instantly.

Create the Installation DVD

I knew that I would need to repeat these steps several times (ended up 8 times), so I downloaded the OpenSuSE Tumbleweed installation DVD. Normally the ISO file would go on the VM host, but I've found it convenient to put one copy on my installation server, and let all the hosts get it by NFS. How I use it:

• ln -s /net/distro/s1/SuSE/SuSE-build/x86_64/99.8/iso/openSUSE-Tumbleweed-DVD-x86_64-current_j.iso ocelot-cd.iso

Create New XML for the VM

I'm creating new XML from scratch using virt-manager, rather than copying an old VM's XML, to get the latest best practices and to leave behind accumulated cruft. Then I will compare the new and old XML, and I'll bring forward a few important details.

• Prerequisite: you will need to install SuSE package qemu-hw-usb-host to pass the Z-Stick through to the guest, as well as to create the XML to do the pass-through. Do that now if not yet installed.

• Prerequisite: Make sure the installation disc can be read. If NFS mounting is flaky, virt-manager will hang until it comes back (i.e. forever) when writing out the XML, and you can't kill it. file has to actually read the file, so use this command. Do it in the background so if it hangs you can just ignore it. -L means to follow the symbolic links.
  file -L ocelot-cd.iso &

• Start up the virt-manager GUI (no command line arguments).

• Create a new VM; use the icon or the file menu. It will pop up a sequence of windows to gather information.

• In the first pop-up, it wants to know where the OS installation media is. Click Local Install Media. The architecture should be x86_64. Click Forward.

• Use the file browser to find the link to the ISO image (ocelot-cd.iso). Click on Browse Local to see outside the default directory of /var/lib/libvirt/images which is too small with my partition layout. Specify the OS type. There's a checkbox for automatically detect from the installation media, but it couldn't figure it out; turn that off. Type openSUSE Tumbleweed; just type the first few letters and a match list will be shown that you can pick from. Click Forward.

• Give it 2048 MiB of memory and 2 CPUs, my usual resources for this kind of VM. Click Forward.

• Enable storage, turn it on. Pick Select Custom Storage. Click on Manage to get a file browser. Go direct to Ocelot's homedir; look in the sidebar. Click on ocelot-disc1.raw, then Choose Volume. Click Forward in the containing popup.

• Ready to begin installation. Change the VM's name (to ocelot). Click on Customize configuration before install. I always use bridge networking and the host already has a bridge set up: br0. Under Network Selection (click on the headline), pick Bridge Device and fill in the device name br0.

• Now we're on the customization page.

  • I generally leave the title and description blank; the title ends up in titlebars, if specified, and is messy. You have a new UUID.
  • Under Hypervisor Details, use the Q35 chipset (from 2007) instead of the i440FX chipset from 1996. See Virtual Win10 — i440FX vs Q35 by Gordan Bobic (2022-01-05) for a useful discussion.
  • An important choice is the firmware: the default is BIOS (seabios), but we're moving away from that to EFI. But which EFI? See QEMU — Which Firmware to use, OP kitman (2021-07-xx). All of them use Intel's TianoCore EFI booter, but they have different certificates installed, used when verifying boot images, The ones including the string ms have Microsoft signing certs, opensuse is for Tumbleweed and Leap, and suse has SLES certs. The ones without a vendor name have no certs and will fail Secure Boot unless you import your own cert (or turn it off). Since this VM will run Tumbleweed, I picked UEFI x86_64: /usr/share/qemu/ovmf-x86_64-opensuse-code.bin.
  • Under Memory in the sidebar, I was tempted to tweak the Current Memory down to 1024MiB (let it use the memballoon feature if it needs more), but I hit a snag later, and didn't do that in the future.
  • Under NIC, set the correct MAC address. On my net all the MACs are in /etc/ethers and the DHCP server has fixed IP assignments based on the MAC. I have a local convention for VMs and bridges: 52:54:00 (the assigned range for KVM) followed by the last 3 octets of the fixed IPv4 address, converted to hex with leading 0's. For this machine: 09:c8:e4. Leave the IP address set to unknown; DHCP will reveal it later.
  • The Home Assistant instructions mention the Guest Agent. But under Channel, we see that it's already set up.
  • Home Assistant is going to communicate with the various IoT devices using a Z-Stick (radio) which plugs into USB, which therefore needs to be passed through to the VM. However, I didn't want to take the Z-Stick away from Home Assistant on Dragon for a week while I set all this up. So I plugged in (on the host) a flash memory drive and left it there. For use in OS installation I copied into its root directory my personal SSH public key (extension of .pub required), which could be added to /root/.ssh/authorized_keys . Use lsusb on the host to discover the vendor and product identifiers (4 hex digits each) for both the flash stick and the Z-Stick.
  • Now click Add Hardware at the bottom of the sidebar. In the resulting dialog pick USB Host Device. A list similar to lsusb output appears. Click on the device you want to pass through. (At present this would be the flash stick.) Click Finish. There's a new item on the sidebar for this device.
  • Re-check that the installation media can be mounted, if using NFS. Twice it got into an infinite loop of polling on 3 FDs which always times out. I'm pretty sure the issue is NFS mounting the installation media.
  • Click Begin Installation (upper left). Cross fingers.

• Steps that virt-manager does autonomously:

  • Writes the new VM's XML.
  • Defines the guest from the XML (virsh define /path/to/XML).
  • Execs virt-viewer, connecting to the (not yet running) guest's console by VNC.
  • Starts the new guest.

Booting the Installer DVD

• The boot process:

  • The TianoCore EFI booter starts up (though its logo goes past too quick to be seen). The DVD is booted with no drama. Hit an up or down arrow key to avoid auto execution of Installation so you can take notes.
  • When ready, navigate (arrow keys) to Installation and press Return.
  • Grub tells the EFI booter to boot the installer. Yes, Secure Boot is enabled.

• Early installation steps:

  • It boots OpenSuSE, and starts the pre-installer. Plymouth blanks your view of things; hit Escape to see the action. Be patient for hardware and network configuration.
  • The installer GUI appears. It goes through some pre-configuration steps including bringing up the NIC, which gets its correct IPv4+6 addresses by DHCP and/or RFC 4864 autoconfig.
  • It will help to expand the virt-viewer window so it shows the whole thing including the Next button in the lower right corner.
  • The license is shown; there are also dropdown lists for the language (en_US default), the keyboard layout (en_US), and the translation for the license. Click Next.
  • Allow it to use the online repos (3 of them). You'll be glad you did because you will get updated packages that came out since the DVD was downloaded.
  • System role: Server. This installs the minimum software. I have package and configuration management, and this choice results in installing the fewest packages that I am just going to remove later. Click Next.

Partitioning and BTRFS

Here we reconcile SuSE and local policies about storage, and develop new policies for btrfs.

• It would be very convenient if Guided Setup could be used. But they propose to give 0.5Gb to the EFI partition. I don't know what they're planning to stuff into 0.5GiB, but that's way overkill. Debian, Raspbian, Ubuntu and Gentoo put the kernel and initrd into the EFI partition, and I found that 64MiB was enough for one kernel at a time (Gentoo on Raspberry Pi). Tumbleweed puts the kernel in /boot which, if not a separate partition, is in the root, and only actual EFI stuff is in EFI: currently just over 10MiB. On a machine dual-booting Windows this went up to 20MiB. Normally I generously allow 64MiB. But you can't shrink partitions like that and transfer the space to other partitions; you have to delete all the partitions and do them over. In the Expert Partitioner.

• Start up Expert Partitioner. Start with Existing Partitions (there aren't any, yet). Select /dev/vda. By default they put the partition table in a GPT (GUID Partition Table), vs. MBR (Master Boot Record). Make sure you're getting a GPT; it's technically much superior.

• This procedure will be repeated for the EFI, swap and root partitions, in that order. After selecting /dev/vda or one of its partitions, click on Add Partition. Go through these pages (illustrated for EFI):

  • Custom Size: 64 MiB for EFI, 2 GiB for swap (the memory size), or for the root partition change to Maximum Size, meaning everything remaining on the disc. Click Next.
  • Role: EFI Boot Partition, Swap, or Operating System. Click Next.
  • On the third page you can accept all their presets, except for one: click on Fstab Options. Change to Mount by Label, and fill in the desired label, which would be EFI-DRAGON, SWAP-DRAGON, or ROOT-DRAGON. In repair activities it's important that the labels should not duplicate those of the host, and adding some unique suffix can ensure that, such as the hostname, or a locally managed chassis number. Upper case is recommended for compatibility with MS-DOS. I don't like to mount by UUID because while the UUID is guaranteed unique, the resulting 256 bit (32 hex) integer is unreadable and isn't recognized as a name by humans. (Click on OK, and Next on the containing page.)
  • When you create the root btrfs filesystem, don't Enable Snapshots unless you want to ascend a learning curve which I'm not ready for yet. However, SuSE splits off about 8 subdirectories into subvolumes, with good reasons described in the docs, and if you ever do enable snapshots, having them out of the root's snapshots will be really helpful. So I left the subvolumes in place.
  • Finally click Accept to accept the finished partition table, and Next on the containing Partitioning page.

More Mundane Setup Steps

• Timezone. Click on the map, or select from the two dropdown lists. You want the clock set to UTC.
• Skip user creation (click the bottom radio button). My users are identified and authorized from a set of directory servers.
• Give the root password (twice). Import your SSH public key if available.

Installation Settings Overview

• Booting: Grub2 EFI, secure boot, update NVRAM (no changes).
• Software: Suppress Help and Support Documentation. You probably should not suppress any other package groups. I have some packages that my configuration management needs, and it's going to be nicer to select them now than to make my script install them. Click on Details and change to the Search tab. The packages are:
  Already selected: rsync curl
  Need to add: m4 expect pam_ssh pam_krb5 krb5-client libcap-progs
  Click Accept, and Continue (for the automatic items).
• Default target, System and Security: no changes. You will have the SuSE firewall, and SSH with its port open, etc.
• Network: First switch to Wicked (from NetworkManager, which is better for a laptop). [Update: H-A wants NetworkManager; belay the switchover.] Then click the Network headline. Under Hostname/DNS, give your static hostname (ocelot), and for Set Hostname via DHCP, change to No. I want to rename the interface from enp1s0 to en0, but don't try it here; it got confused once and killed the net, preventing installation. Click Next.
• Finally, click Install and confirm. You're on your way.
• 823 packages to install, 1.9GiB from the net plus mostly the DVD. I didn't write down timing, but it took 10-15 minutes. The new machine booted successfully, unattended. It got its correct IPv4+6 addresses.

Checkout and Configuration

This particular instance was created just to verify and collect procedures for the initial installation, so I'm not going to go through the full post_jump procedure (install my wanted packages, (re)enable services, remove unwanted packages, online update, install my configuration files). But there are a few items needed before post_jump that should be documented.

• Will it let me log in to the console as root? Yes.

• What SSH host keys is it using? The wrong ones, created randomly. How about SSL host keys? It has none, yet.

• In each VM's homedir I made a directory called ./config into which I could drop host keys and similar important items, and just rsync the whole directory onto the virtual machine.

• To get SSH and SSL right on Ocelot, I did these steps:

  • On Ocelot: echo "PermitRootLogin yes" > /usr/etc/ssh/sshd_config.d/50-rootlogin.conf
    Normally only publickey login is allowed, but I botched providing my SSH public key during installation, so I need to turn on KeyboardInteractive capability. Remove this config file later.
  • systemctl restart sshd #reload isn't enough
  • On Iris: mkdir -p config/etc/ssh config/etc/ssl
  • rsync -a /home/backup/ocelot/etc/ssh/. config/etc/ssh/.
  • rsync -a /home/backup/ocelot/etc/ssl/. config/etc/ssl/.
  • I forgot to put my SSH public key on the flash memory stick. Let's include the complete authorized_keys in config:
    mkdir -p -m 0700 config/root/.ssh
    rsync -a xena:/root/.ssh/authorized_keys config/root/.ssh/
  • rsync -a config/. ocelot:/. #Give root password
    I have SSHFP records for all hosts, and ssh prints a warning that the SSHFP data doesn't match the proffered host key, but it continues anyway to keyboard-interactive authentication, which Ocelot's sshd accepts.
  • On Ocelot: systemctl restart sshd #With host keys from backup
  • On another host with your key agent: ssh ocelot uname -a #Your key is accepted for admission.

• Additional files that are good to have in ./config/ ; most are copied from the host.

  • From /root : .gnupg/ .exrc .bashrc .profile (→.bashrc) .dmrc .forward backup.pln .ssh/
  • From /etc :
    • passwd group shadow hosts
    • hostname (contains ocelot not the host's name (iris))
    • ssh/ : ssh_config sshd_config moduli (just copy these three)
    • ssh/ssh_host_*_key{,.pub} for rsa, dsa, ecdsa, ed25519 from the backup directory for Ocelot (not Iris). If you have to generate new ones, the procedure is documented below. DSA is beyond end of life, but it should be purged in a separate mini-project, not one host at a time.
    • /etc/zypp/repos.d/hass.repo , the repo definition file for Home Assistant (this won't be on the host)
  • From /m1/custom:
    • Copy couchnet.sel
    • Create extra.sel with these packages: keystone-orion home-assistant gpg-pubkey-2e2b22c6-5c3c929d
      The key is for the system:homeautomation OBS Project (2019). Remember to change to keystone-dragon when Orion is renamed.
    • conffiles/etc/zypp/repos.d/hass.repo , the repo definition file for Home Assistant. It needs to be here to keep audit-repos and audit-pkgs from deleting it as cruft.

• How to create the host keys and SSHFP records in ./etc/ssh :

  • This needs to be repeated for each algo: rsa dsa ecdsa ed25519. Illustrated for rsa.
  • ssh-keygen -t rsa -b 3072 -C orion-2023 -f ./ssh_host_rsa_key
    For ecdsa use -b 384. For ed25519 and dsa, omit -b.
    Yes overwrite. Must be unencrypted; hit return when asked for the passphrase. The key fingerprint and ASCII art image will be printed.
  • ssh-keygen -r orion.cft.ca.us -f ./ssh_host_rsa_key.pub > ./ssh_host_rsa.sshfp

• Update jacinth: /home/hostdata/hostdata.db uncommenting the stanza for Orion. In its hostgroup line change it from grub_noefi to grub_efi, add btrfs, and add jacinth_vms. To Jacinth's hostgroups add orion_vhost. It should already have dragon_vhost. Add Orion's SSHFP records, just hash type 2 (the long ones).

  Install the derived files. Copy changed ones (non-DNS like /etc/hosts) into the post_jump storage area.

• An important item that gave me a lot of grief: in /etc/default/grub you need:
  SUSE_BTRFS_SNAPSHOT_BOOTING="true"

  See this forum post about the issue. Basically, grub2-mkconfig runs all (or most?) filenames through a function make_system_path_relative_to_its_root which chops off the mount point of the partition containing the file. But on btrfs filesystems only, it prepends the subvolume path (rooted at /@ on SuSE). Grub at boot time cannot read such a path, with the result that after you rebuild /boot/grub2/grub.cfg (e.g. after getting a new kernel), the machine cannot boot. Except, if /etc/default/grub contains SUSE_BTRFS_SNAPSHOT_BOOTING="true", the -r option is added to the call to grub2-mkrelpath (from btrfsprogs package), which makes it behave like on other filesystem types, i.e. no prepended subvolume path.

  SuSE includes SUSE_BTRFS_SNAPSHOT_BOOTING="true" by default, but I turned it off because I had not turned on snapshots. Bad move. Now that the option is true again, bootable grub.cfg is being produced again, including on hosts with ext4 root filesystems.

• When I set up Orion, before post_jump, nscd was enabled and running, and ssh_keygen couldn't get user info for UID 0 (root), preventing sshd from starting. I've had sooooo much trouble with nscd. Kill it and disable it, or let audit-scripts disable it. Now sshd will start.

Things to Do with Power Off

• Before monkeying with Ocelot any more, I'm powering it off and making a backup copy (took only 2 or 3 mins with this minimal content). Don't try this with the VM running; if you restore it the result will be corrupt.
  pigz -c ocelot-disc1.raw > ocelot-disc1.pristine

• Also we no longer need the DVD, so I renamed it to ocelot-cd.iso.nfs and created a zero-length ISO image to replace it. The booter won't treat it as removable and removed, but just gives a one-line error message and boots /dev/vda. I can bring the DVD back if I need to use it as a rescue disc (see the More option on the initial screen).

• I want the domain XML in the home directory where I can get at it. The domain XML is hiding in /etc/libvirt/qemu/ocelot.xml , and one could copy it from there, but the straight-arrow way to extract it is:
  virsh dumpxml ocelot > ocelot.xml

• I also want my NVRAM in the homedir. It's stored in /var/lib/libvirt/qemu/nvram/ocelot_VARS.fd ; copy that file (2¹⁷ bytes) into the homedir. Then edit the XML to point to the new containing directory; look for the <nvram> element early in the file.

• To install the altered XML file, just do:
  virsh define ./ocelot.xml
  This assumes that you didn't alter the VM's name or UUID. If you did so, e.g. if you're cloning one machine as the basis for another, be sure to give it a unique name, and execute uuidgen (command line options are not necessary) to print out a new UUID, and paste it into the file. If the VM was defined already with a different UUID, before defining it you need to do:
  virsh undefine --nvram ocelot
  The --nvram option gives it permission to toss the machine's nvram; without this, virsh will refuse to undefine.

• I have a script that runs virt-viewer. The command line that it execs, if my laptop is the host, is:
  virt-viewer -c qemu:///system -w -r $guest
  Or in the more common case that the host is remote:
  virt-viewer -c qemu+ssh://root@$host/system -w -r $guest
  To see the early boot process, start virt-viewer before booting the guest. Or you can start it any time later.

• On Iris (the host); virsh start ocelot
  It boots right up, no hassle.