Valid HTML 4.01 Transitional

Upgrading to OpenSuSE Leap 42.1

James F. Carter <jimc@jfcarter.net>, 2015-11-30

This is a history of what jimc did to upgrade the OpenSuSE distro from 13.1 to 42.1.

Index

Quick Links

Table of Contents

Introduction

We are upgrding from OpenSuSE 13.1 dated 2013-09-xx, installed about 2014-01-xx. The new OS is OpenSuSE Leap 42.1. Why the big leap in the version number? If SuSE's historical practice had continued, it would have been 14.0. However, Google's Chromium started omitting the major number and putting minor releases first, to give the impression of rapid development, and Mozilla Firefox followed suit. Maybe SuSE are jumping on the bandwagon of steroidal version numbers.

UCLA-Mathnet and jimc generally try to get about 1.5 years of use out of each new distro version: a balance between the effort to upgrade, and being ready with the new version when support ends for the old one. Here, support means that security patches and major bugfixes are provided. SuSE have changed their support policy: See SuSE's roadmap for the release schedule and a link to the lifetime policy. There is a distinction between releases with long term support (42.1 is one of them) and minor releases. The LTS releases correspond to SLES versions while the minor releases are like service packs. Three months after a minor release is superceded (by another minor one or a major one), it will lose support. Thus with 8 months or so between minor releases, if we install one, we will have to finish upgrading to the next one within a year. We should prefer LTS releases.

But I wonder how frequent the major releases really will be: we don't want to be stuck in a time warp for three years or so, judging from history on SLES.

One possibility is to switch to the Tumbleweed track, which is a rolling distro. I don't know the details on this, but judging from forum postings, you're supposed to do a dist-upgrade frequently, similar to the frequency of online updates (security patches) in a stationary distro. This means the upgrade effort is spread out over the whole year and could be a lot more automated, unless a new version of a package interferes with one of our special configurations (and surprises us when it goes in).

Planning

I always begin a project by stating its goals, issues likely to be encountered, and then the actions to be taken. The distro upgrade is a lot of work, but the goals, issues and action are pretty similar to what we've done in previous years.

Goals

Issues

Actions

Here's an overview of installing a new distro:

Executing the Plan

Infrastructure Setup

This took one day, not a full 8 hour day.

Shell Variable Cruft Check

You can source the file /s1/SuSE/source.me to set useful shell variables for the distro root and the major scripts in distro maintenance. This file was reviewed for being up to date. It auto-sets the architecture (according to the machine it's run on) and the release, taking the lexically highest version in /s1/SuSE/SuSE-build/$ARCH/ , which will be the new version when you create the directory for it. Here is jimc's version of source.me.

Remove Old Distros

We need to keep files for v13.1 until all hosts have been upgraded, but 11.4 and older can definitely be tossed. At home, before cleanup, the repo occupied 167 Gb (oink). After, 73Gb; tossed 94Gb.

At the same time, we need to get rid of the configuration files for old distro versions. These are in /home/post_jump/${RLSE} or on Mathnet, /h1/post_jump/${RLSE}. Being anally retentive I made a directory ./ancient and moved all the obsolete dirs into it. When deleting one of these directories it's a good idea to shred the non-public files. Here's how, illustrated for distro version 10.2:
find 10.2 -type f ! -perm -04 -ls |& less #Are you going to remove the right files?
find 10.2 -type f ! -perm -04 -print | xargs -n 25 shred -u -n 3
rm -r 10.2

Create Directories

Create all the directories for distro components. Owned by root:root, mode 755, except the update dir has to be owned by wwwrun. These are illustrated for the new version, 42.1 and all paths are relative to $di (/s1/SuSE).

Download the Distro

Find it at software.opensuse.org . The x86_64 DVD ISO image goes in /s1/SuSE/SuSE-build/x86_64/42.1/iso/ . Older releases put it in the i586 arch directory and there is a symlink from x86_64/iso, but at least at home, I'm putting the ISOs in the x86_64 directory instead because i586 is deprecated. Also, formerly it was put in e.g. /s1/SuSE/SuSE-distro/x86_64/13.1/iso/ , but (at home) I'm going to mount the ISO image on the empty $sd directory, so the iso directory cannot be inside. This modification is optional at Mathnet.

The line in /etc/fstab to mount the DVD image is:
/s1/SuSE/SuSE-build/x86_64/42.1/iso/openSUSE-Leap-42.1-DVD-x86_64.iso /s1/SuSE/SuSE/x86_64/42.1 iso9660 ro,loop 0 0

Which ISOs to download? Clearly we need the DVD. I generally download the network installer CD; however, more and more machines lack CD drives. You can copy it to a USB flash storage device, but if you're going to do that, it makes a lot more sense, particularly for the rescue system, for the system image to be obtained from the USB storage rather than across the net. Making that work will take some research. Link to jimc's instructions to make a bootable USB flash drive.

Unpack the Distro

The unpacked ISO takes exactly as much space as the ISO itself, about 4.7Gb each. It's very tempting and easy to mount the ISO via a loopback device. However, in the past I put the iso directory, the local distro signing key, and the Packman key, in the root with the SuSE package signing keys, which would be impossible with a mounted ISO because it has to be readonly. At home I will modify post_jump and audit-pkgs to look for keys in all the sub-repos. This is optional at work.

Package keys wanted: The first two come with the main distro DVD; the others go in the CouchNet or Mathnet repo.

These keys are obsolete:

How to identify a public key:
gpg gpg-pubkey-0cc9523f-4a9865cc.asc
It prints: pub 1024D/0CC9523F 2009-08-28 UCLA-Mathnet Distro Signing Key <distro@math.ucla.edu>

Check Update Download Script

This script is $di/bin/rsyncsuse.sh and is executed from /etc/cron.d/distro-maint with this content (I download on Tuesday AM; Mathnet downloads more frequently):
10 0 * * 2 root /home/httpd/htdocs/SuSE/../bin/rsyncsuse.sh > /tmp/system/rsyncsuse.debug 2>&1

The script downloads updates for all versions for which update directories exist, except, it has a list of architectures, and it checks if any hosts have each version and architecture, and skips the download if none do. It's suggested that you edit the hostgroup entry for your virtual machine right away and put it in v42.1 and x86_64 (and not down), so the updates for the new version will be downloaded.

You normally don't have to actually edit the script (except for a change in the URL from which updates are obtained), but you should review it so you know what it's doing. Unfortunately SuSE dumbly inserted a subdirectory called leap in the download path, and I needed to edit the script and to download v13.1 and v42.1 in separate subroutine calls. Hiss, boo! Link to jimc's modified download script.

rsyncsuse downloads on a schedule, but if you run it by hand, give the -f option to download unconditionally. The outcome appears in /tmp/system/rsyncsuse.log .

Virtual Machine

The virtual machine for testing is called Oso, and is hosted on Diamond, the repo site, using KVM as the framework and qemu as the virtual executor. The guest connects itself to the host's network bridge (br0) and thinks that it is directly on the local network. (Look at Simba for an example of bridging.) Setting this up took about 0.5 days. It helped that I already had the XML file for this VM, for testing previous new distro versions.

It turns out that we're going to need to test converting the architecture from i686 to x86_64 at the same time as the version upgrade. I have a virtual machine for that called Petra. It was gotten into shape similarly to Oso.

VM Specifications

Oso is defined in this file. Key parameters of Oso and Petra are:

Petra formerly had i686 architecture, and correspondingly for its image for v13.1. The VM architecture was changed to x86_64 to test upgrading. A i586/i686 image will boot and run on x86_64 hardware.

Formerly for the CD I had a symlink in /s1/kvm to whichever image, named for that image, which was named explicitly in the machine's XML definition file. This time I'm calling it just oso-cd.iso, and I can change the CD if I need to, without editing the definition and un/defining the VM anew. (Requires the machine to be powered off and restarted, though.) But changing the boot order still requires the VM to be un/defined. To undefine and redefine a VM:
virsh undefine oso
virsh define ./oso.xml

The definition file has the CD first in the boot order, and the hard disc second. To permanently not boot from the CD you can switch the boot order, which requires un/redefining the machine. However, I made a discovery: If oso-cd.iso is a symlink to /dev/null, the BIOS will not boot from it, same as if a physical drive has no disc in it. It goes on to the hard disc promptly and with no error messages.

Update VMs

Oso and Petra got all the current updates for v13.1 plus local configuration changes made since they were last used.

Save Disc

Save a copy of Oso's and Petra's disc. Command line:
cd /s1/kvm/oso
gzip -c disc1.raw > disc1.131final.gz

The raw disc ended up with 17Gb; why did I pick that number? Compressed size was 4.4Gb. Compression took 10.5 mins. To restore, make sure the machine is not running, and:
zcat disc1.131final.gz > disc1.raw
Restoration took 7 mins.

Start Oso

Oso is going to boot from the network installer. The command to start a VM is
virsh start oso
Then you need to connect to its console. If you have to do this on the virtualization host (not recommended, see next paragraph), do this:
virt-viewer -w -r oso
-w = wait (vs. dying) for it to start up. -r = reconnect (vs. exiting) if it reboots.

Running virt-viewer on the machine with the display (recommended):
virt-viewer -c qemu+ssh://root@diamond/system -w -r oso
This works as a non-root user provided it has a remote execution privilege as root on the hosting machine.

The VM's installer couldn't communicate with the repo server. See the next paragraph for how to get past this mess.

Untangle Network

When you boot the network installer disc on the virtual machine it uses a network interface that doesn't exist, with baleful effects on its ability to boot over the network. Here is the overly complicated procedure to get it running.

Initial Installation

The first step will be to upgrade Oso using the SuSE installer. This took about 4 hours.

Lurking Dragon

See this bug report about grub. Before beginning the upgrade, edit /etc/default/grub_installdevice and make sure it's installing itself in the MBR. 99% of the time you want the device to be (hd0), not e.g. (hd0,1) which would install in the root partition's boot sector. To detect whether (hd0) is correct you could do:
grub2-probe -t drive /boot/.
and remove the partition number, e.g. it might print (hd0,msdos2).

In hindsight the most trouble-free way to handle this issue would have been: First re-verify that (hd0) refers to the disc you're booting from, on all hosts. Then edit /etc/default/grub_installdevice on the master site and install on all hosts (v13.1 and v42.1). Install it as a new file in both versions' post_jump directories. If some host is found that boots from other than (hd0), edit specially and copy it to /m1/custom/conffiles/etc/default/grub_installdevice . See the bug report for when to use a UNIX device path.

Installation or Upgrade

Formerly the choice of install or upgrade was made partway through the installation. Now you get to choose on the installer's boot screen. Options are: boot from hard disc (default), install, upgrade, rescue system, or memory test.

Boot Screen

It's going to fail to communicate using the nonexistent Red Hat Virtio NIC. See Untangle Network above for how to get it booted and running.

Installation

Here are the steps in an upgrade:

Apache Fixup

Apache would not start. Here are the 2 fixes:

Default Installation

Having saved the disc image from the upgrade, I went back and Installed, taking defaults as much as was reasonable. Here are the details, emphasizing differences from the upgrade.

Package Selection

What should the next step be? I'm very tempted to go forward with setting up the configuration files, so I can have a fully functional machine even if some packages are missing. However, if any new daemon-type packages are installed (like Network Nanny :-) it will be a lot simpler and more complete if I work on all the configuration files as a unit, rather than having to do that step over with the new packages. [Update: package issues continued throughout the following sections.]

Installed Packages

Immediately after the upgrade, Oso had these packages installed. A default installation would be different; in an upgrade it tries to preserve the existing package selection as much as possible. There were 4163 packages installed (oink). See earlier for the results of a default installation.

Package Selection List: Cleanup

In Debian you can specify a list of keystone packages and apt-get (I think) will install them and all their dependencies. SuSE doesn't have this. It's possible that a sysadmin could make a pattern (set of packages) and have it accepted by Zypper, but our list of wanted packages varies by hostgroup, so the equivalent of the list of keystone packages is a little more complicated, managed by the audit-pkgs app.

/m1/custom/couchnet.sel or mathnet.sel is our list of keystone packages. Source is in /src/math/etc/audit-suse. The first step is to clone a new version and get rid of cruft, particularly conditional expressions involving back versions. Now is also a good time to make a judgment for each package whether we really want it to be installed, or if it should be tossed.

Here's a list of points discovered that need to be handled later:

Keystone Packages

After cleanup, couchnet.sel has 590 packages listed. This is rather a lot. Can the list be shortened by listing only keystone packages? That means, we'll rely on keystones to drag in other packages, and not list them explicitly, even if they are important. It turns out that looking at the v13.1 couchnet.sel, there are 697 packages listed (including non-v13.1 packages) and 624 of them are keystones. So this kind of cleanup will have little effect and will not be worth doing.

Command line to distinguish keystone from dependent packages:
rpm -qa | sort | while read pkg ; do \
if rpm -e --test $pkg ; then echo Keystone: $pkg ; fi ; \
done >& /tmp/keystones.ls

New Packages

A major goal is to identify new packages that may be useful in my environment, and to add them to the package list. The package list from the default installation gives useful guidance.

Repo Definitions

Repo definitions are found in /etc/zypp/repos.d . On CouchNet and Mathnet there are four of them:

For v42.1, separate repo files will be needed for updates to open source and non-open packages.

There is a subdir jail which has the original SuSE repos, and Packman and VideoLAN (for VLC). There is also a template subdir in which ARCH represents the architecture. The script audit-repos (called from instsetup and post_jump) retrieves the templates from the post_jump configuration file dirs (/home/post_jump/ or /u/sunset/h1/post_jump), then fills in the correct architecture and puts the result in /etc/zypp/repos.d.

While most configuration files have to wait until after package selection, the repo definitions are needed immediately.

Immediate Configuration Files

Here is how to set up the configuration file directory with enough content that package selection can be finished. The new OS version is 42.1 and the old one is 13.1.

Package Keys

On CouchNet, the non-SuSE package signing keys have to be moved from the main distro root because the main distro was going to be mounted from the DVD ISO image. They are installed by audit-pkgs (-K option), and this script needs to be modified to look for them in the new location. As implemented, the script reads the repo URLs from /etc/zypp/repos.d/*.repo, so keys should be in the sub-repo to which they pertain, e.g. the CouchNet/Mathnet distro signing key should be in the CouchNet/Mathnet sub-repo. Also Packman and VideoLAN.

However, the mksuserepo script wants at least one package signing key in each repo. I copied the SuSE package signing key out of the main distro, to populate the SuSE Build Service repo.

Package List

Here is the package list for CouchNet. It's expected to evolve as problems are found and resolved.

SuSE Build Service, and Packman

Now is the time to try out the new package list. We're going to essentially run post_jump by hand and out of order, to finish upgrading the VM by installing the latest versions of packages not in the main distro. Due to the keeppackages=1 setting, downloaded packages will be kept and we will copy them over to the enterprise mirror.

What is the significance of Packman? SuSE is presently a subsidiary of an American company, Attachmate. (Formerly SuSE was owned by Novell.) Even though SuSE's main physical presence is in the European Union, it needs to comply with the union of EU and American law, specifically regarding sofware patents and licensing. Thus packages that are illegal in the USA may not be hosted on SuSE servers. On the other hand, Packman is wholly under EU jurisdiction, and it can legally ignore software patents, specifically those covering multimedia codecs. That is why Packman is included among the repos in this installation, but why the packages hosted there are kept separate from the SuSE Build Service packages.

Start VM

Start the virtual machine (Oso) first. Make sure you have the upgraded one, not the default installation test.

New Repo Definitions

Emplace the new repo definitions on Oso. Execute this on Diamond (with remote execution privilege to Oso).
audit-repos -n -v -r 42.1 -i oso
And if it looks reasonable, remove the -n. You need to specify the version explicitly because the default is the (old) version currently on the target.
Review the result for being correct.
Toss totally useless SuSE repos from the jail.
Snarf the official definitions for download.opensuse.org-oss.repo and download.opensuse.org-non-oss.repo and replace the ones in post_jump/42.1/etc/zypp/repos.d/jail . Look at the base URL, toss the update repos. Turn on keeppackage=1 .

Send Configuration

Send over the minimal configuration already set up. The most important item here is /m1/custom/couchnet.sel .
/home/post_jump/pushconfig -C -a oso >& /tmp/output
# Execute on the master (diamond/sunset). If the result looks reasonable, change -C to -c to actually push the files.

Erase Unwanted Packages

Better to clear the decks before trying to install SBS packages. The first step is to make a list of what it proposes to erase. Remember to add /usr/diklo/bin:/usr/diklo/sbin to the path (or /usr/math/etc). This is going to take a long time because it needs to make a cache of which packages depend on which, in /var/cache/audit-pkgs . Actually under 2 minutes.
audit-pkgs -v -e >& /tmp/output

It proposes to erase 1725 packages. (In a few cases dependencies are missed and the actually needed packages will not be erased.) Go through the list and try to spot gold among the mud. Maybe clearing the decks was not such a good idea because it's erasing quite a number of packages that are going to reappear in the final installation -- probably because prerequisites and/or requiring packages are not in the main distro, so the packages being tossed are unwanted at present.

These items bear investigation:

Revise couchnet.sel so wanted packages are actually kept, and try again.

When couchnet.sel is correct, add the -c option to audit-pkgs, to actually erase the packages. Complicated dependencies are sometimes missed, and the script will give a reprieve to the needed packages and try again, sometimes several times. Removed 1699 packages; 2464 packages remain.

Up to this point took about 4 or 5 hours.

Something that I removed provided the shutdown and poweroff commands. A substitute is systemctl poweroff, which works. /sbin/shutdown has vanished.

Something that I removed prevents wicked.service (network) from starting. Oopsie! /usr/sbin/wickedd has vanished. Let's recover by bringing the net up manually.
ip link set dev eth0 up
ip addr add 192.9.200.212/26 dev eth0

More issues:

How to initialize a new sub-repo: See $mkr for docs. Step by step:

Recover Deleted Packages

Now I can run zypper without a zillion zingers, and install wicked and systemd-sysvinit (provides shutdown and poweroff commands).

Off-Site Packages

When the VM was upgraded the default SuSE repos were not disabled, and some packages are believed to have been downloaded from them. How to identify them:

Distro Upgrade from SBS

We're going to enable the SuSE remote repos and do zypper dist-upgrade. This means to download and install all packages which are on the machine already and which have a newer version in any known repo. These packages will be saved (not tossed as usual) and then copied to the enterprise mirror, saving a tremendous amount of work, compared with downloading packages individually.

Particularly on Packman, it's important to install everything over a short time, e.g. on the same day, because multimedia packages evolve fairly quickly and may not be installable with back-version supporting libraries, requiring you to re-download and upgrade the whole multimedia suite again later.

Configuration Files

In this section, for each configuration file that CouchNet/Mathnet has altered from the default, we compare with the distro-provided default, with these possible outcomes:

I took a machine with v13.1 on it, upgraded to v42.1, then compared with v13.1 post_jump files, rather than comparing with a pristine v42.1 machine, which means that a lot of v13.1 configuration files are already on the machine. If they weren't altered during the upgrade, I don't have to mess with them (except see *.rpmnew files, below).

rpmnew files

When a package is replaced or removed, configuration files (so designated in the package's spec file) can be disposed of in several ways:

Copy Old Version

I already had the repo definitions set up, so I did these steps:
cd /home/post_jump
mv 42.1 xx-42.1
mkdir 42.1
ssync -a 13.1/ 42.1/
rm -r 42.1/etc/zypp #To lose the v13.1 repos
ssync -a xx-42.1/ 42.1/
rm -r xx-42.1

Compare *.rpmnew Files

There are only 22 rpmnew files. Here are the outcomes:

By Hostgroup Files

In $pj/../byhg there are directories named after hostgroup expressions, and if the target is in that hostgroup, the files therein are installed on the target. It looks like some of them need love.

Compare Conf Files

65 files differ, 434 files don't differ, total of 499 files.

Cruft Removal

I reviewed all the post_jump configuration files (for v42.1) and tossed obsolete ones. Not too many. At the end of this document there is a list of items to be worked on later, and one item is to go over the LSB scripts in /etc/init.d and convert them to systemd units, as much as possible.

Test Upgrading

Supposedly we can now successfully upgrade a machine using the local enterprise mirror and configuration files. The following procedure is slightly edited according to later experience. The first time through, $target refers to Oso, but it's presented that way because this section is going to be followed for the production machines.

Post_jump and Checkout Issues

The first time this was run on Oso, I hit these issues:

Zypper Soap Opera

The dist-upgrade did not go as smoothly as desired. Here are the issues found:

Rescue System Again

User Experience Checkout

The main point here is to check that old packages are still working and newly added packages are useful and functional.

Audio Output

Audio is often hard to get working.

Firefox and Flash

Is Firefox performing its usual content? See Jimc's browser test suite for tests of a range of content.

A particular concern is Shockwave (Adobe) Flash content. Flash is a dinosaur and is totally proprietary; it should have been junked long ago. HTML5 is superior in every way. Nonetheless "should have been" is not the operative word here. The obsolete Flash Player for Linux has been reincarnated as Pepper Flash by Google for Chromium, and the freshplayerplugin acts as middleware to make it useful to Firefox. Check out Youtube and your favorite brokerage firm to see if the flash content is being performed.

GStreamer Multimedia

Check if Meow Music Player (by jimc), Parole for XFCE, and Totem (also called Videos) for Gnome, are functional. These apps all use GStreamer-1.x.

Conclusion: Piki (a real machine) can run any of the three players and can perform all the tested audio and video formats except AVI, with a quality corresponding to the resolution of the video clip. Oso (virtual) was more spotty. The X-Windows display is across the net and OpenGL cannot be used, so the data link is more or less overloaded, and some players and/or codecs react very badly.

Testing on Oso (virtual) with meow.sh, these video formats played: Ogg Theora. Quicktime videos stuffed with these formats played: RPZA, 3GPP, 3GPP2, Sorenson, MPEG-4, MPEG-2, MPEG-4/iPod, H.264/iPod. These video formats failed: AVI, These audio formats played (though no virtual sound was heard from the virtual sound card on the virtual machine): MP3, Ogg Vorbis. They did their thing for the expected length of the track used, whereas my experience is that when they fail, they do it right away. Repeating the test on Piki: these formats actually produced sound, and the listed video formats (except AVI) again played, with audio.

Parole on Oso (virtual) failed to play the MPEG-2, claimed a decoding error. It showed the first frame of the Ogg Theora, then shut down with no error message. It uses GStreamer-1.0 (libgstreamer-1_0-0-1.6.1-61.1.x86_64), why did it have a different outcome from meow.sh, which calls /usr/bin/gst-launch-1.0 (same version)? Repeating the test on Piki (real): all formats played successfully with audio, except AVI.

Totem on Oso (virtual) played Ogg Theora, though it started in nearly full screen mode and overloaded the data link. Other formats not tested. Repeating the test on Piki (real): all formats played successfully with audio, except AVI.

Grivet Music Service

Go through the chain of interdependent services by which CouchNet delivers streaming audio. Aurora has most of this toolchain and is in a hot standby mode. Can't test on Oso.

Jacinth has the full stack of audio components, and they all work together to deliver audio from all sources. As a playback node, Piki performs music hosted on Jacinth, and also the OTA Icecast stream served from Jacinth. Kermit does the same; this is its primary role.

MythTV

Check if MythTV still works. Jacinth records programs on the assigned schedule, using the Hauppague WinTV 950q DVB device, which has touchy and flaky drivers. Both Iris and Aurora can perform these programs.

Tcl/TK

Jimc relies on Tcl/TK; check if the Tcl apps still work. Yes, it works. MicroEmacs also works (i.e. has the required 32bit libraries). Tested on both Oso and Piki.

Wicked and Network Nanny

Needs careful and extensive checking to see if the new network manager behaves itself. In particular, I need to find and read the documentation all the way through. There are man pages for:

In a context of pre-made /etc/sysconfig/network/ifcfg-* files, Wicked and friends have worked with no issues on multiple virtual and real machines. I assume that Wicked would work equally well on a new machine whose network was set up with standard tools, e.g yast2 lan.

My laptop was another story. The admin guide for SLED-12 says you should enable either Wicked or NetworkManager, not both at the same time. Wicked does not have enough brain power to handle the dynamic networking of a typical laptop, and NetworkManager is recommended. My experience confirms this advice: I disabled Wicked and continued with the existing NetworkManager configuration.

Remmina

This is the new remote desktop client. Run through all the protocol variants and see how well they work.

Lightdm

See if we can get lightdm to do better than last time. See: /etc/lightdm/lightdm-gtk-greeter.conf ,.

Simple Scan

Check out this new scanner frontend. Pretty nice. It works. Call as simple-scan net:hostname:pixma. It actually has a manual which is not too lame (via yelp). It appears to be written in Java. The one problem is if your scanner is across the net: you can't just launch it from the menu system; you need to start it from the command line as indicated above.

Sound Juicer

Try this new app for ripping CDs.

Sound Juicer is specified for hostgroup user, which does not include Piki. I installed it specially. Should it be ubiquitous? Only 656kb installed, plus 1.9Mb for the language pack which I don't install.

Ripping a CD: estimated time about 18 minutes, actual 14 mins. It uses about 40% CPU on this AMD Athlon 2650e @1.6GHz (1 core). It was compressing to Ogg Vorbis; also available are FLAC, MP3, AAC. The program works similar to the old grip and has a reasonable user interface and a useful help document (via yelp). It automatically retrieves album and track metadata (artist and title) from musicbrainz.org. By default it delivers tracks to ~/Music with folders for the album artist, the album title, and the track within that. There is a limited set of track filename formats, which are probably fine for standard music players, but I'm going to have to rename them by hand to fit what Grivet wants to see.

The one problem I hit was with an album not on MusicBrainz; Sound Juicer claimed that the disc was not mounted. The help file says you should be able to fill in the metadata by hand (and send it to MusicBrainz).

Shotwell

Organize some photos; see if I like this app.

Instsetup, No Hands Installer

In the jump from SuSE 9.3 to 10.1 (I think), they changed the compression algorithm in the RPM files, and the back-version zypper could not read new RPMs. At that time I created instsetup, which creates an image of a chroot jail stuffed with the new version's zypper. It worked well. However, it's extra work and complication to maintain this program.

I've found that I can successfully make the back version zypper do a dist-upgrade from v13.1 to v42.1, and it's somewhat less scary, and about the same amount of work, if I bypass instsetup and do the key steps by hand.

Best laid plans etc. Several machines need to change architecture from i586/i686 to x86_64 and I really doubt that I can make this happen while the operating system is running. So I am going to have to set up instsetup. I will do the work for this on Petra, my i686 VM for testing distro upgrades.

Create Installer Image

The command has to be executed on a machine with the future architecture (x86_64) and preferably upgraded to the new OS version. That means execute on Oso. You will need a SSH agent on Oso because it is going to mount the distro directory using sshfs (FUSE). The command is:
instsetup -J -v
If it asks you for a password, and later maunders about transport endpoint not connected, the FUSE mount is hosed (due to failure to authenticate). To recover, look in /proc/mounts for the FUSE mount, then
fusermount -u /tmp/instsetup.oso.0/pjdir #(or whatever mount point).

Installing 107 packages in the installer image, which ends up at oso:/var/cache/instsetup.jail.x86_64.42.1/ (176Mb, oink).

Copy that dir to the same name on the master site (Diamond).

Virtual Machine

I prepared Petra similar to what I did for Oso, q.v.

Hostgroup

Put the target host in the hostgroups for its new architecture and OS version.

Emplace Installer

Command line is:
instsetup -h petra -d /m1/boot421 -D hd0,msdos2 -p wouldntyouliketoknow -a x86_64 -r 42.1 -R -E eth0 -v
See /usr/math/etc/instsetup for the command line arguments. The program used to fail to get the -E argument right on a VM and it was recommended to specify it explicitly as -E eth0. But instsetup has been fixed to omit netdevice entirely unless -E is specified, so just omit -E.

Items that need to be fixed up:

Execute Installer

Second Distro Upgrade

The second upgrade reinstalled and changed the vendor of 82 packages, a lot of Packman to OpenSuSE. A repeat of the same command did exactly the same thing to the same packages. This is very strange. I wonder if there was anything wrong with them in the first place.

Post_jump

Added 101 packages, removed 328 packages. Why did post_jump claim that Petra has v13.1? Failed to install chrony (got it from SBS) and libdvdcss2.

Checkout.sh

Both VMs cannot start vsftpd because they cannot find/read the (nonexistent) host certificate (for FTP over TLS). Other discrepancies were fixed with no problems: rpmconfigcheck, and missing chrony package in the repo.

Conclusion: I'm ready to sic instsetup on a real machine, Piki.

Upgrade Production Machines

Here's the schedule to upgrade the production machines. For several of them, see separate sections later with gory details on the upgrade including discrepancies encountered and fixed.

Upgrading Piki: Details

Upgrading Xena: Details

checkout.sh discrepancies:

More discreps:

Trying with NetworkManager* disabled. Bad move, Wicked didn't know to bring up the network despite what should have been sufficient info in /etc/sysconfig/network/ifcfg-wlan0. The admin guide for SLED-12 says use NM or Wicked, not both. OK, Wicked off and NM back on.

Discrepancies reported in daily housekeeping:

Upgrading Jacinth: Details

Upgrading Diamond (Master Site): Details

Diamond is the master site for distro storage, so there are issues of self-reference. For example, I will execute post_jup on Diamond as the master site, and that includes ssh diamond audit-pkgs -v -i (and numerous similar ones) with Diamond as the target to be upgraded. Audit-pkgs on the target will do zypper install whatever, obtaining the packages from Diamond as the repo storage. This will work for post_jump, but I'm afraid that it will work poorly for the dist-upgrade, particularly when Apache and ssh are being upgraded. Instsetup has some notes for what to do when upgrading the distro master: basically, hand edit the repo definitions to refer to the local disc rather than HTTP.

Steps in the upgrade:

Script Index

Here is a brief index of scripts used in the upgrade.

/s1/SuSE/bin/snarf-bs ($sbs)

Downloads a package from the SuSE Build Service and puts it in the correct directory. Also works with Packman and other outside repos. With SBS it downloads both the x86_64 and i586 versions, but if we're decommitting i586 that behavior will have to be suppressed.

/s1/SuSE/bin/mksuserepo ($mkr)

Rebuilds and signs the metadata for the repo, using the key in /s1/SuSE/distro-{pub,sec}.gpg . The private key is encrypted with the current root password.

hostgroup

The various scripts rely on hostgroup to accurately report the architecture and distro version on each machine. Before upgrading each machine, particularly for an architecture swap, you need to edit the hostgroup database.

audit-pkgs

This script adds, removes or updates packages by calling SuSE's zypper with appropriate arguments. For dealing with difficult situations it has a -I option to let you interact with zypper and make choices on the fly about what to install. Your goal should be that such handholding will not be needed when upgrading production machines.

audit-repos

Copies the repo definitions from the post_jump directory to the target, and fills in the target's architecture. Removes unwanted (e.g. back-version) repos.

onlu-check

Translates the output of zypper list-updates and extracts descriptions of the patches from repo metadata. As usual, they changed the metadata format in the new release.

instsetup

It installs in a chroot jail just enough packages to run zypper and do a distro upgrade. When that is set up, it prints out a page of instructions, which you should save so you can copy and paste from it during the upgrade. The installer has a -I option so you can interact with zypper, in case of unanticipated dependency problems.

post_jump

After an upgrade or a fresh installation, it copies over the configuration files in /etc, installs wanted packages, removes unwanted packages, and does an online update (all using audit-pkgs). It finishes by running daily housekeeping (slow, sorry).

audit-scripts

Enables or disables services to be started at boot. It is normally run by post_jump but can be run by hand if needed.

checkout.sh

Does a functional test on many services; others are just checked for being enabled and running. This is a good quick test if an installation or upgrade succeeded. Takes 1 to 2 minutes because it tests if a cron job will be executed and it has to wait for the scheduled time.

ins

Installs files on remote hosts. If you do e.g. ins -src /home/post_jump/42.1 v42.1 /etc/default/grub.m4 it will take this file out of the post_jump directory to install. While this feature has been there since 2009 (but forgotten), I improved the precheck display so the correct file is listed.

Shockwave (Adobe) Flash Player, Hiss, Boo

Adobe's Flash Player has been kicked off SuSE due to licensing problems. I doubt the SuSE staff are shedding tears over its demise, since it is a maintenance nightmare and is a suppurating security hole. But the WAF of SuSE in general and the upgraded OS in particular depends on being able to perform Flash, specifically Youtube videos, and stock market graphs. So what am I going to do?

I've used Chromium extensively and I gave it up because (at that time) it would not present X.509 and Kerberos credentials that I needed. The users here all use Firefox and are not likely to change. I think my solution is going to be to use chromium-pepper-flash and freshplayerplugin. Wish us luck, my user base are going to need it.

Freshplayerplugin and pepper-flash turn out to be satisfactory for the flash sites that we normally use: Youtube and CNN. However, Pepper Flash is the bad old Flash Player, security holes and all, just better maintained (maybe) and available with a slightly more sanitary license. It would be better if we could get Firefox to use HTML5, for which see the next section.

Making Firefox Use HTML5 Vs. Flash

Credit to Ben Carter for making this work. To make Firefox use HTML5, do these steps:

More web references about enabling HTML5 in Firefox:

Large, Gross Botfly in Ointment: Printing

After Diamond was upgraded, the financial manager, executing on the same machine, tried to print hardcopy and failed. The I.T. staff got an earful. On other Linux machines upgraded to v42.1, the command lp -d lp2 $file results in the error message lp: The printer or class does not exist. Executed on the machine with the printer, the same command prints the file (if it's PostScript).

Second symptom: if the file is text/plain, executed on Diamond, the error message is lp: Unsupported document-format "text/plain". The cure for this was to install these filter packages (like codecs). The filters are recommended with the cups package, and it looks like in my dist-upgrade the recommended packages were suppressed.

In v13.1 printers were announced to clients by UDP broadcasts on port 631. But in v42.1 (cups-1.7.5) the UDP protocol (as well as slp) has been taken out of cupsd, and dnssd (DNS Service Discovery) is the only supported browse protocol. Here's an overview of how it's supposed to work:
dig @224.0.0.251 -p 5353 _services._dns_sd._udp.local. PTR
(The ending dot is essential; without it you get nothing. This is a multicast address. For IPv6 it's ff02::fb.) This should yield a set of PTR records to service categories, and when you ask for each of their referents, you should get PTRs to services within each category, together with answers to future queries, i.e. the SRV and TXT records for that service plus the A and AAAA records for the server in the SRV record. Empirically dig only shows the first service even if several are registered on the same machine, because dig reports the first reply it receives and ignores the rest. If services in a category are registered on multiple machines, necessarily each one will send its own packet. In the initial query all categories known to the responding machine are sent in one packet, but if another category is on a different machine, it will be sent in a separate packet.

To check what is registered:

CouchNet has a very frustrating complication: a firewall on every host. Port 5353 had to be opened. And of course avahi-daemon.service and avahi-daemon.socket (both) had to be enabled and started. avahi-dnsconfd.service is apparently optional, and I haven't found a good explanation of what it actually does. You will also need package avahi-utils to get the avahi-browse command.

Another complication is, cupsd does not actually register its print queues with Avahi (dnssd) unless you tell it to. In my setup there is a separate configuration file for the print server, and I dumbly edited the generic file for hosts with no printers, so of course registration didn't work. [Fixed.]

Here is the current setup on CouchNet, also describing extensions needed for Mathnet:

So printing on CouchNet is apparently functional, and we have a way forward for the more complex Mathnet printers.

Another tidbit: there's a file /etc/cups/cups-files.conf and the User, Group, ServerKey and several others are supposed to be in it. Read /var/log/cups/error_log . [Fixed.]

LightDM Display Manager and Webkit Greeter

It seems that in every release something bad happens to the display manager and its greeter, so I have to switch to a different one. By now I have used every one, some several times. In OpenSuSE 13.1 I tried to use LightDM. The GTK greeter was not satisfactory for our use case, and neither were either of the samples that came with the Webkit greeter, so I wrote my own HTML/Javascript document. It didn't work out due to (their) bugs, so I finally put together a satisfactory collection of widgets centered on the ancient XDM (which I am quite familar with). But it's a kludge and is obviously home-built. So this time around I decided to re-try the Webkit greeter. With the current software version, I was finally able to make it work right.

The requirements for the greeter are:

You need to install one of the provided branding variants -- I randomly picked lightdm-webkit-greeter-branding-bevel -- and it will drag in lightdm-webkit-greeter.

Here is a tarball of the finished greeter document. Make a directory under /usr/share/lightdm-webkit/themes and unpack it there. Files included:

These configuration files were modified:

The only deficiency I've found so far is, for the background you would like to specify (on the body) style="background: cover url(file:///m1/custom/background.jpeg)" but the cover property requires CSS-3, which the provided webkit does not have.

Another trap for the unwary: something provides a default session type of twm, which is an ancient ancestor of FVWM and is not installed on my system. It is seen in this role in /usr/bin/startx (package xinit), and in /var/lib/AccountsService/users/jimc (origin unknown but package accountsservice is the obvious culprit). If the lightdm session-child decides to use this session type, it will open /usr/share/xsessions/twm.desktop (ENOENT) and then hang forever, preventing any more logins. To recover, kill the hanging session-child by hand, then systemctl restart display-manager (which would miss the culprit session-child which holds some kind of lock). Then the user should explicitly pick a session type when logging in, creating ~/.dmrc .

Oopsie, in the morning I wake up my laptop, log in, and 13 seconds after wakeup (while I'm typing my password) the display-manager is killed. I have a script that kills orphan dbus and other session daemons, and it decides that the greeter session is not a real session, and kills its dbus daemon. Fixed.

So far, the LightDM Webkit greeter is working out for us.

Tidbits for Later Work

Finished items:

Bug Reports