Valid HTML 4.01 Transitional

Holly Hosed, Tested Backup

James F. Carter <jimc@jfcarter.net>, 2021-06-08

Today SuSE has a big update for aarch64, moving lots of stuff from /bin /lib to /usr/bin /usr/lib. This was done for x86_64 last week and went smoothly. But not on Holly. The result is that either the executor or the shared libraries have become missing, so if you do "ls filename" it says "/usr/bin/ls: no such file or directory" (even though it's there, per "echo /usr/bin/ls*". Specific error message:

Installation of filesystem-15.5-40.2.aarch64 failed:
Error: Subprocess failed. Error: RPM failed: Make a copy of `/bin'.
(It seems to have successfully made a copy of /bin.)

This upgrade was not exactly atomic: last week the firmware migrated from /lib/firmware to /usr/lib/firmware. But the drivers, specifically the out of kernel user compiled driver for (RTL) 88x1bu.ko, are looking for the firmware in /lib/firmware. A simple fix was to make a symlink from /lib/firmware to /usr/lib/firmware. Problem solved — yeah, sure. I suspect without real proof that the posttrans script for the filesystem package was not smart enough to recognize that it should just remove the symlink, not try to copy it to /usr/lib where ./firmware already exists. In any case, only part of the essential infrastructure formerly in /lib got copied over.

So I'm going to have to do something drastic to bring Holly back to life. The intervention will obviously involve restoring things from backup… and now is a very good opportunity to test if my backup system is actually saving everything important, and if access to the backup copy is actually feasible. (Other shops have had shortcomings in both these areas.) Fortunately all hosts including Holly were freshly backed up, according to the standard procedure, just before the system update that went awry.

Goals

Here are some goals for the recovery campaign:

Outline of the Plan

Executing the Plan

Stuffing the SD Card

Initial Boot and Login

Running Post_jump

Restore from Backup

Overall Conclusion

This disaster recovery exercise has been surprisingly successful.