| Planning | Testing | Setup | Collapse | Gentoo | Top |
onthe guest or target)
I have an issue in which 3D GPU accelerated graphics locks up on OpenSuSE Tumbleweed for aarch64, coincident with steadily declining free CMA memory. Is the problem specific to OpenSuSE, which has unique features such as EFI booting, or is it independent of the distro? That determines in which forum or bugzilla I ask for help. To make that decision I need to test GPU acceleration in another distro, preferably as different from OpenSuSE as possible.
According to Gavinmc42 replying to my forum post about turning on GPU acceleration, Gentoo has this working with a 64bit kernel.
Diversion about Gentoo: It's about as far from OpenSuSE as you can get.
While all normal
distros distribute precompiled packages, Gentoo's
biggest unique feature is that, as a matter of policy, each end user is
expected to download the source code and compile it on the spot. There are, of
course, tools to automate this process, which have proven to be quite reliable.
Portage
is the name of the whole package management suite, and
emerge
is the program that the sysadmin invokes to compile and install a
package, analogous to zypper
on SuSE. Precompiled packages are used
occasionally for packages that are too complex, touchy, or resource intensive
for local compilation to be practical; Mozilla Firefox is a prime example
(but is discontinued as of about 2018-08-xx).
For deployment on multiple machines they have tools and documentation so
packages can be compiled once on a server and turned into binary packages,
and these can then be installed on leaf nodes.
Besides political correctness, this being an open source distro, one big advantage of local compilation is that the packages installed on each leaf node can take into account features and optimizations appropriate for that host, even accomodating different architectures, rather than requiring cooperation from the distro, which can only make space for a few precompiled package variants such as i586 vs. i686 vs. x86_64.
Web resources for installing Gentoo:
the Howto.
The first trial was to just steal Gentoo's start.elf file. The foreign start.elf was able to start u-boot, which execed Grub2, which loaded the specified kernel and initrd, but the kernel did not run. There were no useful error messages, but I would bet that the SuSE kernel tried and failed to retrieve the device tree that start.elf would have left somewhere for it.
For further progress I am going to have to actually install Gentoo. There is a chicken and egg issue here: there is no prebuilt installer for aarch64, so you have to compile the kernel and basic packages on an existing working host, which for me will have Intel architecture (x86_64, or as they call it, amd64). This means cross-compiling. The installation guide shows all the steps (cross fingers). But the packages for the cross-compile toolchain are for Gentoo, meaning I will first have to do a full installation of Gentoo for x86_64.
Hostnames: Rather than using phrases like the Raspberry Pi
I'm
using in this writeup the machines' hostnames on my net.
target machine.
In this writeup I often abbreviate Raspberry Pi
as RPi
.
Petra's resources are:
Outline of the steps:
The following procedure mostly follows the Gentoo Handbook; page transitions are noted. The Handbook has a lot of explanation and alternatives, and it should be read concurrently. Much of the procedure is executed 2.5 times, first to go from the minimal installer to a working cross-compiler setup on Petra, then to cross-compile an arm64 kernel, install it on Orion's disc, and do as much setup as Petra can handle, and finally to boot up Orion and finish the setup steps that require running arm64 code on the Raspberry Pi.
Choosing the Media
Before setting up Petra's Gentoo disc I downloaded the minimal installer
ISO and the stage3 tarball for amd64 (Petra) from
the (recommended) download page, dated 2018-10-04. Plus the
DIGESTS.asc file; follow the link under Details
.
Plus
Stage3 for arm64 (no signature, hiss, boo).
I'm doing this now so the procedure to populate the disc wouldn't be
interrupted by the download. With my dates, the full filenames are
install-amd64-minimal-20181004T214502Z.iso and
stage3-amd64-20181004T214502Z.tar.xz . Use the most recent versions, not
these back-version examples.
Verification procedure:
Both machines need their proper host keys, ssh_config and sshd_config, and an authorized_keys file. Also your regular /etc/hosts. On the VM host I made a directory called keys.petra or keys.orion with the needed content, root-relative, e.g. when preparing it: rsync -a /root/.ssh/known_hosts keys.petra/ , which copies into keys.petra/root/.ssh/known_hosts , creating the intermediate directories.
For the ssh configuration files I used the ones I set up for OpenSuSE. But there was a problem of version skew. OpenSuSE has openssh-7.7p1, OpenSSL1.1.0h-fips whereas Gentoo has OpenSSH_7.7p1, OpenSSL 1.0.2p, and there are some features that SuSE has and Gentoo doesn't. Comment these out in the conf files that will go onto the Gentoo machines.
Preparing the Disks
We just finished Choosing the Media
and we're
now skipping to Preparing the Disks
. I'm following the Howto with
improvements
. This procedure applies to both Petra and Orion, and can
be done ahead of time on the host, for both in parallel.
I'm partitioning both discs similarly except for the total size,
which for Petra is 16Gib (234 bytes), while Orion's SD card
has 32Gb (power of 10), actually 32010928128 bytes, but the last
partition ends at 32004564479 bytes. This leaves 6.3Mb vacant at the
end, which would be important with a GPT (though I'm using MBR). The
actually
size is (220)*30528 exactly.
I created Orion's file-backed disc with this command, executing
on the host:
dd if=/dev/zero of=disc2.raw bs=1M count=30528 iflag=fullblock oflag=direct
For Petra's smaller disc use of=disc1.raw count=16384.
I'm using a MBR partition table rather than a GPT, and I'm not using EFI. (OpenSuSE uses EFI to excellent effect.) I use GPT whenever possible, and the Raspberry Pi boot ROM can find the boot partition in a GPT, and bootcode.bin can find start.elf in a GPT boot partition, but a subsequent step fails (maybe finding the kernel?) So I've reverted to the MBR partition table on RPi. If the GPT problems could be worked around, it should begin with a BIOS boot partition of 1Mib (do not format or mount it), whereas with a MBR you just start the first partition 1Mib in from the beginning of the disc.
You can make the partitions safely on the host or on the guest,
once you get it running. I did it on the host, like this:
parted -a minimal disc2.raw
Minimal
alignment is a compromise between optimal
,
which has optimal partition endpoints about 200Mb apart, way too coarse
for the boot partition, and none
, which can produce poor
performance. Even so, it will complain that the partitions are poorly
aligned. Just accept the misalignment and it will do what you told it.
Hard experience shows that 32Mb is not enough for the boot partition since the actual kernel ends up there. About 36Mb is actually used, and I gave it 64Mib.
I'm giving it 0.5Gib of swap space (to supplement 1.0Gib RAM). My eventual application is audiovisual playback, and swapping is not acceptable, so my swap space is skimpy. However, emerging (compiling) software is another story: with only 0.5Gib of swap, several compilations failed because it was all filled. I was not able to find out how much was actually required, but moderately reliabie observations suggest that 1.5Gb swap would have been just barely enough. That's 2.5Gib total, RAM+swap. [Update: one of the Rust modules was seen using 3.0Gib total.]
To get things emerged, I added a swap file of 2.5Gb, for a total of 4.0Gib (1Gib RAM, 0.5Gib swap partition, 2.5Gib supplementary file). This succeeded. I had the thought to delete the swap file after software setup, but the big packages could be updated at any time. 3Gib in the swap partition would be a more sanitary solution.
Thrashing did not seem to be a big problem: the compiler had sufficient locality of reference that the resident set (usually) would fit in 1.0Gib RAM. [Update: the Rust module mentioned above thrashed horribly and took almost 2 hours to compile. But it didn't die.]
Here is the partitioning I ended up with, in units of Mib, shown for Orion. For the endpoint of partition 3 specify -1 which will adapt to the actual size of the media. The numeric suffixes on the labels are 02 for Petra and 04 for Orion (each of my SD cards and VM discs has its own number); differentiation is important because you're going to have both discs mounted at the same time on Petra.
Number Start End Size Label File system Flags 1 1.0Mib 65.0Mib 64.0Mib BOOT-04 fat16 boot, lba, type=0c 2 65.0Mib 577Mib 512MB SWAP-04 linux-swap type=82 3 577Mib 31495Mib 31477Mib ROOT-04 ext4 type=83
Parted commands:
Don't try to create filesystems on the partitions using a loopback device on the host: it's hard to get it right, and particularly, the loopback device extends to the end of the media, and you need to specify the size of the filesystem explicitly lest you end up with overlapping filesystems.
Booting Petra from the minimal installer: In the Handbook this is two
pages back, Choosing the Media
, section 4, Booting.
Connect the minimal installer's ISO image to the guest's CDROM. My CDROM is file-backed using a generic filename on the host, and I make a symlink (with the guest powered off) to the desired ISO image or to an inode in the same filesystem which is a duplicate of /dev/null. QEMU barfs on a symlink to a different filesystem, e.g. the real /dev/null.
I used standard procedures to attach to the console VNC server
first. Here's my command line, executed as myself. I have a key agent
with a key in root's authorized_keys, so it doesn't have to ask me for
a password (which would be rejected anyway), and I run it in the
background. Xena is the VM host and Petra is the guest.
virt-viewer -c qemu+ssh://root@xena/system -w -r petra
Then virsh start petra
(on Xena).
On their ISO, you have to press any key within 15 secs, otherwise it will try to boot from the hard disc, which is great for unattended installation but faked me out. Hit Enter to have it boot from the ISO image with all default options. Soon it will want you to pick a keymap or hit Enter for the default which is US. (It times out and takes the default after about 15sec.)
The network came up cleanly with dhcp for IPv4 and a RFC 4862 address for IPv6. One of the major design criteria for Petra's networking was to support guests that do normal generic network protocols, without needing to tweak the guest.
virt-viewer is supposed to release the cursor when you hit Ctrl-Alt, but somehow this is broken, just on this installer. To recover, log in to the host (where virt-viewer is running) from somewhere else, and kill virt-viewer. You can start virt-viewer again with no harm to the guest.
You need to use ssh, to hang onto your cursor, to cut and paste, and to copy files onto and off of Petra using scp.
PermitRootLogin yes(versus the default which is
prohibit-password).
ssh root@petra. Accept the installer's random host key. Give the root password.
The guest disc (Petra's) was /dev/vda and the target disc (Orion's) was /dev/vdb. (But it will be /dev/mmcblk0 on the target.)
In music this would be a repeat mark: I'm going to first install Gentoo for x86_64 on Petra, then come back here and install Gentoo for arm64 on Orion's disc, then a switch to Orion itself, with most but not all of the steps identical.
Partitioning: I already partitioned both discs on the host, but if you
didn't, now is the time to do it on the guest. See above for
details of the partition sizes and the MBR (not GPT)
partition table. You can do both discs
in parallel the first time through, but be sure to get the
correct disc. Do e.g. parted /dev/vda print
(or /dev/vdb) and
distinguish the discs by their total size: 17Gb (16Gib) for Petra and 32Gb
for Orion.
Follow the Handbook's instructions to format each partition with the appropriate filesystem type. Be sure to get the correct disc and the correct partition on it. I assign a label to each partition and mount that way. Although the device pathnames (like /dev/vdb3) for the SD card are pretty stable, until you actually put it in a RPi, in general you can't rely on consistent major device numbers, so you always should mount by label or by UUID, and I think UUIDs are prolix, meaningless and ugly. Here are the filesystem creation commands, assuming you have the same major number I got. The suffix -02 or -04 refers to the number I marked on the SD card, to distinguish various ones. This is important because (as in this case) you may have similarly named partitions from multiple discs on the same machine. Here, Petra's devices and label suffix are shown.
The mini-installer for Petra could use some swap space and the Handbook line
about that, in the section Activating the Swap Partition
, is easy to
miss: swapon /dev/vda2
, or probably safer,
swapon /dev/disk/by-label/SWAP-02
. When you come back here to populate
Orion's disc, Petra will already have its swap activated.
The system clock needs to be correct, and the VM imports it from the host. But the mini-installer only has UTC. Leave the timezone alone for now and convert times to the local timezone in your head if needed.
In the coming sections you install Gentoo on Petra's or Orion's disc, so you need to mount one or the other now. Petra's label suffixes are shown.
cd /mnt/gentoo. Pathnames beginning with ./ are relative to /mnt/gentoo.
Installing Stage 3(Installation Files)
Deposit on the disc the stage3 that you previously downloaded,
like this (on the host):
scp /s1/kvm/petra/gentoo/stage3-amd64-20181023T214502Z.tar.xz petra:/mnt/gentoo/
(give the root password.) For Orion get the arm64 file, but it still
goes onto petra:/mnt/gentoo/ .
Now un-tar the stage3 tarball. (amd64 for Petra, arm64 for Orion.)
Mind the special options. Since the Gentoo files use standard extensions to
represent the compression algorithm, this need not be specified on the command
line. The Gentoo Handbook specifies verbose output (a list of files copied,
the 'v' option), and you will want to capture the output in a file for easy
perusal. The instructions in the Handbook assume that you will have downloaded
just one tar file in /mnt/gentoo, so ./stage3* will find it (check with
ls
).
On both discs, now that the new disc has some population, edit /mnt/gentoo/etc/fstab to reflect the partitions to be mounted at boot, as they will be when you reboot the machine.
noautoon the boot partition; either remove it, or plan to mount it by hand whenever you want to fiddle with files there. Like during installation.
Here's Orion's /etc/fstab . Petra's is the same except for the label suffix (-02).
# device mountpoint type opts dump/pass LABEL=ROOT-04 / ext4 noatime 0 1 LABEL=BOOT-04 /boot vfat noatime 1 2 LABEL=SWAP-04 none swap defaults 0 0 #/dev/cdrom /mnt/cdrom auto noauto,ro 0 0
Preset some configuration for the net, on both discs. This is also needed when you jump into the chroot jail. If you forgot to do it and have to do it on the booted-up Orion, paths beginning with ./ are relative to Orion's root.
localis generally used.
On the mini-installer you should already be using ssh, which means
you will already have changed the root password. If not, do it now:
passwd root
and set your usual root password. Now:
cp -p /etc/shadow /mnt/gentoo/etc/shadow
Do this for both discs, so root can log in to both Petra and Orion.
I tweaked /etc/portage/make.conf with these additions; see also the next paragraph where GENTOO_MIRRORS is also appended to make.conf. When you install Petra this is the installer's make.conf, but I made the same changes in Petra's and Orion's instance, /mnt/gentoo/etc/portage/make.conf . (Don't mess with the cross-compiler's make.conf.)
Installing the Gentoo Base System
I used mirrorselect -i
to point at two geographically nearby mirrors:
http://gentoo.osuosl.org/ (Oregon State Open Source Lab) and
http://gentoo.cs.utah.edu/ (University of Utah Computer Science Department).
Use up and down arrow to pick a mirror.
Spacebar selects the highlighted mirror; you can select more than one; Enter
(when OK is highlighted) appends the environment variable setting to
/etc/portage/make.conf (the one in the minimal installer's ramdisc).
You will want to append the same setting to
/mnt/gentoo/etc/portage/make.conf as well so it will be used when you
compile toolchain packages. And similarly on Orion's disc.
The list is space separated and quoted.
GENTOO_MIRRORS="http://gentoo.osuosl.org/ http://gentoo.cs.utah.edu/"
mkdir ./etc/portage/repos.conf # That's in /mnt/gentoo
cp ./usr/share/portage/config/repos.conf ./etc/portage/repos.conf/gentoo.conf
Just copied, no changes.
Both machines need their proper host keys, ssh_config and sshd_config, an authorized_keys file, and /etc/hosts with local addresses (if any), which I packed up in advance in the host's keys.petra and keys.orion directories. There are several ways to get the keys onto the new disc. It's possible to shut down Petra, mount the root partition on the host using a loop device, and just copy, but it's a pain to get the correct offset and so I don't recommend using the loop device. Probably the least confusing method is:
PermitRootLogin yeswhich allows root to get on by giving a password. Once the authorized_keys file is installed, revert to
PermitRootLogin prohibit-passwordwhich is the default.
ssh-keygen -R petrato forget its correct host key. When you're setting up Orion, Petra will have its correct host key, and you will need to make the VM host forget the former random key.
The following steps are done in Petra's chroot jail. Except as noted, the same steps are needed on Orion, but you need to defer them until the disc image has been copied to Orion's SD card and Orion has been booted up, because the disc contains arm64 (aarch64) software that needs a RPi to run it. So jump to Cross Compiling, picking up after your backward jump to set up Orion. Follow the provided link back here after you have cross-compiled the kernel etc. and booted up Orion.
When setting up Petra (not on a booted-up Orion), you mount these installer objects. The Handbook's command lines for --make-rslave are for installing systemd which I'm not doing.
Now Petra makes a suicide leap into the chroot jail:
chroot /mnt/gentoo /bin/bash
source /etc/profile
export PS1="(chroot) ${PS1}"
Diversion on how to install packages, called emerge
in Gentoo:
To find all packages whose name contains the search string, do
emerge -s string |& less
(or --search). It also tells you
the available version, and which version (if any) is installed.
Masked
means that there is some reason it cannot be installed,
like a missing keyword or USE flag. Try to install it anyway and
follow instructions, to get the appropriate configuration file updated
(within the limitations of automatic fixups).
To install a package on Petra, or on Orion when it's booted and
running independently, do this (you can list several packages on the
same command line):
emerge --ask category/package |& tee /tmp/logfile
.
(Always use --ask except in non-interactive scripts.) It will propose
what it wants to install including dependencies, which can be
extensive. If you're satisfied, 'y' is a sufficient answer. Or 'n'
to do a more thorough review of what it's going to do. If it says
that keyword fixups are needed, normally tell it 'y', whereupon it
will write revision file(s) and exit. The addition of ~arm64 is
very common; you need to give permission to use the package on this
experimental architecture.
If you need keyword changes, or if an installed package has
deposited a revised configuration file (you will be told if this
happens), execute dispatch-conf
to install the changed file.
The newbie will almost always go along with the proposed revisions.
dispatch-conf needs output to the terminal; don't pipe it through
less
. For each affected file it will show a diff of the old vs.
revised file ('q' gets you out of less
if it's multi-page), then
dispatch-conf wants an action letter. 'u' means use the revised file,
'z' means toss the revision and keep the old one, 'n' means leave both
alone and go on to the next file (you can re-run dispatch-conf later),
and there are others.
Next the ebuild repository snapshot (Portage package index) has to be
downloaded. The command is emerge-webrsync
, no command line arguments.
Afterward do eselect news read |& less
to read news back to
2013-09-27 (only 13 items, none of which you need to respond to).
Do eselect profile list |& less
; the currently (default)
selected profile is marked with a '*'. This is default/linux/amd64/17.0
(stable), and I did not change it. A
systemd profile is available, and on OpenSuSE I use systemd, but I'm not going
to turn it on to avoid waking sleeping dragons. Gentoo utils have a tendency
to barf a lot on the screen, in this case 54 choices. Piping through
less
is a good idea. [Update: later on Orion, not Petra, I switched to
default/linux/arm64/17.0/desktop which needed 162 more packages and 24.7 more
hours to compile.]
@world will now be updated. (Orion too, once running.) This is the set
of keystone packages, that is, packages that are picked by the user versus
being required by other packages. Your profile provides most of these
packages, but any package that you emerge explicitly goes into @world unless
you give an option to keep it out, or remove it later. Command:
emerge --ask --verbose --update --deep --newuse @world
Took 9 mins on Petra, 2 hours on Orion. If I had gone directly to the
desktop profile on Orion, it would have taken 48 hours.
After @world is updated, it's recommended to do emerge --ask
--depclean
to get rid of unneeded packages. The first time through, answer
no
and peruse the proposed deletions (unmerge) carefully. For some
reason, app-editors/nano is provided by the mini-installer but is not a
keystone package in the default profile, so it would be deleted. To select
something as a keystone package (which depclean has a compulsion to preserve),
do e.g. emerge --noreplace app-editors/nano
, and it will be added to the
@world set but not recompiled.
Now you have timezones beyond UTC and can set one (Petra and Orion).
America/Los Angeles.)
Next set the locale (Petra and Orion). Do these steps:
en_US.UTF-8 UTF-8.
locale-gen(no arguments).
locale -a |& lessand verify that your selected locale is there. There are 795 choices and it's essential to pipe the output to
less.
eselect locale list |& lessand remember the number of your locale. There are also 795 choices here… output to
less.
eselect locale set 3(or whatever number) sets the default locale.
env-update && source /etc/profile && export PS1="(chroot) $PS1"sets this default locale for your session, needed to build the kernel.
Configuring the Kernel
This is only for Petra; on Orion you have already cross-compiled and installed it. Do these steps:
emerge --ask sys-kernel/gentoo-sources— Install the kernel source code, together with some dependent packages.
emerge --ask sys-kernel/genkernel— For compiling a generic (vs. custom) kernel. It needs a USE flag update, and there are more dependent packages to build.
genkernel all— Actually compiles the kernel. Took 80 min. Not too bad, but it could have been less if I had done menuconfig and excluded impossible devices. Which the Howto author recommends to not do, because an error could leave you with a machine that's hard to debug and repair.
In the following step on Orion, the X-Server turned out to depend (at build
time I think) on sys-devel/llvm, which is the core of the clang
'C'
compiler. It's an interesting concept, but very resoure-intensive when it's
being emerged. It repeatedly ran the RPi out of memory. I finally figured out
an effective intervention. I had provided 0.5Gb of swap space, which is plenty
for audio-video performance (which is spoiled if swap has to be used), but
which overflowed with llvm stuff. The solution which should have been
obvious is to provide more swap space. I created a 2.5Gb file on Orion and
activated it (swapon /var/spool/swapfile2). Actually I think 1Gb would have
been enough but I wasn't able to capture exactly the amount of swap used.
Success at last! The swap file can be deactivated and deleted when the
compilation is over. Well, maybe not: suppose llvm gets updated? It would
have been more sanitary if I had provided a bigger swap partition in the
beginning. [Update: llvm was not the only package killed by skimpy swap
space. And llvm was updated when I switched to the desktop profile,
taking 7.8 hours for that package alone.]
I emerged these utility programs that I'm going to need or that I like. We're compiling now for the x86_64 virtual machine (Petra); later these will have to be done again on the RPi (Orion). To save retyping and hassle I made a script with the emerge command line and package list. I provided the options --noreplace --keep-going. The first one avoids rebuilding packages that have already been installed, in case the process dies and I have to do it again. The second means, if an individual emerge fails, to compile the rest of them, excluding packages that require the one that failed.
locate.
Testing Orionsection, but the first time through I didn't have the organization quite straight, and I emerged them as part of setup.
clang'C' compiler. It's a giant, and it needs more than the RPi's 1Gb RAM to compile, and it's a prerequisite to emerge the X-Windows server. See the previous section for how I finally got it to compile.
normalinput devices and graphics cards.
Compiling all this stuff on the RPi, 354 packages counting dependencies, took 22:52 (almost 23 hours). These keystone packages could not be emerged:
internal compiler errorin one of the compilations. I'm blaming this on the memory/swap situation. With the big swap file, re-emerging dev-lang/spidermonkey succeeded and x11-misc/lightdm was emerged successfully.
common.cpp:38:1: error: narrowing conversion of ?18446744073709551615? from ?long unsigned int? to ?long long int? inside { } [-Wnarrowing]. Typecasts that lose information can sometimes be exploited for security attacks, so this is an error. Later I'll install net-misc/ntp.
Failed Patch: raspberrypi-userland-9999-gentoo.patch !This provides (among others) /usr/lib64/dri/vc4_dri.so which sends OpenGL primitives to the VC4 graphics processor, so it is required for 3D GPU acceleration. Rather than actually fixing the problem, I stole vc4_dri.so from the RPi running OpenSuSE. This was not effective.
Configuring the System
We've already edited /etc/fstab and set the root password. To start services (network, sshd) at boot, do this. On Petra this works in the chroot jail, but Orion has to be already booted.
If you've just booted Orion, jump back to the section after the chroot jail and do the steps that require executing on the RPi. You will work through to here again. Of course you don't have to re-enable services, nor configure the bootloader; your final step would be to create your ordinary user account (below).
Configuring the Bootloader
Petra needs Grub as its bootloader; Orion uses the native booter (start.elf), not Grub. Do this in Petra's chroot jail:
Finalizing
We already set the root password, and already tossed /stage3-….tgz . On both machines I added my own ordinary user account. Command lines for reference:
For Orion running on the RPi, it is supposed to have all its packages and to be ready to run some tests. For the results, jump ahead to Testing Orion.
Rebooting Petra:
exit(to get out of the chroot jail).
If you have just rebooted Petra, your next job is to set up Orion's new
disc. Pop back to Begin Installation. There you will
mount Orion's disc on (Petra's) /mnt/gentoo and /mnt/gentoo/boot. cd
/mnt/gentoo
. Stuff them with the arm64 stage3 file. Continue until Petra
would have entered the chroot jail, which Orion's disc can't support, then come
back here and cross-compile the kernel.
At this point, you're executing on the real Petra (not the mini-installer) and are going to emerge the cross-compiler, and then to compile the kernel and other ARM64 packages for Orion.
Web Resources:
Crossdev runs on one architecture (x86_64) and compiles for another (aarch64 in this case). Here is how to create the cross-compiler toolchain:
[crossdev] location = /usr/local/portage-crossdev priority = 10 masters = gentoo auto-sync = no
Now I have my toolchain to build aarch64 programs. One strategy is to build just the kernel using Sakaki's companion guide for 64bit kernel building. The other is to do a complete installation of Gentoo. The latter is what I planned to do, since Petra can compile bigger packages and can do it faster than the RPi, but I was not able to get it to work.
The 64bit installation writeup recommends doing the actual compilation as an ordinary user, which is always wise if it's practical.
firmware, i.e. boot files:
git checkout rpi-4.14.yand the running kernel is 4.14.65. It turns out that the downloaded repo is already on that branch. Apparently rpi-4.14.y is the only branch you can check out, and it's already checked out.
See above for how to put partitions and filesystem formats on Orion's disc. If you didn't do it earlier, do it now and mount the appropriate partitions on /mnt/gentoo and /mnt/gentoo/boot .
This is going to be nasty: the Gentoo64 Howto says to install a Portage snapshot by following the Handbook instructions, which tell some preparatory steps (listed above) and then you chroot into /mnt/gentoo followed by execing /bin/bash for ARM on your Intel processor. Fortunately this is not required to boot Orion (the machine on the SD card), and I will have to defer syncing Portage until then.
Installing the kernel: Copy arm64 boot files and the kernel onto Orion's disc.
Additional configuration.
groupadd -r -P /mnt/gentoo -g $GID $GROUP. GIDs for the new groups: i2c 249, spi 248, gpio 247.
touch/etc/udev/rules.d/80-net-name-slot.rules, overriding /lib/udev/rules.d/80-net-name-slot.rules . You could also put net.ifnames=0 on the kernel command line.
fsck.repair=yes. I wish you could use /dev/disk/by-label/ROOT-04, but I tried it and the RPi looked like it was booting but then hung. Here's the command line (all in one line):
I wanted to cross-compile a bunch of packages, because Petra has a faster
CPU, and also more memory which is important for sys-devel/llvm. It looks like
the normal
strategy is to set up distcc. See
discussion here. The package is built on the target host (Orion) except
the compilation itself is handed off to the remote server (Petra). I thought
this sounded like too much work and complication, so I tried to cross-compile
binary packages, the idea being to copy them to Orion (or deposit them on
Orion's disc, which is mounted), and install them once Orion is booted.
Dreamer! Emerge --buildpkgonly does not build dependent packages. If you
build them by hand they are recognized, but in some cases the requiring package
wants to run a program it provides, which of course is impossible. I punted:
I'm going to get a result a lot faster by compiling natively on Orion, assuming
I can work around the memory issue for sys-devel/llvm. See in the emerge
section for how I handled that.
Supposedly Orion is ready to boot.
syncon the VM host just for paranoia.
The first step in testing Orion is to compile X-Windows software. Back in the setup area I emerged x11-base/xorg-server, xfce-base/xfce4-meta and quite a lot of others. (For proper organization, that stuff belongs here, but I'm leaving it in setup to reflect how I actually did the installation.)
Next, the test procedure involves extensive exercises for Mozilla Firefox
(www-client/firefox). So I emerged it. 79 hours later… Really, the
RPi needed 78hr 54min 10sec to compile 28 packages, including the clang
'C' compiler and rustc
for the Rust object oriented language, both of
which are used extensively in Firefox and both of which are giants. Firefox
alone took 26.8 hours to compile.
Positive control: Without KMS, default CMA memory is 8Mib with 1644kb free. I started up the CMA free memory monitor, which every 30 secs prints the date and grep CmaFree /proc/meminfo. Now starting a XFCE session, and going through 1.16Mb of text/html on Firefox. Firefox started. Billable CPU timet:
| Firefox | Web Content | X-Server | |
|---|---|---|---|
| CPU Time (sec) | 405.9 | 54.3 | 96.4 |
| %CPU | 114% | 23% | 16% |
| %memory | 50% (all together) | ||
While Firefox was being started, CMA memory went as low as 0. During the test it went up and down, as high as 2966kb, returned to 0 once, end at 908kb. Conclusion: Firefox is operating normally and can do the test if the video driver is not flaky.
Rebooted with dtoverlay=vc4-fkms-v3d. Default CMA memory is 256Mib (262144 kB really kib), with 237092 kB free. vc4.ko is loaded and we do have /dev/dri/card0. The X-Server did not attempt to load /usr/lib64/dri/vc4_dri.so, rather tried glamoregl which failed to initialize. AIGLX says screen 0 is not DRI2 capable, and loaded swrast instead, successfully. vc4_dri.so would have been used on OpenSuSE.
Repeating the test.
| Firefox | Web Content | X-Server | |
|---|---|---|---|
| CPU Time (sec) | 272.5 | 150.5 | 297.5 |
| %CPU | 80.1% | 33.8% | 91.7% |
| %memory | 60.3% (all together) | ||
Wnen the test started, free CMA memory was 228972 kB (down slightly from 237092 kB). At the finish, 228472 kB; the lowest value seen was 226696 kB but some was freed. It's interesting that different component programs had significantly different CPU usage.
Continuing on to show the big photo essay. At the end, CMA memory was 219356 kB, slightly but not alarmingly reduced. (18:27) Conclusion: with software rendering (swrast_dri.so), the memory leak does not occur and the test can be finished successfully.
Results of emerge -s raspberrypi
.
Userspace tools and libraries. The working directory for the ebuild is /var/tmp/portage/media-libs/raspberrypi-userland-9999/ . In the prepare phase, failed patch raspberrypi-userland-9999-gentoo.patch (in ./files, which is a symlink to /usr/portage/media-libs/raspberrypi-userland/files, which contains 2 patches, a udev rule, and 4 others that I didn't look at closely). See ./temp/raspberrypi-userland-9999-gentoo.patch.out . Collecting files for a bug report.
The patch was attempted with -p0 through -p4, but the file to be patched was missing. $PWD is /var/tmp/portage/media-libs/raspberrypi-userland-9999/work/raspberrypi-userland-9999 . The target file in the patch is raspberrypi-userland-9999/host_applications/linux/apps/dtoverlay/CMakeLists.txt A plausible existing target file is ./work/raspberrypi-userland-9999/host_applications/linux/apps/dtoverlay/CMakeLists.txt . In other words, either work/ should be prepended or it should have cd'd to that directory. If I change dir, the target file is found but the hunk isn't. The context lines are present, but the material to be removed is not there, as if the patch had already been applied.
The other patch looks like it should be OK, cross fingers.
I turned raspberrypi-userland-9999-gentoo.patch into a symlink to
/dev/null. ebuild /usr/portage/media-libs/raspberrypi-userland/raspberrypi-userland-9999.ebuild prepare —
Digest verification failed (duh). Try --skip-manifest . That helped,
but now Cannot find $EPATCH_SOURCE!
(the patch file).
I got through that idiocy and the patch was non-applied.
This turns out to be a duplicate of
Gentoo bug 641046; I appended
a comment to that effect.
Now I'm going to do ebuild file.ebuild merge
.
Gakk, /var/tmp/portage/media-libs/raspberrypi-userland-9999/work/raspberrypi-userland-9999/interface/khronos/common/khrn_int_hash_asm.s:36: Error: unknown architecture `armv6'
And it goes downhill from there. Collecting files for another bug
report.
Mtime is Jul 30 2017.
This is possibly a binary package from the above one.
The source
URL is
http://gentoo.osuosl.org/distfiles/raspberrypi-firmware-1.20170303.tar.gz ,
113Mb (oink). Build directory size is 209Mib, installed size 8.3Mib.
Stuff ended up in opt/bin and opt/vc/lib64.
Compilation phase took zero time.
It provides e.g. libEGL.so and libOpenVG.so and libGLESv2.so, but
not /usr/lib64/dri/vc4_dri.so . When it's installed, GPU acceleration
did not happen.
Command line media player for the Raspberry Pi. Irrelevant.
Raspberry PI boot loader and firmware. Does not have to be installed on the execution host.
Raspberry PI boot loader and firmware, so it says. Is this a precompiled kernel?
Raspberry PI kernel sources. We cross-compile instead.
Raspberry Pi kernel mangling tool mkimage/imagetool-uncompressed.py . I'm not sure what this is, but I doubt that I need it.
Now trying: ebuild /usr/portage/media-libs/raspberrypi-userland/raspberrypi-userland-9999.ebuild unpack . The failed
patch is raspberrypi-userland-9999-gentoo.patch . The first argument has to
be an ebuild file. These saved files appear relevant:
/usr/portage/media-libs/raspberrypi-userland-bin/raspberrypi-userland-bin-1.20170303.ebuild
Mtime is Jul 30 2017.
This is possibly a binary package. The source
URL is
http://gentoo.osuosl.org/distfiles/raspberrypi-firmware-1.20170303.tar.gz ,
113Mb (oink). Build directory size is 209Mib, installed size 8.3Mib.
Stuff ended up in opt/bin and opt/vc/lib64.
Compilation phase took zero time.
It provides e.g. libEGL.so and libOpenVG.so and libGLESv2.so, but
not /usr/lib64/dri/vc4_dri.so .
/usr/portage/media-libs/raspberrypi-userland/raspberrypi-userland-9999.ebuild
Dated Feb 28 2017, has a lot less cleanup interventions than the above.
Trying media-libs/raspberrypi-userland-bin .
Before emerge, /usr/lib64/dri/vc4_dri.so ctime = Nov 9 10:47.
After emerge, same.
The binary package did not provide /usr/lib64/dri/vc4_dri.so .
/var/log/Xorg.0.log was identical: did not load vc4_dri.so and tried to
init glamoregl, failing.
Conclusion: I have not been able to turn on GPU acceleration in Gentoo-64, and so I am not able to test whether the graphics freezeups are fixed in Gentoo.
To cross-compile a package when you're populating Orion's new
disc: Initially, see
Cross Compiling with Portage
and in particular, you need
to do this on Petra (once):
emerge-wrapper --target aarch64-unknown-linux-gnu --init
Now to produce a binary package, do one of these alternatives:
emerge-wrapper --target aarch64-unknown-linux-gnu --ask --buildpkgonly category/package
emerge-aarch64-unknown-linux-gnu --ask --buildpkgonly category/package
But it doesn't do dependencies automatically, and often pkg-config will check on the version of an unknown library, failing. You then have to locate the library's name and category, and build it. Then retry the failed package — but that doesn't help if it's trying to actually execute what the package provides.
The normal way to produce binary packages is as a byproduct of a normal emerge, with the --buildpkg option. Then dependent packages would also be saved, autoconf tests would be satisfied from the already installed dependent packages, and if a compilation command line used an executable program from a build-time dependent package (like rustc), it would already be available.
When you run dispatch_conf you need to set this environment variable: CONFIG_PROTECT="/usr/aarch64-unknown-linux-gnu/etc"
The default PKGDIR=/usr/portage/packages . Cross compiled packages
end up in /usr/aarch64-unknown-linux-gnu/packages/category/pkg.tbz2 .
Copy the package onto the target host (Orion), then install it by:
emerge pkg.tbz2
.
I was never able to actually produce a useful binary package and install it.
When Gentoo is deployed on a group of hosts with similar architecture, a common strategy is to run distcc on one or more powerful hosts, and the less powerful ones hand off compilation to the servers. It would appear (I'm not sure of this) that the 'C' preprocessor is run on the client, where header files are among the unpacked source code, to produce something that the server can run through gcc or g++, sending back the object file. There is also documentation for cross-compilation with distcc.
Initially I bypassed distcc because it looked like a lot of work to set up,
given that cross-compiling was needed. But subsequent experience shows that
the investment of time would have been worthwhile. Except for one minor
detail: distcc has to be handcrafted (by the developers) to use each compiler,
and the lion's share of compilation time, for my installation of XFCE and
Firefox, went to programs compiled by rustc
(for the Rust
object-oriented language) and clang
(a new and improved 'C' and C++
implementation; see also llvm which was the first component to give me
trouble). None of these jobs could be done on the server. So it turned out
that I was correct: distcc would not turn a profit for me.
How to jigger the USE variable for a particular package: The XFCE wiki
page shows an example. Create
/etc/portage/package.use/xfce with the content app-text/poppler -qt5
meaning when building package xfce, get rid of the qt5 support in poppler which
is a pig. The minus sign means to suppress that USE flag; without the minus
you're turning on the flag.
equery is provided byapp-portage/gentoolkit . To find out what USE flags a requiring package sets: equery w sys-fs/lvm2
What I would really like to know is what USE flags the required package is sensitive to.
When a file is auto updated in /etc and certain other dirs, the new version is named ._cfg$NUMBER_original_name. dispatch-conf looks for such files and helps you update them. There is a session log in /var/log/dispatch-conf.log . Need to set up /etc/dispatch-conf.conf and create listed archive directory(s) (default is /etc/config-archive ). Here is a complete list of dispatch-conf's action letters:
There is a dependency tangle; a long list of programs require dev-libs/glib. Another long list require dev-util/gdbus-codegen. They need to be of the same version. dev-libs/glib-2.56.2 is the most recent version. But dev-util/gdbus-codegen-2.56.2-r1 is not recognized as satisfying the dependency of the requiring programs and matching glib in version. I broke the tangle by emerging it explicitly, and the dependency resolver accepted it.
emerge --ask "=dev-util/gdbus-codegen-2.56.2-r1" -- It actually
proposes to do the emerge, including upgrading glib and glib-utils.
Compilation succeeded.
I got this from a search on Google with these keywords:
gentoo emerge arm64 dev-libs/glib-2.56.2 dev-util/gdbus-codegen-2.52.3
.
The useful result was a
Gentoo Browse instance which gives, per package, the available
versions, their keywords, USE flags, dependencies, inverse
dependent packages, and changelogs.
See also https://packages.gentoo.org/packages/dev-lang/spidermonkey
for a package index.
Temperature limiter is working. Often all 4 cores are running compilations. /sys/devices/virtual/thermal/thermal_zone0/temp says 82C to 84C and there's a thermometer symbol overlaying the display. /sys/devices/system/cpu/cpufreq/policy0/cpuinfo_cur_freq = 1.2GHz but evidently this is being throttled. It looks like the threshold for the GPU taking action is 80C; the thermometer is off at 79C.
It finished 81 packages in 2:41:24 elapsed = 161min, then failed
to compile dev-lang/spidermonkey-52.9.1_pre1::gentoo .
Lots of complaints about Unable to trace static ELF:
/sbin/ldconfig: /sbin/ldconfig -p
. There were a lot of warnings
about coding practices that someone should look at. The real
error message was aarch64-unknown-linux-gnu-g++: internal
compiler error: Killed (program cc1plus)
.
Rebuilding just this package separately. Fails equally.
Cutting a long story short: I suppressed parallel make and added
more swap space (both discussed extensively previously), and
dev-lang/spidermonkey compiled successfully.
When I switched the profile to default/linux/arm64/17.0/desktop, it decided to compile Thunar, XFCE's file manager. This failed in the following dependency chain:
So why do I care about thin-provisioning-tools? This
forum post on Google Groups about thin-provisioning-tools (OP Andrew
Lowe, 2017-09-15) gives some insight. LVM handles enlarging
(or shrinking) partitions at runtime and fusing partitions on different
discs into one logical partition. One possibiity is to defer allocating
underlying space until data is actually written to it, called thin
provisioning
. It is very unlikely that a Raspberry Pi will ever use
either LVM or thin provisioning.
Respondent Neil Bothwick says: Add "sys-fs/lvm2 -thin" to /etc/portage/package.use and thin-provisioning-tools will go away. In my case /etc/portage/package.use is already a directory (all files therein are effectively concatenated), and I created a file called sys-fs-lvm2 with just that content (and a #comment). Indeed, sys-block/thin-provisioning-tools was not emerged and Thunar was.
Other respondents discussed why you might keep or toss these packages,
and where the thin
USE flag came from. It is not a global USE
setting; sys-fs/lvm2 specifically has it configured. The explicit
-thin
suppresses lvm2's default.
According to the Gentoo wiki article on Benchmarking, app-benchmarks/glmark2 used to be available but it no longer builds. The source code is available on Github.
| Planning | Testing | Setup | Collapse | Gentoo | Top |