HTC G1 Cellphone
Debian on the G1

Introduction

It's a phone!

The G1 is not a server, nor is it a program development machine. Android is designed to do phone things, and to support its Java applications. It is not POSIX compliant; there isn't even a test command for its shell. Once you get root access you will find that your normal UNIX procedures are not going to get you very far. Here are a few tricks to tide you over:

• There is a UNIX filesystem structure, with a root directory called / and with mount points for the other partitions.
• The proc filesystem is mounted on /proc. cat /proc/mounts works.
• It does have cat. It doesn't have more or less. It has dd also. It doesn't have cp, but does have mv. I wouldn't try moving across filesystems though. It has ln and can make symbolic links (not on FAT filesystems).
• The shell can do glob patterns.
• There is a ls command. It takes -l but not -d.
• To avoid listing directory contents, use echo glob* instead.
• cd works.

However, this limited repertoire is essentially useless for anything a normal system administrator or UNIX hacker would want to do. Therefore a number of people have worked out various schemes to put a real UNIX distro onto the G1.

Goals and Issues

• The main goal is to provide useable utilities and normal desktop-style applications. It would be a cute project to actually come up with device drivers so you could boot a foreign kernel and palmtop distro, like the people trying to run Android on the Nokia N810 but in reverse, but that's not what I'm doing and that's not what most people are after. I want Android to be fully functional, but with supplements.

• The installation is going to have to be done at the command line: the adb shell, a shell reached through Dropbear, or Terminal Emulator if you want to do the whole job on the G1.

• Generally Debian is chosen because it is easy to handle in the situation of only command line access. I have experience with Debian in this role on the N810 and I have a list of packages all worked out from that project, so that's what I'll use. Other people report installing various kinds of Ubuntu, and I believe I remember one instance of Fedora.

• A Linux distro occupies a minimum of 200Mb, and most likely I'll want additional application packages. An aggressively stripped Debian installation will barely fit in main flash (256Mb) provided the existing operating system, Android, is evicted, which is not acceptable. Therefore the distro has to go on the memory card, and specifically, on a real UNIX filesystem on the memory card, since symbolic links and ownership and modes are important.

• However, when you upgrade the software on the G1 you deposit the new image(s) as /sdcard/update.zip, exactly this name, and the recovery utility expects to read it. I have never seen anyone reporting success when upgrading from a memory card formatted as other than VFAT. I'm not saying it's impossible, and it would be really nice to find out, but for this project I'm going to be conservative. The kernel in Android v1.0 lacked the ext2/ext3 driver, so reading a memory card thus formatted was certainly impossible. The key test would be to find out when ext2/ext3 came in, and to try to upgrade from that version to the next.

• There are three ways to make a UNIX filesystem available.

  • An easy solution is to create a big file on the VFAT filesystem and hook up to it via a loopback device (/dev/loop0 for example). It can then be formatted with any filesystem your kernel can read. The problem is, how do you get it mounted and unmounted automatically when the memory card is inserted or removed? Also, when accessing data you first run through the kernel driver code for the UNIX filesystem, then for the loopback device, then for the VFAT filesystem, and finally for the MMC device: double work and double slowness.

  • Most people seem to make two partitions: the first is VFAT for the provided system to treat normally, and the second has the UNIX filesystem. Again, how do you get it mounted and unmounted automatically?

  • Vold is the automatic mounting daemon. If it could be hacked to mount filesystem types other than VFAT, then a card with a single UNIX partition could be used. For upgrades one would put the new image on a second memory card with a VFAT filesystem, insert that, and boot in recovery mode.

Ext3 Filesystem on the Memory Card

Can I mount a non-VFAT memory card at all? The MMC devices appear automatically as /dev/block/mmcblk0p1 etc, or equivalently, /dev/block/vold/179:1 etc. Minor device 0, or mmcblk0 without the partition, refers to the raw device. Neither /dev/block nor /dev/block/vold exist on my kernel 2.6.27 (from OpenSuSE 11.1), and in a Google search I see them mentioned only with Android. Android's automatic mount supervisor fails to mount partition 1 unless it is formatted as VFAT (not too surprising). Although the mount command's usage message says that the filesystem type is optional, in fact an ext3 partition cannot be mounted without -t ext3. So it's time to hack.

Use the source, Luke! I found answers for several of the above points in the sources for vold, installed as /system/bin/vold. This program is the first to start at boot, even before d-bus. It seems to do the same job as udev on a normal distro, but specialized for the palmtop role: when the memory card is hotplugged it creates the /dev/block inodes described previously, it coordinates (I didn't study the details) exporting these devices by the USB connection, and it mounts the partition(s) including running the appropriate file system check program (which, to my knowledge, no desktop distro manages to do, and neither does Maemo for Nokia tablets, and Maemo really needs to because corruption is endemic). Vold also starts the media indexer on each partition.

As installed, vold only knows about VFAT. However, ext3 support is already in the code and to turn it on you only have to uncomment one table row. I did so, and it was able to identify the filesystem on the card and mount it, for both VFAT and ext3. But although the code is written to mount arbitrarily many partitions, it only mounted the first one. I decided to not try to debug it, but instead to put only one partition on the card. For software upgrades I will put the image onto a VFAT card and that into the G1.

In the tale below I'm skipping numerous failed attempts, presenting only what was on the main line to success.

Various Howto's

• This howto by Pavel Machek (2009-03-04) suggests (with command lines shown):

  • Mount the phone's VFAT partition on your desktop machine.
  • Create a big file on it and put a ext2/ext3 filesystem on it.
  • Mount it on the desktop machine using a loop device.
  • Download and execute debootstrap pointing at the mount point. This does the first stage of installing Debian.
  • Unmount everything; remount on the G1.
  • Executing on the G1, mount the Debian partition using loopback.
  • chroot and run the second stage of debootstrap.

  I'm going to have an actual partition, not a loopback file, but the instructions for using debootstrap will be very helpful.

• This forum post describes an installer script by Ghostwalker (2009-03-13). The only fly in the ointment is that it requires JesusFreke's firmware update (hacked image).

• This wiki article describes partitioning and moving apps to the card, posted 2009-04-12 by DarkriftX. It is for the Haykuro hacked image, and it implies that a second partition will be mounted automatically if the first partition is VFAT. But this may (or may not) be one of the hacks in the Haykuro image. A relatively unobtrusive way is shown to install busybox, if needed, but Cupcake is supposed to have cp and rm now, so busybox is no longer needed to do the procedure. In reality it has rm but doesn't have cp.

• This forum post asks if additional partition(s) are automatically mounted if mount points are listed in /system/etc/mount.conf? The post is by Brian dated 2008-12-16. Developer Mike Lockwood replies that it's supposed to work, although the higher level framework provides no support. I assume that means that, for example, there is a settings dialog for the SD card, meaning its first partition, but there would not be any analog for the other partitions.

  Some snooping reveals that /system/etc/mount.conf does not exist, but /system/etc/vold.conf does, and looks useful except for the detail that the filesystem type is mentioned nowhere. vold is running and starts up very early, ahead of dbus-daemon. mountd is not running.

• DarkriftX tries and fails to add a stanza to vold.conf.

• Another howto for splitting the SD card by Martin Fick, 2009-04-07. It needs the JesusFreke image and has a link to it. This split script is said to be high quality.

Jimc's Procedure

• Thanks to developer Mike Lockwood (2009-01-11) for this suggestion on how to make adbd run as root so adb remount will actually do something. Also adb sync, and adb push to a directory owned by root. Edit /default.prop changing ro.secure=1 to ro.secure=0.

  adb pull /default.prop .
  (Edit it on your real machine)
  adb push ./default.prop /sdcard
  adb shell  . . .  su root
  mount -t rootfs -o remount,rw rootfs /
  dd if=/default.prop of=/default.prop.orig
  dd if=/sdcard/default.prop of=/default.prop
  mount -t rootfs -o remount,ro rootfs /
  Reboot the phone.
  Well, that was a nice idea but it was ineffective (on Cupcake).

• When I repartition the production card I will lose all the data, so I first made a backup copy on my laptop.

• The first step was to get the card to mount. I took a spare memory card and partitioned it on my laptop (after unmounting the automatically mounted existing partitions). For production I had to revert to just one partition, but the procedure for partitioning a disc is kept here for completeness.

  parted /dev/mmcblk0
  print (shows existing partitions; I have 1=32Mb fat16, 2=984Mb ext3)
  rm 2 (removing both partitions)
  rm 1
  mkpart primary fat32 32.3kB 128MB (reported as fat16)
  mkpart primary ext2 128MB 1016MB (not ext3)
  print (shows the above partitions)
  quit

• Command lines to format a partition. For production I will want volume labels, and I'm going to use ext3 rather than ext2 (see discussion elsewhere). To make this happen: (sd1a-1 and 2 are the labels)

  mkdosfs -n sd1a-1 -F 32 -v /dev/mmcblk0p1
  mke2fs -L sd1a-2 -j /dev/mmcblk0p2
  tune2fs -c 0 -i 40d /dev/mmcblk0p2

  Command line switches above:

  mkdosfs

  • -n volume_label
  • -F 32 selects a 32-bit FAT (for VFAT)
  • -v for verbose output

  mke2fs

  • -L volume_label
  • -j for a journal, making it ext3

  tune2fs

  • -i 40d means do a full fsck every 40 days; inbetween it will just recover the journal and assume that takes care of any problems from unclean unmounting (crashing).
  • -c 0 means do not consider the number of mounts when deciding whether to do a full fsck. A nonzero number would make it check every so many mounts (or by time, whichever comes first).
  • -j could be given also to tune2fs to upgrade an existing ext2 filesystem to ext3.

• With the memory card prepared as above, Android will mount partition 1 as if it were an authentic memory card, and the content, such as it is, is visible. (I put a test file on each partition with a distinctive name and containing one line saying this is the fat32 partition, etc.) Now I can do (as root, via adb shell):

  mkdir /data/mnt
  mount -t ext3 /dev/block/vold/179:2 /data/mnt (-t is required)
  cat /data/mnt/sd1a-2.txt (shows correct content)
  umount /data/mnt

• Next step is to hack /system/etc/vold.conf. The stanza I'm going to add is:

  volume_sd2 {
      media_path     /devices/platform/msm_sdcc.2/mmc_host/mmc1
      emu_media_path /devices/platform/goldfish_mmc.0/mmc_host/mmc0
  
      media_type     mmc
      mount_point    /sd2
      ums_path       /devices/platform/usb_mass_storage/lun0
  }
  	

  Comments: In /sys/devices/platform there are two possibilities: msm_sdcc.2 and msm_sdcc.1. I'll start with msm_sdcc.2. msm_sdcc.1 seems to have no content: perhaps an additional SD card controller that has no associated socket. I left media_path and emu_media_path untouched, since likely they refer to the raw device (not to the partitions).

  Looking at strings in vold, I found a hardwired mount options string for vfat. vold may (or may not) always run fsck before mounting any volume.

  Do this on the laptop:

  adb pull /system/etc/vold.conf ./vold.conf
  (Make a backup copy on the laptop)
  (Add the stanza using the laptop's editor)
  adb push ./vold.conf /sdcard/vold.conf
  adb remount (operation not allowed . . .)
  adb shell
  mount -o remount,rw -t yaffs2 /dev/block/mtdblock3 /system
  cd /system/etc
  dd if=vold.conf of=vold.conf.orig
  dd if=/sdcard/vold.conf of=./vold.conf
  ps (and find the PID of vold, the 2nd field)
  kill -HUP 31 (that's the PID) (but has no visible effect)
  Reboot (but partition is not mounted).
  /system/etc/vold.conf

  So this attempt was ineffective but at least we now have vold.conf describing two partitions. Unfortunately I was never able to get the second partition into action.

Hacking Vold

• In the likely case that I have to get the entire source.

  • Make sure prerequisite packages are installed.

    • git >= v1.5.4 (installable)
    • gpg (Gnu Privacy Guard) (got it)
    • JDK >= 5.0 update 12 (not Java 6). On SuSE this is back-version. It turns out that you can compile Android with OpenSuSE's java-1_6_0-sun-devel but possibly the emulator may have trouble executing the applications using the host hardware (if it actually does that).
    • flex and bison (got it)
    • aperf (What is it and where do I get it? Not in SuSE build service.) gperf may be an acceptable alternative that's in the distro; this command is what the Makefile wants.
    • libsdl-dev (not in distro) It's called SDL and it's in the distro under that name. You can install SDL-devel with no other dependencies.
    • libesd0-dev (libesd-devel on SuSE, installable, deps libesd0 itself, and audiofile-devel)
    • libwxgtk2.6-dev (optional) (skip this)
    • zip (got it)
    • curl (got it)
    • valgrind (optional, installable, I'm going to skip it)

  • Snarf the repo installer script. Create a directory for Android (1.5Gb sources, 6Gb needed to build). Execute repo using the command line in the howto. It initializes itself.

  • Download everything (1.5Gb download) by doing repo sync. When unpacked, the files occupy 2.4Gb. 5.5Gb after building.

  • Do the needed hacks.

  • Just execute make in the top directory. It will build everything, taking about 3 hours. A new system image is deposited as ./out/target/product/generic/system.img . On the other hand, ./out/target/product/generic/system is the original instance of what's in the system image; for example, the individual application packages will be found in ./out/target/product/generic/system/app/*.apk and vold ends up as ./out/target/product/generic/system/bin/vold .

According to the sources in ./system/core/vold/volmgr.c, the config file vold.conf is read by a generic subroutine (config_load_file). It consists of key-value pairs. The outer layer is a name (must begin with volume_) and a stanza in { }. The keys that may be in the stanza are these:

It is a little strange to see the device mapper getting into the act, since Android doesn't have one. However, I suspect that Solaris' vold was the model for Android's, and on a Solaris server a big RAID and/or LVM setup would be expected.

When a memory card is hotplugged (signalled by HAL), vold iterates over each partition, identifies the filesystem type, runs fsck (filesystem checker), mounts it, and then starts the media inventory scanner. A desktop distro would use object oriented fsck and mount programs, that is, these programs would identify the filesystem type and would run the appropriate variant automatically. But in Android, type identification and switching are handled by vold. It has a virtual table hardcoded, which in the stock configuration has only one row for VFAT. However, the row for ext3 (not ext2) is present but commented out. All it takes to get vold to auto-switch between VFAT and ext3 filesystems is to uncomment that row and recompile.

I never did get vold to actually mount any partitions beyond the first, even though it recognizes that there are multiple partitions and even though the code to do it is present. I started debugging this issue, but decided that my higher priority was to install Debian (on one partition) and to use it for testing and hacking.

Mounting the Memory Card

To get the ext3 partition mounted, you need some prerequisites:

• A memory card formatted with ext3. The command lines to format an ext3 partition were given previously.

• Vold that knows about ext3, as hacked above.

• Kernel support for ext3. Look for it in /proc/filesystems. If present, the module will be hardwired in the kernel (not a loadable module). It was absent in Android v1.0 (the original) and appeared in some revision prior to v1.5 Cupcake. If you are stuck on a back version, I've seen forum posts with links to hacker-built loadable modules.

• An e2fsck program: /usr/sbin/e2fsck. Vold will not mount the partition unless e2fsck gives a successful return code, implying that it has to exist and be executable. Alternatives are:

  • I copied e2fsck from Debian; actually before installing Debian I downloaded just the e2fsprogs package for armel (modern ARM ABI) and ripped it apart manually to get e2fsck. The reason this failed was that the shared object loader for Debian is /lib/ld-linux.so.2 (as in a normal Gnu distro), whereas on Android it is /system/bin/linker. People say that you can make a symbolic link between them and it will work, and this would be worthwhile trying, but I didn't yet. The symbolic link would be in a new /lib directory, and the root filesystem is a ramdisk (probably the initrd), so /lib and the link would have to be re-created at every reboot, and this would have to happen before the memory card was mounted. See the discussion elsewhere of the Android boot concept and how it could be subverted to do this task.

    Another problem is that Debian uses a versioned glibc, specifically /lib/libc.so.6 like a normal desktop distro, and the basename appears in the library list of e2fsck. Android provides /system/lib/libc.so. Again a symbolic link would solve the problem, but it would have to be made at boot time. (And was libc built with all the needed entry points? People say it works, but we'll have to experiment to find out.)

  • Cheat. To get the main job moving I wrote a simple shell script that just does exit 0, nobody here but us chickens. I shouldn't do this permanently, though.

  • The code for e2fsck is in Android's source code, and by far the best solution would be to compile it there. But initially I was not able to find the switch that makes the build system build it. Android has quite a number of unique features, and I'm sure it would be a total bust if I compiled directly in the source directory, even if I did override the output architecture. The same issue applies to getting the source from elsewhere.

    Progress: the source dir contains Android.mk which looks serviceable; in particular, it ends with include $(BUILD_EXECUTABLE) (not commented out). But the containing dir says include $(call all-subdir-makefiles) except commented. I uncommented it and it produced the programs and libraries, total of 345 kbytes. I installed them into /system/lib and /system/bin (first remounting /system read-write), and e2fsck runs.

I restored the backed-up memory card content. A VFAT partition has very feeble permissions. Rather than doing research on improving security, I pretty much replicated the VFAT behavior: changed all files to mode 666 and directories to 777, hiss, boo. Later I need to review these and find out how to tighten up these awful permissions. I checked out all the content on the card (by programs, not every single file individually), and everything still worked.

Installing Debian

As of 2009-02-14 the standard (stable) version of Debian is Lenny. This is the one I used. It is the first to have complete support for the armel architecture. There are over 25000 packages in Lenny, of varying usefulness on the handheld device, or on any device.

A prerequisite for installing Debian turns out to be Busybox, because standard Android lacks a chroot command. See this blog post about Debian on Android by Saurik (Jay Freeman, dated 2008, earlier than 2008-11-19). He has a link to a statically linked, working version of Busybox, which I used.

I followed Pavel Machek's instructions to install Debian. Specifically, on the laptop locate and install a recent version of the debootstrap package that includes an install script for Lenny. If your distro uses RPM, as mine does, you will need to disassemble and install the Debian package by hand (follow link). Mount the memory card, let's say on /mnt. I decided to put my Debian files in a separate directory with sane permissions called .../deb. Do stage 1 of the installation, which downloads the package files:

mkdir /mnt/deb
debootstrap --foreign --arch armel lenny /mnt/deb/

This took me about 15 minutes and put 130Mb of package files onto the card. Unmount it when finished. Connect the USB cable to the phone. Suppress sleeping when on charge; this is Settings - Applications - Development - Stay Awake. (And of course USB Debugging, on the same screen, has to already be turned on.) Even if you're using the Terminal Emulator you need external power because the following process will empty your battery. Mount the memory card on the G1; I'm assuming, as in my case, that it will mount on /sdcard. Gotcha: when mounting the card, vold specifies the options noexec, npdev, nosuid (actually, the MS_NOEXEC (etc.) flags for mount(2)). You will need to remount it by hand omitting those options. (Adjust the device if necessary. The command line is folded for readability.)

cat /proc/mounts #Check what device is used
mount -o remount,rw,nodiratime,errors=continue,data=ordered \
-t ext3 /dev/block/vold/179:1 /sdcard

Now start stage 2 of the installation, which unpacks and installs all the packages. (The command line is folded for readability, so your browser does not fold it at some arbitrary point.)

PATH=/usr/sbin:/usr/bin:/sbin:/bin /system/xbin/busybox \
chroot /sdcard/deb debootstrap/debootstrap --second-stage

You need to set the PATH to where the utility programs are in Debian, and you need to chroot so it installs itself in its own area (and can find the utilities and libraries). The installation process took 18 minutes and put 860Mb on the card. The result is a minimal Debian system that can be used to download and install your keystone packages: the ones you use to accomplish your mission. Quite a bit of the 860Mb can actually be gotten rid of, if space is a problem.

Using Debian

As noted previously with e2fsck, Debian packages expect to find basic infrastructure in standard places like /lib and /usr/lib, which don't exist on Android. There are various maneuvers to deal with this and Saurik's blog post goes into quite a lot of detail on the topic.

Android Paranoid Network

Initially, none of the networking applications in Debian do anything; it is as if the network does not exist. This Google Groups thread by leemgs (2009-04-09) posts "features" of kernel 2.6.29 for Android: Paranoid network (doesn't give details). A subsequent commenter mentions that ANDROID_PARANOID_NETWORK is the kernel config symbol.

The Android source package includes complete kernel sources with this patch present and enabled. ./Documentation/android.txt lists all the required and forbidden configuration options that Android's userspace expects, including this one. It is used in ./net/ipv4/af_inet.c and ./net/ipv6/af_inet6.c and ./net/bluetooth/af_bluetooth.c. All three files are similar: linux/android_aid.h is included, which defines AID_INET = 3003, AID_NET_RAW = 3004, AID_NET_BT = 3002. For inet and inet6, current_has_network() tests for AID_INET or AID_NET_RAW; current_has_cap() tests for AID_NET_RAW. I believe the latter is the limiting factor, at least when root runs the program. Bluetooth is similar except with AID_NET_BT.

So I copied several net programs (ping, traceroute, wget) to /tmp and made them setGID to group 3004. They were then able to function. Clearly this is a workaround for testing, not a longterm solution.

Miscellaneous Tidbits

• "adb logcat" dumps the log and continuously feeds it out, like "tail -f".

• How to disassemble a Debian package by hand: It is an ar archive. Do ar x package.deb. You get three files in the current directory; you want data.tar.gz. Un-tar it: tar xzf data.tar.gz. The payload appears, in directory stacks which for the production installation would begin at the filesystem root.

• This blog post by Brad Fitzpatrick (2008-12-14) has among other things a script so he can inject keystrokes received at a terminal (ssh) session into the Android console datastream, as if typed on the keyboard.