Valid HTML 4.01 Transitional

HTC G1 Cellphone
Upgrade to CyanogenMod 6

Jim Carter, 2010-03-30

Web links:

CyanogenMod is a hacked image actively maintained by Steve Kondik Cyanogen and a voluteer hacker crew. It is widely used, having about 3.5e5 official installations, and is well regarded. But compare to 5.5e5 activations per day of phones running Android. From AndroidCentral quoting Larry Page (Google)'s earnings announcement, dated 2011-07-14.

We can consider saving some development sweat by trying to use CyanogenMod assuming it has the wanted features. CyanogenMod-6 supports the HTC Dream (G1), is based on Froyo (2.2), and is dated 2010-08-28. CyanogenMod-7 is based on Gingerbread (2.3.3), dated 2011-04-11. Unfortunately it does not support the HTC Dream (G1). Already with Eclair it needs to move partition boundaries; the Dream is standardly configured with 64Mb in the /system partition, and CyanogenMod 5 and above won't fit, according to the DangerSPL page. A recent comment poster says he has CyanogenMod (version not stated) on his G1 (HTC Dream).

List of features in CyanogenMod (there are additional features of a non-sexy nature that aren't listed):

What do the CyanogenMod docs say about support for jimc's existing hacks? (See below for the actual situation.)

The Recovery image is a mini-system with just enough brains to write a software package to flash. The one most often mentioned is Clockworkmod Recovery; it is available from the Android Market and includes the payload image. The wiki page for the HTC Dream suggests that the Amon-Ra Recovery works better on the Dream. Clockworkmod Recovery is maintained by Koushik Dutta; could this be the Budins' son-in-law? Photo doesn't look like him.

Koush suggests using (his?) ROM Manager 4.4.0.3. This program is included in CyanogenMod (not sure of the version).

Plan for CyanogenMod

The plan is to put CyanogenMod onto the phone, see what it really is, and then try to get my stuff working.

Upgrade plan:

Upgrading the OS

This is summarized from CyanogenMod's HTC Dream product page.

It's recommended to fully charge the battery before starting these steps. Loss of power in the middle of flashing could cause widespread corruption, requiring JTAG to get out of (good luck!) It seems to be bad mojo to run the charger, most likely because you need working firmware to turn it on.

Step 0: make a normal backup of your phone.

Step 1: r00t your phone. With the Dev Phone 1 this is already accomplished.

According to the wiki page, ClockworkMod Recovery is not recommended for the Dream. Please upgrade to Amon_Ra. Where to get Amon_Ra's recovery program: Read forum posting; the download links are after the usage instructions.

If you have an engineering SPL, as on my Dev Phone 1, here is the procedure to flash using fastboot, according to Amon_Ra. Android Wiki article about fastboot.

If you rooted your phone but didn't put the engineering SPL on it, use this alternative. (CyanogenMod-5.0.7 has a bug precluding this method.) Copy recovery-RA-dream-v1.7.0.img to the root of the SD card. Jimc's suggestion, don't tempt sleeping dragons, use a card formatted with FAT32. Start the phone's terminal app, su root, then flash_image recovery $filename.

If you have a cooperative Recovery program, you can use that to flash the replacement Recovery; see the procedure below. But Amon_Ra recommends to use fastboot if at all possible.

If your radio version is not on the compatibility list for DangerSPL, download the right one (9.2Mb) and install it now using Recovery (procedure shown below). However, people recommend using fastboot instead of Recovery. Unfortunately fastboot doesn't know how to unzip a package. In any case, my radio already has the right firmware so I'm going to not re-flash it. [Phone survived.]

Next, install the DangerSPL, which is the SPL (second program loader, kind of like Grub stage 2) off the closely related HTC Magic. Wiki page for DangerSPL. You need this because the native SPL can only handle 64Mb system images and CyanogenMod-5 and 6 are bigger than this; DangerSPL can handle up to 96Mb. Caveats:

To obtain the various files, see the CyanogenMod Wiki product page for HTC Dream. It has links to howto's and index pages, which is where you find the links to the files themselves. Download them and put them in the root of the SD card. I mounted the SD card on my laptop to do the copying. Here's a list of the file basenames you want, as of this writing (2011-09-05), but it's possible (likely) that versions may become updated. Note: you need the Google Apps file that goes with your CyanogenMod version, and for CyanogenMod-6 and earlier you need to get the correct Google Apps variant for your phone. I got into a horrendous tangle by using gapps-mdpi (not tiny) which is the wrong one.

It's better to use fastboot to install the first two, but I wasn't able to make this happen. Install the zipped images using Recovery, in the order given. It is probably necessary to use the Amon_Ra Recovery program in order to adjust the image size. Android Wiki article about Boot Modifiers including keyboard commands for Recovery. But Amon_Ra Recovery can do a lot more things.

Investigating CyanogenMod

Setting the Settings

Open the Settings app via Menu/Settings. Going through all the settings -- these notes indicate only the ones that I altered, except when the default is notable.

Application Settings:

My Desktop Icons

Page 0 (Center)
Battery Usage Market
Clock Amazon Kindle Music
Jota Text Edit Google Maps Tricorder
Page 1 (Right)
RealCalc
K-9 Mail Contacts Calendar OpenVPN
Out of Milk CadreBible ConnectBot Terminal
Page -1 (Left)
Andoku Lexic Gallery Camera

Setting Up Dropbear

Summarizing the CyanogenMod wiki page on SSH:

Miscellaneous Setup

My Web Home Page

I restored my backed-up web materials, that formerly were on the SDcard of Cupcake. I edited the home page removing the Google Gears hack; the version of Gears that I'm using is obsolete and needs to be upgraded, but not today. A URL of file:///sdcard/jimc-web/index.html was honored by the browser and was functional as my home page -- unlike on Cupcake where the developers, in a fit of unjustified paranoia, suppressed file URLs.

The only problem was that I use some symlinks to indices of contained directories, and FAT32 does not do symlinks. So I duplicated the referent pages instead, hiss, boo.

Setting Up Root Certificate

To import the CFT Root Certificate I simply navigated to the link to the cert (mime-type application/x-pem-file) and clicked on it. A dialog box opened asking for the friendly name of the cert, and certificate storage had no password yet so it asked for it to be set. It reported that the downloaded object had one CA cert (correct), and that was that. But this certificate was not honored for SSL/TLS connections (by the web browser).

To watch out for: how do you change the password for the Java certificate store? (Settings/Location and Security/Certificate Store.) Also, how do the trust vendor certs get updated, e.g. to remove the compromised cert for DigiNotar?

Here's a forum post from omegat on how to import certs the right way (on Donut or Eclair). Here's the procedure (in the 4th unit):

Eventually I repeated the procedure I developed on Cupcake to import certs into the system keystore. That was effective: my home and work certificate authorities are now trusted.

VPN Connection

CyanogenMod-6 comes with OpenVPN preinstalled. (Not sure when it came in; I think CyanogenMod-5.) If you have the stock Android, a rooted phone, and a tun.ko module compatible with your kernel (that's the hard part), you can install OpenVPN from the Market. Recommended to just upgrade to CyanogenMod.

I installed OpenVPN Settings and OpenVPN Installer by Freidrich Schäuffelhut, from the Market (free; the latter is ad supported). The two OpenVPN GUI packages don't look promising; they appear to attempt to install their own binaries, two different ones.

CyanogenMod includes the OpenVPN binary, the tun.ko network tunnel module, and a settings module, found in Settings / Wireless and Net / VPN Settings / Add VPN / Add OpenVPN. I believe that stock Android-2.2 does not have any of this, but you can install them from Market packages. To load tun.ko requires root access, that is, r00t your phone.

I was able to create a configuration that would connect; however, the provided configuration does not pull settings like the MTU from the gateway, and the cipher and HMAC algo must match exactly and cannot be pulled, but there is no way to influence them in the client, so no data can be transferred and the connection eventually dies.

These are the specific complaints, in the unlikely case that you are able to adjust the gateway to match what the stock client wants to do. The client wants a MTU of 1541 bytes; it does not do LZO compression; it uses the BF-CBC cipher (Blowfish?), and it uses SHA1 for HMAC.

I turned to OpenVPN Installer, which you execute as an application. It opens with a settings-type dialog. It wants to see /sdcard/openvpn/ populated with certs and *.ovpn. The latter is the configuration file extension expected by this program. I installed these files from /etc/openvpn on Jacinth, which works:

Procedure:

Here are some details:

XMPP/Jabber Client

On Cupcake I used Jabiru as a XMPP client, but it appears to be discontinued. I replaced it with Xabber by Redsolution (free). The server on CFT reports that it was making connections (probably meaning that TLS negotiation succeeded; it was configured for TLS), but it could not authenticate. It requires a saved password, hiss, boo!

Trying Beem. People say good things about it. But it requires a saved password. It force closed when connecting to my server. Uninstalled.

Trying Yaxim. Same issues as with Beem. Requires a saved password. Could the server be messing it up? Somehow it sets itself up to start at boot, and it tries to connect to the server without being explicitly turned on. That's going to eat battery. Uninstalled.

These three are the only reasonable XMPP clients on the Market. I need to get XMPP working, but I need other things first, so I'm deferring this issue.

Multimedia Checkout

Results of my standard web browser test:

Type Class Result
JPEG Image Shown properly
PNG Image Shown properly
GIF Image Shown properly
PDF Text Downloaded; you start EBookDroid (reader) by clicking the notification
WAV Audio Downloaded, click notification, it plays
OGG Audio URL was sent to the music player, which dies
MP3 Audio URL was sent to the music player, which plays it
Theora Video URL was sent to the music player, which dies
SWF Video Downloaded, click notification, nothing happens
RPZA Video Downloaded, click notification, says this video cannot be played
AVI Video Downloaded, click notification, says this video cannot be played
Java Applet Was not executed
M3U Playlist Unrecognized by browser
ZPL Playlist Unrecognized by browser and music player

Using the Pocket Computer as a Media Player

This is using the stock player, and not through the web browser. It is not possible to type in a URL in the media player; you have to go through the web browser, or else deposit music files on the SD card by other means.

Procedure to copy media without wires: Get a SSH session on Selen, display on laptop. Using the laptop's web browser, get the URL to the TAR file of the desired album; see below for accepted formats. It will be about 60 to 70Mb each. On Selen:


cd /sdcard/Music
wget -O - $URL | tar xf -

How does the music player know where the music is? cd elsewhere, then unmount the SD card (Settings / SD Card / Unmount), then remount it. The music indexer will be run. This hogs the CPU, so give it a minute or two.

To play the music: My albums include pre-made playlists, with tracks in the correct order. Start the music app. Pick the Playlist tab. Long-press on a playlist filename. Pick Play from the context menu.

Alternatively, pick the Album tab. Click on an album's name. The list of tracks is shown, not necessarily sorted lexically, in the form of a playlist. Click on one track and it will be played; or more typical for my usage, hit Menu / Play All. A shortcut for this action is to long-click on the album's name and pick Play.

The playlist is expected (or at least is acceptable) in M3U format (extension .m3u). The content may/must be relative filenames, relative to the playlist. The content may be in Ogg Vorbis or MP3 format; others are promised but I did not test them.

If the playlist contains a URL to a streaming audio site, it is removed! Hiss, boo!

Mounting the SD Card

I partitioned my SD card with (in order) VFAT, swap, and ext4 partitions. The VFAT partition is mounted at boot; the others aren't. I deferred the issue of getting them mounted, to this point.

The intended design in CyanogenMod-6.1 (I am not sure how much of this is inherited from the stock Android-2.2 Froyo) is that the scripts in /system/etc/init.d/* are executed one by one (in lexical order) as a very early step in booting. Once all of these finish, vold starts (though I haven't found the invocation). One of these scripts is 05mountsd. It waits up to 6 seconds for /dev/block/mmcblk0 to appear. It goes through the partitions on this device, and the first one of type 83 (any Linux filesystem) is checked and mounted, if there is one.

The fly in the ointment is that vold creates the inodes for both /dev/block/mmcblk0* etc and /dev/block/vold/179:[0-9] etc (the same referents). Thus 05mountsd waits however long, exits having done nothing, and vold then creates the devices.

I did these hacks:

Once I have CyanogenMod-7.0 running on my new pocket computer, I should verify that the same issues are present, and should send this in as a bug report.

Restoring User Data from Cupcake

Now that I have CyanogenMod stabilized, I want to restore user data from Cupcake, wherever the format is compatible or reasonably hackable. In particular, I want my contacts back. Here is a list of saved databases. On Android, an extension of .db always cues a SQLite database.

alarms.db

I re-created these by hand; no need to restore.

apns-conf.xml

assets.db

browser.db

com.android.settings_preferences.xml

contacts.db

In Cupcake the format is about as distant from vCards as you can get: separate tables for many of the aspects such as the phone numbers, street addresses, and photos. In NCCN (Froyo) the database is found in /data/data/com.android.providers.contacts/databases/contacts2.db which has a different but equally messy format. It won't be possible to just copy the contacts, even by a minorly clever Perl script.

The correct solution is cloud sync, but my Sogo server was not ready to take the Cupcake information.

The job will be done most promptly by stupid re-entry by hand, using the database dump as input.

mmssms.db

user_dict.db

Backup Script

On Cupcake I had a script that copied all relevant stuff to /sdcard/BackupRoot, and a cockamamie procedure would transfer it (by sneakernet) to the backup server. On my normal Linux boxes I have a script called backup-host which I execute from the central site by SSH, and I hope to make that work equivalently on the Android machine.

Untoward Events

No Home - No Receive Syndrome

After I installed CyanogenMod-6.1, I got these very annoying symptoms:

Forum postings suggest that this is a syndrome, and a fix for one element fixes the others too.

Cause and Solution: I was using the non-tiny version of Google Apps. When I did a factory reset, re-flashed CyanogenMod-6, and used gapps-mdpi-tiny-20101020-signed.zip (the correct Google Apps for HTC Dream), it ran the setup wizard as it should, the Home button goes to the home screen, I can receive incoming calls, and the UI has stopped crashing. The following material is kept for historical interest.

In a Google search for cyanogenmod home button not working, this forum post (OP jsam69, 2010-09-01) indicates that re-flashing the Cyanogenmod image will fix it. Works for some people, not for others. Another item in the same thread from (anonymous) says to run the terminal emulator, type bin and press the home button. Doesn't say what's supposed to happen then.

In another thread (2011-06-13), the problem was caused by restoring apps using Titanium Backup and indiscriminately restoring the data for everything. They haven't found out which app's data is poisonous but it appears to be a system app.

At the end of this thread, the person says he went into recovery and fixed permissions and this helped. The Amon_Ra Recovery has an action item to fix permissions; basically this sets a bunch of directories to mode 755, where some of them formerly were not world readable. (Didn't help me.)

OP WaVeS (dated 2010-04-28) says he has lost his home button. One respondent says he had this symptom repeatedly (on a G1). Boot into recovery (power off, hold down home and power on). Do a full wipe, then reboot. Fixed the problem every time. (What you're really after is to wipe cache and data.) It didn't help me.

OP says if he blows cold air onto the home button, it would work. Do a factory reset and reflash the image. It works again. Another procedure: Install AnyCut from the market. Execute it. Run Setup Wizard. Go through the procedure. This should fix your dial (green), home, hangup (red) keys gone wacko.

JoshuaE says (paraphrasing): Make /data/data/com.android.providers.settings/databases/settings.db available on a real machine. Use sqlite to insert a record with name=device_provisioned value=1. Copy it back to the original location. Reboot.

My experience: Booted into Recovery, wiped cache, Dalvik cache, and Factory reset/data (which I think includes the 2 caches). The Home key worked in Recovery (other people's symptoms suggest corrosion in the Home key). This wiped my bookmarks (and other settings) but didn't cure the Home key problem.

More drastic: Boot into recovery. Wipe data factory reset. (Also wipes the cache.) Flash CyanogenMod. Flash Google Apps (MDPI). Reboot. Outcome: it did not run the setup wizard. This doesn't seem to be on the phone. The home key is still inoperative.

Got into Koushik's ROM Manager and hit Fix Permissions. (And rebooted afterward.) It changed 64 files and dirs always to mode 755; some had more restrictive permissions before. Home key is still inoperative.

Comparing JoshuaE's fix and actual phone behavior, the problem is that the setup wizard is not being run, and the software is insufficiently idiotproof against missing values. Another syndrome element is that the market and map apps do not appear in the launcher page. Nonetheless, all of these are present in the Google Apps (MDPI) image (gapps-mdpi-20110501-signed.zip). I had an idea to obtain the APK's and install them separately, but this is obviously bogus.

More Plans to Recover

More on the home key - receive calls syndrome. Doing the right thing, i.e. wipe and reflash, was ineffective on repeated trials. I'm going to try several lines of attack. Of course it's perverse to stick with the G1 when the current OS version won't fit on it, and it's definitely on the agenda to get a new phone, but it's also not prudent, and it's a waste of a good learning experience, to accelerate that step. So, here's what I'm going to try in the short term.

Miscellaneous Tidbits

Top 10 phone models

As of 2011-08-25 by count of CyanogenMod installations (in thousands):

Model Installs Comment
bravo 67.8 Mid level phone by Motorola for AT&T.
umss_jordan 20.9
morrison 26.2
supersonic 26.1
passion 22.6
blade 20.5 Low-end phone by ZTC
sholes 17.5 The OG (original) Droid
buzz 16.9
galaxysmtd 15.8
encore 14.9
Other 109.7
Total 358.9 (All official CyanogenMod installs)

 

Items detected in logcat output:

This was for diagnosing the home key syndrome, but it provided a number of useful tidbits:

How to install Debian on CyanogenMod

You need an ext[34] partition. There's a chroot program called "Run Linux", and you can bind mount. Get the picture?

List of 30 indispensible apps (francophone)

Several that I already picked are on this list, and several on the list look like I will want them.

Android Bootscript System

Link to a writeup about the Android bootscript system.

Boot Init Scripts

At some early point, something kicks off the scripts in /system/etc/init.d (probably using runparts):

00banner

Shows a banner at the start of the logcat information; it spells out cyanogenmod in unreadable ASCII art.

01sysctl

Sets kernel parms from /etc/sysctl.conf, which exists but is empty.

02audio_profile

Sets audio parameters, picking the init file according to the hardware model.

03firstboot

If existing, execs then removes /data/firstboot.sh

04modules

Loads (modprobe) modules listed in the script, currently just ipv6.

05mountsd

Identifies the first partition of type 83 on the SD card (if any), runs e2fsck, and mounts it on /sd-ext . But it did not find /dev/block/mmcblk0 .

20userinit

Executes each of these scripts, if existing:

  • /sd-ext/userinit.sh
  • /sd-ext/userinit.d/* (using run-parts)
  • /data/local/userinit.sh

Open Issues

Formerly Open Issues Now Finished