Asus Transformer Pad Infinity
Upgrading to CM-11.0-M12

Now I'm upgrading the operating system to CyanogenMod CM-11.0-M12 based on Android 4.4 KitKat, from 10.2.1 based on Android-4.3 Hyper Jelly Bean. A big advantage of CyanogenMod is that you get updates like this, which wouldn't happen with a carrier image. However, the CyanogenMod team have changed their naming system, confusing a lot of people; the M-series (monthly) releases are a bit less authoritative than the former stable releases, more like release candidates, but they are a lot more authoritative than the nightly builds and former snapshots. Thus I unnecessarily delayed quite a long time before updating 10.2.1 to 11.0-M12.

It takes some work to discover the eentsy version on which a particular CM version is based. According to a post on Phandroid dated 2014-07-08, images starting with M8 are based on Android-4.4.4. This is significant due to a bugfix in the API that VPNs use to set the DNS server.

My plan for CyanogenMod-12.x based on Android-5.0 Lollipop is to wait for the second M-series release and install that; then to upgrade monthly in the M-series.

I'm upgrading both the Asus TF700T (Mica) and the Galaxy S III (Selen) in parallel. Since this upgrade is between major versions (10 to 11), I will need to wipe the cache and data, and so I will need to rely on backups to get my app information back -- selectively.

Procedure for Upgrade

• Review the procedure for initially installing CyanogenMod and the upgrade notes for the previous upgrade to 10.2.1.

• I made a directory to store backup and image files (on Xena): /scr/android-upgr/asus-xfmr-tf700t/cm-11.0-M12 and /scr/android-upgr/sam-galaxy-s3.121022/cm-11.0-M12 ; with symlinks from ~/upgrade/{mica,selen}/files . These should probably be tossed about a month after the upgrade, i.e. 2015-02-03.

• I made backups in TAR format.
  ssh mica "cd /data ; tar czf - app data local" > mica-bku-20150103.tgz

  Oopsie, tar: write error: Broken pipe. gunzip -t mica-bku-20150103.tgz reports unexpected end of file. To recover (successfully):

  ssh mica "cd /data ; tar cf - app data local" | gzip -c > mica-bku-20150103.tgz

  Speed on Mica: about 8.5e5 bytes/sec (compressed), total size 448Mb (compressed), 740Mb (uncompressed). 60% of the material is APK files which are already compressed. Took about 8 minutes.

  Speed on Selen: about 1.02e6 bytes/sec (uncompressed). Total size 884Mb (uncompressed), took about 14.5 mins,

• ADB did not migrate to my new laptop so I needed to download it again.

  • CyanogenMod wiki page about the SDK.
  • Google's page for downloading Android Studio (the latest IDE) and the SDK tools.
  • Current SDK as of 2015-01-03, 140Mb. This link is to a page where you agree to the license terms; then you get the software. Today the server is delivering 36kb/sec (arthritic snail).
  • The resulting TGZ file's root directory is ./android-sdk-linux. cd there and execute ./tools/android . It gives you a list of packages to install (18 of them). Deselect all, and re-select Android SDK Platform-tools. Hit Install. Agree to license. Download server is twice as fast, oooo! But the package is a lot smaller. To exit, close the GUI window from your window manager.
  • The binaries are in /home/android-sdk-linux/platform-tools/adb and fastboot. Make a symlink to them from some directory on the path, such as ~jimc/bin/ .
  • If you're going to develop apps, you will need many of the recommended 18 packages, plus Android Studio is recommended as an IDE.

• I made a redundant backup using ADB. Connect the USB cable.

  adb devices
  It gives the tablet's serial number plus unauthorized. On the tablet there is a pop-up. Tell it always authorize, then click on OK. Re-do adb devices and it will give the serial number plus device.

  adb backup -f ./mica-bku-20150103.ab -apk -obb -all -system
  These command line arguments select both app data and APK files. You do not want the -shared option which dumps the SD card also. The tablet goes through several security steps:

  • When you do adb backup … it runs an app on the tablet which asks for a password and requests confirmation.
  • On Mica I gave it the standard commercial low security password, probably a bad choice. On Selen I left the password blank.
  • It reports the apps it is backing up, one at a time.
  • On Mica, file size 510Mb, took about 6 minutes, similar to making the TGZ file. On Selen, size is 329Mb, took also about 6 mins. Last time Selen was twice as fast.
  • On the destination I changed the mode of all the backup files to 600 owned by jimc.

• The CyanogenMod instructions suggest backing up your SD card in the remote chance that it gets trashed. I didn't do that for intra-version upgrades, but this time I'm going to be paranoid and do it. Speed: 2.5Mb/sec (compressed). Took about 20 mins, 1.28Gb (compressed), 1.09Mb/sec (compressed), 1.50Gb (uncompressed).

• Downloaded Google Apps for Android 4.4 KitKat; you need the version that goes with the upgraded OS version. See http://goo.im/gapps for a link. I got https://goo.im/gapps/gapps-kk-20140606-signed.zip , 157Mb; check the page to see if there is a newer one. This link is to an info page from which you actually download the file (Javascript only). This server is pretty fast today.

  And I copied it to mica:/sdcard/. The same file serves on both devices.

• Oh, crap, when I did the update, ClockworkMod Recovery reported get_metadata_recursive: some changes failed, installation aborted. Answer: upgrade ClockworkMod Recovery. Evan Caroll (2014-02-08) says you need at least v6.0.4.5. In KitKat there is a change to the API that updater-script needs to use to set permissions.

  On Koush's website I downloaded (to the laptop) the image for my machine, recovery-clockwork-touch-6.0.4.7-tf700t.img (exact match required for the hardware) (6.8Mb).

  Procedure to get this image onto the mobile device:

  • Make sure you have the fastboot binary available. Connect your USB cable. If (unlike me) you are still running on the old Android version, reboot into fastboot. You can do adb reboot bootloader, or (if you have turned on advanced rebooting in developer options) you can hold down the power button, pick Reboot, then pick Download. (These labels are for CM-10.2.1.)

    For Selen, adb reboot bootloader gets you into Odin Mode. You need to first install Heimdall for Linux, or use Odin on Windows, see the S III installation page for the gory details of using Odin.

    • It's looking for a TGZ file; Koush's img file is not the right format.

    • http://wiki.cyanogenmod.org/w/Install_CM_for_d2vzw

    • Tidbit: Starting with CM-11, d2att is deprecated in favor of d2lte, a unified build that should work on all Galaxy S III devices. However, the wiki page for d2vzw (recently updated) has no mention of d2lte.

    • There is a lurid warning that you must be a 4.2.2 or older build; if you try it on a newer version it will probably brick your device. If you have already upgraded, a qfuse has been tripped and any attempt to downgrade will also result in a brick.

    • News article about the Samsung Galaxy S4 bootloader. QFuse refers to a feature of the Qualcomm APQ8064T chip which cryptographically verifies each step in the boot sequence.

    • I'm pretty sure that the reference to 4.2.2 refers to the stock Android version with its accompanying bootloader and radio code. It's not clear what the consequence may be if you already have CM on the phone. In any case I have the mentioned back-version booter already.

    • Tidbit: Tips on how to remember strong passwords.

  • The Odin/Heimdall method to get ClockworkMod Recovery onto the phone may have been fine two years ago, but it almost certainly is the wrong method for today's problem. Instead, I'm going to try Koush's Rom Manager, which runs on the live system. Obtain from the Android Market (Play Store). Piece of cake, except I got the non-touch version because the touch version requires an in-app purchase and I have not yet set up my Google payment method, which takes work given my security rules.

    Boot into Recovery and check the version, which Rom Manager does not announce. Yes it is v6.0.4.5.

  • If you have already trashed your Android image, power off the machine by holding down the power button for 10 secs, until it vibrates, and release the button. Then hold down Volume Down then hold down Power.

  • When the booter screen appears, release the keys, then press Volume Down enough times (for me, twice) to highlight the fastboot row or the USB icon: the third item. Press Volume Up to select it. You have 10 seconds to accomplish this. The bootloader screen stays visible but small writing in the upper left corner says Starting fastboot USB download protocol.

  • On the support computer do fastboot devices; it should show your device ID in hex.

  • Now do:

    fastboot flash recovery recovery-clockwork-touch-6.0.4.7-tf700t.img

    It tells what it is doing and reports OKAY for success. The whole process takes about 10 seconds. There is also a progress bar on the device (which doesn't seem to move).

  • Now reboot into recovery. The procedure is similar to booting fastboot: Power off the machine by holding down the power button for 10 secs, until it vibrates, and release the button. Then hold down Volume Down then hold down Power. When the screen appears press Volume Up to select Recovery. Now you can continue with the CyanogenMod installation.

  • Alternatively if you haven't trashed your old Android image, you can do (on the laptop) fastboot reboot and proceed with the following section.

• In the past I've downloaded the CyanogenMod image, saved and checked it, and then installed it. However, this is kind of anally retentive. This time I'm going to do the update directly, using the ChanogenMod installer. This time I'm upgrading to a new major version, so I will have to wipe the cache and data.

  It would be a really good idea to upgrade to the latest version of ClockworkMod Recovery before trashing your old Android image. The previous section tells how to do it.

  Exit from shell sessions. Stop Dropbear. Stop any other running apps. Unplug USB.

  Go to Settings - About Tablet - CyanogenMod Updates. This is going to be cm-11-20141112-SNAPSHOT-M12-tf700t.zip (for Mica) and cm-11-20140210-SNAPSHOT-M3-d2vzw.zip (for Selen). Note the considerably older date on Selen's version. Click on the download symbol. Watch the progress bar. Server is very slow today.

  When you resolve download.cyanogenmod.org it gives you a list of mirrors in a random order, and the first on the list will be used. If that one is slow, maybe another won't be. I killed the download, and retried on the desktop machine. Took about 2.8 mins, size 218Mb (compressed), speed about 1.35Mbyte/sec. I copied it to mica:/sdcard/cmupdater/cm-11-20141112-SNAPSHOT-M12.zip .

  Now this is embarrassing: for Selen (d2vzw) the download page only lists up to cm-11-20140210-SNAPSHOT-M3-d2vzw.zip, but the settings/updates page has snapshots up to cm-11-20141112-SNAPSHOT-M12-d2lte.zip . Starting with M4, all the Galaxy S III variants have been combined into one build variant, d2lte. Here is some key information about d2lte:

  • Installation page. Remember that d2vzw has a locked bootloader, so if you are moving from the stock ROM to CyanogenMod, you need to follow the instructions on the specific d2vzw page in order to get ClockworkMod Recovery onto the device, then come back here to install CyanogenMod.

  • Device Info Page. It has links to the download and installation pages, a summary of device characteristics, and of special interest, the key combinations for getting into Recovery or Download mode.

  • Known Issues page.

    • GPS never finds satellites (?), people say it's fixed in at least M11 or M12. jimc says: it works great for me on Selen. But there is a problem on Mica, see below.
    • GPS heading is slightly off, your path is crabwise. Affects Samsung stock image as well.
    • Can only record video in 720p, not 1080p.
    • Audio clipping during video recording happens too frequently.
    • Google Wallet Tap & Pay does not work, no open source Host Card Emulation library available. However, the KitKat hype page claims that HCE is in KitKat, so one wonders if their library is closed source and not licensed for use with CyanogenMod.
    • After the upgrade, cellular data may be inoperative and you can't alter APNs. To fix, upgrade the modem (radio software) (gives link). jimc says: I have I535 VRBMF1 and cellular data works fine with it (Selen only).
    • Can't enable hostapd (workaround given). jimc says: Works for me. But it doesn't pass IPv6.
    • Some apps like Google+ generate duplicate entries in the contact list.
    • When on speaker phone, unable to change the volume.
    • For the Java interpreter use Dalvik, not ART (Android Run Time). It will put you in a boot loop. There's a link to a recovery method.

  If the installer had downloaded, it would offer to install the image; tell it Apply Update. In my case it shows a door icon. Click on it and confirm updating.

  On Selen it goes directly into ClockworkMod Recovery, installs the image (with rotating icosahedron and progress bar), and reboots, taking longer than usual for each reboot step. Recompiling 148 apps this time. Remember that you need to wipe cache and data. This is not automatic. It will complain repeatedly that various apps, particularly Google Play Services, cannot authenticate. Blow off the warnings and reboot into Recovery. Wipe Cache and Data, then install the new Google Apps.

  From Recovery, reboot the machine. It boots into the setup wizard. Follow along in Setting Up CyanogenMod for this and subsequent steps.

  On Mica with CM-10.2.1 it wasn't as simple as expected, and with CM-11-M12 it was worse; see the previous section for how to upgrade ClockworkMod Recovery, which should have been done first. For historical interest, here is the actual procedure I followed.

  • Recovery says, failed to open /sdcard/0/cmupdater/cm-11-20141112-SNAPSHOT-M12.zip, signature verification failed, install untrusted package? Tell it no, whereupon it shows a picture of a dead android. Then:
  • Mounts and Storage - Mount /emmc (hit back button)
  • Install from sdcard - choose zip from internal sdcard - This is actually /emmc. Follow the sequence of containing directories in /emmc/0/cmupdater/ and click on cm-11-20141112-SNAPSHOT-M12.zip .
  • Confirm Install - Yes. Now the Android is standing up, but its icosahedron does not rotate on the TF700t. The progress bar moves at an uneven rate; don't panic.
  • get_metadata_recursive: some changes failed, installation aborted. See the previous section for how to upgrade ClockworkMod Recovery, which should have been done first.
  • If your ClockworkMod Recovery is up to date, it announces, installation complete. Repeat the previous steps to install Google Apps. This is gapps-kk-20140606-signed.zip .
  • Hit Go Back. Since this upgrade is to a new major version, Wipe Data and Cache. When that's done, Reboot Sytem Now.
  • All the reboot steps take longer than usual. See the previous notes for timings. This time I have 145 APK files to recompile (vs. 128).
  • If the upgrade were within a major version, we would be back at the lock (authentication) screen. Job finished. However, since this is a major version upgrade and you [should have] wiped data, it boots into the setup wizard. Follow along in Setting Up CyanogenMod for this and subsequent steps.

  The kernel on TF700t (Mica) is 3.1.10 (same as for CM-10.2.1). On d2vzw (Selen, Galaxy S3) it is 3.4.66.

• Super-quick checkout: Kitchen Timer starts and can play its alarm (both machines).

Restoring Apps

See my procedure for Droid-3 to restore apps. It includes two scripts that will make life a lot simpler and less prone to errors. As with the Droid-3, I am going to not overwrite apps already on the device, and will restore the data for only some of the apps.

There are 64 apk files that were saved from the old version, and there are 3 apk files installed during initial setup. The only one in common between the sets is com.android.vending-?.apk (where ?=1 on Mica and ?=2 in the backup). The following apps will not be installed from the backup. Dups means that this app appears to have a similarly named counterpart in /system/app that came with CyanogenMod or Google Apps. 51 apps remain.

• com.android.vending-2.apk -- Rely on the normal update procedure.
• com.google.android.apps.genie.geniewidget-2.apk -- dups GenieWidget.apk
• com.google.android.apps.maps-2.apk -- dups Maps.apk
• com.google.android.apps.plus-1.apk -- dups PlusOne.apk
• com.google.android.gm-2.apk -- dups Gmail2.apk
• com.google.android.gms-2.apk -- dups Gmail2.apk (maybe)
• com.google.android.googlequicksearchbox-1.apk -- Got to be duplicating some instrinsic function
• com.google.android.street-1.apk -- dups Street.apk
• com.google.android.talk-2.apk -- dups Hangouts.apk
• com.google.android.tts-1.apk -- probably dups VoicePlus.apk
• it.nicola_amatucci.android.certificate_installer-1.apk -- probably dups CertInstaller.apk
• jackpal.androidterm-2.apk -- pretty sure this is included in CM as Term.apk
• me.shkschneider.dropbearserver-1.apk -- replaced by web.oss.sshsftpDaemon-1.apk from same author. The new one is titled Rooted SSH/SFTP Daemon.
• org.deadbeef.android-2.apk -- I'm using Apollo instead
• org.deadbeef.android.freeplugins-1.apk -- Ditto
• com.googlecode.andoku-1.apk -- (Selen) Need to set up account to pay for the ad-free version.
• org.thetomahawk.spreadsheet-2.apk -- (Selen) No longer using this

The upload speed to Mica was about 1.3Mb/sec; installation is about half the upload time. adbd on the device wants to check on Google for harmful apps; I allowed this. adbd does not put icons on your desktop like Google Play does. No errors; total time: 12.5 mins.

For restoring data I excluded a lot more directories. 150 were backed up; 100 were excluded; 50 remained. I tossed everything named com.android.* com.cyanogenmod.* and com.google.android.*, and all directories which did not have a corresponding app installed (in /data/app, with very few exceptions like Apollo). These would be from apps that I uninstalled previously.

Elapsed time on Mica was 4 mins and similarly on Selen. These directories had issues:

• com.amazon.kindle -- several font directories were missing.
• org.dmfs.caldav.lib -- missing directory in /data/data, is this app installed?
• org.mozilla.firefox -- several internal directories were missing.
• org.mythtv -- Obviously a filename containing spaces.

Finally, it's probably important to update every app that needs it, to get the latest version that supports KitKat. Certainly an up to date Google Play Services is required for almost every app (not including Kitchen Timer). It wants to update 23 apps on Mica, 16 on Selen. Server's speed resembles an arthritic snail. The complete update took 2.5 hours on Mica.

Very annoying new feature in Google Apps for KitKat: Google News and Weather claims it wants to see your data on Google which requires logging in. Hitting cancel in the notification and dialog will not shut it up. It turns out that it wants to use your existing Google account record (with password) to authenticate on Google to retrieve news and weather. Eventually I tried out the news reader, and finally gave permission. It connected successfully using the saved password and made no more annoying requests.

If I had really wanted to suppress this, I had a lot of trouble to figure out how. Cancel is not sufficient to shut up the requests. Start the Google Settings app. Click on Connected Apps. Not very helpful, it won't let you revoke access from your mobile device. Help says, connect to https://www.google.com/a/example.com/ManageAccount (replace example.com with your domain name, whatever that is). Under Connected Apps and Services, click Account Permissions. Revoke access for the unwanted service.

Application Checkout

Legend: NT = not tested; NI = not installed; WT = wimpy test. This is a very quick checkout; see the next section for further work that needed to be done.

Application Cleanup

These packages need extra work to be fully functional.

• Firefox -- needs careful test with X.509 certs loaded. Yes, it is functioning normally with network and local content. It can load CA and user certs. It can recognize TLS sites certified by those certs. It can present the user cert for admission to web pages requiring it. It is successfully syncing using the deprecated protocol. (Can owncloud handle the latest protocol?)

• Phone -- Selen can originate and receive calls. Voicemail is set up and works. (Mica does not have a cellular modem.)

• *GPS Status -- Mica is taking an awfully long time to get satellites. I tried cache reset, and power off/reboot. Helped a little. After a long time it got 7 to 8 satellites. But it seems to have excessive error like 300 meters, whereupon it immediately starts a new scan. I tried this procedure several times; it does have recent aGPS data and had plenty of time to download the almanac (takes 13.5 mins) if it needed to. The finger of blame points to Mica's hardware, not the app. This may or may not be related to the note that d2lte has trouble finding satellites. But Selen (which is a d2lte) gets a fix promptly, 3 secs to 10 meter accuracy (with recent aGPS, ephemeris and almanac).

• People --

  • Crop the puma photo and add to both profiles [done].
  • Buy the pro version of CalDAV Sync by Marten Gajda [done].
  • I got everything set up and did a test in which Jacinth, Mica, and Selen each created a contact (sync both devices immediately after each one). All were seen. Then each device deleted the test contact created by another device (sync immediately after each one). All were gone.

• Calendar -- I bought the pro version of iCal Import/Export by Lukas Aichbauer. When adding your account (after a data wipe), create the account in the app, not from the New Account line item in settings. I did a similar test as with People, and it worked as desired except that the app reported one sync error from Mica to Jacinth and Mica's event ended up with the wrong date. Mess cleaned up on Jacinth.

• Bluetooth -- Needs hands on with headphone. Paired Mica and Selen to each other and to the headphone. I've been using it on Selen daily.

• Amazon Kindle -- Stops, probably due to missing fonts noted during transferring the data. Needs repairs. I reinstalled it (on Mica having cleared data), and now it works. I got the same successful result on Selen just by clearing data; reinstallation was not necessary. But of course you need to let it store your Amazon password again.

• WiFi Hotspot (hostapd) -- Tested on Selen. Xena and Mica can connect. The first time on Xena I had a fight with WirelessManager and it ended up with no default route. The second time it was successful. But no IPv6 through Android's hostapd. On Selen's setup there's a list of connected devices, and Xena did not appear, but Mica did. Mysterious.

  Hostapd on Selen does not give the client an IPv6 address. Hmm, it could use its own prefix and EUI-64, but modify the last octet for the client. But if Selen sent router advertisements upstream, would Verizon's cell tower honor them? Would Verizon demand that I pay extra to do routing? Better to use IPSec to Jacinth.

• Dropbear -- This package has been renamed to Rooted SSH/SFTP Daemon (with variants e.g. on the icon label), by OSS Mobile Apps. The help text closely resembles that with me.shkschneider.dropbearserver-1.apk (the old, discontinued version), so likely it's the same author renamed. I needed to copy the saved host key and authorized_keys files onto Selen, and to create a new host key for Mica. I used AndFTP in SFTP mode. Other than that, there was no problem getting it to work. One annoyance: when you shut off the server it gets ANR Unfortunately, SSH Server has stopped. Bug report was sent.

• *AndroidVNC (Mica only) -- Works but arrow keys and tab are not sent over.

• Skype -- Success (Mica and Selen) to Skype Test Call Service. Problems encountered along the way:

  • Microsoft account was blocked, probably for inactivity. Need to change PW.
  • Password is limited to 16 bytes, hiss, boo.
  • On Mica it was very sluggish and finally gelled, no UI (window manager) response. I had to power cycle it. Trying again: success.
  • During the above, I tested from Selen and it wouldn't let me log in from two hosts at once.
  • I participated in the MCG weekly meeting over Skype from Mica. It was not totally successful: lots of compression artifacts and the audio was flaky. They could hear me fine. I blame Skype on their iPad.

• *StrongSwan and OpenVPN are complicated and have their own VPN test page.

So this upgrade appears to be successful.