I rebuilt a phone keyboard from the only copy that survived, then moved its speech onto the phone
The source for my Android keyboard was lost when a machine was reformatted. The only thing left was the app installed on my phone. Before I could change a single key, I had to get the source back out of that file and show that what I rebuilt behaved like what people already had. These notes cover how that worked, how the keyboard then learned to recognise speech without sending it anywhere, and the mistakes that taught me the most along the way.
- The original project was lost, so the source was recovered from the released APK with two decompilers and scripts written for the job.
- A parity check names every original name missing from the rebuild, and each one now has a written reason.
- Speech is recognised on the phone by a model the person downloads, running in its own process so the keyboard survives when Android reclaims memory.
- Typos are scored by how far the wrong key physically sits from the right one, over a list of about 190,000 words in frequency order.
- One command runs 21 checks, including 1,725 JVM tests, and a hand-written record lists which builds have actually run on a phone.
01Why I built it
A keyboard sees everything a person writes. Every message, every search, every half-finished sentence goes through it. The usual arrangement is that the typing becomes the product: it is sent somewhere, analysed, and returned as suggestions. I wanted a keyboard that worked the other way round, where the speech model, the dictionary and whatever the keyboard learns about you all stay on the phone.
It began as a simpler thing. I prefer speaking to typing, and the first version recorded my voice and sent it to a speech server I ran myself. That worked well enough that I used it every day. Then the machine holding the Android project was reformatted, and the project went with it. The app on my phone was the only copy left.
So the work had two halves. First, recover the source from the installed app and prove it matched. Second, make the promises real: dictation on the phone, every store visible and clearable, and checks that fail the build when a promise is broken.
02Architecture at a glance
The app is one Android input method with a handful of activities around it. Most of the logic that decides what a key does lives in plain Kotlin classes with no Android in them, which is what makes it testable on an ordinary computer.
The version number is the git commit count. It is automatic, it only goes up, and it ties every build a person installs to exactly one commit.
03Getting the source back
The one piece of luck was that the app had been built without minification, the step that shortens names to shrink an app. Class names, method names and even parameter names survived inside it. That turned recovery from guesswork into translation.
Two standard tools did the first pass. jadx turns compiled Android code back into Java, and apktool turns the packed resources back into readable XML. Neither gives you something that builds. An APK carries every resource from every library it uses, and the decompiled code does not compile.
For the resources, I wrote a script that starts from what the app's own code refers to and follows references outward, stopping at library boundaries. The subtle part was names looked up as strings at runtime, which a plain search for resource references misses. Out of 1,080 decoded files, 151 were the app's own.
For the code, the fix was to stop fighting Java. Kotlin compiles a call with default arguments into a hidden helper method that the Java compiler cannot see, and the app had 68 such call sites. There is no compiler flag for that. Converting the code to Kotlin made the whole class of error disappear, because default arguments are simply legal again.
04Proving the rebuild matched
A rebuild that compiles is not the same as a rebuild that behaves. Decompilers get meaning wrong in places, and I found four such errors by reading the low-level bytecode listing, including two conditions in the grammar rules that had been inverted.
Two scripts close the loop. A parity script lists every name from the original that is missing from the rebuild. Each of the 19 now has a written verdict: removed on purpose, renamed, or folded into something else. Before those verdicts were written down, an audit spent an afternoon tracing one missing name through the bytecode, only to find it had been dropped deliberately and recorded in a code comment. A second script compares a rebuilt APK's classes, resources and components against the original.
Passing every check still did not mean the app worked. Two builds passed everything and then would not start at all, because the manifest still named the classes at their old package. Since then a hand-written record lists the builds a person has actually installed and used, and the public download page only offers builds from that list.
05Typing that forgives a thumb
Correction starts from a list of about 190,000 words ordered by how often each is used. The list is held in a trie, a tree of shared prefixes, so a misspelling is matched in one walk instead of by trying every possible edit. The original app ranked words by length, which let an obscure word outrank a common one. Rebuilding the list in frequency order fixed that.
A typo is scored by how far the wrong key physically sits from the right one. Hitting a neighbouring key is cheap; a random wrong letter is expensive. The keyboard also learns where one person's taps land on each key and moves the key's edges toward them, while the middle of every key always stays its own.
At a full stop, a sentence check reads each word against the words on either side. That catches pairs like their and there, which spelling alone cannot tell apart. Fixes are offered on a check key and never applied by themselves.
06The missing space
In portrait, a thumb aiming for a letter near the bottom row catches the spacebar, or a space lands where a letter should be. The result is not a misspelled word. It is two real words run together, or one word cut in half, and ordinary autocorrect looks at one word at a time, so it never sees it.
I measured which keys overlap the spacebar from the layout itself before building anything. Then I built a version that repaired these automatically and ran it over 18,000 generated cases. It turned correct words like runbook into two. No rule I could write separated those from the cases it was meant to fix.
So the repair is offered, not applied. A suggestion nobody taps costs nothing, and a correct word quietly split in two costs the person their sentence. That conclusion came from the measurement, not from caution.
07Speech on the phone
The first versions uploaded each recording to a speech server and typed in what came back. Moving recognition onto the phone meant solving three problems: the model is large, the phone is short of memory, and the keyboard must never be the thing that dies.
Models are not bundled with the app. A screen lists five English models, from about 120 MB to 630 MB, and each is downloaded once, checked against its hashes, and unpacked whole or not at all. A cut-off download keeps its parts, so it can resume.
The engine runs in its own Android process. A loaded model takes hundreds of megabytes, and under memory pressure Android reclaims the keyboard's process first. With the two apart, losing the engine costs a reload, never the keyboard. The model starts loading the moment a recording starts, so it is usually ready by the time the person stops speaking.
Where the engine runs was measured rather than chosen. A trial transcribes the person's newest recording on the processor and on two accelerator routes, twice each, and keeps the fastest route that still produces the processor's words. On the two phones it was tried on, the plain processor won. Since version 1.0.362 the phone has been the only engine, and the server route is gone.
08Privacy as a check, not a document
A privacy policy is a promise, and promises drift from code. So the promises are enforced by a script that runs with every other check. It fails if a new network destination appears in the source, if a secret is stored unencrypted, or if cloud backup is turned back on. Each of those has been wrong once in this codebase.
Every store that holds text is sealed with a key kept in the Android Keystore. A write that cannot be sealed is dropped rather than saved as plain text. A store that will not open is never overwritten with an empty one; after three failed starts it is set aside, kept rather than deleted, and the person is told which record it was. Password fields are excluded from everything that stores or sends.
Anything the keyboard works out about your typing is shown back to you, with what it was measured over. The keyboard counts its own slips by kind, such as a missed space or a tap given to the neighbouring key, and shows those counts, never the letters or words behind them.
09Testing a keyboard without a phone
For a long time the main keyboard class was too big to change safely, because nothing could verify a change to it short of typing on a phone and hoping. That changed when I found that the real service, the real layout and real touches could all run on the JVM under Robolectric.
Getting there meant four traps, each of which looked like a broken keyboard. Keys ignore programmatic clicks, so typing has to be a touch. Key positions read as zero until the view is attached to a window. Space, shift and backspace fire on touch-down, not on click. And the text written through a bare input connection lives in the connection's own buffer, so it has to be read back through it.
The first run found a real defect. Autocorrect never checked what kind of field it was writing into, so a mistyped password was silently corrected behind the dots, and the only symptom was being told the password was wrong. Five tests now fail if that guard is removed.
- 21 checks in one command. The build, the unit tests, the privacy audit, keyboard fit on screen, and offline checks of correction, missing-space repair, prediction, swipe and touch targets.
- 1,725 JVM tests. Some drive the real keyboard through simulated touches, because a subtle break in the input path costs somebody their sentence.
- A record of what ran on a phone. Written by a person after the fact. The download page reads it, and builds known not to start are listed so they are never offered again.
10Where it stands
The keyboard is in daily use. It types in QWERTY, Dvorak and one-handed layouts, takes dictation on the phone with a downloaded model, offers optional swipe typing, and keeps every store it holds visible and clearable. A public site is generated from the code and publishes only builds recorded as run on a phone.
Not everything stayed. I built four small language models into the sentence check so it could read for sense, and took them out again when their corrections proved inconsistent in use. The check now reads with word pairs alone, which is less ambitious and more predictable.
What remains is mostly measurement: how the corrections behave under more thumbs than mine, and how the speech models perform on phones I do not own.
11Nine lessons
The project keeps a written list of 45 mistakes, each with what now stops it from happening again. These are the ones that changed how I build things, not just this keyboard.
01Words typed: 0
The method that learns a word after you finish typing it had never once run from ordinary typing. It asked for the word the cursor was inside, and after a space the cursor is between words, so the answer was always empty. Learned words, next-word pairs and personal dictionary promotions from typing: none of it, for the life of the app.
Nothing looked broken. Suggestions still appeared, because they came from the shipped dictionary. It surfaced only because a statistics card on the settings screen said Words typed: 0 after real use.
Now: Each session records how many words were learned, and the figure is shown on screen where a person can disbelieve it.
a feature whose failure looks like success needs a number a human can read.
02Three switches that controlled nothing
Three settings were written correctly, survived restarts, and were never read by the keyboard. A later one was read into a field and then never consulted, and its label said a feature ran on the phone when it did not.
Now: A check asserts that every key the settings screen writes is read by the keyboard. Whether reading it does anything is still a person reading the code.
a control has four links to its effect, and stopping at any one of them looks finished.
03Every check green, and the app would not start
Two builds passed every check, were installed, and would not start at all. The code had been reorganised into packages, and the manifest still named every component at its old location, so Android could not create any of them.
Now: A hand-written record lists builds a person has run. The download page offers only those, and the two broken builds are listed so they are never offered again.
a test suite proves the code agrees with itself, not that it runs.
04A rule fitted to my own examples cost five real ones
The missing-space repair had a rule added to stop one word in a test corpus I had written myself from being split. Within a day of shipping, six missed repairs were reported from a phone, and five of them had been refused by that rule. A short word beside a longer one is the commonest shape in English, and the rule refused all of them.
Now: Holding an underlined word offers to report a missed fix, a script says which rule refused it, and the six reported cases are pinned in the tests.
a guard written for a case you invented generalises into a claim about the language that is simply false.
05Learning on the first sighting learned the typo
The keyboard adds every word you finish to your personal dictionary. That is how it learns vocabulary no dictionary has. It also meant a mistyped word was adopted the instant the space landed, which switched off the repair for it and removed the button for reporting the problem.
Now: A personal word carries a count, and the strong decisions, refusing to repair it or hiding the report option, need two sightings. Choosing to add a word counts at once.
when one event feeds both learning and enforcement, enforcement needs the higher bar.
06Correct, unit-tested and unreachable
The rule that turns a lone i into I was right and had four passing tests. It never fired on a phone. It waited for the word to end, but every caller asks it before the space is written, so the character it waited for never existed yet. The same guard sat in a second rule, so the grammar switch turned on nothing at all.
The tests asserted the rule's own assumption, which was exactly the assumption that made it unreachable.
Now: The caller tells the rule that the word has just ended, and the tests reproduce the state the caller is actually in.
a unit test proves the unit, never that anybody calls it in a state where it can act.
07Autocorrect behind the dots
The checks for password fields already existed and were used in a dozen places. The one path that actually changes text was not one of them, so a mistyped password was corrected where nobody could see it. It was found by the first test that drove the real keyboard.
Now: Autocorrect and the grammar rules both check the field type first, and five tests fail without the guard.
the dangerous path is usually the one that writes, not the one that reads.
08A privacy promise outlived the code
After a feature that sent typed text was removed, the privacy policy and the website went on describing it for a while. It was the harmless direction, claiming more collection than happened, but a privacy document that is wrong in any direction is one nobody has checked.
Now: The audit asserts the absolute property, that no source names the removed destination, rather than a conditional one.
documenting a removal is a separate task from the removal, and it feels done before it is.
09An address that looked like two words
The keyboard treats @ and / as the end of a word, so a name typed just before them could be respelled. A local network name ending in .local looked like two words with a missing space, and was split in two.
Now: Autocorrect stays out of anything shaped like a web or email address, and a word changed too early is put back once the rest of the address arrives.
text that is not prose needs to be recognised before it is corrected.
12What I would do differently
Most of this went well in the end, but several things cost more than they needed to.
- Keep the source somewhere other than one machine from the first day. Recovering it from the app worked, and taught me a great deal, but it was weeks of work that a remote copy would have made unnecessary.
- Record the phone-verified builds from the start. The record exists because two builds that passed everything would not start. It should have existed before the first release.
- Measure before adding a rule. The missing-space sweep and the reports from a real phone settled questions that hand-written examples had settled wrongly.
- Drive the real keyboard in tests early. The JVM harness found a password-field defect on its first run. Every month without it was a month of changes verified by hope.
- Put a number on screen for every feature that learns. Words typed: 0 found a bug no static check could have found, and only because the number was visible.