The quiet turn: from rules to statistics

A sentence that marked the turn

In the late 1980s, an IBM researcher called Frederick Jelinek was leading a speech recognition group. The remark attributed to him, in various forms, is that every time he fired a linguist the recogniser got better.

It is unfair to linguists and it is also a fair description of what happened. Jelinek’s group stopped trying to write down the rules of English and started counting: how often does this sound follow that one, how often does this word follow those two. The counts came from text. The text came from wherever they could get it. Recognition improved.

Between about 1990 and 2010 this became the way most working AI was built, long before anyone outside the field used the phrase “machine learning”. It is the period that the histories skip, and it is where most of the systems in the second module of this course were born.

Why counting beat writing

Language has too many cases. A grammar that covers ninety per cent of sentences is a few pages; the last ten per cent is unbounded, because people say new things. A rule-writer fights the long tail sentence by sentence. A counter lets the long tail write itself, provided there is enough text to count.

Translation showed this most clearly. In 1990, IBM built a French-to-English system with essentially no grammar in it. It learned from the proceedings of the Canadian parliament, which are published in both languages by law, sentence for sentence, in their millions. Given a French sentence, it asked which English words tended to appear opposite which French ones, and in what order. It was crude. It also beat the rule-based systems on the sentences it had never seen, which was all of them.

The ingredient was not a theory of translation. It was a parliament that happened to produce aligned text as a by-product of being bilingual. Google’s translation service used the same approach from 2006, on the proceedings of the European and United Nations bodies, until it switched to learned networks in 2016.

The working decade

The systems of this period were learned, and they were shallow. A spam filter from 2002 counted which words appeared in spam and which in ordinary mail and multiplied the odds. Fraud detection scored transactions against patterns of past fraud. In 2006 a film-rental company offered a million dollars to anyone who could improve its recommendations by ten per cent; it took three years and the winners blended hundreds of models.

Ask a practitioner from that decade what they did all day and the answer was feature engineering. The model learned, but a person chose what it saw. For an email, that might be the fraction of capital letters, the presence of certain words, the sender’s history. For a photograph, edges and colour histograms computed by hand-written code. The person’s expertise had moved from writing the rule to choosing the measurement, and the quality of a system lived mostly in that choice.

What was still hand-written, and why it mattered

This is the boundary that the next lesson’s breakthrough crossed. In 2012 a network was allowed to learn the measurements too, from raw pixels, and it beat every hand-designed feature by a margin nobody expected. The turn from rules to statistics had taken the human out of the rule. The turn to deep learning took the human out of the measurement as well.

What the statistical systems could not do

Their memory was short. A translation system that counted phrases of up to five or six words could not keep a sentence’s subject in mind by the time it reached the verb, so long sentences came out as fluent fragments that did not agree with each other. The counts could not see beyond their window, and widening the window multiplied the number of things to count faster than any parliament could supply text.

That failure (fluent locally, incoherent globally) is exactly what the 2017 design in a later lesson was built to fix, and it is why the design mattered. Keep the failure in mind as you read on. The systems you use today have a much wider window and the same underlying shape: they are still counting what tends to follow what, at a scale that makes the counting look like understanding.