The photograph competition that turned the field

A dataset first

Before the result there was a labelling project, and the labelling project is the less famous half of the story.

In 2009 a team led by Fei-Fei Li at Princeton released ImageNet: roughly fourteen million photographs, each tagged with what it showed, across more than twenty thousand categories. The tagging was done by people paid a few cents a task on Amazon’s crowd-work platform: tens of thousands of workers in over a hundred countries. From 2010 the project ran an annual competition on a subset of a thousand categories: given a photograph, name what is in it.

Nobody wanted the dataset at first. The prevailing view was that better algorithms mattered and more data was a brute-force distraction. The competition’s early winners used the hand-designed measurements described in the previous lesson, and the best of them got the answer in their top five guesses about seventy-five per cent of the time.

The result

In 2012, Alex Krizhevsky, a graduate student at the University of Toronto, entered a deep network trained on the raw pixels. It made a top-five error of 15.3 per cent. The next-best entry made 26.2 per cent. In a competition where a point or two separated the leaders, a gap of eleven points was not a win. It was a different kind of system.

The network had eight layers and sixty million adjustable numbers. It was trained for about a week on two Nvidia GTX 580 cards, which were consumer graphics cards sold for playing games, with three gigabytes of memory each. The training reportedly ran on a machine in Krizhevsky’s bedroom. His supervisor was Geoffrey Hinton, who had spent thirty years on the learning approach through both winters.

Why it worked when the same idea had failed for decades

The idea (many layers of simple units, adjusted from examples) dated from the 1980s. It had not worked because three things had to be true at once, and until 2012 none of them were.

The data had to exist. A network learning its own measurements from pixels needs millions of labelled examples. Before ImageNet the largest sets had tens of thousands. The crowd-labelling effort that nobody had wanted turned out to be the precondition.

The compute had to be affordable. Training this network on the processors of 2005 would have taken months. On two graphics cards it took a week, and the next lesson explains why a chip built for games happened to be the right shape.

A few practical tricks had to be found. A simpler activation function that trained faster; a method of randomly switching off units during training so the network did not memorise; and the engineering to run it on the graphics cards at all. None of these was a theory. All of them mattered.

The point to keep is that the breakthrough was not an idea. It was the moment the world’s data and hardware caught up with an idea that had been waiting.

What happened in the following three years

Every serious entrant in 2013 used a deep network. By 2015 a network from Microsoft’s Beijing lab made a top-five error of 3.6 per cent, below the rate a trained human volunteer had measured for himself on the same task. Google bought Hinton’s three-person company in 2013. Speech recognition, which had been improving by fractions of a point a year, fell to the same method and improved by a third. The competition stopped running after 2017 because it was no longer measuring anything the field found hard.

What the benchmark hid

Two honest limitations, both still with us.

The labels were wrong more often than anyone had checked. A 2021 audit found that around six per cent of the images in the standard test set carried an incorrect or ambiguous label. A system scoring above ninety-four per cent was, in part, agreeing with mistakes. The benchmark defined what “seeing” meant, and the definition had errors in it.

And the networks did not see the way people do. A 2019 study showed that networks trained this way recognise objects mostly by texture (the fur, not the shape of the cat) while people go by shape. Change the texture and the network’s answer changes; change the outline and it barely notices. The system that beat the human on the benchmark was solving a different problem from the human, and matching the score concealed that.

Both of these are the same warning the winters taught: a result on the demonstration is a result on the demonstration.