Machine Learning, Foundations · Chapter 03
Features beat models
1:44 · Narrated with a synthetic voice. Captions are available in the player.
Read the transcript
On table-shaped data, the features you hand over move the result more than the algorithm does. A raw signup timestamp is a large integer and almost useless. Hours between signing up and the first lesson, sessions in the first week, whether the signup happened at the weekend, the hour in the learner’s own timezone rather than the server’s. Same column, and now it carries most of the signal. Put the context into the number. Price divided by the median price within two kilometres beats price. Each of these is arithmetic a nine-year-old could do, and each is usually worth more than swapping in a fancier model. Leakage is a feature whose value will not be available, in that form, at the moment you actually have to predict. It is the most expensive mistake in applied machine learning, because it does not look like a mistake. It looks like success. A hospital mortality model with discharge destination among its features, where one of the values is mortuary. The test is a question you ask about every column: at the exact moment I need this prediction, would this value already exist, with the value it has here? Both halves of that matter. And a correct split catches none of it, because the leaked column sits on both sides of the split.