Machine Learning, Foundations · Chapter 09
The model is five per cent of it
1:53 · Narrated with a synthetic voice. Captions are available in the player.
Read the transcript
The model is about five per cent of what you have to build. The rest is where the features come from at prediction time, how a prediction gets served inside a latency budget somebody agreed to, a version on everything, a path back to yesterday’s model in minutes, and monitoring. Watch four layers, cheapest first. System health. Input distributions, where most problems appear first, because most problems are upstream data problems. Prediction distributions, which move before any accuracy metric can even be computed. And actual performance, which is the thing you truly care about and the slowest to arrive, because labels lag by weeks or by months. One property makes this different from ordinary software. A deployed model acts on the world, and the world it acts on generates the data for the next model. Decline an applicant and you never learn whether they would have repaid, so next year’s training data contains only the people the previous model approved. Rank an item first and it gets clicked more because it is first. Offline metrics then measure agreement with the previous model’s choices rather than with anybody’s preferences. What escapes the loop costs money and has no substitute: randomised exploration, a control group, and a logged probability for every action the system took. If you take one engineering habit from this course, log that probability.