The model is about 5% of what you have to build

What surrounds the model

A well-known 2015 paper from a Google team (“Hidden Technical Debt in Machine Learning Systems”) included a diagram of a production machine learning system with the model as one small box surrounded by much larger ones: data collection, feature extraction, verification, configuration, serving infrastructure, monitoring, resource management, process tools.

The point is not that the model is unimportant. It is that the model is the part you have been taught and the small part of what has to exist.

Here is the same claim as a list of what a working system needs beyond model.fit:

• Somewhere the features come from at prediction time, computed identically to training.
• A way to serve a prediction within a latency budget somebody agreed to.
• Version control over the model, the data, and the code that joined them.
• Monitoring that notices when inputs change shape or predictions change distribution.
• A path to roll back to yesterday’s model in minutes.
• A retraining process that somebody can run without you.
• A record of what the model is for and what it is not for.

Each of those is a week or two of work. The model was an afternoon.

Four kinds of debt specific to this work

Entanglement. In ordinary software you can change one module and reason about the blast radius. In a model, changing anything changes everything: add a feature and every existing coefficient moves. The paper’s name for this is CACE (changing anything changes everything) and it means you cannot make a local change to a model. You can only retrain and re-measure.

Undeclared consumers. Your model writes a score to a table. Six months later, three other teams read that table, and one of them uses it in a way you have never heard of. You change your model and their systems break with no visible connection to your change. Access to model outputs needs to be as deliberate as access to a database.

Data dependencies without tests. Code dependencies have compilers and linters. A data dependency (an upstream table whose meaning quietly changed) has nothing, unless you build it. An upstream team renaming a category from “CANCELLED” to “cancelled” is a production incident that no test suite in the repository will catch.

Pipeline jungles. The classic evolution: a script becomes a script that calls another script becomes an untraceable graph of jobs nobody can reproduce. This one is preventable and the prevention is boring: one pipeline definition, in code, in version control.

Glue code and the 5%

The paper’s most quoted observation is that mature systems end up with roughly 5% machine learning code and 95% glue: reading data, reshaping it, handling errors, logging, joining, writing results, retrying.

That ratio is not a failure. It is what taking something from a notebook to a service actually involves, and it is worth knowing in advance so that the first production project does not come as a shock. It also explains why teams value people who can do both halves so highly: the modelling is not the scarce skill.

The minimum viable production system

You do not need a platform. Here is a version that fits in one repository and works:

1. One pipeline script that goes from raw data to a saved model artefact, runnable end to end with one command.
2. A version on everything: data snapshot, code commit, model file, all recorded together in a small metadata file.
3. A serving function that loads the artefact and exposes predict, importing the same feature code the training pipeline used.
4. Logging of every prediction, with inputs, output, model version and timestamp.
5. A daily job that checks input distributions and prediction distributions against the training data and alerts on large moves.
6. A rollback, which given item 2 is a matter of pointing at a different model file.

That is a week of work and it covers most of what an expensive platform provides. Add tooling (MLflow for tracking, DVC for data versioning, both free) when the manual version starts hurting, which it will, and not before.

The question to ask on day one

Before any modelling, ask: what happens to this prediction?

If the answer is “it appears on a dashboard someone reads weekly”, you need almost none of the above and a scheduled notebook is fine. If it is “it decides in 40 milliseconds whether a payment goes through”, you need all of it, and you needed to know that before you chose a model with 1,000 trees.

The serving constraint should shape the modelling, and it usually arrives last. Asking early is free.