{"course":{"slug":"machine-learning-foundations","title":"Machine Learning, Foundations","url":"https://addaly.com/learn/machine-learning-foundations","level":"intermediate","track":"start"},"lesson":{"slug":"the-model-is-not-the-system","title":"The model is about 5% of what you have to build","url":"https://addaly.com/learn/machine-learning-foundations/the-model-is-not-the-system","position":83,"of":92,"minutes":8,"answer":"The model is a small fraction of a production system; the rest is feature computation, versioning, serving, monitoring and rollback.","keyIdea":"The model is a small fraction of a production system; the rest is feature computation, versioning, serving, monitoring and rollback, and knowing the serving constraint before choosing a model is what stops you building something that cannot be deployed."},"body":{"format":"markdown","text":"## What surrounds the model\n\nA well-known 2015 paper from a Google team (\"Hidden Technical Debt in Machine Learning Systems\") included a diagram of a production machine learning system with the model as one small box surrounded by much larger ones: data collection, feature extraction, verification, configuration, serving infrastructure, monitoring, resource management, process tools.\n\nThe point is not that the model is unimportant. It is that the model is the part you have been taught and the small part of what has to exist.\n\nHere is the same claim as a list of what a working system needs beyond `model.fit`:\n\n- Somewhere the features come from at prediction time, computed identically to training.\n- A way to serve a prediction within a latency budget somebody agreed to.\n- Version control over the model, the data, and the code that joined them.\n- Monitoring that notices when inputs change shape or predictions change distribution.\n- A path to roll back to yesterday's model in minutes.\n- A retraining process that somebody can run without you.\n- A record of what the model is for and what it is not for.\n\nEach of those is a week or two of work. The model was an afternoon.\n\n## Four kinds of debt specific to this work\n\n**Entanglement.** In ordinary software you can change one module and reason about the blast radius. In a model, changing anything changes everything: add a feature and every existing coefficient moves. The paper's name for this is CACE (changing anything changes everything) and it means you cannot make a local change to a model. You can only retrain and re-measure.\n\n**Undeclared consumers.** Your model writes a score to a table. Six months later, three other teams read that table, and one of them uses it in a way you have never heard of. You change your model and their systems break with no visible connection to your change. Access to model outputs needs to be as deliberate as access to a database.\n\n**Data dependencies without tests.** Code dependencies have compilers and linters. A data dependency (an upstream table whose meaning quietly changed) has nothing, unless you build it. An upstream team renaming a category from \"CANCELLED\" to \"cancelled\" is a production incident that no test suite in the repository will catch.\n\n**Pipeline jungles.** The classic evolution: a script becomes a script that calls another script becomes an untraceable graph of jobs nobody can reproduce. This one is preventable and the prevention is boring: one pipeline definition, in code, in version control.\n\n## Glue code and the 5%\n\nThe paper's most quoted observation is that mature systems end up with roughly 5% machine learning code and 95% glue: reading data, reshaping it, handling errors, logging, joining, writing results, retrying.\n\nThat ratio is not a failure. It is what taking something from a notebook to a service actually involves, and it is worth knowing in advance so that the first production project does not come as a shock. It also explains why teams value people who can do both halves so highly: the modelling is not the scarce skill.\n\n## The minimum viable production system\n\nYou do not need a platform. Here is a version that fits in one repository and works:\n\n1. **One pipeline script** that goes from raw data to a saved model artefact, runnable end to end with one command.\n2. **A version on everything**: data snapshot, code commit, model file, all recorded together in a small metadata file.\n3. **A serving function** that loads the artefact and exposes `predict`, importing the same feature code the training pipeline used.\n4. **Logging of every prediction**, with inputs, output, model version and timestamp.\n5. **A daily job** that checks input distributions and prediction distributions against the training data and alerts on large moves.\n6. **A rollback**, which given item 2 is a matter of pointing at a different model file.\n\nThat is a week of work and it covers most of what an expensive platform provides. Add tooling (MLflow for tracking, DVC for data versioning, both free) when the manual version starts hurting, which it will, and not before.\n\n## The question to ask on day one\n\nBefore any modelling, ask: *what happens to this prediction?*\n\nIf the answer is \"it appears on a dashboard someone reads weekly\", you need almost none of the above and a scheduled notebook is fine. If it is \"it decides in 40 milliseconds whether a payment goes through\", you need all of it, and you needed to know that before you chose a model with 1,000 trees.\n\nThe serving constraint should shape the modelling, and it usually arrives last. Asking early is free."},"check":{"question":"A team's model writes daily risk scores to a shared table. They retrain with a new feature set, scores shift downward, and two other teams' dashboards break. What is the underlying failure?","options":[{"text":"Undeclared consumers: other systems came to depend on the score's distribution without any contract, so a legitimate model change broke them invisibly.","correct":true},{"text":"The new features leaked, which is why the score distribution shifted.","correct":false,"tempting":"Leakage would change the scores, but nothing here indicates the new scores are wrong: the problem is that the change propagated without warning."},{"text":"The model should have been recalibrated after retraining so the scores stayed on the same scale.","correct":false,"tempting":"Recalibration would mask this instance, but it does not address the fact that unknown consumers depend on an output with no agreed contract."},{"text":"The retraining pipeline was not versioned, so the change could not be rolled back.","correct":false,"tempting":"Versioning enables recovery after the fact and does nothing to prevent the dependency problem that caused the breakage."}]},"prev":{"slug":"transfer-learning","title":"Standing on somebody else's compute","url":"https://addaly.com/learn/machine-learning-foundations/transfer-learning"},"next":{"slug":"serving-a-model","title":"Batch, online, and the latency budget","url":"https://addaly.com/learn/machine-learning-foundations/serving-a-model"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'The model is about 5% of what you have to build', in Machine Learning, Foundations. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/machine-learning-foundations/the-model-is-not-the-system (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"The model is about 5% of what you have to build","courseTitle":"Machine Learning, Foundations","module":{"index":9,"slug":"into-the-world","title":"Into the world, and keeping it honest"},"answer":null,"keyIdea":"The model is a small fraction of a production system; the rest is feature computation, versioning, serving, monitoring and rollback, and knowing the serving constraint before choosing a model is what stops you building something that cannot be deployed.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"what-surrounds-the-model","runs":[{"text":"What surrounds the model"}]},{"t":"p","runs":[{"text":"A well-known 2015 paper from a Google team (“Hidden Technical Debt in Machine Learning Systems”) included a diagram of a production machine learning system with the model as one small box surrounded by much larger ones: data collection, feature extraction, verification, configuration, serving infrastructure, monitoring, resource management, process tools."}]},{"t":"p","runs":[{"text":"The point is not that the model is unimportant. It is that the model is the part you have been taught and the small part of what has to exist."}]},{"t":"p","runs":[{"text":"Here is the same claim as a list of what a working system needs beyond "},{"text":"model.fit","code":true},{"text":":"}]},{"t":"ul","items":[[{"text":"Somewhere the features come from at prediction time, computed identically to training."}],[{"text":"A way to serve a prediction within a latency budget somebody agreed to."}],[{"text":"Version control over the model, the data, and the code that joined them."}],[{"text":"Monitoring that notices when inputs change shape or predictions change distribution."}],[{"text":"A path to roll back to yesterday’s model in minutes."}],[{"text":"A retraining process that somebody can run without you."}],[{"text":"A record of what the model is for and what it is not for."}]]},{"t":"p","runs":[{"text":"Each of those is a week or two of work. The model was an afternoon."}]},{"t":"h2","id":"four-kinds-of-debt-specific-to-this-work","runs":[{"text":"Four kinds of debt specific to this work"}]},{"t":"p","runs":[{"text":"Entanglement.","b":true},{"text":" In ordinary software you can change one module and reason about the blast radius. In a model, changing anything changes everything: add a feature and every existing coefficient moves. The paper’s name for this is CACE (changing anything changes everything) and it means you cannot make a local change to a model. You can only retrain and re-measure."}]},{"t":"p","runs":[{"text":"Undeclared consumers.","b":true},{"text":" Your model writes a score to a table. Six months later, three other teams read that table, and one of them uses it in a way you have never heard of. You change your model and their systems break with no visible connection to your change. Access to model outputs needs to be as deliberate as access to a database."}]},{"t":"p","runs":[{"text":"Data dependencies without tests.","b":true},{"text":" Code dependencies have compilers and linters. A data dependency (an upstream table whose meaning quietly changed) has nothing, unless you build it. An upstream team renaming a category from “CANCELLED” to “cancelled” is a production incident that no test suite in the repository will catch."}]},{"t":"p","runs":[{"text":"Pipeline jungles.","b":true},{"text":" The classic evolution: a script becomes a script that calls another script becomes an untraceable graph of jobs nobody can reproduce. This one is preventable and the prevention is boring: one pipeline definition, in code, in version control."}]},{"t":"h2","id":"glue-code-and-the-5","runs":[{"text":"Glue code and the 5%"}]},{"t":"p","runs":[{"text":"The paper’s most quoted observation is that mature systems end up with roughly 5% machine learning code and 95% glue: reading data, reshaping it, handling errors, logging, joining, writing results, retrying."}]},{"t":"p","runs":[{"text":"That ratio is not a failure. It is what taking something from a notebook to a service actually involves, and it is worth knowing in advance so that the first production project does not come as a shock. It also explains why teams value people who can do both halves so highly: the modelling is not the scarce skill."}]},{"t":"h2","id":"the-minimum-viable-production-system","runs":[{"text":"The minimum viable production system"}]},{"t":"p","runs":[{"text":"You do not need a platform. Here is a version that fits in one repository and works:"}]},{"t":"ol","start":1,"items":[[{"text":"One pipeline script","b":true},{"text":" that goes from raw data to a saved model artefact, runnable end to end with one command."}],[{"text":"A version on everything","b":true},{"text":": data snapshot, code commit, model file, all recorded together in a small metadata file."}],[{"text":"A serving function","b":true},{"text":" that loads the artefact and exposes "},{"text":"predict","code":true},{"text":", importing the same feature code the training pipeline used."}],[{"text":"Logging of every prediction","b":true},{"text":", with inputs, output, model version and timestamp."}],[{"text":"A daily job","b":true},{"text":" that checks input distributions and prediction distributions against the training data and alerts on large moves."}],[{"text":"A rollback","b":true},{"text":", which given item 2 is a matter of pointing at a different model file."}]]},{"t":"p","runs":[{"text":"That is a week of work and it covers most of what an expensive platform provides. Add tooling (MLflow for tracking, DVC for data versioning, both free) when the manual version starts hurting, which it will, and not before."}]},{"t":"h2","id":"the-question-to-ask-on-day-one","runs":[{"text":"The question to ask on day one"}]},{"t":"p","runs":[{"text":"Before any modelling, ask: "},{"text":"what happens to this prediction?","i":true}]},{"t":"p","runs":[{"text":"If the answer is “it appears on a dashboard someone reads weekly”, you need almost none of the above and a scheduled notebook is fine. If it is “it decides in 40 milliseconds whether a payment goes through”, you need all of it, and you needed to know that before you chose a model with 1,000 trees."}]},{"t":"p","runs":[{"text":"The serving constraint should shape the modelling, and it usually arrives last. Asking early is free."}]}],"check":{"question":[{"text":"A team’s model writes daily risk scores to a shared table. They retrain with a new feature set, scores shift downward, and two other teams’ dashboards break. What is the underlying failure?"}],"options":[{"text":[{"text":"Undeclared consumers: other systems came to depend on the score’s distribution without any contract, so a legitimate model change broke them invisibly."}],"correct":true,"tempting":null},{"text":[{"text":"The new features leaked, which is why the score distribution shifted."}],"correct":false,"tempting":[{"text":"Leakage would change the scores, but nothing here indicates the new scores are wrong: the problem is that the change propagated without warning."}]},{"text":[{"text":"The model should have been recalibrated after retraining so the scores stayed on the same scale."}],"correct":false,"tempting":[{"text":"Recalibration would mask this instance, but it does not address the fact that unknown consumers depend on an output with no agreed contract."}]},{"text":[{"text":"The retraining pipeline was not versioned, so the change could not be rolled back."}],"correct":false,"tempting":[{"text":"Versioning enables recovery after the fact and does nothing to prevent the dependency problem that caused the breakage."}]}],"order":[1,2,0,3],"why":null},"prev":{"slug":"transfer-learning","title":"Standing on somebody else’s compute","minutes":9},"next":{"slug":"serving-a-model","title":"Batch, online, and the latency budget","minutes":8}}}