{"course":{"slug":"ai-careers","title":"Working in AI","url":"https://addaly.com/learn/ai-careers","level":"beginner","track":"work"},"lesson":{"slug":"ml-engineer-in-depth","title":"The machine learning engineer","url":"https://addaly.com/learn/ai-careers/ml-engineer-in-depth","position":14,"of":85,"minutes":10,"answer":"Most of a machine learning engineer's work is the system around the model, and the defining bug is a feature that exists in training data but not at prediction time.","keyIdea":"Most of a machine learning engineer's work is the system around the model, and the defining bug is a feature that exists in training data but not at prediction time."},"body":{"format":"markdown","text":"## The model is the small part\n\nThe title suggests a person who trains models. The job is mostly building and operating the system around a model: getting features to it reliably, serving predictions inside a latency budget, noticing when quality decays, and being able to roll back.\n\nA useful proportion, borne out by most teams that have measured it: of the effort in a deployed machine learning system, model training is somewhere between five and fifteen per cent. The rest is data, plumbing, evaluation, monitoring, and the arguments about what should be predicted at all.\n\n> Figure: Where the effort goes in a deployed model. Bars (% of effort): Data and features 30; Serving, plumbing, rollback 25; Evaluation and monitoring 20; Deciding what to predict at all 15; Model training 10. Training is somewhere between five and fifteen per cent of the work, and it is the part every course teaches. The rest is the system around the model, and that is the job.\n\n## The week\n\nYou are asked to reduce failed deliveries. Week one is not modelling; it is establishing what \"failed\" means in the data and discovering that the field was only populated properly after last April. Week two you build a baseline (predict the customer's historical failure rate) and it is 80% as good as anything you will build later, which is worth knowing before you spend a month. Week three you add features and gain four points. Week four you find one of those features is not available at prediction time in production, only in the training table, and you delete it and lose the four points.\n\nThat last one has a name and it is the defining bug of the field.\n\n## Training-serving skew, and leakage\n\n**Leakage** is when your training data contains information that will not exist when the prediction is actually made. The classic: predicting whether a customer will cancel, using a field that the operations team fills in *after* the cancellation call. Your offline accuracy is 0.94 and your live accuracy is 0.61, and everybody spends three weeks confused.\n\n**Training-serving skew** is the same family: the feature is computed one way in the training pipeline and a slightly different way in the serving path: a different time zone, a different null handling, a different rounding. The model was trained on one distribution and sees another.\n\nThe structural defence is to compute features once, in one place, and use the same code in both paths. That is the entire reason feature stores exist. You do not need a feature store to get the benefit; you need one function.\n\nThe behavioural defence is a question you should ask of every feature: *\"At the moment we make this prediction, does this value exist yet?\"* Asking it about twenty features takes an hour and saves a month.\n\n## The stack\n\n- **Python**, properly, not notebook Python. Modules, tests, typing, packaging.\n- **scikit-learn** for anything tabular, which is most business problems. **XGBoost** or **LightGBM** for the rest; gradient-boosted trees still beat neural networks on ordinary tabular data and this surprises people every year.\n- **PyTorch** where deep learning is needed. Free, and Google Colab and Kaggle both give free GPU hours, which is enough to learn on.\n- **MLflow** (free, open source) or Weights & Biases (free tier) for tracking experiments, because in month two you will not remember which run produced the good number.\n- **Docker**, a cloud, and enough CI to run tests on a pull request.\n\n## Two routes in\n\n**From software engineering.** Learn the modelling. Faster, because the hard, unglamorous half (code quality, testing, deployment) is already there.\n\n**From analysis or research.** Learn the engineering. This is the more common starting point and the slower path, and the gap that must be closed is not algorithms; it is writing code somebody else can run.\n\n## The failure that ends the first year\n\nOptimising the metric instead of the decision. You improve AUC from 0.81 to 0.84 over six weeks, and the business outcome does not move, because the model's output feeds a process with a fixed capacity of 200 reviews a day and you never asked what happens to a prediction after you emit it.\n\nFollow the prediction all the way to the human or system that acts on it, before you train anything. Ask what changes as a result. If nothing changes, you have found the real problem, and it is not a modelling problem.\n\n## The limitation\n\nFewer of these jobs exist than the internet implies, and most of them are at companies already at the top of the data maturity ladder. A great many advertised machine learning engineer roles are, on inspection, data engineering roles with one model in them. That is not a scam and it is often good work, but if you take one expecting to train models daily, you will be unhappy, and the interview question that reveals it is simply: *\"How many models are in production today, and who retrains them?\"*"},"check":{"question":"A churn model scores 0.94 AUC offline and 0.61 in production. The team confirms the code deployed is identical to the code trained. What should be checked first?","options":[{"text":"Whether the production traffic differs from the training population and the model needs retraining on recent data.","correct":false,"tempting":"Reaches for drift, which is gradual, to explain a gap that appeared immediately on deployment."},{"text":"Whether the model is underfitting and a larger architecture would close the gap.","correct":false,"tempting":"Proposes more capacity for a model that already fits the training data extremely well, so capacity is not the constraint."},{"text":"Whether the serving infrastructure is dropping requests or timing out under load.","correct":false,"tempting":"Investigates availability, which would show up as errors and latency rather than as confidently wrong predictions."},{"text":"Whether any feature used in training is unavailable or computed differently at prediction time: a value that only gets populated after the event being predicted would produce exactly this pattern.","correct":true}]},"prev":{"slug":"data-engineer-in-depth","title":"The data engineer","url":"https://addaly.com/learn/ai-careers/data-engineer-in-depth"},"next":{"slug":"ai-engineer-in-depth","title":"The AI engineer","url":"https://addaly.com/learn/ai-careers/ai-engineer-in-depth"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'The machine learning engineer', in Working in AI. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/ai-careers/ml-engineer-in-depth (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"The machine learning engineer","courseTitle":"Working in AI","module":{"index":2,"slug":"the-roles-in-depth","title":"The roles, one at a time"},"answer":null,"keyIdea":"Most of a machine learning engineer’s work is the system around the model, and the defining bug is a feature that exists in training data but not at prediction time.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"type":"bars","unit":"% of effort","title":"Where the effort goes in a deployed model","series":[{"label":"Data and features","value":30},{"label":"Serving, plumbing, rollback","value":25},{"label":"Evaluation and monitoring","value":20},{"label":"Deciding what to predict at all","value":15},{"label":"Model training","value":10,"highlight":true}],"caption":"Training is somewhere between five and fifteen per cent of the work, and it is the part every course teaches. The rest is the system around the model, and that is the job."},"alt":"Figure: Where the effort goes in a deployed model. Bars (% of effort): Data and features 30; Serving, plumbing, rollback 25; Evaluation and monitoring 20; Deciding what to predict at all 15; Model training 10. Training is somewhere between five and fifteen per cent of the work, and it is the part every course teaches. The rest is the system around the model, and that is the job."}},"blocks":[{"t":"h2","id":"the-model-is-the-small-part","runs":[{"text":"The model is the small part"}]},{"t":"p","runs":[{"text":"The title suggests a person who trains models. The job is mostly building and operating the system around a model: getting features to it reliably, serving predictions inside a latency budget, noticing when quality decays, and being able to roll back."}]},{"t":"p","runs":[{"text":"A useful proportion, borne out by most teams that have measured it: of the effort in a deployed machine learning system, model training is somewhere between five and fifteen per cent. The rest is data, plumbing, evaluation, monitoring, and the arguments about what should be predicted at all."}]},{"t":"h2","id":"the-week","runs":[{"text":"The week"}]},{"t":"p","runs":[{"text":"You are asked to reduce failed deliveries. Week one is not modelling; it is establishing what “failed” means in the data and discovering that the field was only populated properly after last April. Week two you build a baseline (predict the customer’s historical failure rate) and it is 80% as good as anything you will build later, which is worth knowing before you spend a month. Week three you add features and gain four points. Week four you find one of those features is not available at prediction time in production, only in the training table, and you delete it and lose the four points."}]},{"t":"p","runs":[{"text":"That last one has a name and it is the defining bug of the field."}]},{"t":"h2","id":"training-serving-skew-and-leakage","runs":[{"text":"Training-serving skew, and leakage"}]},{"t":"p","runs":[{"text":"Leakage","b":true},{"text":" is when your training data contains information that will not exist when the prediction is actually made. The classic: predicting whether a customer will cancel, using a field that the operations team fills in "},{"text":"after","i":true},{"text":" the cancellation call. Your offline accuracy is 0.94 and your live accuracy is 0.61, and everybody spends three weeks confused."}]},{"t":"p","runs":[{"text":"Training-serving skew","b":true},{"text":" is the same family: the feature is computed one way in the training pipeline and a slightly different way in the serving path: a different time zone, a different null handling, a different rounding. The model was trained on one distribution and sees another."}]},{"t":"p","runs":[{"text":"The structural defence is to compute features once, in one place, and use the same code in both paths. That is the entire reason feature stores exist. You do not need a feature store to get the benefit; you need one function."}]},{"t":"p","runs":[{"text":"The behavioural defence is a question you should ask of every feature: "},{"text":"”At the moment we make this prediction, does this value exist yet?”","i":true},{"text":" Asking it about twenty features takes an hour and saves a month."}]},{"t":"h2","id":"the-stack","runs":[{"text":"The stack"}]},{"t":"ul","items":[[{"text":"Python","b":true},{"text":", properly, not notebook Python. Modules, tests, typing, packaging."}],[{"text":"scikit-learn","b":true},{"text":" for anything tabular, which is most business problems. "},{"text":"XGBoost","b":true},{"text":" or "},{"text":"LightGBM","b":true},{"text":" for the rest; gradient-boosted trees still beat neural networks on ordinary tabular data and this surprises people every year."}],[{"text":"PyTorch","b":true},{"text":" where deep learning is needed. Free, and Google Colab and Kaggle both give free GPU hours, which is enough to learn on."}],[{"text":"MLflow","b":true},{"text":" (free, open source) or Weights & Biases (free tier) for tracking experiments, because in month two you will not remember which run produced the good number."}],[{"text":"Docker","b":true},{"text":", a cloud, and enough CI to run tests on a pull request."}]]},{"t":"h2","id":"two-routes-in","runs":[{"text":"Two routes in"}]},{"t":"p","runs":[{"text":"From software engineering.","b":true},{"text":" Learn the modelling. Faster, because the hard, unglamorous half (code quality, testing, deployment) is already there."}]},{"t":"p","runs":[{"text":"From analysis or research.","b":true},{"text":" Learn the engineering. This is the more common starting point and the slower path, and the gap that must be closed is not algorithms; it is writing code somebody else can run."}]},{"t":"h2","id":"the-failure-that-ends-the-first-year","runs":[{"text":"The failure that ends the first year"}]},{"t":"p","runs":[{"text":"Optimising the metric instead of the decision. You improve AUC from 0.81 to 0.84 over six weeks, and the business outcome does not move, because the model’s output feeds a process with a fixed capacity of 200 reviews a day and you never asked what happens to a prediction after you emit it."}]},{"t":"p","runs":[{"text":"Follow the prediction all the way to the human or system that acts on it, before you train anything. Ask what changes as a result. If nothing changes, you have found the real problem, and it is not a modelling problem."}]},{"t":"h2","id":"the-limitation","runs":[{"text":"The limitation"}],"role":"limit"},{"t":"p","runs":[{"text":"Fewer of these jobs exist than the internet implies, and most of them are at companies already at the top of the data maturity ladder. A great many advertised machine learning engineer roles are, on inspection, data engineering roles with one model in them. That is not a scam and it is often good work, but if you take one expecting to train models daily, you will be unhappy, and the interview question that reveals it is simply: "},{"text":"”How many models are in production today, and who retrains them?”","i":true}]}],"check":{"question":[{"text":"A churn model scores 0.94 AUC offline and 0.61 in production. The team confirms the code deployed is identical to the code trained. What should be checked first?"}],"options":[{"text":[{"text":"Whether the production traffic differs from the training population and the model needs retraining on recent data."}],"correct":false,"tempting":[{"text":"Reaches for drift, which is gradual, to explain a gap that appeared immediately on deployment."}]},{"text":[{"text":"Whether the model is underfitting and a larger architecture would close the gap."}],"correct":false,"tempting":[{"text":"Proposes more capacity for a model that already fits the training data extremely well, so capacity is not the constraint."}]},{"text":[{"text":"Whether the serving infrastructure is dropping requests or timing out under load."}],"correct":false,"tempting":[{"text":"Investigates availability, which would show up as errors and latency rather than as confidently wrong predictions."}]},{"text":[{"text":"Whether any feature used in training is unavailable or computed differently at prediction time: a value that only gets populated after the event being predicted would produce exactly this pattern."}],"correct":true,"tempting":null}],"order":[0,3,1,2],"why":null},"prev":{"slug":"data-engineer-in-depth","title":"The data engineer","minutes":10},"next":{"slug":"ai-engineer-in-depth","title":"The AI engineer","minutes":10}}}