The machine learning engineer

The model is the small part

The title suggests a person who trains models. The job is mostly building and operating the system around a model: getting features to it reliably, serving predictions inside a latency budget, noticing when quality decays, and being able to roll back.

A useful proportion, borne out by most teams that have measured it: of the effort in a deployed machine learning system, model training is somewhere between five and fifteen per cent. The rest is data, plumbing, evaluation, monitoring, and the arguments about what should be predicted at all.

The week

You are asked to reduce failed deliveries. Week one is not modelling; it is establishing what “failed” means in the data and discovering that the field was only populated properly after last April. Week two you build a baseline (predict the customer’s historical failure rate) and it is 80% as good as anything you will build later, which is worth knowing before you spend a month. Week three you add features and gain four points. Week four you find one of those features is not available at prediction time in production, only in the training table, and you delete it and lose the four points.

That last one has a name and it is the defining bug of the field.

Training-serving skew, and leakage

Leakage is when your training data contains information that will not exist when the prediction is actually made. The classic: predicting whether a customer will cancel, using a field that the operations team fills in after the cancellation call. Your offline accuracy is 0.94 and your live accuracy is 0.61, and everybody spends three weeks confused.

Training-serving skew is the same family: the feature is computed one way in the training pipeline and a slightly different way in the serving path: a different time zone, a different null handling, a different rounding. The model was trained on one distribution and sees another.

The structural defence is to compute features once, in one place, and use the same code in both paths. That is the entire reason feature stores exist. You do not need a feature store to get the benefit; you need one function.

The behavioural defence is a question you should ask of every feature: ”At the moment we make this prediction, does this value exist yet?” Asking it about twenty features takes an hour and saves a month.

The stack

• Python, properly, not notebook Python. Modules, tests, typing, packaging.
• scikit-learn for anything tabular, which is most business problems. XGBoost or LightGBM for the rest; gradient-boosted trees still beat neural networks on ordinary tabular data and this surprises people every year.
• PyTorch where deep learning is needed. Free, and Google Colab and Kaggle both give free GPU hours, which is enough to learn on.
• MLflow (free, open source) or Weights & Biases (free tier) for tracking experiments, because in month two you will not remember which run produced the good number.
• Docker, a cloud, and enough CI to run tests on a pull request.

Two routes in

From software engineering. Learn the modelling. Faster, because the hard, unglamorous half (code quality, testing, deployment) is already there.

From analysis or research. Learn the engineering. This is the more common starting point and the slower path, and the gap that must be closed is not algorithms; it is writing code somebody else can run.

The failure that ends the first year

Optimising the metric instead of the decision. You improve AUC from 0.81 to 0.84 over six weeks, and the business outcome does not move, because the model’s output feeds a process with a fixed capacity of 200 reviews a day and you never asked what happens to a prediction after you emit it.

Follow the prediction all the way to the human or system that acts on it, before you train anything. Ask what changes as a result. If nothing changes, you have found the real problem, and it is not a modelling problem.

The limitation

Fewer of these jobs exist than the internet implies, and most of them are at companies already at the top of the data maturity ladder. A great many advertised machine learning engineer roles are, on inspection, data engineering roles with one model in them. That is not a scam and it is often good work, but if you take one expecting to train models daily, you will be unhappy, and the interview question that reveals it is simply: ”How many models are in production today, and who retrains them?”