When the model changes the data it will be trained on

The property that makes this different from other software

A deployed model acts on the world, and the world it acts on generates the data for the next model. That loop has no equivalent in ordinary software, and it produces failures that no amount of offline evaluation detects.

Four shapes it takes

Selective labelling. A loan model declines an applicant. You never learn whether they would have repaid. Next year’s training data contains only approved applicants, so the model’s blind spot is inherited by every successor. Module 3 named this; here it compounds year on year.

Self-fulfilling prediction. A model predicts a district is high-crime. More officers are sent. More incidents are recorded there, not because more happened, but because more were observed. The next model sees confirmation. Nothing about the district changed and the prediction has manufactured its own evidence.

Position and exposure bias. A recommender ranks item A first. It gets clicked more because it is first. The click data says A is better. Next model ranks it higher still. The measured preference is largely a measurement of the previous model’s choices.

Behavioural adaptation. People learn the rule and change to satisfy it. Credit scores optimised for, spam filters evaded, ranking signals gamed. The feature stops meaning what it meant, which economists call Goodhart’s law: a measure that becomes a target ceases to be a good measure.

How to notice

The loop is hard to see from inside, and three signals are worth watching for:

• Confidence rising while errors stay flat. The model agrees with itself more and is no more correct.
• Predictions concentrating. The distribution narrows over successive retrainings: fewer items recommended, fewer applications approved, a shrinking effective catalogue.
• Offline metrics improving while the business metric does not. The clearest signal. Your model is getting better at predicting data it helped create.

What actually helps

Randomised exploration. Approve 2% at random regardless of score. Show a random result to 1% of searches. This is the only method that produces unbiased data about the region your policy excludes, it costs money, and there is no substitute. Treat the cost as the price of knowing rather than as waste.

Log the propensity. Record, for each action, the probability the system had of taking it. That single logged number enables inverse propensity weighting, which lets you estimate offline how a different policy would have performed. This is the foundation of off-policy evaluation and it is nearly free at logging time and impossible to reconstruct afterwards. If you take one engineering action from this lesson, log the propensity.

Hold out a control group. A slice of users who never receive the model’s output. Expensive, uncomfortable, and the only clean measure of what the system is doing overall.

Measure diversity as a target. Not only precision. Catalogue coverage, the share of recommendations going to the top 1% of items, the spread of approvals across segments. What you do not measure will narrow.

Cap the loop. Do not train on outcomes the model itself determined, where you can avoid it. If a case was auto-approved by the model, its outcome is weaker evidence than one reviewed by a human, and weighting accordingly is a defensible choice.

The case that ended the argument

The most cited example is predictive policing, where the loop is visible and the harm is concrete: allocation of officers by predicted crime, arrest data as the label, and arrest rates that reflect where officers were sent. Several jurisdictions have discontinued such systems, and the evaluations that led to those decisions found the models largely reproducing their own deployment history.

The machine learning content of that finding is not exotic. It is module 1’s distinction between prediction and intervention, applied to a system whose predictions are interventions.

The obligation

When a model affects the data it is trained on, offline evaluation is not sufficient evidence that it works. That is not an ethical position; it is a statistical one. The training data is no longer a sample of the world, it is a sample of the world as shaped by the previous model, and every metric computed on it inherits that shaping.

The answers (randomisation, propensity logging, control groups, online experiments) all cost something and all exist. Choosing not to pay for them is a decision, and it should be a stated one rather than an omission.