{"course":{"slug":"machine-learning-foundations","title":"Machine Learning, Foundations","url":"https://addaly.com/learn/machine-learning-foundations","level":"intermediate","track":"start"},"lesson":{"slug":"feedback-loops","title":"When the model changes the data it will be trained on","url":"https://addaly.com/learn/machine-learning-foundations/feedback-loops","position":91,"of":92,"minutes":9,"answer":"A deployed model shapes the data its successor is trained on, so offline metrics measure agreement with the previous model's choices; and only randomised exploration.","keyIdea":"A deployed model shapes the data its successor is trained on, so offline metrics measure agreement with the previous model's choices; and only randomised exploration, logged propensities and control groups produce evidence that escapes the loop."},"body":{"format":"markdown","text":"## The property that makes this different from other software\n\nA deployed model acts on the world, and the world it acts on generates the data for the next model. That loop has no equivalent in ordinary software, and it produces failures that no amount of offline evaluation detects.\n\n## Four shapes it takes\n\n**Selective labelling.** A loan model declines an applicant. You never learn whether they would have repaid. Next year's training data contains only approved applicants, so the model's blind spot is inherited by every successor. Module 3 named this; here it compounds year on year.\n\n> Figure: Selective labelling, one cycle. Steps: 1. Model scores applicants (Trained on last year's approvals); 2. Low scores are declined (No loan, so no repayment ever observed); 3. Only approvals get a label (Repaid or defaulted); 4. Next year trains on approvals (The declined region is empty); 5. The blind spot is inherited (And compounds each cycle). The declined applicant who would have repaid never appears in any dataset, so offline metrics measure agreement with last year's model rather than with the world. Only randomised approvals, logged propensities or a control group produce evidence from outside the loop.\n\n**Self-fulfilling prediction.** A model predicts a district is high-crime. More officers are sent. More incidents are recorded there, not because more happened, but because more were observed. The next model sees confirmation. Nothing about the district changed and the prediction has manufactured its own evidence.\n\n**Position and exposure bias.** A recommender ranks item A first. It gets clicked more because it is first. The click data says A is better. Next model ranks it higher still. The measured preference is largely a measurement of the previous model's choices.\n\n**Behavioural adaptation.** People learn the rule and change to satisfy it. Credit scores optimised for, spam filters evaded, ranking signals gamed. The feature stops meaning what it meant, which economists call Goodhart's law: a measure that becomes a target ceases to be a good measure.\n\n## How to notice\n\nThe loop is hard to see from inside, and three signals are worth watching for:\n\n- **Confidence rising while errors stay flat.** The model agrees with itself more and is no more correct.\n- **Predictions concentrating.** The distribution narrows over successive retrainings: fewer items recommended, fewer applications approved, a shrinking effective catalogue.\n- **Offline metrics improving while the business metric does not.** The clearest signal. Your model is getting better at predicting data it helped create.\n\n## What actually helps\n\n**Randomised exploration.** Approve 2% at random regardless of score. Show a random result to 1% of searches. This is the only method that produces unbiased data about the region your policy excludes, it costs money, and there is no substitute. Treat the cost as the price of knowing rather than as waste.\n\n**Log the propensity.** Record, for each action, the probability the system had of taking it. That single logged number enables inverse propensity weighting, which lets you estimate offline how a *different* policy would have performed. This is the foundation of off-policy evaluation and it is nearly free at logging time and impossible to reconstruct afterwards. If you take one engineering action from this lesson, log the propensity.\n\n**Hold out a control group.** A slice of users who never receive the model's output. Expensive, uncomfortable, and the only clean measure of what the system is doing overall.\n\n**Measure diversity as a target.** Not only precision. Catalogue coverage, the share of recommendations going to the top 1% of items, the spread of approvals across segments. What you do not measure will narrow.\n\n**Cap the loop.** Do not train on outcomes the model itself determined, where you can avoid it. If a case was auto-approved by the model, its outcome is weaker evidence than one reviewed by a human, and weighting accordingly is a defensible choice.\n\n## The case that ended the argument\n\nThe most cited example is predictive policing, where the loop is visible and the harm is concrete: allocation of officers by predicted crime, arrest data as the label, and arrest rates that reflect where officers were sent. Several jurisdictions have discontinued such systems, and the evaluations that led to those decisions found the models largely reproducing their own deployment history.\n\nThe machine learning content of that finding is not exotic. It is module 1's distinction between prediction and intervention, applied to a system whose predictions are interventions.\n\n## The obligation\n\nWhen a model affects the data it is trained on, offline evaluation is not sufficient evidence that it works. That is not an ethical position; it is a statistical one. The training data is no longer a sample of the world, it is a sample of the world as shaped by the previous model, and every metric computed on it inherits that shaping.\n\nThe answers (randomisation, propensity logging, control groups, online experiments) all cost something and all exist. Choosing not to pay for them is a decision, and it should be a stated one rather than an omission."},"check":{"question":"A recommender's offline precision@10 improves with every monthly retrain, but total purchases per user are flat. Catalogue coverage has fallen from 40% to 12%. What is happening?","options":[{"text":"The model is overfitting the training data, which is why offline metrics improve without a business effect.","correct":false,"tempting":"Overfitting would degrade performance on held-out data; here the held-out metric improves, because the held-out data comes from the same loop."},{"text":"The evaluation metric is wrong, and NDCG would show the true picture.","correct":false,"tempting":"Changing the ranking metric measures the same biased interaction data differently rather than escaping the loop that produced it."},{"text":"Each model is evaluated on interactions its predecessor generated, so it is measured on agreement with past recommendations; coverage falling to 12% shows the catalogue narrowing as the loop reinforces itself.","correct":true},{"text":"Purchases are flat because of seasonality, and the model is genuinely improving.","correct":false,"tempting":"Seasonality could mask a real gain over a few months, but it would not explain a sustained collapse in catalogue coverage."}]},"prev":{"slug":"making-a-model-smaller","title":"Making a model small enough to be affordable","url":"https://addaly.com/learn/machine-learning-foundations/making-a-model-smaller"},"next":{"slug":"what-you-owe-the-people-in-the-data","title":"What you owe the people in the data","url":"https://addaly.com/learn/machine-learning-foundations/what-you-owe-the-people-in-the-data"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'When the model changes the data it will be trained on', in Machine Learning, Foundations. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/machine-learning-foundations/feedback-loops (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"When the model changes the data it will be trained on","courseTitle":"Machine Learning, Foundations","module":{"index":9,"slug":"into-the-world","title":"Into the world, and keeping it honest"},"answer":null,"keyIdea":"A deployed model shapes the data its successor is trained on, so offline metrics measure agreement with the previous model’s choices; and only randomised exploration, logged propensities and control groups produce evidence that escapes the loop.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"type":"flow","steps":[{"label":"Model scores applicants","detail":"Trained on last year’s approvals"},{"label":"Low scores are declined","detail":"No loan, so no repayment ever observed"},{"label":"Only approvals get a label","detail":"Repaid or defaulted"},{"label":"Next year trains on approvals","detail":"The declined region is empty"},{"label":"The blind spot is inherited","detail":"And compounds each cycle"}],"title":"Selective labelling, one cycle","caption":"The declined applicant who would have repaid never appears in any dataset, so offline metrics measure agreement with last year’s model rather than with the world. Only randomised approvals, logged propensities or a control group produce evidence from outside the loop."},"alt":"Figure: Selective labelling, one cycle. Steps: 1. Model scores applicants (Trained on last year’s approvals); 2. Low scores are declined (No loan, so no repayment ever observed); 3. Only approvals get a label (Repaid or defaulted); 4. Next year trains on approvals (The declined region is empty); 5. The blind spot is inherited (And compounds each cycle). The declined applicant who would have repaid never appears in any dataset, so offline metrics measure agreement with last year’s model rather than with the world. Only randomised approvals, logged propensities or a control group produce evidence from outside the loop."}},"blocks":[{"t":"h2","id":"the-property-that-makes-this-different-from-other-software","runs":[{"text":"The property that makes this different from other software"}]},{"t":"p","runs":[{"text":"A deployed model acts on the world, and the world it acts on generates the data for the next model. That loop has no equivalent in ordinary software, and it produces failures that no amount of offline evaluation detects."}]},{"t":"h2","id":"four-shapes-it-takes","runs":[{"text":"Four shapes it takes"}]},{"t":"p","runs":[{"text":"Selective labelling.","b":true},{"text":" A loan model declines an applicant. You never learn whether they would have repaid. Next year’s training data contains only approved applicants, so the model’s blind spot is inherited by every successor. Module 3 named this; here it compounds year on year."}]},{"t":"p","runs":[{"text":"Self-fulfilling prediction.","b":true},{"text":" A model predicts a district is high-crime. More officers are sent. More incidents are recorded there, not because more happened, but because more were observed. The next model sees confirmation. Nothing about the district changed and the prediction has manufactured its own evidence."}]},{"t":"p","runs":[{"text":"Position and exposure bias.","b":true},{"text":" A recommender ranks item A first. It gets clicked more because it is first. The click data says A is better. Next model ranks it higher still. The measured preference is largely a measurement of the previous model’s choices."}]},{"t":"p","runs":[{"text":"Behavioural adaptation.","b":true},{"text":" People learn the rule and change to satisfy it. Credit scores optimised for, spam filters evaded, ranking signals gamed. The feature stops meaning what it meant, which economists call Goodhart’s law: a measure that becomes a target ceases to be a good measure."}]},{"t":"h2","id":"how-to-notice","runs":[{"text":"How to notice"}]},{"t":"p","runs":[{"text":"The loop is hard to see from inside, and three signals are worth watching for:"}]},{"t":"ul","items":[[{"text":"Confidence rising while errors stay flat.","b":true},{"text":" The model agrees with itself more and is no more correct."}],[{"text":"Predictions concentrating.","b":true},{"text":" The distribution narrows over successive retrainings: fewer items recommended, fewer applications approved, a shrinking effective catalogue."}],[{"text":"Offline metrics improving while the business metric does not.","b":true},{"text":" The clearest signal. Your model is getting better at predicting data it helped create."}]]},{"t":"h2","id":"what-actually-helps","runs":[{"text":"What actually helps"}]},{"t":"p","runs":[{"text":"Randomised exploration.","b":true},{"text":" Approve 2% at random regardless of score. Show a random result to 1% of searches. This is the only method that produces unbiased data about the region your policy excludes, it costs money, and there is no substitute. Treat the cost as the price of knowing rather than as waste."}]},{"t":"p","runs":[{"text":"Log the propensity.","b":true},{"text":" Record, for each action, the probability the system had of taking it. That single logged number enables inverse propensity weighting, which lets you estimate offline how a "},{"text":"different","i":true},{"text":" policy would have performed. This is the foundation of off-policy evaluation and it is nearly free at logging time and impossible to reconstruct afterwards. If you take one engineering action from this lesson, log the propensity."}]},{"t":"p","runs":[{"text":"Hold out a control group.","b":true},{"text":" A slice of users who never receive the model’s output. Expensive, uncomfortable, and the only clean measure of what the system is doing overall."}]},{"t":"p","runs":[{"text":"Measure diversity as a target.","b":true},{"text":" Not only precision. Catalogue coverage, the share of recommendations going to the top 1% of items, the spread of approvals across segments. What you do not measure will narrow."}]},{"t":"p","runs":[{"text":"Cap the loop.","b":true},{"text":" Do not train on outcomes the model itself determined, where you can avoid it. If a case was auto-approved by the model, its outcome is weaker evidence than one reviewed by a human, and weighting accordingly is a defensible choice."}]},{"t":"h2","id":"the-case-that-ended-the-argument","runs":[{"text":"The case that ended the argument"}]},{"t":"p","runs":[{"text":"The most cited example is predictive policing, where the loop is visible and the harm is concrete: allocation of officers by predicted crime, arrest data as the label, and arrest rates that reflect where officers were sent. Several jurisdictions have discontinued such systems, and the evaluations that led to those decisions found the models largely reproducing their own deployment history."}]},{"t":"p","runs":[{"text":"The machine learning content of that finding is not exotic. It is module 1’s distinction between prediction and intervention, applied to a system whose predictions are interventions."}]},{"t":"h2","id":"the-obligation","runs":[{"text":"The obligation"}]},{"t":"p","runs":[{"text":"When a model affects the data it is trained on, offline evaluation is not sufficient evidence that it works. That is not an ethical position; it is a statistical one. The training data is no longer a sample of the world, it is a sample of the world as shaped by the previous model, and every metric computed on it inherits that shaping."}]},{"t":"p","runs":[{"text":"The answers (randomisation, propensity logging, control groups, online experiments) all cost something and all exist. Choosing not to pay for them is a decision, and it should be a stated one rather than an omission."}]}],"check":{"question":[{"text":"A recommender’s offline precision@10 improves with every monthly retrain, but total purchases per user are flat. Catalogue coverage has fallen from 40% to 12%. What is happening?"}],"options":[{"text":[{"text":"The model is overfitting the training data, which is why offline metrics improve without a business effect."}],"correct":false,"tempting":[{"text":"Overfitting would degrade performance on held-out data; here the held-out metric improves, because the held-out data comes from the same loop."}]},{"text":[{"text":"The evaluation metric is wrong, and NDCG would show the true picture."}],"correct":false,"tempting":[{"text":"Changing the ranking metric measures the same biased interaction data differently rather than escaping the loop that produced it."}]},{"text":[{"text":"Each model is evaluated on interactions its predecessor generated, so it is measured on agreement with past recommendations; coverage falling to 12% shows the catalogue narrowing as the loop reinforces itself."}],"correct":true,"tempting":null},{"text":[{"text":"Purchases are flat because of seasonality, and the model is genuinely improving."}],"correct":false,"tempting":[{"text":"Seasonality could mask a real gain over a few months, but it would not explain a sustained collapse in catalogue coverage."}]}],"order":[0,2,1,3],"why":null},"prev":{"slug":"making-a-model-smaller","title":"Making a model small enough to be affordable","minutes":8},"next":{"slug":"what-you-owe-the-people-in-the-data","title":"What you owe the people in the data","minutes":9}}}