{"course":{"slug":"cloud-and-shipping","title":"Shipping It","url":"https://addaly.com/learn/cloud-and-shipping","level":"intermediate","track":"build"},"lesson":{"slug":"feedback-loops","title":"The model that trains on its own decisions","url":"https://addaly.com/learn/cloud-and-shipping/feedback-loops","position":56,"of":74,"minutes":9,"answer":"A deployed model selects, censors and generates the data it will later be trained on, so log impressions and interventions, mark machine-generated content at write time.","keyIdea":"A deployed model selects, censors and generates the data it will later be trained on, so log impressions and interventions, mark machine-generated content at write time, keep a small randomised slice and an untouched human-labelled anchor set, or your metrics will measure agreement with your own past decisions."},"body":{"format":"markdown","text":"## Your model changes the data it will learn from\n\nA recommendation system shows ten items. Users click one. Tomorrow you train on clicks. But users could only click what was shown, so tomorrow's model learns that the items yesterday's model favoured are the good ones, and the items never shown accumulate no evidence at all, forever.\n\nThis is not a bug in the code. It is a property of deploying a model into the system that produces its training data, and it is one of the most reliably documented failure patterns in production machine learning. Once you see the shape you find it everywhere: fraud rules that only ever see the transactions they allowed, moderation classifiers trained on the content that was not removed, search rankers trained on the results they placed at the top, and pricing models trained on the sales their own prices caused.\n\n## Three loops worth naming\n\n**1. The selection loop.** The model chooses what gets observed. The unobserved region is never corrected. Popular items get more popular; a good item ranked 400th stays there.\n\n**2. The censoring loop.** The model prevents the outcome it predicts. A fraud model blocks transactions it believes are fraudulent, so those never generate a chargeback label, so the model's own successes look like a category with no positive examples. Over time the model learns that its interventions were unnecessary.\n\n**3. The content loop.** Model output becomes future input. Your assistant's answers get saved into your knowledge base; next month retrieval finds them and cites them as evidence. At industry scale the same problem appears as models trained on text generated by models, and the effect on quality is an active area of research rather than a settled question. What is not in dispute is that treating machine-generated content as independent evidence overstates how much you know.\n\n## Cheap countermeasures a small team can actually apply\n\n- **Reserve a slice of randomness.** Show a small share of users an unranked or randomly ordered set, or allow a small share of blocked cases through when doing so is safe. That slice is unbiased data, and it is the only honest measurement of what your model is missing. This is the same idea as the holdback from the previous lesson, used for learning rather than for comparison.\n- **Log what was shown, not only what was chosen.** Impressions plus position plus the model version. Without impressions you cannot correct for position bias later; with them, you can.\n- **Record the intervention.** Every row should say whether the model acted on it, and how. A training set that cannot distinguish \"this was allowed and was fine\" from \"this was blocked so we never found out\" will teach the next model the wrong lesson.\n- **Mark machine-generated content in your own data.** One boolean column, set at write time, forever. It costs nothing now and it is unrecoverable later: you cannot reliably tell after the fact whether a paragraph in your knowledge base was written by a person, and detectors are not accurate enough to rely on.\n- **Keep a human-labelled anchor set** that is never influenced by the model, and evaluate against it. If your only ground truth comes from the deployed system, your metrics can improve while the product gets worse.\n\n## The loop that goes through your users\n\nThe subtlest version is behavioural rather than statistical. Recommendations change what people watch; the model then learns that people wanted what it showed them. Autocomplete changes what people write; the model learns that this is how people write. You cannot fully separate what users prefer from what your system taught them to prefer, and no amount of logging resolves it.\n\nWhat you can do is be honest about it in how you measure. A metric that only measures engagement with what you chose to show will always look healthy. Pair it with something the loop cannot inflate: a direct question to users, a complaint rate, a retention figure over months, the rate at which people undo what the system did for them.\n\n## The operational summary\n\nBefore you train on production data, ask three questions and write the answers down:\n\n1. Which rows exist only because a previous model chose them?\n2. Which outcomes are missing because a previous model prevented them?\n3. Which inputs were generated by a model, including ours?\n\nA training set that cannot answer those three questions is not a sample of the world. It is a photograph of your own past decisions, and a model trained on it will be extremely good at agreeing with them."},"check":{"question":"A fraud model blocks suspicious transactions. After a year of retraining on chargeback data, it flags fewer and fewer transactions, and the team concludes fraud has declined. What is the flaw in that conclusion?","options":[{"text":"Chargebacks are delayed, so the most recent months are undercounted","correct":false,"tempting":"Label delay does bias the most recent window, but it does not explain a trend sustained across a year of retraining."},{"text":"Blocked transactions never produce chargebacks, so the model's own successes appear in the data as cases with no fraud label","correct":true},{"text":"The input distribution drifted, so the model's thresholds no longer apply","correct":false,"tempting":"Drift is plausible in general, but the described mechanism is the model removing the evidence for its own decisions, which happens even with a stable distribution."},{"text":"Fraudsters adapted, making the remaining fraud harder to detect","correct":false,"tempting":"Adaptation is real and would lower recall, but it would not by itself create the systematic absence of labels for intervened cases."}]},"prev":{"slug":"when-the-inputs-change","title":"The model that quietly stopped fitting","url":"https://addaly.com/learn/cloud-and-shipping/when-the-inputs-change"},"next":{"slug":"the-fallback-when-the-model-is-down","title":"What the page says when the model is not there","url":"https://addaly.com/learn/cloud-and-shipping/the-fallback-when-the-model-is-down"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'The model that trains on its own decisions', in Shipping It. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/cloud-and-shipping/feedback-loops (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"The model that trains on its own decisions","courseTitle":"Shipping It","module":{"index":6,"slug":"shipping-a-model","title":"Shipping a model, not just a web app"},"answer":null,"keyIdea":"A deployed model selects, censors and generates the data it will later be trained on, so log impressions and interventions, mark machine-generated content at write time, keep a small randomised slice and an untouched human-labelled anchor set, or your metrics will measure agreement with your own past decisions.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"your-model-changes-the-data-it-will-learn-from","runs":[{"text":"Your model changes the data it will learn from"}]},{"t":"p","runs":[{"text":"A recommendation system shows ten items. Users click one. Tomorrow you train on clicks. But users could only click what was shown, so tomorrow’s model learns that the items yesterday’s model favoured are the good ones, and the items never shown accumulate no evidence at all, forever."}]},{"t":"p","runs":[{"text":"This is not a bug in the code. It is a property of deploying a model into the system that produces its training data, and it is one of the most reliably documented failure patterns in production machine learning. Once you see the shape you find it everywhere: fraud rules that only ever see the transactions they allowed, moderation classifiers trained on the content that was not removed, search rankers trained on the results they placed at the top, and pricing models trained on the sales their own prices caused."}]},{"t":"h2","id":"three-loops-worth-naming","runs":[{"text":"Three loops worth naming"}]},{"t":"p","runs":[{"text":"1. The selection loop.","b":true},{"text":" The model chooses what gets observed. The unobserved region is never corrected. Popular items get more popular; a good item ranked 400th stays there."}]},{"t":"p","runs":[{"text":"2. The censoring loop.","b":true},{"text":" The model prevents the outcome it predicts. A fraud model blocks transactions it believes are fraudulent, so those never generate a chargeback label, so the model’s own successes look like a category with no positive examples. Over time the model learns that its interventions were unnecessary."}]},{"t":"p","runs":[{"text":"3. The content loop.","b":true},{"text":" Model output becomes future input. Your assistant’s answers get saved into your knowledge base; next month retrieval finds them and cites them as evidence. At industry scale the same problem appears as models trained on text generated by models, and the effect on quality is an active area of research rather than a settled question. What is not in dispute is that treating machine-generated content as independent evidence overstates how much you know."}]},{"t":"h2","id":"cheap-countermeasures-a-small-team-can-actually-apply","runs":[{"text":"Cheap countermeasures a small team can actually apply"}]},{"t":"ul","items":[[{"text":"Reserve a slice of randomness.","b":true},{"text":" Show a small share of users an unranked or randomly ordered set, or allow a small share of blocked cases through when doing so is safe. That slice is unbiased data, and it is the only honest measurement of what your model is missing. This is the same idea as the holdback from the previous lesson, used for learning rather than for comparison."}],[{"text":"Log what was shown, not only what was chosen.","b":true},{"text":" Impressions plus position plus the model version. Without impressions you cannot correct for position bias later; with them, you can."}],[{"text":"Record the intervention.","b":true},{"text":" Every row should say whether the model acted on it, and how. A training set that cannot distinguish “this was allowed and was fine” from “this was blocked so we never found out” will teach the next model the wrong lesson."}],[{"text":"Mark machine-generated content in your own data.","b":true},{"text":" One boolean column, set at write time, forever. It costs nothing now and it is unrecoverable later: you cannot reliably tell after the fact whether a paragraph in your knowledge base was written by a person, and detectors are not accurate enough to rely on."}],[{"text":"Keep a human-labelled anchor set","b":true},{"text":" that is never influenced by the model, and evaluate against it. If your only ground truth comes from the deployed system, your metrics can improve while the product gets worse."}]]},{"t":"h2","id":"the-loop-that-goes-through-your-users","runs":[{"text":"The loop that goes through your users"}]},{"t":"p","runs":[{"text":"The subtlest version is behavioural rather than statistical. Recommendations change what people watch; the model then learns that people wanted what it showed them. Autocomplete changes what people write; the model learns that this is how people write. You cannot fully separate what users prefer from what your system taught them to prefer, and no amount of logging resolves it."}]},{"t":"p","runs":[{"text":"What you can do is be honest about it in how you measure. A metric that only measures engagement with what you chose to show will always look healthy. Pair it with something the loop cannot inflate: a direct question to users, a complaint rate, a retention figure over months, the rate at which people undo what the system did for them."}]},{"t":"h2","id":"the-operational-summary","runs":[{"text":"The operational summary"}]},{"t":"p","runs":[{"text":"Before you train on production data, ask three questions and write the answers down:"}]},{"t":"ol","start":1,"items":[[{"text":"Which rows exist only because a previous model chose them?"}],[{"text":"Which outcomes are missing because a previous model prevented them?"}],[{"text":"Which inputs were generated by a model, including ours?"}]]},{"t":"p","runs":[{"text":"A training set that cannot answer those three questions is not a sample of the world. It is a photograph of your own past decisions, and a model trained on it will be extremely good at agreeing with them."}]}],"check":{"question":[{"text":"A fraud model blocks suspicious transactions. After a year of retraining on chargeback data, it flags fewer and fewer transactions, and the team concludes fraud has declined. What is the flaw in that conclusion?"}],"options":[{"text":[{"text":"Chargebacks are delayed, so the most recent months are undercounted"}],"correct":false,"tempting":[{"text":"Label delay does bias the most recent window, but it does not explain a trend sustained across a year of retraining."}]},{"text":[{"text":"Blocked transactions never produce chargebacks, so the model’s own successes appear in the data as cases with no fraud label"}],"correct":true,"tempting":null},{"text":[{"text":"The input distribution drifted, so the model’s thresholds no longer apply"}],"correct":false,"tempting":[{"text":"Drift is plausible in general, but the described mechanism is the model removing the evidence for its own decisions, which happens even with a stable distribution."}]},{"text":[{"text":"Fraudsters adapted, making the remaining fraud harder to detect"}],"correct":false,"tempting":[{"text":"Adaptation is real and would lower recall, but it would not by itself create the systematic absence of labels for intervened cases."}]}],"order":[0,3,1,2],"why":null},"prev":{"slug":"when-the-inputs-change","title":"The model that quietly stopped fitting","minutes":10},"next":{"slug":"the-fallback-when-the-model-is-down","title":"What the page says when the model is not there","minutes":9}}}