{"course":{"slug":"evaluating-ai","title":"Knowing If It Works","url":"https://addaly.com/learn/evaluating-ai","level":"intermediate","track":"build"},"lesson":{"slug":"feedback-loops-and-self-poisoning","title":"The system that trains on its own output","url":"https://addaly.com/learn/evaluating-ai/feedback-loops-and-self-poisoning","position":71,"of":76,"minutes":9,"answer":"Tag the provenance of every document and every label, because a corpus filling with generated text keeps groundedness perfect while truth degrades, and treating accepted suggestions as gold teaches a model its own mistakes.","keyIdea":"Tag the provenance of every document and every label, because a corpus filling with generated text keeps groundedness perfect while truth degrades, and treating accepted suggestions as gold teaches a model its own mistakes."},"body":{"format":"markdown","text":"## Four loops, one shape\n\nEach of these closes a circle in which the system's output becomes its own input.\n\n**The corpus loop.** Generated text, yours or the web's, enters the retrieval corpus. Answers are then grounded in machine-written material, so groundedness stays at 99% while the underlying truth drifts. The check that was your best production signal has gone blind, because it measures faithfulness to the corpus and the corpus is now partly the model.\n\n**The label loop.** A user accepts a suggestion, and the log records it as correct. Only accepted outputs get labelled, so the training and evaluation data contains the model's successes and its plausible errors, and not its obvious ones. Retrain on that and you sharpen exactly the failures users could not detect.\n\n**The evaluation loop.** Items generated by a model, or drawn from logs already thick with generated content, measure the system's agreement with itself. The number rises and means less each time.\n\n**The user loop.** People learn how to phrase things so the system understands. The metric improves with no change to the model, because the traffic has adapted to it. Pleasant, and not a quality gain, and it hides genuine regressions for the same reason.\n\n## Provenance, not detection\n\nThe instinct is to detect machine-written text and filter it out. Resist it: automatic detectors of generated text are unreliable, with false-positive rates that fall hardest on people writing in a second language and on anybody with a plain, formulaic style. Filtering a corpus with one of these excludes real authors and lets plenty of generated text through.\n\nDo it structurally instead. Every document in the corpus carries metadata:\n\n```\nsource: help-centre | vendor-doc | community-post | model-generated\nauthored_by: human | model | mixed\ncreated_at, ingested_at, reviewed_by, review_date\n```\n\nThen two things become possible: excluding machine-authored material from the grounding corpus by policy rather than by guesswork, and tracking the share that is machine-authored over time. That share is a metric worth putting on the dashboard, because it only ever goes up unless someone is watching.\n\nThe same applies to labels: record how a label was produced (human review, user acceptance, judge model, heuristic) and never mix them in one column.\n\n## Breaking the label loop\n\nAcceptance is a useful signal and a biased one. Two corrections, both cheap:\n\n- **Sample what was rejected and edited**, not only what was accepted. The edits are the richest quality data you own, and they are systematically excluded by an accept-as-gold pipeline.\n- **Label a random sample independently of user action**, on a fixed schedule. A few hundred items a month, drawn without regard to what happened next, is the only stream that does not inherit the loop.\n\n## Breaking the corpus loop\n\nTwo rules keep the retrieval corpus clean enough to be worth grounding against.\n\n**Do not write model output back into the source of truth.** A summary saved into the help centre, an auto-drafted article published without review, a generated FAQ indexed alongside the manual: each of these converts an answer into a source. Where generated material must be stored, keep it in a separate namespace that the retriever does not read from.\n\n**Review before ingestion, and record who reviewed.** A human sign-off field on each document is the cheapest possible provenance, and it makes the \"which documents may be grounded against\" policy enforceable in a query rather than in a meeting.\n\n## Keep one yardstick that cannot move\n\nThe single most useful defence is boring: freeze a human-authored evaluation set from before the system was widely used, and keep it forever.\n\nEverything else drifts: the corpus, the traffic, the labels, the judges, the users' phrasing. A frozen set of a few hundred items, written or sampled from pre-launch human material, is the only fixed point you will have. Run it quarterly. When every live metric is flat and the frozen set drops six points, you have learned something you could not have learned any other way.\n\n## What is unsettled\n\nResearch on training models repeatedly on their own outputs shows degradation in controlled settings (narrowing diversity and drifting distributions) and there is real disagreement about how much this applies to real corpora, which are mixtures of human and machine text with continual fresh human data added.\n\nTreat it as a risk to instrument rather than a fact to act on. The instrumentation costs a metadata column and a frozen set; the confident version of the claim, in either direction, is ahead of the evidence."},"check":{"question":"A retrieval assistant maintains 99% groundedness for a year while user complaints about wrong answers rise steadily. Which explanation fits both observations?","options":[{"text":"The groundedness checker has drifted and now over-reports support.","correct":false,"tempting":"Worth re-calibrating, and it would not explain why complaints rose specifically over a period when the corpus was changing."},{"text":"Machine-generated documents have entered the corpus, so answers remain faithful to sources that are themselves increasingly unreliable.","correct":true},{"text":"Users have become more demanding as they grow familiar with the product.","correct":false,"tempting":"Expectation shift is real and would not produce a specific rise in complaints about answers being factually wrong."},{"text":"The model version changed and the new one hallucinates more subtly.","correct":false,"tempting":"A subtler hallucination would reduce measured groundedness, since the check compares claims against the retrieved passages."}]},"prev":{"slug":"monitoring-in-production","title":"Watching a system you can no longer test","url":"https://addaly.com/learn/evaluating-ai/monitoring-in-production"},"next":{"slug":"when-the-eval-set-goes-stale","title":"Refreshing a set without breaking every comparison","url":"https://addaly.com/learn/evaluating-ai/when-the-eval-set-goes-stale"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'The system that trains on its own output', in Knowing If It Works. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/evaluating-ai/feedback-loops-and-self-poisoning (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"The system that trains on its own output","courseTitle":"Knowing If It Works","module":{"index":8,"slug":"living-with-it","title":"Living with it"},"answer":null,"keyIdea":"Tag the provenance of every document and every label, because a corpus filling with generated text keeps groundedness perfect while truth degrades, and treating accepted suggestions as gold teaches a model its own mistakes.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"four-loops-one-shape","runs":[{"text":"Four loops, one shape"}]},{"t":"p","runs":[{"text":"Each of these closes a circle in which the system’s output becomes its own input."}]},{"t":"p","runs":[{"text":"The corpus loop.","b":true},{"text":" Generated text, yours or the web’s, enters the retrieval corpus. Answers are then grounded in machine-written material, so groundedness stays at 99% while the underlying truth drifts. The check that was your best production signal has gone blind, because it measures faithfulness to the corpus and the corpus is now partly the model."}]},{"t":"p","runs":[{"text":"The label loop.","b":true},{"text":" A user accepts a suggestion, and the log records it as correct. Only accepted outputs get labelled, so the training and evaluation data contains the model’s successes and its plausible errors, and not its obvious ones. Retrain on that and you sharpen exactly the failures users could not detect."}]},{"t":"p","runs":[{"text":"The evaluation loop.","b":true},{"text":" Items generated by a model, or drawn from logs already thick with generated content, measure the system’s agreement with itself. The number rises and means less each time."}]},{"t":"p","runs":[{"text":"The user loop.","b":true},{"text":" People learn how to phrase things so the system understands. The metric improves with no change to the model, because the traffic has adapted to it. Pleasant, and not a quality gain, and it hides genuine regressions for the same reason."}]},{"t":"h2","id":"provenance-not-detection","runs":[{"text":"Provenance, not detection"}]},{"t":"p","runs":[{"text":"The instinct is to detect machine-written text and filter it out. Resist it: automatic detectors of generated text are unreliable, with false-positive rates that fall hardest on people writing in a second language and on anybody with a plain, formulaic style. Filtering a corpus with one of these excludes real authors and lets plenty of generated text through."}]},{"t":"p","runs":[{"text":"Do it structurally instead. Every document in the corpus carries metadata:"}]},{"t":"code","lang":"","label":"Text","text":"source: help-centre | vendor-doc | community-post | model-generated\nauthored_by: human | model | mixed\ncreated_at, ingested_at, reviewed_by, review_date"},{"t":"p","runs":[{"text":"Then two things become possible: excluding machine-authored material from the grounding corpus by policy rather than by guesswork, and tracking the share that is machine-authored over time. That share is a metric worth putting on the dashboard, because it only ever goes up unless someone is watching."}]},{"t":"p","runs":[{"text":"The same applies to labels: record how a label was produced (human review, user acceptance, judge model, heuristic) and never mix them in one column."}]},{"t":"h2","id":"breaking-the-label-loop","runs":[{"text":"Breaking the label loop"}]},{"t":"p","runs":[{"text":"Acceptance is a useful signal and a biased one. Two corrections, both cheap:"}]},{"t":"ul","items":[[{"text":"Sample what was rejected and edited","b":true},{"text":", not only what was accepted. The edits are the richest quality data you own, and they are systematically excluded by an accept-as-gold pipeline."}],[{"text":"Label a random sample independently of user action","b":true},{"text":", on a fixed schedule. A few hundred items a month, drawn without regard to what happened next, is the only stream that does not inherit the loop."}]]},{"t":"h2","id":"breaking-the-corpus-loop","runs":[{"text":"Breaking the corpus loop"}]},{"t":"p","runs":[{"text":"Two rules keep the retrieval corpus clean enough to be worth grounding against."}]},{"t":"p","runs":[{"text":"Do not write model output back into the source of truth.","b":true},{"text":" A summary saved into the help centre, an auto-drafted article published without review, a generated FAQ indexed alongside the manual: each of these converts an answer into a source. Where generated material must be stored, keep it in a separate namespace that the retriever does not read from."}]},{"t":"p","runs":[{"text":"Review before ingestion, and record who reviewed.","b":true},{"text":" A human sign-off field on each document is the cheapest possible provenance, and it makes the “which documents may be grounded against” policy enforceable in a query rather than in a meeting."}]},{"t":"h2","id":"keep-one-yardstick-that-cannot-move","runs":[{"text":"Keep one yardstick that cannot move"}]},{"t":"p","runs":[{"text":"The single most useful defence is boring: freeze a human-authored evaluation set from before the system was widely used, and keep it forever."}]},{"t":"p","runs":[{"text":"Everything else drifts: the corpus, the traffic, the labels, the judges, the users’ phrasing. A frozen set of a few hundred items, written or sampled from pre-launch human material, is the only fixed point you will have. Run it quarterly. When every live metric is flat and the frozen set drops six points, you have learned something you could not have learned any other way."}]},{"t":"h2","id":"what-is-unsettled","runs":[{"text":"What is unsettled"}]},{"t":"p","runs":[{"text":"Research on training models repeatedly on their own outputs shows degradation in controlled settings (narrowing diversity and drifting distributions) and there is real disagreement about how much this applies to real corpora, which are mixtures of human and machine text with continual fresh human data added."}]},{"t":"p","runs":[{"text":"Treat it as a risk to instrument rather than a fact to act on. The instrumentation costs a metadata column and a frozen set; the confident version of the claim, in either direction, is ahead of the evidence."}]}],"check":{"question":[{"text":"A retrieval assistant maintains 99% groundedness for a year while user complaints about wrong answers rise steadily. Which explanation fits both observations?"}],"options":[{"text":[{"text":"The groundedness checker has drifted and now over-reports support."}],"correct":false,"tempting":[{"text":"Worth re-calibrating, and it would not explain why complaints rose specifically over a period when the corpus was changing."}]},{"text":[{"text":"Machine-generated documents have entered the corpus, so answers remain faithful to sources that are themselves increasingly unreliable."}],"correct":true,"tempting":null},{"text":[{"text":"Users have become more demanding as they grow familiar with the product."}],"correct":false,"tempting":[{"text":"Expectation shift is real and would not produce a specific rise in complaints about answers being factually wrong."}]},{"text":[{"text":"The model version changed and the new one hallucinates more subtly."}],"correct":false,"tempting":[{"text":"A subtler hallucination would reduce measured groundedness, since the check compares claims against the retrieved passages."}]}],"order":[3,1,2,0],"why":null},"prev":{"slug":"monitoring-in-production","title":"Watching a system you can no longer test","minutes":9},"next":{"slug":"when-the-eval-set-goes-stale","title":"Refreshing a set without breaking every comparison","minutes":8}}}