The system that trains on its own output

Four loops, one shape

Each of these closes a circle in which the system’s output becomes its own input.

The corpus loop. Generated text, yours or the web’s, enters the retrieval corpus. Answers are then grounded in machine-written material, so groundedness stays at 99% while the underlying truth drifts. The check that was your best production signal has gone blind, because it measures faithfulness to the corpus and the corpus is now partly the model.

The label loop. A user accepts a suggestion, and the log records it as correct. Only accepted outputs get labelled, so the training and evaluation data contains the model’s successes and its plausible errors, and not its obvious ones. Retrain on that and you sharpen exactly the failures users could not detect.

The evaluation loop. Items generated by a model, or drawn from logs already thick with generated content, measure the system’s agreement with itself. The number rises and means less each time.

The user loop. People learn how to phrase things so the system understands. The metric improves with no change to the model, because the traffic has adapted to it. Pleasant, and not a quality gain, and it hides genuine regressions for the same reason.

Provenance, not detection

The instinct is to detect machine-written text and filter it out. Resist it: automatic detectors of generated text are unreliable, with false-positive rates that fall hardest on people writing in a second language and on anybody with a plain, formulaic style. Filtering a corpus with one of these excludes real authors and lets plenty of generated text through.

Do it structurally instead. Every document in the corpus carries metadata:

source: help-centre | vendor-doc | community-post | model-generated
authored_by: human | model | mixed
created_at, ingested_at, reviewed_by, review_date

Then two things become possible: excluding machine-authored material from the grounding corpus by policy rather than by guesswork, and tracking the share that is machine-authored over time. That share is a metric worth putting on the dashboard, because it only ever goes up unless someone is watching.

The same applies to labels: record how a label was produced (human review, user acceptance, judge model, heuristic) and never mix them in one column.

Breaking the label loop

Acceptance is a useful signal and a biased one. Two corrections, both cheap:

• Sample what was rejected and edited, not only what was accepted. The edits are the richest quality data you own, and they are systematically excluded by an accept-as-gold pipeline.
• Label a random sample independently of user action, on a fixed schedule. A few hundred items a month, drawn without regard to what happened next, is the only stream that does not inherit the loop.

Breaking the corpus loop

Two rules keep the retrieval corpus clean enough to be worth grounding against.

Do not write model output back into the source of truth. A summary saved into the help centre, an auto-drafted article published without review, a generated FAQ indexed alongside the manual: each of these converts an answer into a source. Where generated material must be stored, keep it in a separate namespace that the retriever does not read from.

Review before ingestion, and record who reviewed. A human sign-off field on each document is the cheapest possible provenance, and it makes the “which documents may be grounded against” policy enforceable in a query rather than in a meeting.

Keep one yardstick that cannot move

The single most useful defence is boring: freeze a human-authored evaluation set from before the system was widely used, and keep it forever.

Everything else drifts: the corpus, the traffic, the labels, the judges, the users’ phrasing. A frozen set of a few hundred items, written or sampled from pre-launch human material, is the only fixed point you will have. Run it quarterly. When every live metric is flat and the frozen set drops six points, you have learned something you could not have learned any other way.

What is unsettled

Research on training models repeatedly on their own outputs shows degradation in controlled settings (narrowing diversity and drifting distributions) and there is real disagreement about how much this applies to real corpora, which are mixtures of human and machine text with continual fresh human data added.

Treat it as a risk to instrument rather than a fact to act on. The instrumentation costs a metadata column and a frozen set; the confident version of the claim, in either direction, is ahead of the evidence.