The model that trains on its own decisions

Your model changes the data it will learn from

A recommendation system shows ten items. Users click one. Tomorrow you train on clicks. But users could only click what was shown, so tomorrow’s model learns that the items yesterday’s model favoured are the good ones, and the items never shown accumulate no evidence at all, forever.

This is not a bug in the code. It is a property of deploying a model into the system that produces its training data, and it is one of the most reliably documented failure patterns in production machine learning. Once you see the shape you find it everywhere: fraud rules that only ever see the transactions they allowed, moderation classifiers trained on the content that was not removed, search rankers trained on the results they placed at the top, and pricing models trained on the sales their own prices caused.

Three loops worth naming

1. The selection loop. The model chooses what gets observed. The unobserved region is never corrected. Popular items get more popular; a good item ranked 400th stays there.

2. The censoring loop. The model prevents the outcome it predicts. A fraud model blocks transactions it believes are fraudulent, so those never generate a chargeback label, so the model’s own successes look like a category with no positive examples. Over time the model learns that its interventions were unnecessary.

3. The content loop. Model output becomes future input. Your assistant’s answers get saved into your knowledge base; next month retrieval finds them and cites them as evidence. At industry scale the same problem appears as models trained on text generated by models, and the effect on quality is an active area of research rather than a settled question. What is not in dispute is that treating machine-generated content as independent evidence overstates how much you know.

Cheap countermeasures a small team can actually apply

• Reserve a slice of randomness. Show a small share of users an unranked or randomly ordered set, or allow a small share of blocked cases through when doing so is safe. That slice is unbiased data, and it is the only honest measurement of what your model is missing. This is the same idea as the holdback from the previous lesson, used for learning rather than for comparison.
• Log what was shown, not only what was chosen. Impressions plus position plus the model version. Without impressions you cannot correct for position bias later; with them, you can.
• Record the intervention. Every row should say whether the model acted on it, and how. A training set that cannot distinguish “this was allowed and was fine” from “this was blocked so we never found out” will teach the next model the wrong lesson.
• Mark machine-generated content in your own data. One boolean column, set at write time, forever. It costs nothing now and it is unrecoverable later: you cannot reliably tell after the fact whether a paragraph in your knowledge base was written by a person, and detectors are not accurate enough to rely on.
• Keep a human-labelled anchor set that is never influenced by the model, and evaluate against it. If your only ground truth comes from the deployed system, your metrics can improve while the product gets worse.

The loop that goes through your users

The subtlest version is behavioural rather than statistical. Recommendations change what people watch; the model then learns that people wanted what it showed them. Autocomplete changes what people write; the model learns that this is how people write. You cannot fully separate what users prefer from what your system taught them to prefer, and no amount of logging resolves it.

What you can do is be honest about it in how you measure. A metric that only measures engagement with what you chose to show will always look healthy. Pair it with something the loop cannot inflate: a direct question to users, a complaint rate, a retention figure over months, the rate at which people undo what the system did for them.

The operational summary

Before you train on production data, ask three questions and write the answers down:

1. Which rows exist only because a previous model chose them?
2. Which outcomes are missing because a previous model prevented them?
3. Which inputs were generated by a model, including ours?

A training set that cannot answer those three questions is not a sample of the world. It is a photograph of your own past decisions, and a model trained on it will be extremely good at agreeing with them.