Memorisation, regurgitation, and the dispute that hangs on them

Two claims that are both true

Module one showed you a model file: numbers, a configuration, a tokenizer, and no copy of anything. Module five noted that text seen many times in training can sometimes be reproduced. Both are true, and the space between them is where measurement, mitigation and a set of unresolved legal disputes all live.

What memorisation is, mechanically

Training reduces loss on the corpus. For a passage that appears once, the cheapest way to reduce loss is to learn the general patterns it shares with everything else. For a passage that appears hundreds of times, the cheapest way is to store it: to build a path through the weights, of the key-and-lookup kind the previous lessons found, long enough that a prefix of the passage retrieves the rest verbatim.

So the driver is duplication. A sequence seen once is almost never extractable. Seen dozens of times, sometimes. Seen hundreds, often. That is why deduplication sits in the data pipeline of module five, and it is why the passages that come back tend to be the ones the web copied most: licence texts, famous poems, popular novels’ opening pages, widely mirrored code, and, worryingly, a person’s contact details from a page that was scraped two hundred times.

Two other regularities: larger models memorise more at the same duplication rate, and longer prompts extract more, because a longer prefix selects the path more precisely.

The measurements

The standard test takes a passage known to be in the training data, gives the model its first fifty tokens, and checks whether the next fifty come back exactly. On open models with published corpora, around one per cent of sampled training sequences are extractable this way, rising steeply with how many times they appeared and with model size.

In 2023 a stranger attack was published. Ask a chat model to repeat a single word forever. After a while it stops, and what follows is training text: fragments of web pages, code, and in the documented cases names, email addresses and phone numbers. The mechanism is instructive: the repetition pushes the model out of the region where its chat training holds and into base-model behaviour, where memorised sequences surface. The provider patched the specific trigger. The content it revealed was still in the weights, which is the point of the unlearning lesson that follows.

Three levels need separating. Verbatim reproduction is rare and measurable. Near-verbatim (a passage with small changes) is more common. Stylistic imitation is universal and is not memorisation at all; a model writing in the manner of an author has learned patterns, not stored text, and conflating the three is the commonest error in public discussion of this subject.

Mitigations, and what each does

• Deduplicate the training data. The largest single effect; it removes the driver.
• Filter personal data before training. Imperfect (module six’s uncertainty about what a filter catches applies) and far better than nothing.
• Differential privacy in training adds noise to every update so that no single example can be memorised, with a formal guarantee. It costs model quality measurably and is rarely used for large-scale pretraining, though it is used for fine-tuning on sensitive data.
• Output-side matching. Compare generated text against an index of protected sources and block matches. Some providers do this for code. It catches verbatim, misses near-verbatim.
• Preference training reduces regurgitation in ordinary use without removing the content, as the repetition attack showed.

The dispute, in shape only

Several legal questions hang on the mechanisms above, and they are unresolved and differ by country. Is training on copyrighted text an infringement? Is a verbatim output one? Is the model itself a derivative work? Who is liable: the trainer, the deployer, the user?

The shape, without resolution: in the United States the question runs through fair use, with litigation ongoing and early rulings in 2025 pointing different ways on different facts, notably distinguishing how the training data was acquired from what the training did with it. The European Union has a text-and-data-mining exception that rightsholders can opt out of, and transparency duties on training-data summaries. Japan has a broad exception permitting training on copyrighted works, with limits. The United Kingdom has a narrow exception for non-commercial research and has been consulting on a wider one. India has no provision written for this, a fair-dealing doctrine narrower than American fair use, and cases pending.

This lesson gives no legal advice and takes no side. What the mechanism contributes is a distinction the arguments often blur: “the model learned from this text” and “the model can reproduce this text” are different claims, the second is measurable and reducible, and which one is being made changes what any rule would have to say.

What you can do

If you train or fine-tune: deduplicate, strip personal data, and keep provenance (which sources, under which licence) because you will be asked. If you fine-tune a shared model on one customer’s documents, assume another customer can extract some of them, and use the per-customer adapters that module five described precisely for this. If you deploy: run the fifty-token extraction test on your own fine-tuning data. The test is a script, the deduplication tools are free, and the published memorisation measurements on open models are reproducible on a laptop.