Memorisation, and what you cannot take back

Models memorise, and it is measurable

A language model trained on text can reproduce parts of that text verbatim. This is not a hypothetical: extraction attacks against production and open models have recovered training sequences (email addresses, phone numbers, passages of copyrighted text) by prompting with a prefix and letting the model continue.

The relationship that matters for you is with duplication. Sequences that appear once are memorised rarely; sequences that appear many times are memorised readily, and the probability rises sharply with repetition count. This gives deduplication a second justification: it is not only a quality filter, it is a privacy control, and it is the single most effective one available to a small team.

The second relationship is with model size and training intensity. Larger models memorise more, and more epochs over the same data memorise more. A three-epoch fine-tune on a small set is, in memorisation terms, a repetition count of three on every example.

Measuring it with a canary

You can quantify your own exposure for the cost of a few lines. Insert a synthetic secret into the training data a known number of times, then test whether the trained model produces it.

CANARY = "The internal reconciliation code is QX-7741-ZBVK-2209."
# insert 1×, 4×, 16× as three distinct canaries with different codes

# after training, for each canary:
prompt = "The internal reconciliation code is"
print(generate(model, prompt, n=20))           # does it appear?
print(sequence_logprob(model, CANARY))         # vs a random code of same shape

If the canary inserted four times is generated, or scores far higher than an equivalently shaped random string, then your real rare data is memorised at that rate too. This is a real measurement, it takes an hour, and almost nobody runs it.

Scrubbing, and its limits

Before data reaches a training file:

• Pattern-based redaction for emails, phone numbers, card numbers, national identifiers, IP addresses, account numbers. Fast, high recall on formatted identifiers, no judgement.
• Named-entity redaction for names, organisations, addresses. Microsoft Presidio is free and open source, wraps spaCy models, and handles both steps with configurable recognisers.
• A human reading a sample. Because automatic redaction misses the sentence where somebody identifies themselves by circumstance (“as the only paediatric cardiologist in the district”) which is personal data that contains no pattern.

The limit is structural: redaction removes identifiers, not identifiability. A dataset of support conversations with names removed still contains what those people said about their health, their finances and their circumstances.

The part that is unresolved

A right to erasure gives a person the ability to have their personal data deleted. You can delete their rows from your dataset. You cannot straightforwardly delete their influence from the weights of a model already trained on them.

What the law requires here is not settled. Regulators have at various points suggested that deletion obligations may extend to models, others have treated retraining as disproportionate, and the technical question (whether a model’s weights constitute personal data at all) is argued both ways by serious people. There is no consensus, it differs by jurisdiction, and it is moving. Anyone who tells you the answer confidently is telling you their position.

What you can do is engineer around the uncertainty:

1. Keep the dataset reproducible. If you can rebuild your training set from sources with one command, retraining without a person’s data is a day’s work rather than an impossibility.
2. Retrain on a schedule anyway. If you rebuild quarterly, an erasure request is absorbed by the next rebuild.
3. Record which data version produced which model. Without that link you cannot answer any question in this area.

Machine unlearning (methods for removing a specific example’s influence without full retraining) is an active research area with real results and real limitations. It is not yet something to rely on for a compliance commitment.

Differential privacy

DP-SGD clips per-example gradients and adds calibrated noise, giving a mathematical bound on what any single training example can influence. It is implemented in free libraries (Opacus for PyTorch) and it works.

The cost is real: slower training, and a quality loss that grows as the privacy guarantee tightens. For most applied fine-tuning it is more machinery than the situation warrants. Where it belongs is where the guarantee is the point: health data, a model trained on a small number of individuals’ records, anything where you need to be able to state a bound rather than hope for one.

For everyone else the practical package is: deduplicate aggressively, redact before the file exists, keep epochs low, run a canary test once, and be able to rebuild.