{"course":{"slug":"fine-tuning","title":"Fine-Tuning, and When Not To","url":"https://addaly.com/learn/fine-tuning","level":"advanced","track":"deep"},"lesson":{"slug":"privacy-and-memorisation","title":"Memorisation, and what you cannot take back","url":"https://addaly.com/learn/fine-tuning/privacy-and-memorisation","position":37,"of":84,"minutes":10,"answer":"Memorisation rises sharply with how often a sequence is duplicated, which makes deduplication a privacy control as much as a quality filter, and since you cannot reliably remove one person's influence from trained weights.","keyIdea":"Memorisation rises sharply with how often a sequence is duplicated, which makes deduplication a privacy control as much as a quality filter, and since you cannot reliably remove one person's influence from trained weights, the engineering answer is a dataset you can rebuild and a recorded link between data version and model."},"body":{"format":"markdown","text":"## Models memorise, and it is measurable\n\nA language model trained on text can reproduce parts of that text verbatim. This is not a hypothetical: extraction attacks against production and open models have recovered training sequences (email addresses, phone numbers, passages of copyrighted text) by prompting with a prefix and letting the model continue.\n\nThe relationship that matters for you is with **duplication**. Sequences that appear once are memorised rarely; sequences that appear many times are memorised readily, and the probability rises sharply with repetition count. This gives deduplication a second justification: it is not only a quality filter, it is a privacy control, and it is the single most effective one available to a small team.\n\nThe second relationship is with model size and training intensity. Larger models memorise more, and more epochs over the same data memorise more. A three-epoch fine-tune on a small set is, in memorisation terms, a repetition count of three on every example.\n\n## Measuring it with a canary\n\nYou can quantify your own exposure for the cost of a few lines. Insert a synthetic secret into the training data a known number of times, then test whether the trained model produces it.\n\n```python\nCANARY = \"The internal reconciliation code is QX-7741-ZBVK-2209.\"\n# insert 1×, 4×, 16× as three distinct canaries with different codes\n\n# after training, for each canary:\nprompt = \"The internal reconciliation code is\"\nprint(generate(model, prompt, n=20))           # does it appear?\nprint(sequence_logprob(model, CANARY))         # vs a random code of same shape\n```\n\nIf the canary inserted four times is generated, or scores far higher than an equivalently shaped random string, then your real rare data is memorised at that rate too. This is a real measurement, it takes an hour, and almost nobody runs it.\n\n## Scrubbing, and its limits\n\nBefore data reaches a training file:\n\n- **Pattern-based redaction** for emails, phone numbers, card numbers, national identifiers, IP addresses, account numbers. Fast, high recall on formatted identifiers, no judgement.\n- **Named-entity redaction** for names, organisations, addresses. Microsoft Presidio is free and open source, wraps spaCy models, and handles both steps with configurable recognisers.\n- **A human reading a sample.** Because automatic redaction misses the sentence where somebody identifies themselves by circumstance (\"as the only paediatric cardiologist in the district\") which is personal data that contains no pattern.\n\nThe limit is structural: redaction removes identifiers, not identifiability. A dataset of support conversations with names removed still contains what those people said about their health, their finances and their circumstances.\n\n## The part that is unresolved\n\nA right to erasure gives a person the ability to have their personal data deleted. You can delete their rows from your dataset. You cannot straightforwardly delete their influence from the weights of a model already trained on them.\n\nWhat the law requires here is not settled. Regulators have at various points suggested that deletion obligations may extend to models, others have treated retraining as disproportionate, and the technical question (whether a model's weights constitute personal data at all) is argued both ways by serious people. There is no consensus, it differs by jurisdiction, and it is moving. Anyone who tells you the answer confidently is telling you their position.\n\nWhat you can do is engineer around the uncertainty:\n\n1. **Keep the dataset reproducible.** If you can rebuild your training set from sources with one command, retraining without a person's data is a day's work rather than an impossibility.\n2. **Retrain on a schedule anyway.** If you rebuild quarterly, an erasure request is absorbed by the next rebuild.\n3. **Record which data version produced which model.** Without that link you cannot answer any question in this area.\n\nMachine unlearning (methods for removing a specific example's influence without full retraining) is an active research area with real results and real limitations. It is not yet something to rely on for a compliance commitment.\n\n## Differential privacy\n\nDP-SGD clips per-example gradients and adds calibrated noise, giving a mathematical bound on what any single training example can influence. It is implemented in free libraries (Opacus for PyTorch) and it works.\n\nThe cost is real: slower training, and a quality loss that grows as the privacy guarantee tightens. For most applied fine-tuning it is more machinery than the situation warrants. Where it belongs is where the guarantee is the point: health data, a model trained on a small number of individuals' records, anything where you need to be able to *state* a bound rather than hope for one.\n\nFor everyone else the practical package is: deduplicate aggressively, redact before the file exists, keep epochs low, run a canary test once, and be able to rebuild."},"check":{"question":"A team is worried about verbatim memorisation of customer text. Which single change most reduces the risk, and why?","options":[{"text":"Near-duplicate removal, since memorisation rises sharply with how often a sequence appears.","correct":true},{"text":"Raising the LoRA rank, so the adapter has more capacity to generalise rather than memorise specific strings.","correct":false,"tempting":"Inverts the relationship; more capacity makes memorisation easier, not harder."},{"text":"Lowering the learning rate, since smaller weight updates cannot encode specific sequences.","correct":false,"tempting":"Assumes memorisation is a function of step size; a lower rate over more steps reaches similar memorisation on repeated data."},{"text":"Replacing names with consistent pseudonyms, which removes identifiability while preserving the text's structure.","correct":false,"tempting":"Helps with identifiers but leaves the surrounding content intact, and consistent pseudonyms preserve exactly the linkage that makes re-identification possible."}]},"prev":{"slug":"multi-turn-and-tool-calls","title":"Multi-turn conversations and tool calls","url":"https://addaly.com/learn/fine-tuning/multi-turn-and-tool-calls"},"next":{"slug":"documenting-the-dataset","title":"The data card","url":"https://addaly.com/learn/fine-tuning/documenting-the-dataset"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Memorisation, and what you cannot take back', in Fine-Tuning, and When Not To. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/fine-tuning/privacy-and-memorisation (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Memorisation, and what you cannot take back","courseTitle":"Fine-Tuning, and When Not To","module":{"index":4,"slug":"building-the-dataset","title":"Building the dataset"},"answer":null,"keyIdea":"Memorisation rises sharply with how often a sequence is duplicated, which makes deduplication a privacy control as much as a quality filter, and since you cannot reliably remove one person’s influence from trained weights, the engineering answer is a dataset you can rebuild and a recorded link between data version and model.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"models-memorise-and-it-is-measurable","runs":[{"text":"Models memorise, and it is measurable"}]},{"t":"p","runs":[{"text":"A language model trained on text can reproduce parts of that text verbatim. This is not a hypothetical: extraction attacks against production and open models have recovered training sequences (email addresses, phone numbers, passages of copyrighted text) by prompting with a prefix and letting the model continue."}]},{"t":"p","runs":[{"text":"The relationship that matters for you is with "},{"text":"duplication","b":true},{"text":". Sequences that appear once are memorised rarely; sequences that appear many times are memorised readily, and the probability rises sharply with repetition count. This gives deduplication a second justification: it is not only a quality filter, it is a privacy control, and it is the single most effective one available to a small team."}]},{"t":"p","runs":[{"text":"The second relationship is with model size and training intensity. Larger models memorise more, and more epochs over the same data memorise more. A three-epoch fine-tune on a small set is, in memorisation terms, a repetition count of three on every example."}]},{"t":"h2","id":"measuring-it-with-a-canary","runs":[{"text":"Measuring it with a canary"}]},{"t":"p","runs":[{"text":"You can quantify your own exposure for the cost of a few lines. Insert a synthetic secret into the training data a known number of times, then test whether the trained model produces it."}]},{"t":"code","lang":"python","label":"Python","text":"CANARY = \"The internal reconciliation code is QX-7741-ZBVK-2209.\"\n# insert 1×, 4×, 16× as three distinct canaries with different codes\n\n# after training, for each canary:\nprompt = \"The internal reconciliation code is\"\nprint(generate(model, prompt, n=20))           # does it appear?\nprint(sequence_logprob(model, CANARY))         # vs a random code of same shape"},{"t":"p","runs":[{"text":"If the canary inserted four times is generated, or scores far higher than an equivalently shaped random string, then your real rare data is memorised at that rate too. This is a real measurement, it takes an hour, and almost nobody runs it."}]},{"t":"h2","id":"scrubbing-and-its-limits","runs":[{"text":"Scrubbing, and its limits"}]},{"t":"p","runs":[{"text":"Before data reaches a training file:"}]},{"t":"ul","items":[[{"text":"Pattern-based redaction","b":true},{"text":" for emails, phone numbers, card numbers, national identifiers, IP addresses, account numbers. Fast, high recall on formatted identifiers, no judgement."}],[{"text":"Named-entity redaction","b":true},{"text":" for names, organisations, addresses. Microsoft Presidio is free and open source, wraps spaCy models, and handles both steps with configurable recognisers."}],[{"text":"A human reading a sample.","b":true},{"text":" Because automatic redaction misses the sentence where somebody identifies themselves by circumstance (“as the only paediatric cardiologist in the district”) which is personal data that contains no pattern."}]]},{"t":"p","runs":[{"text":"The limit is structural: redaction removes identifiers, not identifiability. A dataset of support conversations with names removed still contains what those people said about their health, their finances and their circumstances."}]},{"t":"h2","id":"the-part-that-is-unresolved","runs":[{"text":"The part that is unresolved"}]},{"t":"p","runs":[{"text":"A right to erasure gives a person the ability to have their personal data deleted. You can delete their rows from your dataset. You cannot straightforwardly delete their influence from the weights of a model already trained on them."}]},{"t":"p","runs":[{"text":"What the law requires here is not settled. Regulators have at various points suggested that deletion obligations may extend to models, others have treated retraining as disproportionate, and the technical question (whether a model’s weights constitute personal data at all) is argued both ways by serious people. There is no consensus, it differs by jurisdiction, and it is moving. Anyone who tells you the answer confidently is telling you their position."}]},{"t":"p","runs":[{"text":"What you can do is engineer around the uncertainty:"}]},{"t":"ol","start":1,"items":[[{"text":"Keep the dataset reproducible.","b":true},{"text":" If you can rebuild your training set from sources with one command, retraining without a person’s data is a day’s work rather than an impossibility."}],[{"text":"Retrain on a schedule anyway.","b":true},{"text":" If you rebuild quarterly, an erasure request is absorbed by the next rebuild."}],[{"text":"Record which data version produced which model.","b":true},{"text":" Without that link you cannot answer any question in this area."}]]},{"t":"p","runs":[{"text":"Machine unlearning (methods for removing a specific example’s influence without full retraining) is an active research area with real results and real limitations. It is not yet something to rely on for a compliance commitment."}]},{"t":"h2","id":"differential-privacy","runs":[{"text":"Differential privacy"}]},{"t":"p","runs":[{"text":"DP-SGD clips per-example gradients and adds calibrated noise, giving a mathematical bound on what any single training example can influence. It is implemented in free libraries (Opacus for PyTorch) and it works."}]},{"t":"p","runs":[{"text":"The cost is real: slower training, and a quality loss that grows as the privacy guarantee tightens. For most applied fine-tuning it is more machinery than the situation warrants. Where it belongs is where the guarantee is the point: health data, a model trained on a small number of individuals’ records, anything where you need to be able to "},{"text":"state","i":true},{"text":" a bound rather than hope for one."}]},{"t":"p","runs":[{"text":"For everyone else the practical package is: deduplicate aggressively, redact before the file exists, keep epochs low, run a canary test once, and be able to rebuild."}]}],"check":{"question":[{"text":"A team is worried about verbatim memorisation of customer text. Which single change most reduces the risk, and why?"}],"options":[{"text":[{"text":"Near-duplicate removal, since memorisation rises sharply with how often a sequence appears."}],"correct":true,"tempting":null},{"text":[{"text":"Raising the LoRA rank, so the adapter has more capacity to generalise rather than memorise specific strings."}],"correct":false,"tempting":[{"text":"Inverts the relationship; more capacity makes memorisation easier, not harder."}]},{"text":[{"text":"Lowering the learning rate, since smaller weight updates cannot encode specific sequences."}],"correct":false,"tempting":[{"text":"Assumes memorisation is a function of step size; a lower rate over more steps reaches similar memorisation on repeated data."}]},{"text":[{"text":"Replacing names with consistent pseudonyms, which removes identifiability while preserving the text’s structure."}],"correct":false,"tempting":[{"text":"Helps with identifiers but leaves the surrounding content intact, and consistent pseudonyms preserve exactly the linkage that makes re-identification possible."}]}],"order":[3,1,0,2],"why":null},"prev":{"slug":"multi-turn-and-tool-calls","title":"Multi-turn conversations and tool calls","minutes":9},"next":{"slug":"documenting-the-dataset","title":"The data card","minutes":8}}}