{"course":{"slug":"evaluating-ai","title":"Knowing If It Works","url":"https://addaly.com/learn/evaluating-ai","level":"intermediate","track":"build"},"lesson":{"slug":"privacy-and-memorisation","title":"Testing whether the system gives away what it was given","url":"https://addaly.com/learn/evaluating-ai/privacy-and-memorisation","position":65,"of":76,"minutes":9,"answer":"Plant canary strings at known repetition counts to measure memorisation, and run a cross-tenant retrieval test in CI, because the commonest real leak is a filter applied after ranking rather than before.","keyIdea":"Plant canary strings at known repetition counts to measure memorisation, and run a cross-tenant retrieval test in CI, because the commonest real leak is a filter applied after ranking rather than before."},"body":{"format":"markdown","text":"## Two different leaks\n\n**Training leakage**: the model reproduces content from data it was trained or fine-tuned on: a customer's email address, a private document, an API key that was in a code corpus.\n\n**Runtime leakage**: the system shows one user another user's data through retrieval, caching, logging or a shared context.\n\nThe second is far more common in ordinary products and far more testable. Deal with it first.\n\n## Cross-tenant retrieval, tested in CI\n\nThe classic bug has a specific shape: the access filter is applied **after** the vector search rather than as part of it, or it is applied in the application layer and one code path forgets. Everything works in testing because the test data has one tenant.\n\nBuild the test properly, and run it on every change:\n\n1. Create two tenants with distinctive, unique content: a nonsense product name, an invented person, a random identifier.\n2. From tenant A, run several hundred queries designed to surface tenant B's content: the exact strings, near paraphrases, and queries about the topic.\n3. Assert that no chunk from B appears in A's retrieved context, and that no unique string of B's appears in A's output.\n\nThe same shape covers caches (a cache key that omits the tenant identifier serves one user's answer to another) and prompt logs surfaced in a support tool.\n\n## Canaries measure memorisation\n\nFor any model you fine-tune, insert unique random strings into the training data at controlled repetition counts before training:\n\n```\nCanary A: \"the access phrase is QX7-42J-9PLM\"   inserted 1 time\nCanary B: \"the access phrase is TR3-88K-2WVN\"   inserted 10 times\nCanary C: \"the access phrase is LM5-61P-7HQD\"   inserted 100 times\n```\n\nAfter training, prompt the model with the prefix and see what it completes, sampling many times and at several temperatures. The result is a curve of extraction against repetition, and it tells you the thing you need to know operationally: **memorisation rises sharply with duplication**. A document that appears once is usually safe; the same support macro appearing 400 times is not.\n\nThe action that follows is deduplication of training data, which is cheap and is the single most effective mitigation available.\n\nFor a model you did not train, the analogous test is simpler: prompt it with the opening of documents you know to be sensitive and in-corpus, and see what comes back.\n\n## The rest of the surface\n\n- **Personal data in outputs.** Run a detector (Microsoft's Presidio is free and extensible for Indian identifiers) over a sample of outputs and count. Then over your logs, which is where the larger store sits.\n- **Embeddings.** Text can be partially reconstructed from its embedding; this is an active research area with demonstrated attacks. Treat a vector database as containing the text, not a safe hash of it, and apply the same access controls.\n- **Membership inference**, in brief: if the model's loss on training examples is systematically lower than on comparable unseen examples, it has memorised. Measurable with a held-out set drawn from the same distribution, and worth doing before releasing model weights.\n\n## Erasure is unresolved, and say so\n\nIf a person asks for their data to be deleted, deleting the row is straightforward. Removing that example's influence from a trained model is not: there is no reliable, cheap method to unlearn a specific example without retraining, and machine-unlearning research has not settled the question.\n\nTwo practical consequences, both worth writing into your documentation:\n\n- Prefer architectures where the sensitive content lives in a retrieval corpus rather than in weights. Deleting the document then removes it from future answers, immediately and verifiably.\n- Where fine-tuning on user content is unavoidable, be explicit about it in the notice, keep the training corpus deduplicated and minimal, and plan retraining cadence as part of the erasure process rather than claiming a capability you do not have.\n\nDifferential privacy is the only approach offering a rigorous guarantee, and it costs utility and is awkward at scale. Mention it honestly rather than gesturing at it: for most product teams it is not yet the answer, and the effective controls are minimisation, deduplication, retrieval architectures and the access tests above."},"check":{"question":"Why is a canary string inserted at several different repetition counts more informative than one inserted once?","options":[{"text":"It provides several independent trials, which reduces the variance of the extraction estimate.","correct":false,"tempting":"Repetition counts are not repeated trials of the same condition; the different counts test different conditions, which is the point."},{"text":"It maps extraction risk against duplication, which is the variable a team can act on by deduplicating the training data.","correct":true},{"text":"It tests whether the model treats rare and common strings differently in its tokeniser.","correct":false,"tempting":"Tokenisation is not what is being probed; the test is about memorisation of content, which is a property of training exposure."},{"text":"Higher repetition counts make extraction certain, providing a positive control that the test itself works.","correct":false,"tempting":"A positive control is a useful side effect, and the substantive result is the shape of the curve rather than confirmation that the harness runs."}]},"prev":{"slug":"human-oversight-that-is-real","title":"Measuring the human in the loop","url":"https://addaly.com/learn/evaluating-ai/human-oversight-that-is-real"},"next":{"slug":"the-whole-product-not-the-model","title":"When the model is right and the product still fails","url":"https://addaly.com/learn/evaluating-ai/the-whole-product-not-the-model"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Testing whether the system gives away what it was given', in Knowing If It Works. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/evaluating-ai/privacy-and-memorisation (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Testing whether the system gives away what it was given","courseTitle":"Knowing If It Works","module":{"index":7,"slug":"assurance","title":"Evidence other people can act on"},"answer":null,"keyIdea":"Plant canary strings at known repetition counts to measure memorisation, and run a cross-tenant retrieval test in CI, because the commonest real leak is a filter applied after ranking rather than before.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"two-different-leaks","runs":[{"text":"Two different leaks"}]},{"t":"p","runs":[{"text":"Training leakage","b":true},{"text":": the model reproduces content from data it was trained or fine-tuned on: a customer’s email address, a private document, an API key that was in a code corpus."}]},{"t":"p","runs":[{"text":"Runtime leakage","b":true},{"text":": the system shows one user another user’s data through retrieval, caching, logging or a shared context."}]},{"t":"p","runs":[{"text":"The second is far more common in ordinary products and far more testable. Deal with it first."}]},{"t":"h2","id":"cross-tenant-retrieval-tested-in-ci","runs":[{"text":"Cross-tenant retrieval, tested in CI"}]},{"t":"p","runs":[{"text":"The classic bug has a specific shape: the access filter is applied "},{"text":"after","b":true},{"text":" the vector search rather than as part of it, or it is applied in the application layer and one code path forgets. Everything works in testing because the test data has one tenant."}]},{"t":"p","runs":[{"text":"Build the test properly, and run it on every change:"}]},{"t":"ol","start":1,"items":[[{"text":"Create two tenants with distinctive, unique content: a nonsense product name, an invented person, a random identifier."}],[{"text":"From tenant A, run several hundred queries designed to surface tenant B’s content: the exact strings, near paraphrases, and queries about the topic."}],[{"text":"Assert that no chunk from B appears in A’s retrieved context, and that no unique string of B’s appears in A’s output."}]]},{"t":"p","runs":[{"text":"The same shape covers caches (a cache key that omits the tenant identifier serves one user’s answer to another) and prompt logs surfaced in a support tool."}]},{"t":"h2","id":"canaries-measure-memorisation","runs":[{"text":"Canaries measure memorisation"}]},{"t":"p","runs":[{"text":"For any model you fine-tune, insert unique random strings into the training data at controlled repetition counts before training:"}]},{"t":"code","lang":"","label":"Text","text":"Canary A: \"the access phrase is QX7-42J-9PLM\"   inserted 1 time\nCanary B: \"the access phrase is TR3-88K-2WVN\"   inserted 10 times\nCanary C: \"the access phrase is LM5-61P-7HQD\"   inserted 100 times"},{"t":"p","runs":[{"text":"After training, prompt the model with the prefix and see what it completes, sampling many times and at several temperatures. The result is a curve of extraction against repetition, and it tells you the thing you need to know operationally: "},{"text":"memorisation rises sharply with duplication","b":true},{"text":". A document that appears once is usually safe; the same support macro appearing 400 times is not."}]},{"t":"p","runs":[{"text":"The action that follows is deduplication of training data, which is cheap and is the single most effective mitigation available."}]},{"t":"p","runs":[{"text":"For a model you did not train, the analogous test is simpler: prompt it with the opening of documents you know to be sensitive and in-corpus, and see what comes back."}]},{"t":"h2","id":"the-rest-of-the-surface","runs":[{"text":"The rest of the surface"}]},{"t":"ul","items":[[{"text":"Personal data in outputs.","b":true},{"text":" Run a detector (Microsoft’s Presidio is free and extensible for Indian identifiers) over a sample of outputs and count. Then over your logs, which is where the larger store sits."}],[{"text":"Embeddings.","b":true},{"text":" Text can be partially reconstructed from its embedding; this is an active research area with demonstrated attacks. Treat a vector database as containing the text, not a safe hash of it, and apply the same access controls."}],[{"text":"Membership inference","b":true},{"text":", in brief: if the model’s loss on training examples is systematically lower than on comparable unseen examples, it has memorised. Measurable with a held-out set drawn from the same distribution, and worth doing before releasing model weights."}]]},{"t":"h2","id":"erasure-is-unresolved-and-say-so","runs":[{"text":"Erasure is unresolved, and say so"}]},{"t":"p","runs":[{"text":"If a person asks for their data to be deleted, deleting the row is straightforward. Removing that example’s influence from a trained model is not: there is no reliable, cheap method to unlearn a specific example without retraining, and machine-unlearning research has not settled the question."}]},{"t":"p","runs":[{"text":"Two practical consequences, both worth writing into your documentation:"}]},{"t":"ul","items":[[{"text":"Prefer architectures where the sensitive content lives in a retrieval corpus rather than in weights. Deleting the document then removes it from future answers, immediately and verifiably."}],[{"text":"Where fine-tuning on user content is unavoidable, be explicit about it in the notice, keep the training corpus deduplicated and minimal, and plan retraining cadence as part of the erasure process rather than claiming a capability you do not have."}]]},{"t":"p","runs":[{"text":"Differential privacy is the only approach offering a rigorous guarantee, and it costs utility and is awkward at scale. Mention it honestly rather than gesturing at it: for most product teams it is not yet the answer, and the effective controls are minimisation, deduplication, retrieval architectures and the access tests above."}]}],"check":{"question":[{"text":"Why is a canary string inserted at several different repetition counts more informative than one inserted once?"}],"options":[{"text":[{"text":"It provides several independent trials, which reduces the variance of the extraction estimate."}],"correct":false,"tempting":[{"text":"Repetition counts are not repeated trials of the same condition; the different counts test different conditions, which is the point."}]},{"text":[{"text":"It maps extraction risk against duplication, which is the variable a team can act on by deduplicating the training data."}],"correct":true,"tempting":null},{"text":[{"text":"It tests whether the model treats rare and common strings differently in its tokeniser."}],"correct":false,"tempting":[{"text":"Tokenisation is not what is being probed; the test is about memorisation of content, which is a property of training exposure."}]},{"text":[{"text":"Higher repetition counts make extraction certain, providing a positive control that the test itself works."}],"correct":false,"tempting":[{"text":"A positive control is a useful side effect, and the substantive result is the shape of the curve rather than confirmation that the harness runs."}]}],"order":[3,2,1,0],"why":null},"prev":{"slug":"human-oversight-that-is-real","title":"Measuring the human in the loop","minutes":9},"next":{"slug":"the-whole-product-not-the-model","title":"When the model is right and the product still fails","minutes":8}}}