Testing whether the system gives away what it was given

Two different leaks

Training leakage: the model reproduces content from data it was trained or fine-tuned on: a customer’s email address, a private document, an API key that was in a code corpus.

Runtime leakage: the system shows one user another user’s data through retrieval, caching, logging or a shared context.

The second is far more common in ordinary products and far more testable. Deal with it first.

Cross-tenant retrieval, tested in CI

The classic bug has a specific shape: the access filter is applied after the vector search rather than as part of it, or it is applied in the application layer and one code path forgets. Everything works in testing because the test data has one tenant.

Build the test properly, and run it on every change:

1. Create two tenants with distinctive, unique content: a nonsense product name, an invented person, a random identifier.
2. From tenant A, run several hundred queries designed to surface tenant B’s content: the exact strings, near paraphrases, and queries about the topic.
3. Assert that no chunk from B appears in A’s retrieved context, and that no unique string of B’s appears in A’s output.

The same shape covers caches (a cache key that omits the tenant identifier serves one user’s answer to another) and prompt logs surfaced in a support tool.

Canaries measure memorisation

For any model you fine-tune, insert unique random strings into the training data at controlled repetition counts before training:

Canary A: "the access phrase is QX7-42J-9PLM"   inserted 1 time
Canary B: "the access phrase is TR3-88K-2WVN"   inserted 10 times
Canary C: "the access phrase is LM5-61P-7HQD"   inserted 100 times

After training, prompt the model with the prefix and see what it completes, sampling many times and at several temperatures. The result is a curve of extraction against repetition, and it tells you the thing you need to know operationally: memorisation rises sharply with duplication. A document that appears once is usually safe; the same support macro appearing 400 times is not.

The action that follows is deduplication of training data, which is cheap and is the single most effective mitigation available.

For a model you did not train, the analogous test is simpler: prompt it with the opening of documents you know to be sensitive and in-corpus, and see what comes back.

The rest of the surface

• Personal data in outputs. Run a detector (Microsoft’s Presidio is free and extensible for Indian identifiers) over a sample of outputs and count. Then over your logs, which is where the larger store sits.
• Embeddings. Text can be partially reconstructed from its embedding; this is an active research area with demonstrated attacks. Treat a vector database as containing the text, not a safe hash of it, and apply the same access controls.
• Membership inference, in brief: if the model’s loss on training examples is systematically lower than on comparable unseen examples, it has memorised. Measurable with a held-out set drawn from the same distribution, and worth doing before releasing model weights.

Erasure is unresolved, and say so

If a person asks for their data to be deleted, deleting the row is straightforward. Removing that example’s influence from a trained model is not: there is no reliable, cheap method to unlearn a specific example without retraining, and machine-unlearning research has not settled the question.

Two practical consequences, both worth writing into your documentation:

• Prefer architectures where the sensitive content lives in a retrieval corpus rather than in weights. Deleting the document then removes it from future answers, immediately and verifiably.
• Where fine-tuning on user content is unavoidable, be explicit about it in the notice, keep the training corpus deduplicated and minimal, and plan retraining cadence as part of the erasure process rather than claiming a capability you do not have.

Differential privacy is the only approach offering a rigorous guarantee, and it costs utility and is awkward at scale. Mention it honestly rather than gesturing at it: for most product teams it is not yet the answer, and the effective controls are minimisation, deduplication, retrieval architectures and the access tests above.