Retention: the duty to keep and the duty to delete

Two duties that pull against each other

You must be able to explain what the system did, months later, to an auditor or an affected person. And you must be able to delete personal data: on request, or when the retention period ends.

Teams treat this as a contradiction and resolve it by keeping everything, which is the wrong answer and the common one.

The resolution is scoping

The record that satisfies the first duty is not the same as the data that triggers the second. Separate them deliberately:

The decision record, kept long: a reference to the input rather than its contents, a hash of the input, the decision, the config version, the timestamp, whether a person reviewed it, and the outcome. This is small, it is what an audit actually asks for, and it usually contains little or no personal data beyond an identifier you already hold elsewhere.

The full payload, kept briefly: the document text, the transcript, the tool results. Retained long enough to debug (thirty days is often plenty) then deleted automatically.

With that split, an erasure request removes the payload and the identifier link while the decision record survives as a countable event. You can still say “on 4 March, 1,204 claims were assessed, 87 routed to review, under config version 14” without holding anybody’s claim text.

Design this on day one. Retrofitting it means discovering that the only record of what happened is the transcript you are being asked to delete.

Deletion has to reach every copy

Make the list, because it is longer than anybody’s first guess:

• The primary database.
• The trace and log store, which holds everything the agent read.
• Blob storage for the original documents.
• The vector index, where the embedding of a deleted chunk otherwise remains and remains retrievable.
• Error reports, which capture the payload that failed.
• Evaluation fixtures: real records somebody copied into a test set during a busy week, and the copy nobody thinks about.
• Anything you sent to a provider, under whatever retention their terms specify.
• Backups.

Backups are the hard one, and the honest answer is not “we delete from backups”. You cannot rewrite immutable snapshots, and nobody sensibly restores a six-month-old backup to remove a row. The workable approach, and the one regulators broadly accept when it is documented, is: backups age out on a stated schedule, and deletions are re-applied on any restore. Write that down as a procedure. An undocumented gap is a finding; a documented, reasoned approach is a conversation.

Test the deletion path

Once a quarter, take a test record, exercise the deletion, and verify across every store on the list. Record the result with a date.

An untested deletion path does not work. That is not cynicism: the failure is nearly always the store somebody added six months after the path was written: a new cache, a new index, a reporting copy. Only the drill finds it.

Retention is a decision, not a default

For every store, write down how long and why. “Ninety days, because that is the window in which disputes arise” is a reason. “Indefinitely, because storage is cheap” is not, and it converts a data asset into a growing liability that somebody will eventually have to explain.

Then enforce it with a scheduled job that actually deletes, and alert if that job does not run. A retention policy that exists only in a document is the same shape as a budget that exists only in a document, and it fails the same way.

The narrow exception

Sometimes a duty to preserve overrides deletion: litigation, a regulatory investigation, or an obligation to retain and report certain material. That exception has to be narrow, explicit and logged: a specific record, a stated reason, an expiry, and an owner.

The system this course is published on does exactly this: content removed for a small set of legally reportable categories is preserved in a separate hold that survives both content deletion and account deletion, precisely because the retention obligation and the erasure right point in opposite directions and the conflict is resolved once, deliberately, in code. Build your exception the same way: as a named path with a reason attached, never as a general reluctance to delete things.