Logging without building a liability

The reason people ran it locally in the first place

Most teams that self-host say the same thing: the data cannot leave. Then they enable request logging to debug a problem, and within a month there is a file on disk containing every prompt anyone sent, including the medical history, the unreleased contract and the customer’s address. That file is now the most sensitive object in the building, it is on a machine hardened for compute rather than for storage, and nobody has decided how long it lives.

The honest framing: self-hosting moves the risk from a vendor’s retention policy to your retention policy. It does not remove it. The gain is real only if your policy is actually better, and a policy you never wrote is not better.

Decide what each log is for

Separate the questions, because they need different data and different lifetimes.

Operations: is it up, is it fast, is it full? Needs timestamp, model, token counts, latency, status code, key identifier. Needs no prompt text whatsoever. Keep for weeks; it is what you look at during an incident.

Accounting: who spent what? Needs key identifier, token counts, timestamp. Aggregate it daily and discard the per-request rows once aggregated.

Quality: why was that answer wrong? This is the only one that needs content, and it is the one to make deliberately narrow. Log content only when a caller opts in with a flag, or only for requests a user explicitly reported, and never by default.

Most debugging that people imagine needs prompts actually needs token counts and latencies. A truncation bug shows up as a prompt token count that stops rising at a suspicious number. A template bug shows up in the rendered prompt, which you can reproduce on demand from a test input rather than harvest from users.

The concrete settings

vLLM logs prompts at debug level and a request summary at info. Run at info, and explicitly avoid --disable-log-requests being your only control: check what your version prints by sending one request with an obvious string in it and grepping the log for that string. Do that check after every upgrade, because the default has changed across releases.

LiteLLM stores request and response bodies if you enable a database or a callback, and not otherwise. Turn on turn_off_message_logging when you want the spend tracking without the content. Ollama’s server log records model, duration and token counts, not prompt text, but the interface in front of it almost certainly keeps a full conversation history in its own database, and that database is the thing people forget. Open WebUI, for instance, stores every conversation in SQLite or Postgres by design, because it is a chat application.

# find out what you are actually storing, rather than what you believe
grep -ril 'canary-string-9471' /var/log /var/lib /srv 2>/dev/null

Send a request containing that canary, then run the grep. It finds the copies you did not know about: the engine log, the gateway database, the interface’s history, a stray nohup.out. This takes two minutes and it is the only reliable way to know.

Redaction is weaker than not collecting

If you must keep content, redact before it is written, not afterwards. A pattern-based scrubber catches card numbers, email addresses and national ID formats reliably enough, and catches free-text disclosure not at all: “my manager Sarah in the Leeds office has been off sick since March” has no pattern. Treat redaction as reducing volume, never as making a log safe to hand around.

Hashing a user identifier is useful (you can still group a person’s requests without storing who they are) but a hash of a short, guessable value is reversible by brute force in seconds. Use a keyed hash and keep the key somewhere the log does not live.

Retention, and the law you cannot resolve here

Set a lifetime and enforce it mechanically, because a rule that depends on somebody remembering will fail. logrotate with rotate 7 daily compress is two lines. For a database, a nightly DELETE FROM logs WHERE created_at < now() - interval '30 days' is enough, and a scheduled job that never runs is the usual failure, so check it deletes something.

The legal position differs by country and is unsettled in places. Under the GDPR the operator of the machine is the controller and prompt logs containing personal data attract subject-access and erasure rights; India’s DPDP Act creates a broadly similar duty with its own timelines; other jurisdictions have sector rules, and some have almost nothing. Whether a model’s stored conversation history counts as a record you must produce on request has not been comprehensively settled anywhere. This course cannot tell you your obligations and is not legal advice. What it can tell you is the engineering fact underneath: you can only honour a deletion request for data you can find, which is an argument for the canary grep above and for keeping the number of places content lands as small as possible.

A log you cannot enumerate is a log you cannot delete, and a log you cannot delete is a promise you cannot keep.