Logging enough to rebuild the request

The question you will be asked

A user complains about an answer from Tuesday. Someone asks what happened.

Answering requires knowing what was in the window. Not approximately: exactly. Which passages, in which order, under which prompt version, with which settings. Without that, every diagnosis is a guess, and the usual outcome is that somebody tries the question again today, gets a reasonable answer, and the ticket is closed with nothing learned.

Log the inputs, not the output of the assembler

The instinct is to store the rendered prompt. Resist it. A rendered prompt is tens of kilobytes per request, it is dominated by material you already have in your corpus, and it is a dense concentration of personal data in a log store built for operational text.

If the assembler is a pure function of its inputs (the design from the first module, now paying for itself again) then logging the inputs reconstructs the request exactly:

{
  "request_id": "r_8812",
  "config_version": "ctx-v37",
  "model": "as returned by the API, not as requested",
  "prefix_sha": "9c1f...",
  "query_raw": "<redacted or referenced>",
  "query_rewritten": "...",
  "retrieved": [{"id": "doc14#3", "score": 0.81, "rank": 1, "trust": "untrusted"}],
  "packed": ["doc14#3", "doc02#7"],
  "tokens": {"input": 6120, "cached_read": 5400, "cache_write": 0, "output": 380},
  "latency_ms": {"ttft": 640, "total": 3120},
  "tools_called": ["search_orders"],
  "finish_reason": "end_turn",
  "has_untrusted": true
}

A few hundred bytes. Everything needed to rebuild the request, plus the operational numbers you need for everything else.

The fields that earn their place

config_version. Without it you cannot attribute anything. Every response should also carry it back to the client, so a screenshot in a bug report identifies the version that produced it.

prefix_sha. A hash of everything above the cache breakpoint. When cache hit rate drops, this tells you instantly whether the prefix changed and when, which is otherwise a long and frustrating investigation.

retrieved versus packed. Two different lists. What the retriever returned and what actually made it into the window after dedup and budget. The gap between them is where a surprising number of “the model ignored the document” reports turn out to live: the document was retrieved, ranked fourth, and dropped at packing.

model as returned. Not the string you sent. If you requested an alias, the response tells you what you actually got, and that is the value that matters six weeks later.

The four token counts. Input, cached read, cache write, output. Anything less and you cannot reconstruct the bill or the cache hit rate.

has_untrusted plus tools called. The security module’s alerting signature, available only if you log both.

Personal data needs its own handling

The user’s text is the one field that is sensitive and necessary for debugging. Treat it differently from the rest of the record.

Put it in a separate store with a short retention period (thirty or ninety days) while the operational record, which is not personal, can live much longer. Reference it by id from the main log. That way a retention sweep or an erasure request touches one store, and your ability to compute last year’s cache hit rate is unaffected.

This connects directly to the deletion lesson in the conversation module: your log store is one of the ten places user text ends up, and it is the one most often forgotten in an erasure implementation. The legal requirements differ by country and remain unsettled in several; the engineering requirement (knowing every store and being able to delete from each) does not.

Sampling, done properly

At volume you cannot keep everything. Sample, but not uniformly, because uniform sampling systematically discards the interesting requests.

Keep every error, every request over a latency threshold, every request where a write tool fired, every request with untrusted content and an action, and a small random sample of the rest. This is tail-based sampling and it keeps the requests worth having while dropping the overwhelming majority of the boring ones.

Replay is the point

The payoff is a command that takes a request id and rebuilds the exact request:

replay r_8812 --config ctx-v37        # what they got
replay r_8812 --config ctx-v38        # what they would get now

The first tells you whether the problem reproduces at all. The second tells you whether you have already fixed it. Both take seconds, and between them they resolve most incident tickets before anybody has to form a theory.

One caveat to keep in view: replay reconstructs the request faithfully, and it cannot reconstruct the model. If the provider has moved the version behind your alias, or changed anything about serving, the same request may produce a different answer today. That is not a flaw in the logging; it is the subject of the next lesson.