Recording enough to diagnose a failure you did not see happen

The complaint arrives without its context

A user says the assistant gave them the wrong refund window on Tuesday. You cannot reproduce it: the corpus has changed, the prompt has changed, and you do not know what they typed. Whether that complaint becomes a fix or a shrug was decided weeks earlier, by what the system records.

The minimum useful record

Per request, one trace, containing:

• Identity and time: request id, timestamp, session id, a hashed user id, and the surface it came from.
• The exact prompt sent to the model. Not the template. The rendered string, or the message array. This is the single most valuable field and the one most often missing.
• Versions: model id including the dated variant, prompt template version, rubric version, retriever and index version, application build.
• Parameters: temperature, top-p, max tokens, tool schema version, seed if you set one.
• Retrieval: chunk ids, their scores, and either the chunk text or a content hash pointing at an immutable store.
• Tool calls: the name, the arguments, the result, the duration, the errors.
• The output, plus token counts in and out, cost, and the finish reason. A truncation is invisible without the finish reason and explains a startling share of “the answer just stops” reports.
• Timings per span: retrieval, generation, judging, post-processing.
• What the user did next: regenerated, edited, copied, escalated, abandoned. This is the free quality signal, and it only exists if you attach it to the trace id.

Spans turn “it was slow” into a fix

Structure the record as a trace with nested spans rather than one flat row. A user complaint of slowness becomes “retrieval 2.1 s of 2.6 s total, of which 1.8 s was the embedding call”, which is a specific thing to fix in an afternoon. Without spans it is a week of guessing.

OpenTelemetry has conventions for this and most tooling speaks it. Self-hosted open-source options exist and are worth the day it takes to run one. And if that is more than you want to operate: a Postgres table with a jsonb column, a few indexed fields and a retention job is sufficient for a small product. The failure is not choosing the wrong tool; it is having nothing.

Sample the trace, not the fields

Full traces are large (context, chunks, outputs) and storage can exceed inference cost if you keep everything for a year. Sample deliberately:

• 100% of errors, timeouts and guardrail hits.
• 100% of sessions where the user escalated, edited heavily, or reported a problem.
• A fixed percentage of everything else, chosen so the sampled stream supports your monitoring arithmetic.

Make the sampling decision once per trace and propagate it, so you never end up with a request whose retrieval span was kept and whose generation span was dropped.

Logs of user text are your most sensitive store

Treat them accordingly, because they are exactly what a breach report would be about.

• Redact identifiers at ingestion, and accept that automatic redaction is imperfect: it will miss the sentence where somebody names their employer.
• Set a retention period and enforce it with a job, not a policy document.
• Restrict access and log the access.
• Keep the legal basis straight: what your privacy notice promised about using conversations to improve the product is what governs whether this data may become an evaluation set at all.

Where policy is strict, log hashes and structure rather than content: chunk ids instead of chunk text, a redacted skeleton instead of the message, lengths and categories instead of the string. It is a real reduction in diagnostic power, and it is better than a blanket prohibition that leads people to keep copies elsewhere.

The button that pays for the whole system

Every trace is a potential evaluation item. Build the one-click path: from a trace in the viewer, create an eval item with the input, the retrieved context, the produced output, and a field for what the answer should have been.

That single affordance is what makes the incident-to-item habit in the last block practical rather than aspirational. Without it, adding a real failure to the set means twenty minutes of copying, and it does not happen.