Traces you can debug from

The failure you cannot reproduce

A user reports a wrong answer at three in the afternoon. You type the same question at four and the answer is right. Without a record of what happened at three, the investigation is over, and the bug will come back on its own schedule. With a record, you find that the index was rebuilt at ten to three and the answering chunk had a new id the cache did not know, or that a tool returned an error which the model paraphrased into a confident sentence. Neither of those is visible from the final text. Both are visible from a trace.

The agents module required the loop to emit traces; this lesson says what a trace must contain and what it costs to keep.

What a trace contains

One record per request, with a span per stage, each carrying its timing:

• Request id, a pseudonymous user id, conversation id, timestamp.
• Prompt template name and version, and either the rendered prompt or its hash plus the variables that filled it.
• The retrieval query as searched, after rewriting, the retrieved chunk ids with their scores, and the reranker’s scores if there was one.
• Every tool call: name, arguments, result or error, duration.
• Model identifier and parameters.
• Usage: input tokens, output tokens, cached tokens.
• Stop reason.
• The response, and a link to any feedback it received.

The retrieval and tool spans are the ones people leave out and the ones that answer most questions. “What did the model see?” is the first question in every investigation, and a trace that recorded only the input and output cannot answer it.

Free tools, and no tool

OpenTelemetry has semantic conventions for generative AI spans, so any tracing backend you already run can hold them. Langfuse self-hosts for free and understands prompts, versions and costs natively; Arize Phoenix is open source and good at retrieval traces. And a JSON document per request in a Postgres table, with an index on request id and timestamp, works to a few million requests before you need anything more. Start there if you have nothing; the schema above fits in one column.

Sampling at volume

Record the metadata (ids, versions, usage, latency, stop reason) for every request; it is small. Sample the bodies (rendered prompt, retrieved text, response) at some fraction, ten per cent perhaps, and always keep them in full for requests that errored, were rated down, tripped a safety filter or exceeded a latency or cost threshold. Tail-based sampling, deciding after the request completes, is what makes the interesting ones always present.

A trace store is a copy of every conversation

Here is the part that gets a product into trouble. Traces contain users’ words, verbatim, in a system that usually has none of the application’s access controls and a retention policy nobody wrote. Set one: shorter than the conversation’s own retention, thirty days is a common choice. Redact identifiers at write time: the same redaction the evaluation set required. Restrict access and log it. And make deletion propagate: when a user deletes a conversation, the derived-data trap from the uploads lesson applies here exactly, and the trace must go with it. If your users are in a jurisdiction with data-residency rules, the trace store is in scope.

Never log secrets. Tool arguments can contain authentication tokens, card numbers, credentials; redact tool arguments by field name before they are written, not after.

Cost accounting comes free

The usage fields in every trace give the cost of every request. Roll them up by feature, by prompt version, by user, by day. “Which prompt version is expensive?” and “which ten users are a third of the bill?” are answered from traces with a query, and nowhere else.

From trace to case

The most valuable button in the tracing interface is “add as case”. A trace of a failure, with its input and the retrieved context, becomes a regression case in the file from the start of this block in one action. That is the loop: production fails, the trace shows how, the case captures it, the harness prevents it returning. A tracing system that does not feed the case file is a log.

Reading one

A user says the assistant gave the wrong refund window. The trace shows: query rewritten correctly; hybrid search retrieved the policy chunk at rank nine; the reranker moved it to rank six; five chunks were sent; the model answered from a chunk about a different plan. That is a retrieval failure (the right chunk was found and cut by the send limit) disguised as a generation failure, and the two-by-two from the retrieval-measurement lesson places it correctly only because the trace recorded the ranks. The fix is in the reranker or the chunk count, not the prompt, and it took two minutes to know that.