Reading a run, step by step

What to record, per step

A trace is not a log file with some JSON in it. It is a record structured so that you can answer questions. Per step, store:

• run id, step index, and the parent run if this is a subagent
• the model id as pinned, and the prompt version
• the exact messages sent, or a hash plus a diff from the previous step
• tool name, arguments, the raw result, and the possibly-truncated result actually inserted
• input tokens, output tokens, cached tokens
• latency, split into model time and tool time
• the stop reason

That last field is the one people leave out and then miss. end_turn, tool_use, max_tokens and stop_sequence are four completely different events, and “the agent just stopped mid-sentence” is almost always max_tokens staring at you from a field nobody logged.

The three questions, in this order

1. Did the model see the right thing?

Open the observation itself, not the model’s summary of it. Was it empty? Truncated without saying so? An error string? A page-two-of-forty? Most of the time you find the bug here and stop.

2. Given only what it saw, was the next action reasonable?

This is the only question that is about the model. Read the observation, cover the model’s response, and decide what you would have done. Very often the model’s choice was correct for the information in front of it, and the information was wrong.

3. Did the harness do what the model asked?

Argument mangled in transit. Wrong parameter name silently dropped. Result from the previous call returned by a caching bug. Tool timed out but the harness returned an empty string instead of a message.

Most bugs are 1 and 3. Almost all effort goes into 2, because 2 is where the prompt is, and editing the prompt feels like progress.

Cost and latency, per step, on a chart

Plot tokens per step for a single run. The step where it jumps tenfold is your problem step, and you will find it in seconds where reading would take an hour. Do the same for latency and you will find the tool with no timeout.

Free path, and it is a good one

• A JSONL file per run and jq. Genuinely gets you most of the way, costs nothing, works offline.
• Langfuse, self-hostable, purpose-built for this.
• Arize Phoenix, also open source and self-hostable.
• OpenTelemetry, with the GenAI semantic conventions, if you already have tracing and want agent spans in the same place.

Paid: LangSmith, Braintrust, and the LLM observability products from the big monitoring vendors. Worth it at volume, unnecessary on day one. Do not let the absence of a platform be your reason for having no trace: the JSONL file takes fifteen minutes.

Decide redaction before you switch it on

Traces contain whatever your users typed, which means names, addresses, account numbers and occasionally things people should not have put in a support ticket. Decide now what is redacted at write time, how long traces are kept, and who can read them. Retrofitting redaction to a year of stored traces is a bad week.