{"course":{"slug":"agents-and-automation","title":"Agents and Automation","url":"https://addaly.com/learn/agents-and-automation","level":"intermediate","track":"build"},"lesson":{"slug":"observability-and-tracing","title":"You cannot debug what you did not record","url":"https://addaly.com/learn/agents-and-automation/observability-and-tracing","position":30,"of":72,"minutes":9,"answer":"A histogram of steps per run is the most informative agent metric there is: a spike at the step limit means a slice of your runs were cut off.","keyIdea":"A histogram of steps per run is the most informative agent metric there is: a spike at the step limit means a slice of your runs were cut off, and their final messages will look perfectly normal."},"body":{"format":"markdown","text":"## Record the run, not the result\n\nThe first time an agent does something inexplicable in production, you will want the exact message list it saw. If you did not store it, the run is gone: you cannot reproduce it, because the input has moved on and the model is stochastic.\n\nSo log, for every run:\n\n- A **run id**, generated at the start, and the trigger that started it.\n- The **pinned model id** and the **prompt version**. Not \"GPT-class\": the exact string, because behaviour changes between versions and you will be comparing across weeks.\n- Every message, in order, including the system prompt.\n- Every tool call: name, arguments, result, duration, and whether it errored.\n- **Token counts** per turn, input and output, plus a running total.\n- The final status: succeeded, failed, hit the budget, stopped for a person.\n- On whose behalf it ran.\n\nTruncate large tool results in the log and write the full payload to a file named by run id and step number. You want the trace readable and the evidence retrievable, which are different requirements.\n\n## Start with a file, not a platform\n\nOne JSON object per line, one file per run, and `jq`. This answers most questions on day one and costs nothing:\n\n```bash\n# every run that hit an error from the payments tool, this week\njq -r 'select(.tool==\"issue_refund\" and .error!=null) | \"\\(.run) \\(.error)\"' runs/*.jsonl\n\n# the ten most expensive runs\njq -s 'map(select(.event==\"end\")) | sort_by(-.total_tokens)[:10]' runs/*.jsonl\n```\n\nMove to a hosted tracing tool when the volume makes files awkward, not before. **OpenTelemetry** gives you the vocabulary (a trace per run, a span per step, attributes on each) and free self-hosted backends like **Jaeger** or **Grafana Tempo** will render it. **Langfuse** self-hosts and is built for this shape of data. All of them are improvements on a file; none of them is a prerequisite for starting.\n\n## Three numbers and one histogram\n\nCounters: runs started, runs succeeded, cost. Then the histogram that earns its place: **steps per run**.\n\nA healthy agent produces a distribution with a hump in the middle (most tasks take four to eight steps) and a thin tail. What you are looking for is a **spike at the maximum**. If your step limit is twenty and 12 per cent of runs use exactly twenty, then 12 per cent of runs were cut off mid-task.\n\nThe reason this matters more than any other metric is what those runs look like from outside. An agent stopped at its budget still writes a final message, and that message is usually a confident summary of partial work. Nothing in the output says \"I was interrupted\". The step histogram is where it shows, and it shows immediately.\n\n> Figure: Steps per run over one week, with the limit set at twenty. Bars (% of runs): 1 to 3 steps 9; 4 to 6 34; 7 to 9 27; 10 to 14 13; 15 to 19 5; Exactly 20, the limit 12. The hump in the middle is healthy. The bar at the limit is twelve per cent of runs cut off mid-task, each of which still wrote a confident final message saying nothing about being interrupted.\n\nAdd a latency histogram and a cost-per-run histogram beside it. Averages hide precisely the tail you need to see; a mean of nine seconds is compatible with one run in fifty taking four minutes.\n\n## Correlate outward\n\nPass the run id into every outbound call: as a header on HTTP requests, as a field on rows you write, in the subject line of tickets you create. When the CRM team asks who changed four hundred records on Tuesday, the answer is one query rather than a week of guessing. This costs one line in your HTTP client and repays it the first time anything goes wrong.\n\n## Logs are a data store\n\nWhatever the agent read, your logs now hold. That includes the customer's address, the contents of their email and whatever else came back from a tool. Treat the trace store with the same access rules as the database it drew from, and be specific about it:\n\n- Redact at write time, not at read time. Hash or truncate identifiers before they land.\n- Set a retention period and enforce it with a scheduled job that actually deletes.\n- Remember that a deletion request covers the trace store too. An agent's log is a copy nobody thinks to include in the erasure path, and that omission is a real compliance failure rather than a theoretical one.\n\n## Sampling, when volume arrives\n\nAt low volume, keep everything. At high volume: keep every failure, keep every run that took an irreversible action, and keep a random sample (one in twenty is plenty) of ordinary successes.\n\nThe random sample is the part teams drop, and it is the part that tells you what normal looks like. Without it, you only ever read broken runs, and after a month you have a vivid and completely unrepresentative picture of how the system behaves.\n\n## What a trace cannot give you\n\nIt records what the model saw and what it did. It does not record why. A reasoning trace is text the model generated, not an account of its computation, and treating it as an explanation will mislead you: models produce plausible justifications for choices made on other grounds.\n\nSo use traces for the questions they can answer, which are the useful ones anyway: did it see the right thing, was the next action reasonable given only what was in front of it, and did the harness do what it was told."},"check":{"question":"An agent’s step limit is 20. Its step-per-run histogram shows a normal hump around six steps and a sharp spike at exactly 20, covering 12 per cent of runs. Users have not complained. What does this most likely mean?","options":[{"text":"Twelve per cent of tasks are genuinely more complex and are using the full budget as designed.","correct":false,"tempting":"Real complexity produces a spread across 15 to 20, not a spike exactly at the limit: a hard edge in a distribution is almost always the limit itself."},{"text":"The limit is set too low and should be raised until the spike disappears.","correct":false,"tempting":"Raising the limit moves the spike rather than diagnosing it; runs that need forty steps usually have a loop or a bad observation behind them."},{"text":"The absence of complaints indicates the truncated runs still produced acceptable answers.","correct":false,"tempting":"Treats silence as evidence of quality, when the whole point is that a truncated run is indistinguishable from a complete one in the output."},{"text":"Twelve per cent of runs are being cut off mid-task, and their final messages read as confident summaries of partial work, so nobody reports them.","correct":true}]},"prev":{"slug":"how-agents-fail","title":"The six ways a run goes wrong","url":"https://addaly.com/learn/agents-and-automation/how-agents-fail"},"next":{"slug":"reading-a-trace","title":"Reading a run, step by step","url":"https://addaly.com/learn/agents-and-automation/reading-a-trace"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'You cannot debug what you did not record', in Agents and Automation. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/agents-and-automation/observability-and-tracing (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"You cannot debug what you did not record","courseTitle":"Agents and Automation","module":{"index":4,"slug":"making-it-reliable","title":"Surviving real data"},"answer":null,"keyIdea":"A histogram of steps per run is the most informative agent metric there is: a spike at the step limit means a slice of your runs were cut off, and their final messages will look perfectly normal.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"type":"bars","unit":"% of runs","title":"Steps per run over one week, with the limit set at twenty","series":[{"label":"1 to 3 steps","value":9},{"label":"4 to 6","value":34},{"label":"7 to 9","value":27},{"label":"10 to 14","value":13},{"label":"15 to 19","value":5},{"label":"Exactly 20, the limit","value":12,"highlight":true}],"caption":"The hump in the middle is healthy. The bar at the limit is twelve per cent of runs cut off mid-task, each of which still wrote a confident final message saying nothing about being interrupted."},"alt":"Figure: Steps per run over one week, with the limit set at twenty. Bars (% of runs): 1 to 3 steps 9; 4 to 6 34; 7 to 9 27; 10 to 14 13; 15 to 19 5; Exactly 20, the limit 12. The hump in the middle is healthy. The bar at the limit is twelve per cent of runs cut off mid-task, each of which still wrote a confident final message saying nothing about being interrupted."}},"blocks":[{"t":"h2","id":"record-the-run-not-the-result","runs":[{"text":"Record the run, not the result"}]},{"t":"p","runs":[{"text":"The first time an agent does something inexplicable in production, you will want the exact message list it saw. If you did not store it, the run is gone: you cannot reproduce it, because the input has moved on and the model is stochastic."}]},{"t":"p","runs":[{"text":"So log, for every run:"}]},{"t":"ul","items":[[{"text":"A "},{"text":"run id","b":true},{"text":", generated at the start, and the trigger that started it."}],[{"text":"The "},{"text":"pinned model id","b":true},{"text":" and the "},{"text":"prompt version","b":true},{"text":". Not “GPT-class”: the exact string, because behaviour changes between versions and you will be comparing across weeks."}],[{"text":"Every message, in order, including the system prompt."}],[{"text":"Every tool call: name, arguments, result, duration, and whether it errored."}],[{"text":"Token counts","b":true},{"text":" per turn, input and output, plus a running total."}],[{"text":"The final status: succeeded, failed, hit the budget, stopped for a person."}],[{"text":"On whose behalf it ran."}]]},{"t":"p","runs":[{"text":"Truncate large tool results in the log and write the full payload to a file named by run id and step number. You want the trace readable and the evidence retrievable, which are different requirements."}]},{"t":"h2","id":"start-with-a-file-not-a-platform","runs":[{"text":"Start with a file, not a platform"}]},{"t":"p","runs":[{"text":"One JSON object per line, one file per run, and "},{"text":"jq","code":true},{"text":". This answers most questions on day one and costs nothing:"}]},{"t":"code","lang":"bash","label":"Bash","text":"# every run that hit an error from the payments tool, this week\njq -r 'select(.tool==\"issue_refund\" and .error!=null) | \"\\(.run) \\(.error)\"' runs/*.jsonl\n\n# the ten most expensive runs\njq -s 'map(select(.event==\"end\")) | sort_by(-.total_tokens)[:10]' runs/*.jsonl"},{"t":"p","runs":[{"text":"Move to a hosted tracing tool when the volume makes files awkward, not before. "},{"text":"OpenTelemetry","b":true},{"text":" gives you the vocabulary (a trace per run, a span per step, attributes on each) and free self-hosted backends like "},{"text":"Jaeger","b":true},{"text":" or "},{"text":"Grafana Tempo","b":true},{"text":" will render it. "},{"text":"Langfuse","b":true},{"text":" self-hosts and is built for this shape of data. All of them are improvements on a file; none of them is a prerequisite for starting."}]},{"t":"h2","id":"three-numbers-and-one-histogram","runs":[{"text":"Three numbers and one histogram"}]},{"t":"p","runs":[{"text":"Counters: runs started, runs succeeded, cost. Then the histogram that earns its place: "},{"text":"steps per run","b":true},{"text":"."}]},{"t":"p","runs":[{"text":"A healthy agent produces a distribution with a hump in the middle (most tasks take four to eight steps) and a thin tail. What you are looking for is a "},{"text":"spike at the maximum","b":true},{"text":". If your step limit is twenty and 12 per cent of runs use exactly twenty, then 12 per cent of runs were cut off mid-task."}]},{"t":"p","runs":[{"text":"The reason this matters more than any other metric is what those runs look like from outside. An agent stopped at its budget still writes a final message, and that message is usually a confident summary of partial work. Nothing in the output says “I was interrupted”. The step histogram is where it shows, and it shows immediately."}]},{"t":"p","runs":[{"text":"Add a latency histogram and a cost-per-run histogram beside it. Averages hide precisely the tail you need to see; a mean of nine seconds is compatible with one run in fifty taking four minutes."}]},{"t":"h2","id":"correlate-outward","runs":[{"text":"Correlate outward"}]},{"t":"p","runs":[{"text":"Pass the run id into every outbound call: as a header on HTTP requests, as a field on rows you write, in the subject line of tickets you create. When the CRM team asks who changed four hundred records on Tuesday, the answer is one query rather than a week of guessing. This costs one line in your HTTP client and repays it the first time anything goes wrong."}]},{"t":"h2","id":"logs-are-a-data-store","runs":[{"text":"Logs are a data store"}]},{"t":"p","runs":[{"text":"Whatever the agent read, your logs now hold. That includes the customer’s address, the contents of their email and whatever else came back from a tool. Treat the trace store with the same access rules as the database it drew from, and be specific about it:"}]},{"t":"ul","items":[[{"text":"Redact at write time, not at read time. Hash or truncate identifiers before they land."}],[{"text":"Set a retention period and enforce it with a scheduled job that actually deletes."}],[{"text":"Remember that a deletion request covers the trace store too. An agent’s log is a copy nobody thinks to include in the erasure path, and that omission is a real compliance failure rather than a theoretical one."}]]},{"t":"h2","id":"sampling-when-volume-arrives","runs":[{"text":"Sampling, when volume arrives"}]},{"t":"p","runs":[{"text":"At low volume, keep everything. At high volume: keep every failure, keep every run that took an irreversible action, and keep a random sample (one in twenty is plenty) of ordinary successes."}]},{"t":"p","runs":[{"text":"The random sample is the part teams drop, and it is the part that tells you what normal looks like. Without it, you only ever read broken runs, and after a month you have a vivid and completely unrepresentative picture of how the system behaves."}]},{"t":"h2","id":"what-a-trace-cannot-give-you","runs":[{"text":"What a trace cannot give you"}]},{"t":"p","runs":[{"text":"It records what the model saw and what it did. It does not record why. A reasoning trace is text the model generated, not an account of its computation, and treating it as an explanation will mislead you: models produce plausible justifications for choices made on other grounds."}]},{"t":"p","runs":[{"text":"So use traces for the questions they can answer, which are the useful ones anyway: did it see the right thing, was the next action reasonable given only what was in front of it, and did the harness do what it was told."}]}],"check":{"question":[{"text":"An agent’s step limit is 20. Its step-per-run histogram shows a normal hump around six steps and a sharp spike at exactly 20, covering 12 per cent of runs. Users have not complained. What does this most likely mean?"}],"options":[{"text":[{"text":"Twelve per cent of tasks are genuinely more complex and are using the full budget as designed."}],"correct":false,"tempting":[{"text":"Real complexity produces a spread across 15 to 20, not a spike exactly at the limit: a hard edge in a distribution is almost always the limit itself."}]},{"text":[{"text":"The limit is set too low and should be raised until the spike disappears."}],"correct":false,"tempting":[{"text":"Raising the limit moves the spike rather than diagnosing it; runs that need forty steps usually have a loop or a bad observation behind them."}]},{"text":[{"text":"The absence of complaints indicates the truncated runs still produced acceptable answers."}],"correct":false,"tempting":[{"text":"Treats silence as evidence of quality, when the whole point is that a truncated run is indistinguishable from a complete one in the output."}]},{"text":[{"text":"Twelve per cent of runs are being cut off mid-task, and their final messages read as confident summaries of partial work, so nobody reports them."}],"correct":true,"tempting":null}],"order":[0,3,2,1],"why":null},"prev":{"slug":"how-agents-fail","title":"The six ways a run goes wrong","minutes":9},"next":{"slug":"reading-a-trace","title":"Reading a run, step by step","minutes":9}}}