{"course":{"slug":"building-with-ai","title":"Building with AI","url":"https://addaly.com/learn/building-with-ai","level":"intermediate","track":"build"},"lesson":{"slug":"traces-you-can-debug-from","title":"Traces you can debug from","url":"https://addaly.com/learn/building-with-ai/traces-you-can-debug-from","position":58,"of":80,"minutes":9,"answer":"A wrong answer you cannot reproduce is diagnosable only from a trace that recorded, at the time, the prompt version, the retrieved ids and scores, every tool call with its arguments and result.","keyIdea":"A wrong answer you cannot reproduce is diagnosable only from a trace that recorded, at the time, the prompt version, the retrieved ids and scores, every tool call with its arguments and result, and the usage and latency per stage, and because that trace is a copy of the user's words, it needs a retention rule and a deletion path of its own."},"body":{"format":"markdown","text":"## The failure you cannot reproduce\n\nA user reports a wrong answer at three in the afternoon. You type the same question at four and the answer is right. Without a record of what happened at three, the investigation is over, and the bug will come back on its own schedule. With a record, you find that the index was rebuilt at ten to three and the answering chunk had a new id the cache did not know, or that a tool returned an error which the model paraphrased into a confident sentence. Neither of those is visible from the final text. Both are visible from a trace.\n\nThe agents module required the loop to emit traces; this lesson says what a trace must contain and what it costs to keep.\n\n## What a trace contains\n\nOne record per request, with a span per stage, each carrying its timing:\n\n- Request id, a pseudonymous user id, conversation id, timestamp.\n- Prompt template name and version, and either the rendered prompt or its hash plus the variables that filled it.\n- The retrieval query as searched, after rewriting, the retrieved chunk ids with their scores, and the reranker's scores if there was one.\n- Every tool call: name, arguments, result or error, duration.\n- Model identifier and parameters.\n- Usage: input tokens, output tokens, cached tokens.\n- Stop reason.\n- The response, and a link to any feedback it received.\n\nThe retrieval and tool spans are the ones people leave out and the ones that answer most questions. \"What did the model see?\" is the first question in every investigation, and a trace that recorded only the input and output cannot answer it.\n\n## Free tools, and no tool\n\nOpenTelemetry has semantic conventions for generative AI spans, so any tracing backend you already run can hold them. Langfuse self-hosts for free and understands prompts, versions and costs natively; Arize Phoenix is open source and good at retrieval traces. And a JSON document per request in a Postgres table, with an index on request id and timestamp, works to a few million requests before you need anything more. Start there if you have nothing; the schema above fits in one column.\n\n## Sampling at volume\n\nRecord the metadata (ids, versions, usage, latency, stop reason) for every request; it is small. Sample the bodies (rendered prompt, retrieved text, response) at some fraction, ten per cent perhaps, and always keep them in full for requests that errored, were rated down, tripped a safety filter or exceeded a latency or cost threshold. Tail-based sampling, deciding after the request completes, is what makes the interesting ones always present.\n\n## A trace store is a copy of every conversation\n\nHere is the part that gets a product into trouble. Traces contain users' words, verbatim, in a system that usually has none of the application's access controls and a retention policy nobody wrote. Set one: shorter than the conversation's own retention, thirty days is a common choice. Redact identifiers at write time: the same redaction the evaluation set required. Restrict access and log it. And make deletion propagate: when a user deletes a conversation, the derived-data trap from the uploads lesson applies here exactly, and the trace must go with it. If your users are in a jurisdiction with data-residency rules, the trace store is in scope.\n\nNever log secrets. Tool arguments can contain authentication tokens, card numbers, credentials; redact tool arguments by field name before they are written, not after.\n\n## Cost accounting comes free\n\nThe usage fields in every trace give the cost of every request. Roll them up by feature, by prompt version, by user, by day. \"Which prompt version is expensive?\" and \"which ten users are a third of the bill?\" are answered from traces with a query, and nowhere else.\n\n## From trace to case\n\nThe most valuable button in the tracing interface is \"add as case\". A trace of a failure, with its input and the retrieved context, becomes a regression case in the file from the start of this block in one action. That is the loop: production fails, the trace shows how, the case captures it, the harness prevents it returning. A tracing system that does not feed the case file is a log.\n\n## Reading one\n\nA user says the assistant gave the wrong refund window. The trace shows: query rewritten correctly; hybrid search retrieved the policy chunk at rank nine; the reranker moved it to rank six; five chunks were sent; the model answered from a chunk about a different plan. That is a retrieval failure (the right chunk was found and cut by the send limit) disguised as a generation failure, and the two-by-two from the retrieval-measurement lesson places it correctly only because the trace recorded the ranks. The fix is in the reranker or the chunk count, not the prompt, and it took two minutes to know that."},"check":{"question":"A user reports a wrong answer that the team cannot reproduce an hour later. Which trace field would most directly explain the difference?","options":[{"text":"The response text at the time, so it can be compared with the later correct one.","correct":false,"tempting":"Confirms that the answers differ, which is already known, and says nothing about why."},{"text":"The retrieved chunk ids and scores at the time, showing what the model was given before it answered.","correct":true},{"text":"The user's feedback record, since the thumbs-down category names the failure.","correct":false,"tempting":"A category such as \"wrong\" is a symptom label, not an explanation of what changed between the two requests."},{"text":"The model's temperature setting, since a non-zero value would explain why the same question produced a different answer an hour later.","correct":false,"tempting":"Sampling variation rarely turns a correct answer into a confidently wrong one; the retrieved context is the usual difference."}]},"prev":{"slug":"the-harness-in-ci","title":"The harness in CI, and why temperature zero is not deterministic","url":"https://addaly.com/learn/building-with-ai/the-harness-in-ci"},"next":{"slug":"error-analysis-as-a-weekly-habit","title":"Error analysis as a weekly habit","url":"https://addaly.com/learn/building-with-ai/error-analysis-as-a-weekly-habit"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Traces you can debug from', in Building with AI. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/building-with-ai/traces-you-can-debug-from (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Traces you can debug from","courseTitle":"Building with AI","module":{"index":6,"slug":"evaluating-and-iterating","title":"Evaluating and iterating"},"answer":null,"keyIdea":"A wrong answer you cannot reproduce is diagnosable only from a trace that recorded, at the time, the prompt version, the retrieved ids and scores, every tool call with its arguments and result, and the usage and latency per stage, and because that trace is a copy of the user’s words, it needs a retention rule and a deletion path of its own.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"the-failure-you-cannot-reproduce","runs":[{"text":"The failure you cannot reproduce"}]},{"t":"p","runs":[{"text":"A user reports a wrong answer at three in the afternoon. You type the same question at four and the answer is right. Without a record of what happened at three, the investigation is over, and the bug will come back on its own schedule. With a record, you find that the index was rebuilt at ten to three and the answering chunk had a new id the cache did not know, or that a tool returned an error which the model paraphrased into a confident sentence. Neither of those is visible from the final text. Both are visible from a trace."}]},{"t":"p","runs":[{"text":"The agents module required the loop to emit traces; this lesson says what a trace must contain and what it costs to keep."}]},{"t":"h2","id":"what-a-trace-contains","runs":[{"text":"What a trace contains"}]},{"t":"p","runs":[{"text":"One record per request, with a span per stage, each carrying its timing:"}]},{"t":"ul","items":[[{"text":"Request id, a pseudonymous user id, conversation id, timestamp."}],[{"text":"Prompt template name and version, and either the rendered prompt or its hash plus the variables that filled it."}],[{"text":"The retrieval query as searched, after rewriting, the retrieved chunk ids with their scores, and the reranker’s scores if there was one."}],[{"text":"Every tool call: name, arguments, result or error, duration."}],[{"text":"Model identifier and parameters."}],[{"text":"Usage: input tokens, output tokens, cached tokens."}],[{"text":"Stop reason."}],[{"text":"The response, and a link to any feedback it received."}]]},{"t":"p","runs":[{"text":"The retrieval and tool spans are the ones people leave out and the ones that answer most questions. “What did the model see?” is the first question in every investigation, and a trace that recorded only the input and output cannot answer it."}]},{"t":"h2","id":"free-tools-and-no-tool","runs":[{"text":"Free tools, and no tool"}]},{"t":"p","runs":[{"text":"OpenTelemetry has semantic conventions for generative AI spans, so any tracing backend you already run can hold them. Langfuse self-hosts for free and understands prompts, versions and costs natively; Arize Phoenix is open source and good at retrieval traces. And a JSON document per request in a Postgres table, with an index on request id and timestamp, works to a few million requests before you need anything more. Start there if you have nothing; the schema above fits in one column."}]},{"t":"h2","id":"sampling-at-volume","runs":[{"text":"Sampling at volume"}]},{"t":"p","runs":[{"text":"Record the metadata (ids, versions, usage, latency, stop reason) for every request; it is small. Sample the bodies (rendered prompt, retrieved text, response) at some fraction, ten per cent perhaps, and always keep them in full for requests that errored, were rated down, tripped a safety filter or exceeded a latency or cost threshold. Tail-based sampling, deciding after the request completes, is what makes the interesting ones always present."}]},{"t":"h2","id":"a-trace-store-is-a-copy-of-every-conversation","runs":[{"text":"A trace store is a copy of every conversation"}]},{"t":"p","runs":[{"text":"Here is the part that gets a product into trouble. Traces contain users’ words, verbatim, in a system that usually has none of the application’s access controls and a retention policy nobody wrote. Set one: shorter than the conversation’s own retention, thirty days is a common choice. Redact identifiers at write time: the same redaction the evaluation set required. Restrict access and log it. And make deletion propagate: when a user deletes a conversation, the derived-data trap from the uploads lesson applies here exactly, and the trace must go with it. If your users are in a jurisdiction with data-residency rules, the trace store is in scope."}]},{"t":"p","runs":[{"text":"Never log secrets. Tool arguments can contain authentication tokens, card numbers, credentials; redact tool arguments by field name before they are written, not after."}]},{"t":"h2","id":"cost-accounting-comes-free","runs":[{"text":"Cost accounting comes free"}]},{"t":"p","runs":[{"text":"The usage fields in every trace give the cost of every request. Roll them up by feature, by prompt version, by user, by day. “Which prompt version is expensive?” and “which ten users are a third of the bill?” are answered from traces with a query, and nowhere else."}]},{"t":"h2","id":"from-trace-to-case","runs":[{"text":"From trace to case"}]},{"t":"p","runs":[{"text":"The most valuable button in the tracing interface is “add as case”. A trace of a failure, with its input and the retrieved context, becomes a regression case in the file from the start of this block in one action. That is the loop: production fails, the trace shows how, the case captures it, the harness prevents it returning. A tracing system that does not feed the case file is a log."}]},{"t":"h2","id":"reading-one","runs":[{"text":"Reading one"}]},{"t":"p","runs":[{"text":"A user says the assistant gave the wrong refund window. The trace shows: query rewritten correctly; hybrid search retrieved the policy chunk at rank nine; the reranker moved it to rank six; five chunks were sent; the model answered from a chunk about a different plan. That is a retrieval failure (the right chunk was found and cut by the send limit) disguised as a generation failure, and the two-by-two from the retrieval-measurement lesson places it correctly only because the trace recorded the ranks. The fix is in the reranker or the chunk count, not the prompt, and it took two minutes to know that."}]}],"check":{"question":[{"text":"A user reports a wrong answer that the team cannot reproduce an hour later. Which trace field would most directly explain the difference?"}],"options":[{"text":[{"text":"The response text at the time, so it can be compared with the later correct one."}],"correct":false,"tempting":[{"text":"Confirms that the answers differ, which is already known, and says nothing about why."}]},{"text":[{"text":"The retrieved chunk ids and scores at the time, showing what the model was given before it answered."}],"correct":true,"tempting":null},{"text":[{"text":"The user’s feedback record, since the thumbs-down category names the failure."}],"correct":false,"tempting":[{"text":"A category such as “wrong” is a symptom label, not an explanation of what changed between the two requests."}]},{"text":[{"text":"The model’s temperature setting, since a non-zero value would explain why the same question produced a different answer an hour later."}],"correct":false,"tempting":[{"text":"Sampling variation rarely turns a correct answer into a confidently wrong one; the retrieved context is the usual difference."}]}],"order":[2,1,3,0],"why":null},"prev":{"slug":"the-harness-in-ci","title":"The harness in CI, and why temperature zero is not deterministic","minutes":9},"next":{"slug":"error-analysis-as-a-weekly-habit","title":"Error analysis as a weekly habit","minutes":8}}}