{"course":{"slug":"evaluating-ai","title":"Knowing If It Works","url":"https://addaly.com/learn/evaluating-ai","level":"intermediate","track":"build"},"lesson":{"slug":"tracing-and-logging","title":"Recording enough to diagnose a failure you did not see happen","url":"https://addaly.com/learn/evaluating-ai/tracing-and-logging","position":52,"of":76,"minutes":9,"answer":"Log the whole trace (the exact prompt sent, the retrieved chunks, the tool calls, the versions.","keyIdea":"Log the whole trace (the exact prompt sent, the retrieved chunks, the tool calls, the versions, the timings and the user's next action) because every production failure is diagnosed from what was recorded and never from what you can reproduce."},"body":{"format":"markdown","text":"## The complaint arrives without its context\n\nA user says the assistant gave them the wrong refund window on Tuesday. You cannot reproduce it: the corpus has changed, the prompt has changed, and you do not know what they typed. Whether that complaint becomes a fix or a shrug was decided weeks earlier, by what the system records.\n\n## The minimum useful record\n\nPer request, one trace, containing:\n\n- **Identity and time**: request id, timestamp, session id, a hashed user id, and the surface it came from.\n- **The exact prompt sent to the model.** Not the template. The rendered string, or the message array. This is the single most valuable field and the one most often missing.\n- **Versions**: model id including the dated variant, prompt template version, rubric version, retriever and index version, application build.\n- **Parameters**: temperature, top-p, max tokens, tool schema version, seed if you set one.\n- **Retrieval**: chunk ids, their scores, and either the chunk text or a content hash pointing at an immutable store.\n- **Tool calls**: the name, the arguments, the result, the duration, the errors.\n- **The output**, plus token counts in and out, cost, and the finish reason. A truncation is invisible without the finish reason and explains a startling share of \"the answer just stops\" reports.\n- **Timings per span**: retrieval, generation, judging, post-processing.\n- **What the user did next**: regenerated, edited, copied, escalated, abandoned. This is the free quality signal, and it only exists if you attach it to the trace id.\n\n## Spans turn \"it was slow\" into a fix\n\nStructure the record as a trace with nested spans rather than one flat row. A user complaint of slowness becomes \"retrieval 2.1 s of 2.6 s total, of which 1.8 s was the embedding call\", which is a specific thing to fix in an afternoon. Without spans it is a week of guessing.\n\nOpenTelemetry has conventions for this and most tooling speaks it. Self-hosted open-source options exist and are worth the day it takes to run one. And if that is more than you want to operate: a Postgres table with a `jsonb` column, a few indexed fields and a retention job is sufficient for a small product. The failure is not choosing the wrong tool; it is having nothing.\n\n## Sample the trace, not the fields\n\nFull traces are large (context, chunks, outputs) and storage can exceed inference cost if you keep everything for a year. Sample deliberately:\n\n- 100% of errors, timeouts and guardrail hits.\n- 100% of sessions where the user escalated, edited heavily, or reported a problem.\n- A fixed percentage of everything else, chosen so the sampled stream supports your monitoring arithmetic.\n\nMake the sampling decision once per trace and propagate it, so you never end up with a request whose retrieval span was kept and whose generation span was dropped.\n\n## Logs of user text are your most sensitive store\n\nTreat them accordingly, because they are exactly what a breach report would be about.\n\n- Redact identifiers at ingestion, and accept that automatic redaction is imperfect: it will miss the sentence where somebody names their employer.\n- Set a retention period and enforce it with a job, not a policy document.\n- Restrict access and log the access.\n- Keep the legal basis straight: what your privacy notice promised about using conversations to improve the product is what governs whether this data may become an evaluation set at all.\n\nWhere policy is strict, log hashes and structure rather than content: chunk ids instead of chunk text, a redacted skeleton instead of the message, lengths and categories instead of the string. It is a real reduction in diagnostic power, and it is better than a blanket prohibition that leads people to keep copies elsewhere.\n\n## The button that pays for the whole system\n\nEvery trace is a potential evaluation item. Build the one-click path: from a trace in the viewer, create an eval item with the input, the retrieved context, the produced output, and a field for what the answer should have been.\n\nThat single affordance is what makes the incident-to-item habit in the last block practical rather than aspirational. Without it, adding a real failure to the set means twenty minutes of copying, and it does not happen."},"check":{"question":"A team logs the prompt template id and the variables, but not the rendered prompt. What goes wrong first?","options":[{"text":"Storage costs rise, because templates and variables together are larger than the rendered string.","correct":false,"tempting":"Reverses the sizes: template plus variables is smaller, which is part of why teams choose it."},{"text":"A failure cannot be reproduced once the template has changed, because the recorded id now renders differently or no longer exists.","correct":true},{"text":"Token counts become impossible to compute, since they depend on the rendered text.","correct":false,"tempting":"Token counts are normally returned by the provider and logged directly, so they survive this choice."},{"text":"The judge cannot be run retrospectively on the logged output, since it needs the template.","correct":false,"tempting":"A retrospective judge needs the input, context and output rather than the template, so this is not the first thing to break."}]},"prev":{"slug":"evals-in-ci","title":"An evaluation that runs on every change without going red at random","url":"https://addaly.com/learn/evaluating-ai/evals-in-ci"},"next":{"slug":"regression-testing-a-prompt","title":"Regression testing a prompt","url":"https://addaly.com/learn/evaluating-ai/regression-testing-a-prompt"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Recording enough to diagnose a failure you did not see happen', in Knowing If It Works. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/evaluating-ai/tracing-and-logging (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Recording enough to diagnose a failure you did not see happen","courseTitle":"Knowing If It Works","module":{"index":6,"slug":"evaluation-in-the-loop","title":"Evaluation in the working loop"},"answer":null,"keyIdea":"Log the whole trace (the exact prompt sent, the retrieved chunks, the tool calls, the versions, the timings and the user’s next action) because every production failure is diagnosed from what was recorded and never from what you can reproduce.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"the-complaint-arrives-without-its-context","runs":[{"text":"The complaint arrives without its context"}]},{"t":"p","runs":[{"text":"A user says the assistant gave them the wrong refund window on Tuesday. You cannot reproduce it: the corpus has changed, the prompt has changed, and you do not know what they typed. Whether that complaint becomes a fix or a shrug was decided weeks earlier, by what the system records."}]},{"t":"h2","id":"the-minimum-useful-record","runs":[{"text":"The minimum useful record"}]},{"t":"p","runs":[{"text":"Per request, one trace, containing:"}]},{"t":"ul","items":[[{"text":"Identity and time","b":true},{"text":": request id, timestamp, session id, a hashed user id, and the surface it came from."}],[{"text":"The exact prompt sent to the model.","b":true},{"text":" Not the template. The rendered string, or the message array. This is the single most valuable field and the one most often missing."}],[{"text":"Versions","b":true},{"text":": model id including the dated variant, prompt template version, rubric version, retriever and index version, application build."}],[{"text":"Parameters","b":true},{"text":": temperature, top-p, max tokens, tool schema version, seed if you set one."}],[{"text":"Retrieval","b":true},{"text":": chunk ids, their scores, and either the chunk text or a content hash pointing at an immutable store."}],[{"text":"Tool calls","b":true},{"text":": the name, the arguments, the result, the duration, the errors."}],[{"text":"The output","b":true},{"text":", plus token counts in and out, cost, and the finish reason. A truncation is invisible without the finish reason and explains a startling share of “the answer just stops” reports."}],[{"text":"Timings per span","b":true},{"text":": retrieval, generation, judging, post-processing."}],[{"text":"What the user did next","b":true},{"text":": regenerated, edited, copied, escalated, abandoned. This is the free quality signal, and it only exists if you attach it to the trace id."}]]},{"t":"h2","id":"spans-turn-it-was-slow-into-a-fix","runs":[{"text":"Spans turn “it was slow” into a fix"}]},{"t":"p","runs":[{"text":"Structure the record as a trace with nested spans rather than one flat row. A user complaint of slowness becomes “retrieval 2.1 s of 2.6 s total, of which 1.8 s was the embedding call”, which is a specific thing to fix in an afternoon. Without spans it is a week of guessing."}]},{"t":"p","runs":[{"text":"OpenTelemetry has conventions for this and most tooling speaks it. Self-hosted open-source options exist and are worth the day it takes to run one. And if that is more than you want to operate: a Postgres table with a "},{"text":"jsonb","code":true},{"text":" column, a few indexed fields and a retention job is sufficient for a small product. The failure is not choosing the wrong tool; it is having nothing."}]},{"t":"h2","id":"sample-the-trace-not-the-fields","runs":[{"text":"Sample the trace, not the fields"}]},{"t":"p","runs":[{"text":"Full traces are large (context, chunks, outputs) and storage can exceed inference cost if you keep everything for a year. Sample deliberately:"}]},{"t":"ul","items":[[{"text":"100% of errors, timeouts and guardrail hits."}],[{"text":"100% of sessions where the user escalated, edited heavily, or reported a problem."}],[{"text":"A fixed percentage of everything else, chosen so the sampled stream supports your monitoring arithmetic."}]]},{"t":"p","runs":[{"text":"Make the sampling decision once per trace and propagate it, so you never end up with a request whose retrieval span was kept and whose generation span was dropped."}]},{"t":"h2","id":"logs-of-user-text-are-your-most-sensitive-store","runs":[{"text":"Logs of user text are your most sensitive store"}]},{"t":"p","runs":[{"text":"Treat them accordingly, because they are exactly what a breach report would be about."}]},{"t":"ul","items":[[{"text":"Redact identifiers at ingestion, and accept that automatic redaction is imperfect: it will miss the sentence where somebody names their employer."}],[{"text":"Set a retention period and enforce it with a job, not a policy document."}],[{"text":"Restrict access and log the access."}],[{"text":"Keep the legal basis straight: what your privacy notice promised about using conversations to improve the product is what governs whether this data may become an evaluation set at all."}]]},{"t":"p","runs":[{"text":"Where policy is strict, log hashes and structure rather than content: chunk ids instead of chunk text, a redacted skeleton instead of the message, lengths and categories instead of the string. It is a real reduction in diagnostic power, and it is better than a blanket prohibition that leads people to keep copies elsewhere."}]},{"t":"h2","id":"the-button-that-pays-for-the-whole-system","runs":[{"text":"The button that pays for the whole system"}]},{"t":"p","runs":[{"text":"Every trace is a potential evaluation item. Build the one-click path: from a trace in the viewer, create an eval item with the input, the retrieved context, the produced output, and a field for what the answer should have been."}]},{"t":"p","runs":[{"text":"That single affordance is what makes the incident-to-item habit in the last block practical rather than aspirational. Without it, adding a real failure to the set means twenty minutes of copying, and it does not happen."}]}],"check":{"question":[{"text":"A team logs the prompt template id and the variables, but not the rendered prompt. What goes wrong first?"}],"options":[{"text":[{"text":"Storage costs rise, because templates and variables together are larger than the rendered string."}],"correct":false,"tempting":[{"text":"Reverses the sizes: template plus variables is smaller, which is part of why teams choose it."}]},{"text":[{"text":"A failure cannot be reproduced once the template has changed, because the recorded id now renders differently or no longer exists."}],"correct":true,"tempting":null},{"text":[{"text":"Token counts become impossible to compute, since they depend on the rendered text."}],"correct":false,"tempting":[{"text":"Token counts are normally returned by the provider and logged directly, so they survive this choice."}]},{"text":[{"text":"The judge cannot be run retrospectively on the logged output, since it needs the template."}],"correct":false,"tempting":[{"text":"A retrospective judge needs the input, context and output rather than the template, so this is not the first thing to break."}]}],"order":[0,2,3,1],"why":null},"prev":{"slug":"evals-in-ci","title":"An evaluation that runs on every change without going red at random","minutes":9},"next":{"slug":"regression-testing-a-prompt","title":"Regression testing a prompt","minutes":8}}}