{"course":{"slug":"evaluating-ai","title":"Knowing If It Works","url":"https://addaly.com/learn/evaluating-ai","level":"intermediate","track":"build"},"lesson":{"slug":"evaluating-conversation","title":"Multi-turn: score the session, not the turn","url":"https://addaly.com/learn/evaluating-ai/evaluating-conversation","position":34,"of":76,"minutes":9,"answer":"One mistake at turn two corrupts every turn after it, so measure task success per session and record the turn where it first went wrong, then test retention, correction and truncation deliberately.","keyIdea":"One mistake at turn two corrupts every turn after it, so measure task success per session and record the turn where it first went wrong, then test retention, correction and truncation deliberately."},"body":{"format":"markdown","text":"## Why per-turn averages mislead\n\nAn assistant misunderstands at turn two and everything after that is built on the misunderstanding. Score the ten turns individually and you record eight failures from one mistake, in a set where the ten rows are anything but independent: exactly the clustering problem from the statistics block, in its most common disguise.\n\nTwo consequences. The number is wrong: long broken sessions dominate the average. And the diagnosis is wrong: it looks like a widespread quality problem rather than one specific breakdown.\n\nScore the session. Then record, for each failed session, **the turn where it first went wrong**. The distribution of that single field is the most useful diagnostic in conversational evaluation. A pile-up at turn one is a comprehension or routing problem. A pile-up around turn twenty is a context-window problem. A flat spread is a general quality problem, and only then does the average mean anything.\n\n> Figure: One mistake, eight failed turns. Timeline: Turn 1, Customer asks, in Hindi, for a refund on order A-8891.; Turn 2, Assistant looks up order A-8819 instead. Nobody notices yet.; Turns 3 to 8, Every reply is correct about the wrong order: amount, date, policy window.; Turn 9, Customer escalates. Per-turn scoring records eight failures; session scoring records one, first wrong at turn two.. The distribution of 'first wrong turn' across failed sessions is the most useful diagnostic there is: a pile-up at turn one is routing or comprehension, a pile-up around turn twenty is the context window.\n\n## Define success as an outcome, not a feeling\n\n\"Was the reply good\" is unanswerable and unstable. \"Did the user end up with a refund initiated, and was the amount correct\" is answerable by reading the transcript or, better, by looking at whether the refund event fired in your product.\n\nWhere the outcome exists as an event in your system, use it: it costs nothing and cannot be argued with. Where it does not, write a session-level rubric with three or four binary checks, and apply it to the whole transcript.\n\n## Five failures worth testing on purpose\n\nScripted probes find these; random sampling finds them slowly and expensively.\n\n- **Instruction retention.** The user says \"reply in Hindi\" at turn one. Check turn seven. Also standing constraints: \"never mention competitors\", \"always give the price in rupees\".\n- **Correction handling.** The user changes their mind mid-conversation: \"actually, make it Thursday\". A large share of real failures are systems that keep using the first value. Build twenty sessions where turn four contradicts turn two, and check which value survives.\n- **Context truncation.** Plant a fact at turn one, run the conversation past the point where history is trimmed, then ask about the fact. You will discover what your truncation strategy actually drops, which is rarely what the code comment claims.\n- **Loops and repetition.** The same reply twice in a row is a distinct failure. It is detectable with a similarity check over consecutive turns, no model needed.\n- **Escalation and refusal at the wrong moment.** Measure how often the system hands off, and sample those transcripts. A rising handoff rate can look like caution and be a regression.\n\n## Simulated users, and their known bias\n\nTo get hundreds of sessions you need a simulated user: a persona, a goal, a stopping rule, and a model playing the part. It works, it is cheap, and it is systematically too easy.\n\nSimulated users are cooperative. They answer the question asked, they do not go silent for four minutes, they rarely misspell, they do not paste a screenshot, they do not change goal halfway. The result is a task-success rate several points above reality.\n\nCalibrate it. Run twenty real transcripts through the same session rubric and compare. If simulated success is 88% and real success is 71%, the simulator is a regression detector (good for comparing versions) and not a source of headline numbers. Write both numbers in the report.\n\n## The rest of the numbers\n\nAlongside task success, three cheap session-level measures usually earn their place:\n\n- **Turns to success.** A system that gets there in three turns beats one that gets there in nine, and the average conceals it.\n- **Escalation rate**, split by whether the escalation was appropriate.\n- **Cost and latency per session**, not per call. A session-level view is what finance and users both experience, and a change that halves per-call cost while doubling the number of turns has made things worse.\n\n## What stays contested\n\nWhether a reply is \"warm enough\", whether an assistant should push back, how much hedging is honest rather than annoying: these are genuine disagreements about what good means, not measurement failures. Report them as a stated position with examples rather than as a score, and revisit the position deliberately. The last block of this course returns to the question of what to do when reasonable people want different outputs."},"check":{"question":"A support assistant is scored by rating every turn independently. Sessions average 9 turns and the reported quality is 63%. Why is that figure likely to understate the system?","options":[{"text":"Because turn-level ratings are harsher than session-level ones, as raters see less context per rating.","correct":false,"tempting":"Rater context is a real effect but works in both directions, and it does not explain the specific arithmetic distortion here."},{"text":"Because a single early failure makes every later turn in that session wrong, so one breakdown is counted eight or nine times.","correct":true},{"text":"Because turn-level scoring counts the system's clarifying questions as failures.","correct":false,"tempting":"A rubric design flaw that some teams do have, but it is a definition problem rather than the structural double-counting the design creates."},{"text":"Because sessions of different lengths are averaged together without weighting.","correct":false,"tempting":"Length weighting does distort the mean, though the dominant effect is the correlation between turns after a failure rather than unequal session lengths."}]},"prev":{"slug":"evaluating-extraction","title":"Structured extraction, where normalisation is most of the work","url":"https://addaly.com/learn/evaluating-ai/evaluating-extraction"},"next":{"slug":"evaluating-search-and-recommendation","title":"Ranking systems, where the labels are clicks and the clicks are biased","url":"https://addaly.com/learn/evaluating-ai/evaluating-search-and-recommendation"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Multi-turn: score the session, not the turn', in Knowing If It Works. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/evaluating-ai/evaluating-conversation (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Multi-turn: score the session, not the turn","courseTitle":"Knowing If It Works","module":{"index":4,"slug":"task-shapes","title":"The systems you will actually be handed"},"answer":null,"keyIdea":"One mistake at turn two corrupts every turn after it, so measure task success per session and record the turn where it first went wrong, then test retention, correction and truncation deliberately.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"type":"timeline","title":"One mistake, eight failed turns","events":[{"when":"Turn 1","what":"Customer asks, in Hindi, for a refund on order A-8891."},{"when":"Turn 2","what":"Assistant looks up order A-8819 instead. Nobody notices yet."},{"when":"Turns 3 to 8","what":"Every reply is correct about the wrong order: amount, date, policy window."},{"when":"Turn 9","what":"Customer escalates. Per-turn scoring records eight failures; session scoring records one, first wrong at turn two."}],"caption":"The distribution of ‘first wrong turn’ across failed sessions is the most useful diagnostic there is: a pile-up at turn one is routing or comprehension, a pile-up around turn twenty is the context window."},"alt":"Figure: One mistake, eight failed turns. Timeline: Turn 1, Customer asks, in Hindi, for a refund on order A-8891.; Turn 2, Assistant looks up order A-8819 instead. Nobody notices yet.; Turns 3 to 8, Every reply is correct about the wrong order: amount, date, policy window.; Turn 9, Customer escalates. Per-turn scoring records eight failures; session scoring records one, first wrong at turn two.. The distribution of ‘first wrong turn’ across failed sessions is the most useful diagnostic there is: a pile-up at turn one is routing or comprehension, a pile-up around turn twenty is the context window."}},"blocks":[{"t":"h2","id":"why-per-turn-averages-mislead","runs":[{"text":"Why per-turn averages mislead"}]},{"t":"p","runs":[{"text":"An assistant misunderstands at turn two and everything after that is built on the misunderstanding. Score the ten turns individually and you record eight failures from one mistake, in a set where the ten rows are anything but independent: exactly the clustering problem from the statistics block, in its most common disguise."}]},{"t":"p","runs":[{"text":"Two consequences. The number is wrong: long broken sessions dominate the average. And the diagnosis is wrong: it looks like a widespread quality problem rather than one specific breakdown."}]},{"t":"p","runs":[{"text":"Score the session. Then record, for each failed session, "},{"text":"the turn where it first went wrong","b":true},{"text":". The distribution of that single field is the most useful diagnostic in conversational evaluation. A pile-up at turn one is a comprehension or routing problem. A pile-up around turn twenty is a context-window problem. A flat spread is a general quality problem, and only then does the average mean anything."}]},{"t":"h2","id":"define-success-as-an-outcome-not-a-feeling","runs":[{"text":"Define success as an outcome, not a feeling"}]},{"t":"p","runs":[{"text":"“Was the reply good” is unanswerable and unstable. “Did the user end up with a refund initiated, and was the amount correct” is answerable by reading the transcript or, better, by looking at whether the refund event fired in your product."}]},{"t":"p","runs":[{"text":"Where the outcome exists as an event in your system, use it: it costs nothing and cannot be argued with. Where it does not, write a session-level rubric with three or four binary checks, and apply it to the whole transcript."}]},{"t":"h2","id":"five-failures-worth-testing-on-purpose","runs":[{"text":"Five failures worth testing on purpose"}]},{"t":"p","runs":[{"text":"Scripted probes find these; random sampling finds them slowly and expensively."}]},{"t":"ul","items":[[{"text":"Instruction retention.","b":true},{"text":" The user says “reply in Hindi” at turn one. Check turn seven. Also standing constraints: “never mention competitors”, “always give the price in rupees”."}],[{"text":"Correction handling.","b":true},{"text":" The user changes their mind mid-conversation: “actually, make it Thursday”. A large share of real failures are systems that keep using the first value. Build twenty sessions where turn four contradicts turn two, and check which value survives."}],[{"text":"Context truncation.","b":true},{"text":" Plant a fact at turn one, run the conversation past the point where history is trimmed, then ask about the fact. You will discover what your truncation strategy actually drops, which is rarely what the code comment claims."}],[{"text":"Loops and repetition.","b":true},{"text":" The same reply twice in a row is a distinct failure. It is detectable with a similarity check over consecutive turns, no model needed."}],[{"text":"Escalation and refusal at the wrong moment.","b":true},{"text":" Measure how often the system hands off, and sample those transcripts. A rising handoff rate can look like caution and be a regression."}]]},{"t":"h2","id":"simulated-users-and-their-known-bias","runs":[{"text":"Simulated users, and their known bias"}]},{"t":"p","runs":[{"text":"To get hundreds of sessions you need a simulated user: a persona, a goal, a stopping rule, and a model playing the part. It works, it is cheap, and it is systematically too easy."}]},{"t":"p","runs":[{"text":"Simulated users are cooperative. They answer the question asked, they do not go silent for four minutes, they rarely misspell, they do not paste a screenshot, they do not change goal halfway. The result is a task-success rate several points above reality."}]},{"t":"p","runs":[{"text":"Calibrate it. Run twenty real transcripts through the same session rubric and compare. If simulated success is 88% and real success is 71%, the simulator is a regression detector (good for comparing versions) and not a source of headline numbers. Write both numbers in the report."}]},{"t":"h2","id":"the-rest-of-the-numbers","runs":[{"text":"The rest of the numbers"}]},{"t":"p","runs":[{"text":"Alongside task success, three cheap session-level measures usually earn their place:"}]},{"t":"ul","items":[[{"text":"Turns to success.","b":true},{"text":" A system that gets there in three turns beats one that gets there in nine, and the average conceals it."}],[{"text":"Escalation rate","b":true},{"text":", split by whether the escalation was appropriate."}],[{"text":"Cost and latency per session","b":true},{"text":", not per call. A session-level view is what finance and users both experience, and a change that halves per-call cost while doubling the number of turns has made things worse."}]]},{"t":"h2","id":"what-stays-contested","runs":[{"text":"What stays contested"}]},{"t":"p","runs":[{"text":"Whether a reply is “warm enough”, whether an assistant should push back, how much hedging is honest rather than annoying: these are genuine disagreements about what good means, not measurement failures. Report them as a stated position with examples rather than as a score, and revisit the position deliberately. The last block of this course returns to the question of what to do when reasonable people want different outputs."}]}],"check":{"question":[{"text":"A support assistant is scored by rating every turn independently. Sessions average 9 turns and the reported quality is 63%. Why is that figure likely to understate the system?"}],"options":[{"text":[{"text":"Because turn-level ratings are harsher than session-level ones, as raters see less context per rating."}],"correct":false,"tempting":[{"text":"Rater context is a real effect but works in both directions, and it does not explain the specific arithmetic distortion here."}]},{"text":[{"text":"Because a single early failure makes every later turn in that session wrong, so one breakdown is counted eight or nine times."}],"correct":true,"tempting":null},{"text":[{"text":"Because turn-level scoring counts the system’s clarifying questions as failures."}],"correct":false,"tempting":[{"text":"A rubric design flaw that some teams do have, but it is a definition problem rather than the structural double-counting the design creates."}]},{"text":[{"text":"Because sessions of different lengths are averaged together without weighting."}],"correct":false,"tempting":[{"text":"Length weighting does distort the mean, though the dominant effect is the correlation between turns after a failure rather than unequal session lengths."}]}],"order":[0,3,2,1],"why":null},"prev":{"slug":"evaluating-extraction","title":"Structured extraction, where normalisation is most of the work","minutes":9},"next":{"slug":"evaluating-search-and-recommendation","title":"Ranking systems, where the labels are clicks and the clicks are biased","minutes":10}}}