Multi-turn: score the session, not the turn

Why per-turn averages mislead

An assistant misunderstands at turn two and everything after that is built on the misunderstanding. Score the ten turns individually and you record eight failures from one mistake, in a set where the ten rows are anything but independent: exactly the clustering problem from the statistics block, in its most common disguise.

Two consequences. The number is wrong: long broken sessions dominate the average. And the diagnosis is wrong: it looks like a widespread quality problem rather than one specific breakdown.

Score the session. Then record, for each failed session, the turn where it first went wrong. The distribution of that single field is the most useful diagnostic in conversational evaluation. A pile-up at turn one is a comprehension or routing problem. A pile-up around turn twenty is a context-window problem. A flat spread is a general quality problem, and only then does the average mean anything.

Define success as an outcome, not a feeling

“Was the reply good” is unanswerable and unstable. “Did the user end up with a refund initiated, and was the amount correct” is answerable by reading the transcript or, better, by looking at whether the refund event fired in your product.

Where the outcome exists as an event in your system, use it: it costs nothing and cannot be argued with. Where it does not, write a session-level rubric with three or four binary checks, and apply it to the whole transcript.

Five failures worth testing on purpose

Scripted probes find these; random sampling finds them slowly and expensively.

• Instruction retention. The user says “reply in Hindi” at turn one. Check turn seven. Also standing constraints: “never mention competitors”, “always give the price in rupees”.
• Correction handling. The user changes their mind mid-conversation: “actually, make it Thursday”. A large share of real failures are systems that keep using the first value. Build twenty sessions where turn four contradicts turn two, and check which value survives.
• Context truncation. Plant a fact at turn one, run the conversation past the point where history is trimmed, then ask about the fact. You will discover what your truncation strategy actually drops, which is rarely what the code comment claims.
• Loops and repetition. The same reply twice in a row is a distinct failure. It is detectable with a similarity check over consecutive turns, no model needed.
• Escalation and refusal at the wrong moment. Measure how often the system hands off, and sample those transcripts. A rising handoff rate can look like caution and be a regression.

Simulated users, and their known bias

To get hundreds of sessions you need a simulated user: a persona, a goal, a stopping rule, and a model playing the part. It works, it is cheap, and it is systematically too easy.

Simulated users are cooperative. They answer the question asked, they do not go silent for four minutes, they rarely misspell, they do not paste a screenshot, they do not change goal halfway. The result is a task-success rate several points above reality.

Calibrate it. Run twenty real transcripts through the same session rubric and compare. If simulated success is 88% and real success is 71%, the simulator is a regression detector (good for comparing versions) and not a source of headline numbers. Write both numbers in the report.

The rest of the numbers

Alongside task success, three cheap session-level measures usually earn their place:

• Turns to success. A system that gets there in three turns beats one that gets there in nine, and the average conceals it.
• Escalation rate, split by whether the escalation was appropriate.
• Cost and latency per session, not per call. A session-level view is what finance and users both experience, and a change that halves per-call cost while doubling the number of turns has made things worse.

What stays contested

Whether a reply is “warm enough”, whether an assistant should push back, how much hedging is honest rather than annoying: these are genuine disagreements about what good means, not measurement failures. Report them as a stated position with examples rather than as a score, and revisit the position deliberately. The last block of this course returns to the question of what to do when reasonable people want different outputs.