{"course":{"slug":"evaluating-ai","title":"Knowing If It Works","url":"https://addaly.com/learn/evaluating-ai","level":"intermediate","track":"build"},"lesson":{"slug":"online-metrics-and-proxies","title":"The signals users give you without being asked","url":"https://addaly.com/learn/evaluating-ai/online-metrics-and-proxies","position":57,"of":76,"minutes":9,"answer":"Regeneration rate, edit distance and escalation are free quality proxies you probably already log, but they only become metrics once you have checked them against human labels on a sample.","keyIdea":"Regeneration rate, edit distance and escalation are free quality proxies you probably already log, but they only become metrics once you have checked them against human labels on a sample."},"body":{"format":"markdown","text":"## Your offline set cannot see everything\n\nA fixed set of a hundred items measures the inputs you collected, against the criteria you wrote, at the moment you wrote them. It cannot see a question nobody has asked yet, a use you did not anticipate, or the slow shift in what people bring to the product. Production can.\n\nThe trouble is that production has no labels. So you use proxies: behaviours that correlate with quality and cost nothing to collect.\n\n## Implicit signals, best first\n\n**Regeneration or retry rate.** The fraction of responses where the user immediately asks again, rephrases, or hits regenerate. This is the single best free signal most products already have in their logs and do not look at. It needs no survey and no annotation, it is unambiguous, and it moves quickly when quality moves.\n\n**Edit distance between suggestion and what shipped.** For anything that drafts text a human sends, compare the draft to the sent version. Accepted verbatim, lightly edited, heavily edited, discarded. This is a graded quality signal collected automatically, and it is the metric that convinces a sceptical team, because it is measured in their own work.\n\n**Copy rate, and what gets copied.** If people copy the code block and never the explanation, the explanation is not earning its tokens.\n\n**Abandonment.** Session ends without the task completing. Noisy (people are interrupted) but a sharp change in it is real.\n\n**Escalation to a human.** For anything with a support path, the fraction of conversations that end with a handover is close to a direct quality measure, and the business already tracks it.\n\n**Time to resolution.** The lagging metric that usually is the actual point.\n\n**Follow-up sentiment.** The user's *next* message after a response. \"no, I meant\", \"that's not right\", \"thanks\": a small classifier over next messages gives a usable dissatisfaction rate, and it is far more representative than a thumbs-down widget.\n\n## Explicit feedback, and its limits\n\nThumbs up and down get very low response rates (in one published developer assistant, ratings came on 0.6% of turns and about 2% of conversations), and the responders skew: people click when annoyed or delighted, rarely when satisfied. Treat the *rate* as a trend, never as a level, and read the free-text comments as a source of eval items rather than as a metric.\n\nOne thing that improves them a lot: ask at the moment of consequence rather than after every message. A single \"did this solve it?\" at the end of a resolved conversation gets a far better response rate and a much less skewed sample than a widget on every turn.\n\n## Guardrails\n\nEvery online experiment needs metrics that are not allowed to get worse, watched independently of the metric you are trying to move:\n\n- p95 latency\n- cost per session\n- escalation rate\n- error and timeout rate\n- refusal rate\n\nRefusal rate belongs on that list for the reason module three gave: it is where a safety improvement silently becomes a product regression, and it is the one nobody thinks to watch.\n\n## Connect the online signal to the offline set\n\nThis is the step that turns proxies into a working system. Take a week of production traffic. For a sample of 100 sessions, compute the online signal (regenerated or not) and also have a human label the quality. Then check whether the two agree.\n\nIf regeneration correlates with your quality label, you have earned the right to treat regeneration rate as a quality dashboard, and you can watch it hourly. If it does not (sometimes people regenerate because they want a different *style*, not because the answer was wrong) you have learned that too, cheaply, and before you built a dashboard on it.\n\nThe same sample gives you the other direction: sessions where the offline criteria would have passed but the user clearly did not get what they needed. Those become new eval items and new rubric rules. This is the main channel by which an eval set stays relevant to the product rather than to its own history.\n\n## Log for it, and log carefully\n\nNone of these signals exist unless someone logs them. The minimum, per interaction: a session id, a turn index, the model and prompt version, latency, tokens, cost, whether it was regenerated, and whether the session escalated.\n\nAnd log with the privacy rules in mind from the first day: redact or hash identifiers, set a retention window, and make sure the people whose messages you are storing have been told. A logging scheme retrofitted for privacy after launch is a much more expensive project than one designed that way, and it is not only a legal question: a team that is nervous about its own logs stops looking at them, which costs you every signal in this lesson."},"check":{"question":"A team wants an ongoing quality signal for a drafting assistant without paying for continuous human review. Which is the strongest starting point?","options":[{"text":"A thumbs-up/thumbs-down widget on every response, with the daily average as the dashboard metric.","correct":false,"tempting":"Builds the dashboard on a one-to-two per cent response rate from a self-selected group, so the level is uninterpretable and the trend is noisy."},{"text":"The edit distance between each generated draft and the version the user actually sent, validated against human labels on a sample of 100.","correct":true},{"text":"Daily token volume and session counts, as a measure of how much value the feature is delivering.","correct":false,"tempting":"Tracks usage, which can rise precisely because the feature is failing and people are trying again."},{"text":"An LLM judge scoring every production response against the rubric, with the mean score charted hourly.","correct":false,"tempting":"Scales the judge to all traffic without addressing that judge scores drift and are only meaningful as same-run relative comparisons."}]},"prev":{"slug":"shadow-and-canary-releases","title":"Getting evidence from live traffic without exposing users","url":"https://addaly.com/learn/evaluating-ai/shadow-and-canary-releases"},"next":{"slug":"human-review-that-scales","title":"Sampling live traffic, and keeping reviewers honest","url":"https://addaly.com/learn/evaluating-ai/human-review-that-scales"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'The signals users give you without being asked', in Knowing If It Works. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/evaluating-ai/online-metrics-and-proxies (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"The signals users give you without being asked","courseTitle":"Knowing If It Works","module":{"index":6,"slug":"evaluation-in-the-loop","title":"Evaluation in the working loop"},"answer":null,"keyIdea":"Regeneration rate, edit distance and escalation are free quality proxies you probably already log, but they only become metrics once you have checked them against human labels on a sample.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"your-offline-set-cannot-see-everything","runs":[{"text":"Your offline set cannot see everything"}]},{"t":"p","runs":[{"text":"A fixed set of a hundred items measures the inputs you collected, against the criteria you wrote, at the moment you wrote them. It cannot see a question nobody has asked yet, a use you did not anticipate, or the slow shift in what people bring to the product. Production can."}]},{"t":"p","runs":[{"text":"The trouble is that production has no labels. So you use proxies: behaviours that correlate with quality and cost nothing to collect."}]},{"t":"h2","id":"implicit-signals-best-first","runs":[{"text":"Implicit signals, best first"}]},{"t":"p","runs":[{"text":"Regeneration or retry rate.","b":true},{"text":" The fraction of responses where the user immediately asks again, rephrases, or hits regenerate. This is the single best free signal most products already have in their logs and do not look at. It needs no survey and no annotation, it is unambiguous, and it moves quickly when quality moves."}]},{"t":"p","runs":[{"text":"Edit distance between suggestion and what shipped.","b":true},{"text":" For anything that drafts text a human sends, compare the draft to the sent version. Accepted verbatim, lightly edited, heavily edited, discarded. This is a graded quality signal collected automatically, and it is the metric that convinces a sceptical team, because it is measured in their own work."}]},{"t":"p","runs":[{"text":"Copy rate, and what gets copied.","b":true},{"text":" If people copy the code block and never the explanation, the explanation is not earning its tokens."}]},{"t":"p","runs":[{"text":"Abandonment.","b":true},{"text":" Session ends without the task completing. Noisy (people are interrupted) but a sharp change in it is real."}]},{"t":"p","runs":[{"text":"Escalation to a human.","b":true},{"text":" For anything with a support path, the fraction of conversations that end with a handover is close to a direct quality measure, and the business already tracks it."}]},{"t":"p","runs":[{"text":"Time to resolution.","b":true},{"text":" The lagging metric that usually is the actual point."}]},{"t":"p","runs":[{"text":"Follow-up sentiment.","b":true},{"text":" The user’s "},{"text":"next","i":true},{"text":" message after a response. “no, I meant”, “that’s not right”, “thanks”: a small classifier over next messages gives a usable dissatisfaction rate, and it is far more representative than a thumbs-down widget."}]},{"t":"h2","id":"explicit-feedback-and-its-limits","runs":[{"text":"Explicit feedback, and its limits"}]},{"t":"p","runs":[{"text":"Thumbs up and down get very low response rates (in one published developer assistant, ratings came on 0.6% of turns and about 2% of conversations), and the responders skew: people click when annoyed or delighted, rarely when satisfied. Treat the "},{"text":"rate","i":true},{"text":" as a trend, never as a level, and read the free-text comments as a source of eval items rather than as a metric."}]},{"t":"p","runs":[{"text":"One thing that improves them a lot: ask at the moment of consequence rather than after every message. A single “did this solve it?” at the end of a resolved conversation gets a far better response rate and a much less skewed sample than a widget on every turn."}]},{"t":"h2","id":"guardrails","runs":[{"text":"Guardrails"}]},{"t":"p","runs":[{"text":"Every online experiment needs metrics that are not allowed to get worse, watched independently of the metric you are trying to move:"}]},{"t":"ul","items":[[{"text":"p95 latency"}],[{"text":"cost per session"}],[{"text":"escalation rate"}],[{"text":"error and timeout rate"}],[{"text":"refusal rate"}]]},{"t":"p","runs":[{"text":"Refusal rate belongs on that list for the reason module three gave: it is where a safety improvement silently becomes a product regression, and it is the one nobody thinks to watch."}]},{"t":"h2","id":"connect-the-online-signal-to-the-offline-set","runs":[{"text":"Connect the online signal to the offline set"}]},{"t":"p","runs":[{"text":"This is the step that turns proxies into a working system. Take a week of production traffic. For a sample of 100 sessions, compute the online signal (regenerated or not) and also have a human label the quality. Then check whether the two agree."}]},{"t":"p","runs":[{"text":"If regeneration correlates with your quality label, you have earned the right to treat regeneration rate as a quality dashboard, and you can watch it hourly. If it does not (sometimes people regenerate because they want a different "},{"text":"style","i":true},{"text":", not because the answer was wrong) you have learned that too, cheaply, and before you built a dashboard on it."}]},{"t":"p","runs":[{"text":"The same sample gives you the other direction: sessions where the offline criteria would have passed but the user clearly did not get what they needed. Those become new eval items and new rubric rules. This is the main channel by which an eval set stays relevant to the product rather than to its own history."}]},{"t":"h2","id":"log-for-it-and-log-carefully","runs":[{"text":"Log for it, and log carefully"}]},{"t":"p","runs":[{"text":"None of these signals exist unless someone logs them. The minimum, per interaction: a session id, a turn index, the model and prompt version, latency, tokens, cost, whether it was regenerated, and whether the session escalated."}]},{"t":"p","runs":[{"text":"And log with the privacy rules in mind from the first day: redact or hash identifiers, set a retention window, and make sure the people whose messages you are storing have been told. A logging scheme retrofitted for privacy after launch is a much more expensive project than one designed that way, and it is not only a legal question: a team that is nervous about its own logs stops looking at them, which costs you every signal in this lesson."}]}],"check":{"question":[{"text":"A team wants an ongoing quality signal for a drafting assistant without paying for continuous human review. Which is the strongest starting point?"}],"options":[{"text":[{"text":"A thumbs-up/thumbs-down widget on every response, with the daily average as the dashboard metric."}],"correct":false,"tempting":[{"text":"Builds the dashboard on a one-to-two per cent response rate from a self-selected group, so the level is uninterpretable and the trend is noisy."}]},{"text":[{"text":"The edit distance between each generated draft and the version the user actually sent, validated against human labels on a sample of 100."}],"correct":true,"tempting":null},{"text":[{"text":"Daily token volume and session counts, as a measure of how much value the feature is delivering."}],"correct":false,"tempting":[{"text":"Tracks usage, which can rise precisely because the feature is failing and people are trying again."}]},{"text":[{"text":"An LLM judge scoring every production response against the rubric, with the mean score charted hourly."}],"correct":false,"tempting":[{"text":"Scales the judge to all traffic without addressing that judge scores drift and are only meaningful as same-run relative comparisons."}]}],"order":[0,1,3,2],"why":null},"prev":{"slug":"shadow-and-canary-releases","title":"Getting evidence from live traffic without exposing users","minutes":8},"next":{"slug":"human-review-that-scales","title":"Sampling live traffic, and keeping reviewers honest","minutes":9}}}