The signals users give you without being asked

Your offline set cannot see everything

A fixed set of a hundred items measures the inputs you collected, against the criteria you wrote, at the moment you wrote them. It cannot see a question nobody has asked yet, a use you did not anticipate, or the slow shift in what people bring to the product. Production can.

The trouble is that production has no labels. So you use proxies: behaviours that correlate with quality and cost nothing to collect.

Implicit signals, best first

Regeneration or retry rate. The fraction of responses where the user immediately asks again, rephrases, or hits regenerate. This is the single best free signal most products already have in their logs and do not look at. It needs no survey and no annotation, it is unambiguous, and it moves quickly when quality moves.

Edit distance between suggestion and what shipped. For anything that drafts text a human sends, compare the draft to the sent version. Accepted verbatim, lightly edited, heavily edited, discarded. This is a graded quality signal collected automatically, and it is the metric that convinces a sceptical team, because it is measured in their own work.

Copy rate, and what gets copied. If people copy the code block and never the explanation, the explanation is not earning its tokens.

Abandonment. Session ends without the task completing. Noisy (people are interrupted) but a sharp change in it is real.

Escalation to a human. For anything with a support path, the fraction of conversations that end with a handover is close to a direct quality measure, and the business already tracks it.

Time to resolution. The lagging metric that usually is the actual point.

Follow-up sentiment. The user’s next message after a response. “no, I meant”, “that’s not right”, “thanks”: a small classifier over next messages gives a usable dissatisfaction rate, and it is far more representative than a thumbs-down widget.

Explicit feedback, and its limits

Thumbs up and down get very low response rates (in one published developer assistant, ratings came on 0.6% of turns and about 2% of conversations), and the responders skew: people click when annoyed or delighted, rarely when satisfied. Treat the rate as a trend, never as a level, and read the free-text comments as a source of eval items rather than as a metric.

One thing that improves them a lot: ask at the moment of consequence rather than after every message. A single “did this solve it?” at the end of a resolved conversation gets a far better response rate and a much less skewed sample than a widget on every turn.

Guardrails

Every online experiment needs metrics that are not allowed to get worse, watched independently of the metric you are trying to move:

• p95 latency
• cost per session
• escalation rate
• error and timeout rate
• refusal rate

Refusal rate belongs on that list for the reason module three gave: it is where a safety improvement silently becomes a product regression, and it is the one nobody thinks to watch.

Connect the online signal to the offline set

This is the step that turns proxies into a working system. Take a week of production traffic. For a sample of 100 sessions, compute the online signal (regenerated or not) and also have a human label the quality. Then check whether the two agree.

If regeneration correlates with your quality label, you have earned the right to treat regeneration rate as a quality dashboard, and you can watch it hourly. If it does not (sometimes people regenerate because they want a different style, not because the answer was wrong) you have learned that too, cheaply, and before you built a dashboard on it.

The same sample gives you the other direction: sessions where the offline criteria would have passed but the user clearly did not get what they needed. Those become new eval items and new rubric rules. This is the main channel by which an eval set stays relevant to the product rather than to its own history.

Log for it, and log carefully

None of these signals exist unless someone logs them. The minimum, per interaction: a session id, a turn index, the model and prompt version, latency, tokens, cost, whether it was regenerated, and whether the session escalated.

And log with the privacy rules in mind from the first day: redact or hash identifiers, set a retention window, and make sure the people whose messages you are storing have been told. A logging scheme retrofitted for privacy after launch is a much more expensive project than one designed that way, and it is not only a legal question: a team that is nervous about its own logs stops looking at them, which costs you every signal in this lesson.