Thumbs up, thumbs down, and what the buttons actually measure

The response rate

Put a thumbs-up and a thumbs-down under every answer and watch the numbers. Typically one to three per cent of responses get a click. The rest, nothing. The people who click are not a sample of your users: some products see mostly negative clicks from people who were annoyed enough to act, others mostly positive from people who were delighted or who click approvingly on everything. Either way the clicking population is a self-selected few per cent, and a number computed from them describes them, not your users. The earlier evaluation course has a whole lesson on why the easiest hundred items to grab are the least representative; feedback buttons are that lesson in a widget.

What a thumbs-down means

The answer was wrong. Or slow. Or the tone was off. Or the model refused. Or the user misread it. Or they meant to click the other one. A bare thumbs-down is a signal that something happened, with the something unknown.

Two changes make it usable. Ask a category at the moment of the click (wrong, unhelpful, refused, harmful, other) as four buttons rather than a text box, because a text box is filled in by roughly one clicker in ten and four buttons by most. And record the full context with the click: the conversation, the retrieved chunks, the prompt version, the model, the latency. A thumbs-down with its trace attached is a case that can be examined. Without the trace it is a count.

Implicit signals cover everything

The buttons cover two per cent. The behaviour around every response covers all of it. Regenerate is a negative signal. Editing the previous message and resending is a negative signal about the answer or about the question, and looking at the edit tells you which. Copying the answer to the clipboard is positive. A follow-up that restates the question is negative; a follow-up that builds on the answer is positive. For inline features, acceptance and edit distance, from the previous lesson, are the strongest signals available anywhere. Ending the conversation after an answer is ambiguous (satisfied or given up) and should not be scored on its own.

Log these for every response, as events with the response id, and you have a signal with complete coverage that costs the user nothing.

From button to test case

The button is not the product. The pipeline behind it is: every thumbs-down with a category lands in a review queue; a person reads a sample each week; each real failure becomes a test case in the evaluation set the next module builds; and the set is run on every change. This is how a product gets better from feedback. A dashboard with a thumbs-down rate that nobody turns into cases is a number that goes up and down while nothing changes.

Do not optimise for the thumbs

It is tempting to train or tune towards thumbs-up, or to pick the prompt variant with the higher rate. Two mechanisms make this dangerous. The first is selection bias, already covered: you are optimising for the clickers. The second is worse. The easiest way to earn approval is to agree with the person, praise their question and tell them what they hoped to hear. A model tuned on approval drifts towards sycophancy, and a widely reported case in 2025 saw a major provider roll back a model update within days after exactly this drift became visible to users. Approval is not correctness, and a system that learns to be liked stops being useful in the situations where usefulness means disagreement.

Use thumbs as a source of cases. Use held-out evaluation, with labels from people paid to be careful, as the measure of quality.

Privacy of feedback

A feedback record attaches to conversation content, and therefore inherits its retention and deletion rules. When the conversation is deleted, the feedback record must lose its content too, or your review queue becomes the place where deleted conversations live on.

What to report

Weekly, by feature and by prompt version: the thumbs-down rate with its categories, the regenerate rate, the edit-and-resend rate, and for inline features the acceptance rate and the edit-distance distribution. Report each with an interval, not a bare percentage, because a week of two per cent clicks on a few thousand responses is a few dozen events, and a change from twelve to nine of them is not a trend. The evaluation course’s lesson on whether a difference is real is the arithmetic; this lesson’s job is to make sure you are collecting something worth doing arithmetic on.