When people disagree about what good means

Some disagreements are not measurement errors

Through this course, disagreement between labellers has been treated as a problem to fix: sharpen the guide, define the terms, re-label. Most of the time that works, and low agreement is a symptom of a vague definition.

Some disagreements are not that. What counts as offensive depends on who is being spoken to and by whom. What counts as a helpful explanation depends on what the reader already knows. What counts as neutral framing on a contested subject depends on where you stand. Whether an assistant should push back or comply is a question about what a product is for.

You can drive agreement up on these by writing a stricter rule, and what you have done is chosen a position, not discovered a fact. That is legitimate. Hiding it inside a metric is not.

What the annotation research shows

Studies of toxicity and hate-speech labelling have repeatedly found that annotators’ backgrounds predict their labels: people from a targeted group and people outside it rate the same text differently, and so do annotators of different ages. The direction of these findings is consistent enough to plan around, even though the magnitudes vary between datasets and study designs.

The mechanism is what matters for your work. Majority vote erases the minority reading. Aggregate five annotators, take the majority, call it the gold label, and the resulting file encodes the dominant group’s interpretation as truth. Train on it, evaluate against it, and every number you produce afterwards inherits that choice invisibly.

Four practices that keep it honest

Keep per-annotator labels. Store every individual judgement with an annotator identifier, not just the aggregate. You can always aggregate later; you can never disaggregate. This one habit costs a column and preserves every option.

Report agreement and composition, not only the label. “Labelled by six reviewers, three of whom are speakers of the language in question, with a kappa of 0.52” tells a reader far more than a clean gold set does.

Where perspectives diverge systematically, report more than one number. Score the system against each group’s labels and publish both. A model at 88% against one group’s judgements and 61% against another’s is a result, an important one, and a single 79% is a way of not saying it.

State the position in the policy. “We treat reclaimed slurs used by members of the group as allowed. Here is why, here is who decided, and here is how to contest it.” A written position can be argued with. A threshold buried in a rubric cannot.

Some disagreements should be settings

Not every difference needs a global answer. Where two reasonable communities want different behaviour, the honest product answer is often a control rather than a decision: a per-community threshold, a per-user preference for directness, a room that sets its own norms.

This has an evaluation consequence worth planning for. Once behaviour is configurable, the metric is no longer “is the output good” but “does the output match the setting that was chosen”, which is a much easier question to measure and a much harder one to design.

Telling the two kinds of disagreement apart

You will not always know at the outset whether a low agreement figure is a vague guide or a genuine difference in values. One diagnostic separates them cheaply: take twenty disagreements and ask each labeller to write why, then read the reasons.

• If the reasons show people applying different readings of an ambiguous word (“we each thought ‘threat’ meant something different”) it is a definition problem. Fix the guide.
• If the reasons show people applying the same definition and reaching different conclusions about whether the thing is acceptable, the guide is not the issue and no rewrite will close the gap.

The second case is where the practices above apply. Mistaking it for the first produces a guide that grows longer every quarter, agreement that never improves, and labellers who quietly conclude that their judgement is unwelcome.

What measurement can and cannot do

It is worth being exact about the boundary, because this is where evaluation is most often oversold.

Measurement can tell you whether you apply your own rule consistently, whether the rule falls more heavily on one group than another, how often people contest a decision and how often those contests succeed, and whether your stated policy and your actual behaviour match.

Measurement cannot tell you the rule is right. That is a judgement somebody has to make, own and be accountable for.

The best evaluation practice therefore ends where it began: a number is a servant of a decision, and the decision belongs to a person. What this course buys you is that the person making it can see what they are choosing, instead of believing the choice was made for them by an average.