{"course":{"slug":"evaluating-ai","title":"Knowing If It Works","url":"https://addaly.com/learn/evaluating-ai","level":"intermediate","track":"build"},"lesson":{"slug":"contested-quality","title":"When people disagree about what good means","url":"https://addaly.com/learn/evaluating-ai/contested-quality","position":67,"of":76,"minutes":9,"answer":"Keep every annotator's individual labels rather than only the majority vote, because aggregation converts one group's reading into ground truth, and measurement can show whether you apply your rule consistently.","keyIdea":"Keep every annotator's individual labels rather than only the majority vote, because aggregation converts one group's reading into ground truth, and measurement can show whether you apply your rule consistently, never whether the rule is right."},"body":{"format":"markdown","text":"## Some disagreements are not measurement errors\n\nThrough this course, disagreement between labellers has been treated as a problem to fix: sharpen the guide, define the terms, re-label. Most of the time that works, and low agreement is a symptom of a vague definition.\n\nSome disagreements are not that. What counts as offensive depends on who is being spoken to and by whom. What counts as a helpful explanation depends on what the reader already knows. What counts as neutral framing on a contested subject depends on where you stand. Whether an assistant should push back or comply is a question about what a product is for.\n\nYou can drive agreement up on these by writing a stricter rule, and what you have done is chosen a position, not discovered a fact. That is legitimate. Hiding it inside a metric is not.\n\n## What the annotation research shows\n\nStudies of toxicity and hate-speech labelling have repeatedly found that annotators' backgrounds predict their labels: people from a targeted group and people outside it rate the same text differently, and so do annotators of different ages. The direction of these findings is consistent enough to plan around, even though the magnitudes vary between datasets and study designs.\n\nThe mechanism is what matters for your work. **Majority vote erases the minority reading.** Aggregate five annotators, take the majority, call it the gold label, and the resulting file encodes the dominant group's interpretation as truth. Train on it, evaluate against it, and every number you produce afterwards inherits that choice invisibly.\n\n## Four practices that keep it honest\n\n**Keep per-annotator labels.** Store every individual judgement with an annotator identifier, not just the aggregate. You can always aggregate later; you can never disaggregate. This one habit costs a column and preserves every option.\n\n**Report agreement and composition, not only the label.** \"Labelled by six reviewers, three of whom are speakers of the language in question, with a kappa of 0.52\" tells a reader far more than a clean gold set does.\n\n**Where perspectives diverge systematically, report more than one number.** Score the system against each group's labels and publish both. A model at 88% against one group's judgements and 61% against another's is a result, an important one, and a single 79% is a way of not saying it.\n\n**State the position in the policy.** \"We treat reclaimed slurs used by members of the group as allowed. Here is why, here is who decided, and here is how to contest it.\" A written position can be argued with. A threshold buried in a rubric cannot.\n\n## Some disagreements should be settings\n\nNot every difference needs a global answer. Where two reasonable communities want different behaviour, the honest product answer is often a control rather than a decision: a per-community threshold, a per-user preference for directness, a room that sets its own norms.\n\nThis has an evaluation consequence worth planning for. Once behaviour is configurable, the metric is no longer \"is the output good\" but \"does the output match the setting that was chosen\", which is a much easier question to measure and a much harder one to design.\n\n## Telling the two kinds of disagreement apart\n\nYou will not always know at the outset whether a low agreement figure is a vague guide or a genuine difference in values. One diagnostic separates them cheaply: take twenty disagreements and ask each labeller to write why, then read the reasons.\n\n- If the reasons show people applying different readings of an ambiguous word (\"we each thought 'threat' meant something different\") it is a definition problem. Fix the guide.\n- If the reasons show people applying the same definition and reaching different conclusions about whether the thing is acceptable, the guide is not the issue and no rewrite will close the gap.\n\nThe second case is where the practices above apply. Mistaking it for the first produces a guide that grows longer every quarter, agreement that never improves, and labellers who quietly conclude that their judgement is unwelcome.\n\n## What measurement can and cannot do\n\nIt is worth being exact about the boundary, because this is where evaluation is most often oversold.\n\nMeasurement **can** tell you whether you apply your own rule consistently, whether the rule falls more heavily on one group than another, how often people contest a decision and how often those contests succeed, and whether your stated policy and your actual behaviour match.\n\nMeasurement **cannot** tell you the rule is right. That is a judgement somebody has to make, own and be accountable for.\n\nThe best evaluation practice therefore ends where it began: a number is a servant of a decision, and the decision belongs to a person. What this course buys you is that the person making it can see what they are choosing, instead of believing the choice was made for them by an average."},"check":{"question":"Why keep individual annotator labels rather than only the aggregated majority verdict?","options":[{"text":"Individual labels allow you to identify and remove unreliable annotators from the dataset.","correct":false,"tempting":"A real quality-control use that treats systematic disagreement as noise, which is exactly the reading this lesson warns against."},{"text":"Aggregation is irreversible, so keeping individual labels preserves the ability to analyse systematic differences between groups of raters later.","correct":true},{"text":"Majority voting requires an odd number of annotators, and individual labels let you recompute when the count changes.","correct":false,"tempting":"A mechanical detail of the voting rule rather than a reason to preserve the underlying judgements."},{"text":"Individual labels increase the effective sample size, since each judgement counts as an observation.","correct":false,"tempting":"Miscounts the statistics: several judgements of the same item are clustered observations of one item, not independent observations."}]},"prev":{"slug":"the-whole-product-not-the-model","title":"When the model is right and the product still fails","url":"https://addaly.com/learn/evaluating-ai/the-whole-product-not-the-model"},"next":{"slug":"cost-latency-and-drift","title":"Cost, latency, and the model that changed under you","url":"https://addaly.com/learn/evaluating-ai/cost-latency-and-drift"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'When people disagree about what good means', in Knowing If It Works. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/evaluating-ai/contested-quality (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"When people disagree about what good means","courseTitle":"Knowing If It Works","module":{"index":7,"slug":"assurance","title":"Evidence other people can act on"},"answer":null,"keyIdea":"Keep every annotator’s individual labels rather than only the majority vote, because aggregation converts one group’s reading into ground truth, and measurement can show whether you apply your rule consistently, never whether the rule is right.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"some-disagreements-are-not-measurement-errors","runs":[{"text":"Some disagreements are not measurement errors"}]},{"t":"p","runs":[{"text":"Through this course, disagreement between labellers has been treated as a problem to fix: sharpen the guide, define the terms, re-label. Most of the time that works, and low agreement is a symptom of a vague definition."}]},{"t":"p","runs":[{"text":"Some disagreements are not that. What counts as offensive depends on who is being spoken to and by whom. What counts as a helpful explanation depends on what the reader already knows. What counts as neutral framing on a contested subject depends on where you stand. Whether an assistant should push back or comply is a question about what a product is for."}]},{"t":"p","runs":[{"text":"You can drive agreement up on these by writing a stricter rule, and what you have done is chosen a position, not discovered a fact. That is legitimate. Hiding it inside a metric is not."}]},{"t":"h2","id":"what-the-annotation-research-shows","runs":[{"text":"What the annotation research shows"}]},{"t":"p","runs":[{"text":"Studies of toxicity and hate-speech labelling have repeatedly found that annotators’ backgrounds predict their labels: people from a targeted group and people outside it rate the same text differently, and so do annotators of different ages. The direction of these findings is consistent enough to plan around, even though the magnitudes vary between datasets and study designs."}]},{"t":"p","runs":[{"text":"The mechanism is what matters for your work. "},{"text":"Majority vote erases the minority reading.","b":true},{"text":" Aggregate five annotators, take the majority, call it the gold label, and the resulting file encodes the dominant group’s interpretation as truth. Train on it, evaluate against it, and every number you produce afterwards inherits that choice invisibly."}]},{"t":"h2","id":"four-practices-that-keep-it-honest","runs":[{"text":"Four practices that keep it honest"}]},{"t":"p","runs":[{"text":"Keep per-annotator labels.","b":true},{"text":" Store every individual judgement with an annotator identifier, not just the aggregate. You can always aggregate later; you can never disaggregate. This one habit costs a column and preserves every option."}]},{"t":"p","runs":[{"text":"Report agreement and composition, not only the label.","b":true},{"text":" “Labelled by six reviewers, three of whom are speakers of the language in question, with a kappa of 0.52” tells a reader far more than a clean gold set does."}]},{"t":"p","runs":[{"text":"Where perspectives diverge systematically, report more than one number.","b":true},{"text":" Score the system against each group’s labels and publish both. A model at 88% against one group’s judgements and 61% against another’s is a result, an important one, and a single 79% is a way of not saying it."}]},{"t":"p","runs":[{"text":"State the position in the policy.","b":true},{"text":" “We treat reclaimed slurs used by members of the group as allowed. Here is why, here is who decided, and here is how to contest it.” A written position can be argued with. A threshold buried in a rubric cannot."}]},{"t":"h2","id":"some-disagreements-should-be-settings","runs":[{"text":"Some disagreements should be settings"}]},{"t":"p","runs":[{"text":"Not every difference needs a global answer. Where two reasonable communities want different behaviour, the honest product answer is often a control rather than a decision: a per-community threshold, a per-user preference for directness, a room that sets its own norms."}]},{"t":"p","runs":[{"text":"This has an evaluation consequence worth planning for. Once behaviour is configurable, the metric is no longer “is the output good” but “does the output match the setting that was chosen”, which is a much easier question to measure and a much harder one to design."}]},{"t":"h2","id":"telling-the-two-kinds-of-disagreement-apart","runs":[{"text":"Telling the two kinds of disagreement apart"}]},{"t":"p","runs":[{"text":"You will not always know at the outset whether a low agreement figure is a vague guide or a genuine difference in values. One diagnostic separates them cheaply: take twenty disagreements and ask each labeller to write why, then read the reasons."}]},{"t":"ul","items":[[{"text":"If the reasons show people applying different readings of an ambiguous word (“we each thought ‘threat’ meant something different”) it is a definition problem. Fix the guide."}],[{"text":"If the reasons show people applying the same definition and reaching different conclusions about whether the thing is acceptable, the guide is not the issue and no rewrite will close the gap."}]]},{"t":"p","runs":[{"text":"The second case is where the practices above apply. Mistaking it for the first produces a guide that grows longer every quarter, agreement that never improves, and labellers who quietly conclude that their judgement is unwelcome."}]},{"t":"h2","id":"what-measurement-can-and-cannot-do","runs":[{"text":"What measurement can and cannot do"}]},{"t":"p","runs":[{"text":"It is worth being exact about the boundary, because this is where evaluation is most often oversold."}]},{"t":"p","runs":[{"text":"Measurement "},{"text":"can","b":true},{"text":" tell you whether you apply your own rule consistently, whether the rule falls more heavily on one group than another, how often people contest a decision and how often those contests succeed, and whether your stated policy and your actual behaviour match."}]},{"t":"p","runs":[{"text":"Measurement "},{"text":"cannot","b":true},{"text":" tell you the rule is right. That is a judgement somebody has to make, own and be accountable for."}]},{"t":"p","runs":[{"text":"The best evaluation practice therefore ends where it began: a number is a servant of a decision, and the decision belongs to a person. What this course buys you is that the person making it can see what they are choosing, instead of believing the choice was made for them by an average."}]}],"check":{"question":[{"text":"Why keep individual annotator labels rather than only the aggregated majority verdict?"}],"options":[{"text":[{"text":"Individual labels allow you to identify and remove unreliable annotators from the dataset."}],"correct":false,"tempting":[{"text":"A real quality-control use that treats systematic disagreement as noise, which is exactly the reading this lesson warns against."}]},{"text":[{"text":"Aggregation is irreversible, so keeping individual labels preserves the ability to analyse systematic differences between groups of raters later."}],"correct":true,"tempting":null},{"text":[{"text":"Majority voting requires an odd number of annotators, and individual labels let you recompute when the count changes."}],"correct":false,"tempting":[{"text":"A mechanical detail of the voting rule rather than a reason to preserve the underlying judgements."}]},{"text":[{"text":"Individual labels increase the effective sample size, since each judgement counts as an observation."}],"correct":false,"tempting":[{"text":"Miscounts the statistics: several judgements of the same item are clustered observations of one item, not independent observations."}]}],"order":[2,1,3,0],"why":null},"prev":{"slug":"the-whole-product-not-the-model","title":"When the model is right and the product still fails","minutes":8},"next":{"slug":"cost-latency-and-drift","title":"Cost, latency, and the model that changed under you","minutes":9}}}