{"course":{"slug":"evaluating-ai","title":"Knowing If It Works","url":"https://addaly.com/learn/evaluating-ai","level":"intermediate","track":"build"},"lesson":{"slug":"agreement-arithmetic","title":"How much your labellers actually agree","url":"https://addaly.com/learn/evaluating-ai/agreement-arithmetic","position":6,"of":76,"minutes":9,"answer":"Raw agreement flatters you whenever one class is common, so compute kappa as well, and read a low kappa as a fault in the definition rather than in the people.","keyIdea":"Raw agreement flatters you whenever one class is common, so compute kappa as well, and read a low kappa as a fault in the definition rather than in the people."},"body":{"format":"markdown","text":"## Ninety-two per cent agreement can be worth almost nothing\n\nTwo people label 100 posts as harmful or fine. They agree on 92. That sounds like a settled definition.\n\nNow look at the marginals. Labeller A called 12 posts harmful, labeller B called 10. Both spend most of their time saying \"fine\", and two people who both say \"fine\" nine times out of ten will agree a lot by accident alone.\n\n> Figure: Two labellers, 100 posts, 92 agreements. A two by two grid, rows A: harmful and A: fine, columns B: harmful and B: fine: A: harmful and B: harmful: 7; A: harmful and B: fine: 5; A: fine and B: harmful: 3; A: fine and B: fine: 85. Agreement is 7 + 85 = 92. But two people who both say 'fine' about nine times in ten would agree on roughly 80 posts by accident alone, which is why kappa comes out at 0.59 rather than 0.92.\n\nCohen's kappa measures how much of the agreement is left once you subtract the accidental part.\n\n```\np_o = observed agreement            = 0.92\np_e = agreement expected by chance\n    = (0.12 * 0.10) + (0.88 * 0.90)\n    = 0.012 + 0.792                 = 0.804\n\nkappa = (p_o - p_e) / (1 - p_e)\n      = (0.92 - 0.804) / 0.196      = 0.59\n```\n\nNinety-two per cent agreement is a kappa of 0.59. The conventional bands (fair 0.21 to 0.40, moderate 0.41 to 0.60, substantial 0.61 to 0.80) put that at the top of \"moderate\". These bands are a 1977 convention by Landis and Koch, not a law of nature; quote them as a shared vocabulary, never as a pass mark.\n\nIn Python that is one line, and it is free:\n\n```python\nfrom sklearn.metrics import cohen_kappa_score\ncohen_kappa_score(labels_a, labels_b)          # 0.59\n```\n\nIf you have no Python, the four counts and the formula above fit in a spreadsheet in five minutes.\n\n## The paradox that will confuse you at some point\n\nPush the imbalance further. Suppose only 2% of posts are harmful, and the two labellers agree on 95 of 100 items. Chance agreement is now around 0.94, so kappa collapses towards zero even though the raw agreement went up. This is the well-known kappa paradox: when one class dominates, `p_e` approaches `p_o` and the ratio becomes unstable.\n\nThe mechanism matters more than the name. Kappa is asking \"how much better than guessing at these rates\", and when guessing is already 94% correct there is very little room left to be better in. So:\n\n- Always report the raw agreement, kappa, and the two marginals together. Any one of them alone can mislead.\n- On very rare classes, prefer agreement measured only on the positives: of the items either person flagged, how many did both flag. That is the Jaccard overlap, and it does not have a chance-correction term to blow up.\n\n## Low kappa is a fault in the guide\n\nThe instinct on seeing 0.31 is to retrain the labellers. That is almost always the wrong move. Print the disagreements and read them. They cluster, and usually into two or three causes:\n\n- **A missing rule.** Nobody said what to do with quoted abuse, so one person removes and the other allows.\n- **A hidden dimension.** Half the disagreements are in Hinglish, where one labeller reads a word as an insult and the other as ordinary slang.\n- **A scale nobody can hold.** A 1 to 5 quality score with no anchors becomes each person's personal mood. Binary questions agree far better than scales, which is why a rubric of several binary checks beats one holistic rating.\n\nFix the guide with a written ruling for each cause, then relabel the affected items. Kappa is a diagnostic instrument pointed at your definition.\n\n## More than two people, or a scale\n\nCohen's kappa handles exactly two raters on categories. Beyond that:\n\n- **Fleiss' kappa** for three or more raters on categories.\n- **Krippendorff's alpha** for any number of raters, missing labels, and ordinal or interval scales. The `krippendorff` package on PyPI is free and about four lines to use.\n- **Weighted kappa** when adjacent categories are nearly-agreements: 4 versus 5 on a five-point scale should not be punished as hard as 1 versus 5.\n\n## Why this number governs everything downstream\n\nTwo consequences follow immediately, and both come up later in the course.\n\nFirst, the labels are your ground truth, and their noise sets a limit on any measurement made against them. If two careful people agree at kappa 0.6, a model scored against one person's labels cannot be meaningfully separated from a model scored against the other's when the gap between models is small.\n\nSecond, when you build an automated judge, the question is never \"does the judge agree with a human\". It is \"does the judge agree with a human about as well as two humans agree with each other\". Without the human-human figure, the judge's agreement score has no scale to sit on. Compute this number early, write it at the top of the labelling guide, and re-compute it whenever the guide changes."},"check":{"question":"A safety label is applied to 2% of items. Two reviewers agree on 96 of 100 items, and Cohen's kappa comes out at 0.09. What is the most defensible reading?","options":[{"text":"The reviewers are unreliable and the labelling task should be abandoned until they are retrained.","correct":false,"tempting":"Reads a low kappa as a verdict on the people, when the arithmetic here is dominated by the class imbalance rather than by reviewer behaviour."},{"text":"Kappa is the wrong statistic for this task, so raw agreement of 96% should be reported instead.","correct":false,"tempting":"Fixes the problem by dropping the inconvenient statistic; 96% agreement is exactly what two people who almost always say \"no\" would get by accident."},{"text":"Chance agreement is already about 96%, so there is almost no headroom for kappa to measure, and agreement should be judged on the flagged items only.","correct":true},{"text":"The two reviewers disagreed on four items, which at this base rate is roughly two items' worth of real disagreement, so the definition is fine.","correct":false,"tempting":"Assumes half the disagreements are noise on no evidence; with only two true positives expected in 100 items, four disagreements is a large signal about the definition, not a small one."}]},"prev":{"slug":"labelling-without-lying","title":"Writing a labelling guide two people can agree on","url":"https://addaly.com/learn/evaluating-ai/labelling-without-lying"},"next":{"slug":"synthetic-test-cases","title":"Test cases you generate, and what they cannot tell you","url":"https://addaly.com/learn/evaluating-ai/synthetic-test-cases"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'How much your labellers actually agree', in Knowing If It Works. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/evaluating-ai/agreement-arithmetic (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"How much your labellers actually agree","courseTitle":"Knowing If It Works","module":{"index":1,"slug":"evidence","title":"What counts as evidence"},"answer":null,"keyIdea":"Raw agreement flatters you whenever one class is common, so compute kappa as well, and read a low kappa as a fault in the definition rather than in the people.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"cols":["B: harmful","B: fine"],"rows":["A: harmful","A: fine"],"type":"matrix","cells":[["7","5"],["3","85"]],"title":"Two labellers, 100 posts, 92 agreements","caption":"Agreement is 7 + 85 = 92. But two people who both say ‘fine’ about nine times in ten would agree on roughly 80 posts by accident alone, which is why kappa comes out at 0.59 rather than 0.92.","cellLabels":[["both harmful","A only"],["B only","both fine"]]},"alt":"Figure: Two labellers, 100 posts, 92 agreements. A two by two grid, rows A: harmful and A: fine, columns B: harmful and B: fine: A: harmful and B: harmful: 7; A: harmful and B: fine: 5; A: fine and B: harmful: 3; A: fine and B: fine: 85. Agreement is 7 + 85 = 92. But two people who both say ‘fine’ about nine times in ten would agree on roughly 80 posts by accident alone, which is why kappa comes out at 0.59 rather than 0.92."}},"blocks":[{"t":"h2","id":"ninety-two-per-cent-agreement-can-be-worth-almost-nothing","runs":[{"text":"Ninety-two per cent agreement can be worth almost nothing"}]},{"t":"p","runs":[{"text":"Two people label 100 posts as harmful or fine. They agree on 92. That sounds like a settled definition."}]},{"t":"p","runs":[{"text":"Now look at the marginals. Labeller A called 12 posts harmful, labeller B called 10. Both spend most of their time saying “fine”, and two people who both say “fine” nine times out of ten will agree a lot by accident alone."}]},{"t":"p","runs":[{"text":"Cohen’s kappa measures how much of the agreement is left once you subtract the accidental part."}]},{"t":"code","lang":"","label":"Text","text":"p_o = observed agreement            = 0.92\np_e = agreement expected by chance\n    = (0.12 * 0.10) + (0.88 * 0.90)\n    = 0.012 + 0.792                 = 0.804\n\nkappa = (p_o - p_e) / (1 - p_e)\n      = (0.92 - 0.804) / 0.196      = 0.59"},{"t":"p","runs":[{"text":"Ninety-two per cent agreement is a kappa of 0.59. The conventional bands (fair 0.21 to 0.40, moderate 0.41 to 0.60, substantial 0.61 to 0.80) put that at the top of “moderate”. These bands are a 1977 convention by Landis and Koch, not a law of nature; quote them as a shared vocabulary, never as a pass mark."}]},{"t":"p","runs":[{"text":"In Python that is one line, and it is free:"}]},{"t":"code","lang":"python","label":"Python","text":"from sklearn.metrics import cohen_kappa_score\ncohen_kappa_score(labels_a, labels_b)          # 0.59"},{"t":"p","runs":[{"text":"If you have no Python, the four counts and the formula above fit in a spreadsheet in five minutes."}]},{"t":"h2","id":"the-paradox-that-will-confuse-you-at-some-point","runs":[{"text":"The paradox that will confuse you at some point"}]},{"t":"p","runs":[{"text":"Push the imbalance further. Suppose only 2% of posts are harmful, and the two labellers agree on 95 of 100 items. Chance agreement is now around 0.94, so kappa collapses towards zero even though the raw agreement went up. This is the well-known kappa paradox: when one class dominates, "},{"text":"p_e","code":true},{"text":" approaches "},{"text":"p_o","code":true},{"text":" and the ratio becomes unstable."}]},{"t":"p","runs":[{"text":"The mechanism matters more than the name. Kappa is asking “how much better than guessing at these rates”, and when guessing is already 94% correct there is very little room left to be better in. So:"}]},{"t":"ul","items":[[{"text":"Always report the raw agreement, kappa, and the two marginals together. Any one of them alone can mislead."}],[{"text":"On very rare classes, prefer agreement measured only on the positives: of the items either person flagged, how many did both flag. That is the Jaccard overlap, and it does not have a chance-correction term to blow up."}]]},{"t":"h2","id":"low-kappa-is-a-fault-in-the-guide","runs":[{"text":"Low kappa is a fault in the guide"}]},{"t":"p","runs":[{"text":"The instinct on seeing 0.31 is to retrain the labellers. That is almost always the wrong move. Print the disagreements and read them. They cluster, and usually into two or three causes:"}]},{"t":"ul","items":[[{"text":"A missing rule.","b":true},{"text":" Nobody said what to do with quoted abuse, so one person removes and the other allows."}],[{"text":"A hidden dimension.","b":true},{"text":" Half the disagreements are in Hinglish, where one labeller reads a word as an insult and the other as ordinary slang."}],[{"text":"A scale nobody can hold.","b":true},{"text":" A 1 to 5 quality score with no anchors becomes each person’s personal mood. Binary questions agree far better than scales, which is why a rubric of several binary checks beats one holistic rating."}]]},{"t":"p","runs":[{"text":"Fix the guide with a written ruling for each cause, then relabel the affected items. Kappa is a diagnostic instrument pointed at your definition."}]},{"t":"h2","id":"more-than-two-people-or-a-scale","runs":[{"text":"More than two people, or a scale"}]},{"t":"p","runs":[{"text":"Cohen’s kappa handles exactly two raters on categories. Beyond that:"}]},{"t":"ul","items":[[{"text":"Fleiss’ kappa","b":true},{"text":" for three or more raters on categories."}],[{"text":"Krippendorff’s alpha","b":true},{"text":" for any number of raters, missing labels, and ordinal or interval scales. The "},{"text":"krippendorff","code":true},{"text":" package on PyPI is free and about four lines to use."}],[{"text":"Weighted kappa","b":true},{"text":" when adjacent categories are nearly-agreements: 4 versus 5 on a five-point scale should not be punished as hard as 1 versus 5."}]]},{"t":"h2","id":"why-this-number-governs-everything-downstream","runs":[{"text":"Why this number governs everything downstream"}]},{"t":"p","runs":[{"text":"Two consequences follow immediately, and both come up later in the course."}]},{"t":"p","runs":[{"text":"First, the labels are your ground truth, and their noise sets a limit on any measurement made against them. If two careful people agree at kappa 0.6, a model scored against one person’s labels cannot be meaningfully separated from a model scored against the other’s when the gap between models is small."}]},{"t":"p","runs":[{"text":"Second, when you build an automated judge, the question is never “does the judge agree with a human”. It is “does the judge agree with a human about as well as two humans agree with each other”. Without the human-human figure, the judge’s agreement score has no scale to sit on. Compute this number early, write it at the top of the labelling guide, and re-compute it whenever the guide changes."}]}],"check":{"question":[{"text":"A safety label is applied to 2% of items. Two reviewers agree on 96 of 100 items, and Cohen’s kappa comes out at 0.09. What is the most defensible reading?"}],"options":[{"text":[{"text":"The reviewers are unreliable and the labelling task should be abandoned until they are retrained."}],"correct":false,"tempting":[{"text":"Reads a low kappa as a verdict on the people, when the arithmetic here is dominated by the class imbalance rather than by reviewer behaviour."}]},{"text":[{"text":"Kappa is the wrong statistic for this task, so raw agreement of 96% should be reported instead."}],"correct":false,"tempting":[{"text":"Fixes the problem by dropping the inconvenient statistic; 96% agreement is exactly what two people who almost always say “no” would get by accident."}]},{"text":[{"text":"Chance agreement is already about 96%, so there is almost no headroom for kappa to measure, and agreement should be judged on the flagged items only."}],"correct":true,"tempting":null},{"text":[{"text":"The two reviewers disagreed on four items, which at this base rate is roughly two items’ worth of real disagreement, so the definition is fine."}],"correct":false,"tempting":[{"text":"Assumes half the disagreements are noise on no evidence; with only two true positives expected in 100 items, four disagreements is a large signal about the definition, not a small one."}]}],"order":[3,0,1,2],"why":null},"prev":{"slug":"labelling-without-lying","title":"Writing a labelling guide two people can agree on","minutes":9},"next":{"slug":"synthetic-test-cases","title":"Test cases you generate, and what they cannot tell you","minutes":8}}}