How much your labellers actually agree

Ninety-two per cent agreement can be worth almost nothing

Two people label 100 posts as harmful or fine. They agree on 92. That sounds like a settled definition.

Now look at the marginals. Labeller A called 12 posts harmful, labeller B called 10. Both spend most of their time saying “fine”, and two people who both say “fine” nine times out of ten will agree a lot by accident alone.

Cohen’s kappa measures how much of the agreement is left once you subtract the accidental part.

p_o = observed agreement            = 0.92
p_e = agreement expected by chance
    = (0.12 * 0.10) + (0.88 * 0.90)
    = 0.012 + 0.792                 = 0.804

kappa = (p_o - p_e) / (1 - p_e)
      = (0.92 - 0.804) / 0.196      = 0.59

Ninety-two per cent agreement is a kappa of 0.59. The conventional bands (fair 0.21 to 0.40, moderate 0.41 to 0.60, substantial 0.61 to 0.80) put that at the top of “moderate”. These bands are a 1977 convention by Landis and Koch, not a law of nature; quote them as a shared vocabulary, never as a pass mark.

In Python that is one line, and it is free:

from sklearn.metrics import cohen_kappa_score
cohen_kappa_score(labels_a, labels_b)          # 0.59

If you have no Python, the four counts and the formula above fit in a spreadsheet in five minutes.

The paradox that will confuse you at some point

Push the imbalance further. Suppose only 2% of posts are harmful, and the two labellers agree on 95 of 100 items. Chance agreement is now around 0.94, so kappa collapses towards zero even though the raw agreement went up. This is the well-known kappa paradox: when one class dominates, p_e approaches p_o and the ratio becomes unstable.

The mechanism matters more than the name. Kappa is asking “how much better than guessing at these rates”, and when guessing is already 94% correct there is very little room left to be better in. So:

• Always report the raw agreement, kappa, and the two marginals together. Any one of them alone can mislead.
• On very rare classes, prefer agreement measured only on the positives: of the items either person flagged, how many did both flag. That is the Jaccard overlap, and it does not have a chance-correction term to blow up.

Low kappa is a fault in the guide

The instinct on seeing 0.31 is to retrain the labellers. That is almost always the wrong move. Print the disagreements and read them. They cluster, and usually into two or three causes:

• A missing rule. Nobody said what to do with quoted abuse, so one person removes and the other allows.
• A hidden dimension. Half the disagreements are in Hinglish, where one labeller reads a word as an insult and the other as ordinary slang.
• A scale nobody can hold. A 1 to 5 quality score with no anchors becomes each person’s personal mood. Binary questions agree far better than scales, which is why a rubric of several binary checks beats one holistic rating.

Fix the guide with a written ruling for each cause, then relabel the affected items. Kappa is a diagnostic instrument pointed at your definition.

More than two people, or a scale

Cohen’s kappa handles exactly two raters on categories. Beyond that:

• Fleiss’ kappa for three or more raters on categories.
• Krippendorff’s alpha for any number of raters, missing labels, and ordinal or interval scales. The krippendorff package on PyPI is free and about four lines to use.
• Weighted kappa when adjacent categories are nearly-agreements: 4 versus 5 on a five-point scale should not be punished as hard as 1 versus 5.

Why this number governs everything downstream

Two consequences follow immediately, and both come up later in the course.

First, the labels are your ground truth, and their noise sets a limit on any measurement made against them. If two careful people agree at kappa 0.6, a model scored against one person’s labels cannot be meaningfully separated from a model scored against the other’s when the gap between models is small.

Second, when you build an automated judge, the question is never “does the judge agree with a human”. It is “does the judge agree with a human about as well as two humans agree with each other”. Without the human-human figure, the judge’s agreement score has no scale to sit on. Compute this number early, write it at the top of the labelling guide, and re-compute it whenever the guide changes.