{"course":{"slug":"evaluating-ai","title":"Knowing If It Works","url":"https://addaly.com/learn/evaluating-ai","level":"intermediate","track":"build"},"lesson":{"slug":"labelling-without-lying","title":"Writing a labelling guide two people can agree on","url":"https://addaly.com/learn/evaluating-ai/labelling-without-lying","position":5,"of":76,"minutes":9,"answer":"Two people labelling the same twenty items blind, and a written ruling for every disagreement, is what turns an opinion into a criterion.","keyIdea":"Two people labelling the same twenty items blind, and a written ruling for every disagreement, is what turns an opinion into a criterion."},"body":{"format":"markdown","text":"## The guide is the real artefact\n\nWhen you label a hundred examples, the labels are not the valuable output. The document that made two people produce the *same* labels is. That document is what lets you add 100 more items next quarter, hand the task to somebody new, or write a judge prompt in module three. Without it, your set is a private opinion that decays as your memory of it fades.\n\nA labelling guide is short. One page per label. It contains:\n\n1. **A definition in one sentence**, stated as something observable in the output rather than something felt about it.\n2. **Three examples that pass**, copied verbatim from real data.\n3. **Three that fail**, with the specific span that fails it underlined in words.\n4. **Two boundary cases with a ruling**: the ones you argued about, and the decision you made, even if the decision was arbitrary.\n\nThe fourth item is the one people skip and the one that does the work. Boundary rulings are how a criterion survives contact with data nobody imagined.\n\n## From adjective to observable\n\nMost first drafts are unlabellable:\n\n> The reply should be helpful and professional.\n\nNeither word points at anything in the text. Rewrite until each rule names something you could highlight with a cursor:\n\n> - The reply answers the question that was asked, rather than a related one.\n> - The reply contains no commitment about dates, money or eligibility that is absent from the policy text supplied.\n> - The reply is in the same language as the customer's message.\n> - If required information is missing, the reply asks for it instead of assuming a value.\n\nFour rules, each separately true or false. \"Helpful and professional\" is now four columns in a spreadsheet, and a disagreement can be located: you and your colleague differ on rule two, not on the vibe.\n\n## Measure agreement, and do not be fooled by the raw number\n\nHave a second person label the same 20 items independently. Then compute agreement, but be careful which agreement you compute.\n\nSuppose 94 of every 100 support replies are fine. Two labellers who both mark everything \"fine\" agree 94% of the time and have learned nothing. Raw agreement flatters you exactly when the classes are imbalanced, which is most of the time in safety and quality work.\n\n**Cohen's kappa** corrects for the agreement you would get by chance. Roughly:\n\n```\nkappa = (observed_agreement - chance_agreement)\n        / (1 - chance_agreement)\n```\n\nIn the example above, chance agreement is already about 0.89, so 94% raw becomes a kappa near 0.45: poor. Useful reference points, and they are conventions rather than laws:\n\n- **0.8 and up**: the criterion is solid; automate against it.\n- **0.6 to 0.8**: workable, but every measurement you take carries that much slack.\n- **below 0.6**: you do not yet have a criterion. Stop labelling and rewrite the guide.\n\nThree lines of Python get you there: `sklearn.metrics.cohen_kappa_score(a, b)`. If you would rather not install anything, the formula above runs fine in a spreadsheet.\n\n## Disagreements are data, not friction\n\nWhen the two of you differ on six of twenty, sit down and go through the six. Each one resolves into exactly one of three things:\n\n- **The guide is silent.** Add a boundary ruling. This is the good case and it is most of them.\n- **One person misread the item.** Fine. No change needed.\n- **The two of you actually want different things from the product.** This is the expensive discovery, and it is far cheaper found now than after launch. It is not a labelling problem at all; it is a product decision wearing a labelling problem's clothes.\n\nRe-label a fresh 20 after each guide revision. Two rounds is usually enough. Three means the criterion is hard, and you should consider splitting it into two simpler ones.\n\n## Label noise is a ceiling\n\nHere is the consequence people miss. If your labels are themselves 92% correct, then a perfect model scores 92% on your set; and worse, a model that has learned your labellers' particular mistakes scores *higher* than one that is right. No amount of prompt engineering climbs above the quality of the ruler.\n\nSo when a score plateaus around 90%, the next honest step is not another prompt variant. It is to take the 30 items the system failed, re-label them blind, and find out how many of those \"failures\" were your own labels being wrong. Teams routinely find a third.\n\n## Tools, including the free ones\n\nAt 100 items, a spreadsheet with one row per item and one column per rule is not a compromise; it is the right tool, and it exports to CSV. Past a few hundred items, or with more than two labellers, **Label Studio** and **Argilla** are both open source, self-hostable and free, and they give you the two things a spreadsheet does not: blind labelling, so nobody sees the other person's answer, and a built-in agreement report. Commercial platforms such as Scale or Labelbox do the same job with a sales call attached.\n\nWhatever you use, keep the guide in version control beside the data, and put the date on it. A guide with no history is a guide nobody can trust six months later, including you."},"check":{"question":"Two reviewers independently label 200 support replies for a rare policy violation. They agree on 191 of them, which sounds excellent. Only 12 of the 200 were actually violations. What should they conclude?","options":[{"text":"Agreement is high enough to proceed; 95.5% is comfortably above any reasonable bar.","correct":false,"tempting":"Reads raw agreement at face value when almost all items are negatives, so most of that 95.5% is agreement about the easy cases."},{"text":"Because violations are rare, most of that agreement is chance; they should compute kappa, which will be far lower, before trusting the criterion.","correct":true},{"text":"The nine disagreements are noise from reviewer fatigue and can be resolved by whichever reviewer is more senior.","correct":false,"tempting":"Treats disagreements as an inconvenience to be settled by authority rather than as the signal that shows where the guide is silent."},{"text":"They need a larger sample, since 200 items give too few violations to measure anything.","correct":false,"tempting":"Reaches for more data when the immediate problem is which statistic is being read, not how much of it there is."}]},"prev":{"slug":"where-examples-come-from","title":"Where the examples come from","url":"https://addaly.com/learn/evaluating-ai/where-examples-come-from"},"next":{"slug":"agreement-arithmetic","title":"How much your labellers actually agree","url":"https://addaly.com/learn/evaluating-ai/agreement-arithmetic"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Writing a labelling guide two people can agree on', in Knowing If It Works. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/evaluating-ai/labelling-without-lying (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Writing a labelling guide two people can agree on","courseTitle":"Knowing If It Works","module":{"index":1,"slug":"evidence","title":"What counts as evidence"},"answer":null,"keyIdea":"Two people labelling the same twenty items blind, and a written ruling for every disagreement, is what turns an opinion into a criterion.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"the-guide-is-the-real-artefact","runs":[{"text":"The guide is the real artefact"}]},{"t":"p","runs":[{"text":"When you label a hundred examples, the labels are not the valuable output. The document that made two people produce the "},{"text":"same","i":true},{"text":" labels is. That document is what lets you add 100 more items next quarter, hand the task to somebody new, or write a judge prompt in module three. Without it, your set is a private opinion that decays as your memory of it fades."}]},{"t":"p","runs":[{"text":"A labelling guide is short. One page per label. It contains:"}]},{"t":"ol","start":1,"items":[[{"text":"A definition in one sentence","b":true},{"text":", stated as something observable in the output rather than something felt about it."}],[{"text":"Three examples that pass","b":true},{"text":", copied verbatim from real data."}],[{"text":"Three that fail","b":true},{"text":", with the specific span that fails it underlined in words."}],[{"text":"Two boundary cases with a ruling","b":true},{"text":": the ones you argued about, and the decision you made, even if the decision was arbitrary."}]]},{"t":"p","runs":[{"text":"The fourth item is the one people skip and the one that does the work. Boundary rulings are how a criterion survives contact with data nobody imagined."}]},{"t":"h2","id":"from-adjective-to-observable","runs":[{"text":"From adjective to observable"}]},{"t":"p","runs":[{"text":"Most first drafts are unlabellable:"}]},{"t":"quote","runs":[{"text":"The reply should be helpful and professional."}]},{"t":"p","runs":[{"text":"Neither word points at anything in the text. Rewrite until each rule names something you could highlight with a cursor:"}]},{"t":"quote","runs":[{"text":"- The reply answers the question that was asked, rather than a related one."}]},{"t":"quote","runs":[{"text":"- The reply contains no commitment about dates, money or eligibility that is absent from the policy text supplied."}]},{"t":"quote","runs":[{"text":"- The reply is in the same language as the customer’s message."}]},{"t":"quote","runs":[{"text":"- If required information is missing, the reply asks for it instead of assuming a value."}]},{"t":"p","runs":[{"text":"Four rules, each separately true or false. “Helpful and professional” is now four columns in a spreadsheet, and a disagreement can be located: you and your colleague differ on rule two, not on the vibe."}]},{"t":"h2","id":"measure-agreement-and-do-not-be-fooled-by-the-raw-number","runs":[{"text":"Measure agreement, and do not be fooled by the raw number"}]},{"t":"p","runs":[{"text":"Have a second person label the same 20 items independently. Then compute agreement, but be careful which agreement you compute."}]},{"t":"p","runs":[{"text":"Suppose 94 of every 100 support replies are fine. Two labellers who both mark everything “fine” agree 94% of the time and have learned nothing. Raw agreement flatters you exactly when the classes are imbalanced, which is most of the time in safety and quality work."}]},{"t":"p","runs":[{"text":"Cohen’s kappa","b":true},{"text":" corrects for the agreement you would get by chance. Roughly:"}]},{"t":"code","lang":"","label":"Text","text":"kappa = (observed_agreement - chance_agreement)\n        / (1 - chance_agreement)"},{"t":"p","runs":[{"text":"In the example above, chance agreement is already about 0.89, so 94% raw becomes a kappa near 0.45: poor. Useful reference points, and they are conventions rather than laws:"}]},{"t":"ul","items":[[{"text":"0.8 and up","b":true},{"text":": the criterion is solid; automate against it."}],[{"text":"0.6 to 0.8","b":true},{"text":": workable, but every measurement you take carries that much slack."}],[{"text":"below 0.6","b":true},{"text":": you do not yet have a criterion. Stop labelling and rewrite the guide."}]]},{"t":"p","runs":[{"text":"Three lines of Python get you there: "},{"text":"sklearn.metrics.cohen_kappa_score(a, b)","code":true},{"text":". If you would rather not install anything, the formula above runs fine in a spreadsheet."}]},{"t":"h2","id":"disagreements-are-data-not-friction","runs":[{"text":"Disagreements are data, not friction"}]},{"t":"p","runs":[{"text":"When the two of you differ on six of twenty, sit down and go through the six. Each one resolves into exactly one of three things:"}]},{"t":"ul","items":[[{"text":"The guide is silent.","b":true},{"text":" Add a boundary ruling. This is the good case and it is most of them."}],[{"text":"One person misread the item.","b":true},{"text":" Fine. No change needed."}],[{"text":"The two of you actually want different things from the product.","b":true},{"text":" This is the expensive discovery, and it is far cheaper found now than after launch. It is not a labelling problem at all; it is a product decision wearing a labelling problem’s clothes."}]]},{"t":"p","runs":[{"text":"Re-label a fresh 20 after each guide revision. Two rounds is usually enough. Three means the criterion is hard, and you should consider splitting it into two simpler ones."}]},{"t":"h2","id":"label-noise-is-a-ceiling","runs":[{"text":"Label noise is a ceiling"}]},{"t":"p","runs":[{"text":"Here is the consequence people miss. If your labels are themselves 92% correct, then a perfect model scores 92% on your set; and worse, a model that has learned your labellers’ particular mistakes scores "},{"text":"higher","i":true},{"text":" than one that is right. No amount of prompt engineering climbs above the quality of the ruler."}]},{"t":"p","runs":[{"text":"So when a score plateaus around 90%, the next honest step is not another prompt variant. It is to take the 30 items the system failed, re-label them blind, and find out how many of those “failures” were your own labels being wrong. Teams routinely find a third."}]},{"t":"h2","id":"tools-including-the-free-ones","runs":[{"text":"Tools, including the free ones"}]},{"t":"p","runs":[{"text":"At 100 items, a spreadsheet with one row per item and one column per rule is not a compromise; it is the right tool, and it exports to CSV. Past a few hundred items, or with more than two labellers, "},{"text":"Label Studio","b":true},{"text":" and "},{"text":"Argilla","b":true},{"text":" are both open source, self-hostable and free, and they give you the two things a spreadsheet does not: blind labelling, so nobody sees the other person’s answer, and a built-in agreement report. Commercial platforms such as Scale or Labelbox do the same job with a sales call attached."}]},{"t":"p","runs":[{"text":"Whatever you use, keep the guide in version control beside the data, and put the date on it. A guide with no history is a guide nobody can trust six months later, including you."}]}],"check":{"question":[{"text":"Two reviewers independently label 200 support replies for a rare policy violation. They agree on 191 of them, which sounds excellent. Only 12 of the 200 were actually violations. What should they conclude?"}],"options":[{"text":[{"text":"Agreement is high enough to proceed; 95.5% is comfortably above any reasonable bar."}],"correct":false,"tempting":[{"text":"Reads raw agreement at face value when almost all items are negatives, so most of that 95.5% is agreement about the easy cases."}]},{"text":[{"text":"Because violations are rare, most of that agreement is chance; they should compute kappa, which will be far lower, before trusting the criterion."}],"correct":true,"tempting":null},{"text":[{"text":"The nine disagreements are noise from reviewer fatigue and can be resolved by whichever reviewer is more senior."}],"correct":false,"tempting":[{"text":"Treats disagreements as an inconvenience to be settled by authority rather than as the signal that shows where the guide is silent."}]},{"text":[{"text":"They need a larger sample, since 200 items give too few violations to measure anything."}],"correct":false,"tempting":[{"text":"Reaches for more data when the immediate problem is which statistic is being read, not how much of it there is."}]}],"order":[3,2,0,1],"why":null},"prev":{"slug":"where-examples-come-from","title":"Where the examples come from","minutes":9},"next":{"slug":"agreement-arithmetic","title":"How much your labellers actually agree","minutes":9}}}