Writing a labelling guide two people can agree on

The guide is the real artefact

When you label a hundred examples, the labels are not the valuable output. The document that made two people produce the same labels is. That document is what lets you add 100 more items next quarter, hand the task to somebody new, or write a judge prompt in module three. Without it, your set is a private opinion that decays as your memory of it fades.

A labelling guide is short. One page per label. It contains:

1. A definition in one sentence, stated as something observable in the output rather than something felt about it.
2. Three examples that pass, copied verbatim from real data.
3. Three that fail, with the specific span that fails it underlined in words.
4. Two boundary cases with a ruling: the ones you argued about, and the decision you made, even if the decision was arbitrary.

The fourth item is the one people skip and the one that does the work. Boundary rulings are how a criterion survives contact with data nobody imagined.

From adjective to observable

Most first drafts are unlabellable:

The reply should be helpful and professional.

Neither word points at anything in the text. Rewrite until each rule names something you could highlight with a cursor:

- The reply answers the question that was asked, rather than a related one.

- The reply contains no commitment about dates, money or eligibility that is absent from the policy text supplied.

- The reply is in the same language as the customer’s message.

- If required information is missing, the reply asks for it instead of assuming a value.

Four rules, each separately true or false. “Helpful and professional” is now four columns in a spreadsheet, and a disagreement can be located: you and your colleague differ on rule two, not on the vibe.

Measure agreement, and do not be fooled by the raw number

Have a second person label the same 20 items independently. Then compute agreement, but be careful which agreement you compute.

Suppose 94 of every 100 support replies are fine. Two labellers who both mark everything “fine” agree 94% of the time and have learned nothing. Raw agreement flatters you exactly when the classes are imbalanced, which is most of the time in safety and quality work.

Cohen’s kappa corrects for the agreement you would get by chance. Roughly:

kappa = (observed_agreement - chance_agreement)
        / (1 - chance_agreement)

In the example above, chance agreement is already about 0.89, so 94% raw becomes a kappa near 0.45: poor. Useful reference points, and they are conventions rather than laws:

• 0.8 and up: the criterion is solid; automate against it.
• 0.6 to 0.8: workable, but every measurement you take carries that much slack.
• below 0.6: you do not yet have a criterion. Stop labelling and rewrite the guide.

Three lines of Python get you there: sklearn.metrics.cohen_kappa_score(a, b). If you would rather not install anything, the formula above runs fine in a spreadsheet.

Disagreements are data, not friction

When the two of you differ on six of twenty, sit down and go through the six. Each one resolves into exactly one of three things:

• The guide is silent. Add a boundary ruling. This is the good case and it is most of them.
• One person misread the item. Fine. No change needed.
• The two of you actually want different things from the product. This is the expensive discovery, and it is far cheaper found now than after launch. It is not a labelling problem at all; it is a product decision wearing a labelling problem’s clothes.

Re-label a fresh 20 after each guide revision. Two rounds is usually enough. Three means the criterion is hard, and you should consider splitting it into two simpler ones.

Label noise is a ceiling

Here is the consequence people miss. If your labels are themselves 92% correct, then a perfect model scores 92% on your set; and worse, a model that has learned your labellers’ particular mistakes scores higher than one that is right. No amount of prompt engineering climbs above the quality of the ruler.

So when a score plateaus around 90%, the next honest step is not another prompt variant. It is to take the 30 items the system failed, re-label them blind, and find out how many of those “failures” were your own labels being wrong. Teams routinely find a third.

Tools, including the free ones

At 100 items, a spreadsheet with one row per item and one column per rule is not a compromise; it is the right tool, and it exports to CSV. Past a few hundred items, or with more than two labellers, Label Studio and Argilla are both open source, self-hostable and free, and they give you the two things a spreadsheet does not: blind labelling, so nobody sees the other person’s answer, and a built-in agreement report. Commercial platforms such as Scale or Labelbox do the same job with a sales call attached.

Whatever you use, keep the guide in version control beside the data, and put the date on it. A guide with no history is a guide nobody can trust six months later, including you.