Writing the answers, and keeping them consistent

Rewriting beats writing

Producing a gold answer from a blank page takes most people four to eight minutes. Taking a model’s draft and correcting it takes one to two. The quality is comparable and often better, because the draft reminds the annotator of details they would have omitted.

So the standard workflow is: generate a draft with the best model you have, then have a person fix it. Do this and you will produce three to five times more data for the same human hours.

The risk is anchoring. A reviewer presented with a fluent, confident, wrong answer accepts it more often than they would reject a blank page. Two countermeasures, both cheap:

• Have the reviewer state the answer’s key claim before reading the draft on a sample of items, and compare.
• Track the edit rate. If more than about 70% of drafts are accepted unchanged, either the model is good (check by having a second person audit 30 of them) or your reviewers have stopped reading.

Consistency is the thing you are teaching

This is the point people miss. The model is not learning “the right answer”; it is learning the distribution of your answers. If half your annotators start with a greeting and half do not, the model learns that greetings are a coin flip, and it will flip that coin in production.

Everything that varies across your examples and is not meant to be information becomes noise the model faithfully reproduces:

• Greeting or no greeting
• Bullet points or prose
• Whether the reasoning is shown
• British or American spelling
• Whether uncertainty is expressed, and in what words
• Length

Pick one of each, write them down, and enforce them.

The style guide is a page, not a document

## Answer style — ticket triage

1. No greeting, no sign-off. Start with the substance.
2. British spelling.
3. One paragraph under 60 words, unless listing steps; then a numbered list.
4. Never apologise for the product. State the position.
5. If the answer is not in the provided context: "I don't have that
   information — I'll pass this to the team." Exactly those words.
6. Never invent an order number, a date or a policy figure.
7. Amounts as "£12.50", never "12.50 GBP" or "twelve pounds fifty".
8. If the customer is angry, acknowledge once in the first clause, then move on.

Eight rules. Every annotator reads it before their first example and re-reads it before their hundredth, because drift is real and mostly unconscious.

Rule 5 is the pattern worth copying: exactly those words. If you want a model to have a reliable way of declining, give it one form of words, not eleven variations on the sentiment.

Measuring agreement, cheaply

Give two people the same 30 items. Have them write answers independently. Then diff.

You are not computing a statistic (30 free-text answers do not support one) you are finding the rules your style guide is missing. Every systematic disagreement is a decision nobody has made yet. Make it, add it to the guide, and continue.

Do this again at 300 examples and at 1,000. Annotator drift over a long labelling session is substantial and nobody notices it in themselves.

The adjudication rule

When two annotators disagree and neither is wrong, do not average and do not alternate. Pick one and record why. An averaged style is a style the model cannot learn because it never sees it.

What to do with the disagreements you cannot resolve

Some items are ambiguous: the customer’s request could reasonably be read two ways. These are not annotation failures, they are product questions, and they should be escalated rather than resolved by whoever happens to be labelling.

Meanwhile, take them out of the training set. An ambiguous item with an arbitrary answer teaches the model to be arbitrary at exactly the point where you most want it to ask a clarifying question. And if that is the behaviour you want, then write that as the gold answer, consistently, and you will have taught something useful instead.

The number that tells you when to stop

Stop hand-writing when a fresh batch of 50 examples produces no new style-guide rules and no new disagreements. At that point the remaining variance is in the inputs, not in your understanding of the task, and volume from cheaper sources will help more than another week of careful writing.