A hundred examples, made by hand

Where the examples come from

Real traffic, if you have it. Pull a week of production inputs, sample 100, and label them. That is the whole method and it beats every clever alternative.

If you have no traffic yet, you still have sources that are not your imagination: support tickets, the search box on your existing site, WhatsApp messages your sales person forwards you, the spreadsheet the ops team maintains by hand today. A used-car marketplace in Bogotá built its first eval set from 100 rows of the WhatsApp export its two agents had been answering manually for a year. Those 100 rows contained things nobody would have invented: customers sending voice-note transcriptions, prices written as “18 palos”, three different spellings of the same neighbourhood.

Stratify, then label

Do not sample 100 items and hope the hard cases show up. Decide the slices you care about first, then fill each one:

• the common case, in proportion (usually 50 to 60 items)
• the known-hard cases: other languages, very long inputs, missing information, ambiguous requests
• the ones that already broke in production, every single one
• a few that should be refused: abuse, off-topic, requests you do not serve

Tag every item with its slice. Overall pass rate is the headline; pass rate per slice is where you actually learn something.

Keep it in the repository as JSONL, next to the code, in version control:

{"id":"addr-014",
 "input":"opposite blue mosque, after 2nd speed bump, ikeja",
 "expected":{"area":"Ikeja","landmark":"blue mosque","street":null},
 "tags":["landmark-only","no-street-name"]}

Why a hundred

Because of what it can and cannot see. A 100-item set measures a pass rate to roughly plus or minus ten points. It will reliably show you a 15-point change. It will not show you a 3-point change, and you should stop pretending otherwise: teams routinely announce improvements far smaller than their set can resolve.

So: 100 is enough to make decisions about prompt rewrites, model swaps, and “is this feature ready”. It is not enough to tune. If you need to detect small differences, you need thousands of items, which means automated labelling, which means the next lesson’s problems. Most teams never need that.

A hundred items also takes about three hours to label properly. Three hours is a price you can pay this week.

Two people, twenty items

Before you label the rest, have a colleague independently label the same 20 items. Then compare.

If you disagree on 6 of 20, your criterion is broken, not the model. Two competent people cannot agree on what “a helpful reply” means, so no model can be scored against it and no judge can automate it. Rewrite the criterion until two humans agree at least 90% of the time, then label the rest. This single step catches more bad evaluation than anything else in this course.

Freeze it, then grow it

Once labelled, the set is frozen. You do not edit it to make a number look better. When production produces a failure your set did not predict, that failure becomes a new item: this is how the set earns its keep over a year.

One legitimate reason to touch existing items: your own labels were wrong. That happens, often to 5 or 10 items in a first set. Fix them, write down what you changed, and re-run every previous version against the corrected set so your history stays comparable. A corrected label with a note is honest. A quietly deleted hard item is how a set slowly becomes a set of things you already pass.