The set of examples you are allowed to trust

Nothing else in this module works without it

Every decision in this course (what k should be, whether the reranker earns its latency, whether the examples beat the instruction, whether a delimiter change was safe) reduces to comparing two variants. A comparison needs a set of cases. The quality of that set determines the quality of every conclusion you will draw for the next two years.

It is also the part teams skip, because writing twenty example questions takes an hour and building a real one takes a week.

Sample it, do not invent it

Imagined test cases are systematically wrong in the same direction. They are well-formed, single-intent, correctly spelled, in one language, and about the thing the product is for.

Real traffic is none of those. It contains half-sentences, two questions in one message, a pasted error, a follow-up with an unresolved pronoun, a question about last year, transliterated Hindi in Latin script, and a substantial number of messages that are not questions at all.

A system evaluated only on the first kind looks fine and behaves badly, and the gap is invisible until someone complains. So take the set from logs. If you have no traffic yet, take it from the support queue, from the questions colleagues actually ask, or from the first fifty real sessions, and replace it with sampled traffic as soon as you have any.

Label the requirement, not the answer

The instinct is to store the correct answer text and compare. This fails almost immediately, because there are many correct wordings and string comparison rejects all but one.

Store the requirement instead: the property that makes an answer acceptable:

{
  "id": "gs-0042",
  "query": "can i return a sale item after 30 days",
  "must_cite": ["returns-policy#sale-items"],
  "must_contain": ["14 days"],
  "must_not_contain": ["30 days"],
  "must_abstain": false,
  "notes": "Sale items are 14 days, not the standard 30. Common error."
}

Most requirements are checkable by code, for free: a substring, a cited identifier, a refusal, the presence of a number. Only the open-ended ones need a judge, and pushing as much as possible into mechanical checks is what keeps an evaluation cheap enough to run on every change.

Stratify deliberately

A random sample of traffic is dominated by easy, frequent questions, and a change that helps the hard tail will not move the number at all. Build the set in named groups and report each separately:

• Common. The frequent, straightforward cases. This is the regression guard: the group you are protecting, not improving.
• Known failures. Every case a user complained about, every bug you fixed. This group only grows, and it is the most valuable one you own.
• Abstentions. Questions the corpus does not answer, where the only correct behaviour is to say so. If this group is missing, every change that makes the model more willing to guess will look like an improvement.
• Edge and adversarial. Ambiguous phrasing, multi-part questions, questions about the wrong entity, injections from the security module.
• Languages and scripts you serve. Separately, because an average across languages hides a collapse in one.

Reporting one aggregate number across these groups is how a change that improves the common case by two points and destroys abstention by twenty gets shipped.

Split it, and mean it

If you tune against the whole set, your score stops predicting production. You have not measured the system; you have measured how many times you adjusted the prompt until this particular set went up.

So split: a development portion you iterate against freely, and a held-out portion you look at rarely: before a release, not during an afternoon of tinkering. The held-out number is the one you report and the one you believe.

The discipline is harder than the mechanism. The temptation to peek is constant and every peek costs a little of the number’s meaning. A workable compromise many teams use: the held-out set is run by CI on release candidates only, and nobody looks at individual failures in it until after the release decision.

How many

There is no universal answer, and the shape of it is: below about 50 cases you can only detect very large differences, a few hundred is where most teams land, and beyond a thousand the cost of maintaining labels starts to dominate.

The next lesson but two is about what a given size can actually resolve, which is the question underneath “how many”. Start with 100 real, well-labelled cases rather than 500 invented ones. A small honest set beats a large fictional one, and it is also the one you will actually keep up to date.

It rots

Traffic drifts, the corpus changes, the product adds features. A set built eighteen months ago is measuring a system that no longer exists, and the most dangerous version of this is a set whose answers have gone stale: where the policy changed and the labels did not, so the evaluation now punishes correct behaviour.

Put a review in the calendar quarterly. Re-sample a slice of recent traffic, re-check the labels against current documents, and add everything that went wrong since. Record the date the set was last reviewed next to the score, because a number from an unreviewed set deserves less confidence and nothing else will tell you that.