Where the examples come from

The last hundred requests are not a sample

The quickest way to build an evaluation set is to open the logs, copy the most recent hundred rows, and start labelling. It feels like real data because it is real data. It is still a bad sample, for four reasons you can name before you look.

• Recency. The last hundred rows cover roughly the last twenty minutes. If you pull them at 11am on a Tuesday you have a working-hours, weekday, one-time-zone sample.
• Volume bias. One integration customer running a nightly script can be forty of your hundred rows. You will tune the system for that script.
• Survivor bias. Logs contain what people asked a system that already exists. Questions your product visibly cannot answer stopped being asked months ago.
• Head dominance. Most traffic is a handful of intents. A uniform sample of 300 from 40,000 weekly queries might contain two refund requests, which is not enough to measure anything about refunds.

Draw the sample on purpose

Decide the strata first: the dimensions along which the system plausibly behaves differently. For a support assistant that is usually intent, language, and input length. Then sample within each stratum.

# 25 items per intent, reproducible, from a week of logs
sample = (df
          .drop_duplicates(subset="normalised_text")
          .groupby("intent", group_keys=False)
          .apply(lambda g: g.sample(n=min(25, len(g)), random_state=0)))

This is stratified sampling, and it buys you the ability to say something about refunds. It also breaks your headline number, and you have to fix that deliberately.

Weight back, or do not quote a headline

If refunds are 1% of traffic and 12% of your set, then a set-wide pass rate is answering a question about a world that does not exist. Two honest options:

1. Report per stratum and never average. “94% on billing, 71% on refunds, 88% on account changes.” Most useful in practice.
2. Weight back to production proportions. Multiply each stratum’s rate by its real traffic share and sum.

overall = sum(rate[s] * traffic_share[s] for s in strata)

The weighted number has wider uncertainty than its item count suggests, because the rare strata carry heavy weights on few items. Say so when you quote it.

Deduplicate before you label, not after

Near-duplicates are the silent inflater. Four hundred copies of the same automated message make a set that looks like 400 items and behaves like one. Normalise (lowercase, strip identifiers and timestamps), hash, and keep one, but record how often it occurred, because that frequency is exactly what you need for the weighting above.

The check that catches this: if any single normalised string is more than 2% of your set, it is not a set, it is a template.

Real user text is personal data

Logs are people’s words. Before any of it lands in an evaluation file:

• Redact identifiers automatically first (emails, phone numbers, account numbers, addresses) and then have a human read the sample, because automatic redaction misses the sentence where somebody names their employer.
• Check what your privacy notice actually promised, and keep the set inside whatever retention window applies.
• Store it where the rest of your production data lives, not in a spreadsheet on a laptop. An eval set is a copy of your users’ most sensitive text with the access controls removed.

Where policy forbids using real content at all, you fall back to team-written and synthetic items, and you say in the report that the set is not drawn from traffic. That is a real limitation, not a footnote.

When you have no logs at all

A feature that has not shipped has no traffic. You then have three sources, in descending order of trustworthiness:

1. Adjacent real text: support tickets, search queries, forum posts about the same job, even if they were never typed at your system.
2. Team-written items, produced by people who have watched users work. Have two people write independently and compare; the overlap is the obvious case, the difference is the interesting one.
3. Synthetic items, which are the next lesson and come with the largest caveats.

Write the provenance of every item into the file (source: logs | adjacent | team | synthetic) because six months later nobody remembers, and every argument about a surprising number starts with where the items came from.

The habit

Sample once a quarter, not once. Traffic moves: a marketing campaign, a new market, a competitor’s outage, a festival. A set drawn in March and still quoted in November is measuring a product that no longer exists. Keep the March set frozen for comparability, and add a fresh sample beside it: the gap between the two is itself a measurement.