{"course":{"slug":"evaluating-ai","title":"Knowing If It Works","url":"https://addaly.com/learn/evaluating-ai","level":"intermediate","track":"build"},"lesson":{"slug":"where-examples-come-from","title":"Where the examples come from","url":"https://addaly.com/learn/evaluating-ai/where-examples-come-from","position":4,"of":76,"minutes":9,"answer":"Sample your traffic deliberately (stratified, deduplicated, and weighted back to real proportions) because the easiest hundred items to grab are the least representative hundred you own.","keyIdea":"Sample your traffic deliberately (stratified, deduplicated, and weighted back to real proportions) because the easiest hundred items to grab are the least representative hundred you own."},"body":{"format":"markdown","text":"## The last hundred requests are not a sample\n\nThe quickest way to build an evaluation set is to open the logs, copy the most recent hundred rows, and start labelling. It feels like real data because it is real data. It is still a bad sample, for four reasons you can name before you look.\n\n- **Recency.** The last hundred rows cover roughly the last twenty minutes. If you pull them at 11am on a Tuesday you have a working-hours, weekday, one-time-zone sample.\n- **Volume bias.** One integration customer running a nightly script can be forty of your hundred rows. You will tune the system for that script.\n- **Survivor bias.** Logs contain what people asked a system that already exists. Questions your product visibly cannot answer stopped being asked months ago.\n- **Head dominance.** Most traffic is a handful of intents. A uniform sample of 300 from 40,000 weekly queries might contain two refund requests, which is not enough to measure anything about refunds.\n\n## Draw the sample on purpose\n\nDecide the strata first: the dimensions along which the system plausibly behaves differently. For a support assistant that is usually intent, language, and input length. Then sample within each stratum.\n\n```python\n# 25 items per intent, reproducible, from a week of logs\nsample = (df\n          .drop_duplicates(subset=\"normalised_text\")\n          .groupby(\"intent\", group_keys=False)\n          .apply(lambda g: g.sample(n=min(25, len(g)), random_state=0)))\n```\n\nThis is stratified sampling, and it buys you the ability to say something about refunds. It also breaks your headline number, and you have to fix that deliberately.\n\n## Weight back, or do not quote a headline\n\nIf refunds are 1% of traffic and 12% of your set, then a set-wide pass rate is answering a question about a world that does not exist. Two honest options:\n\n1. **Report per stratum and never average.** \"94% on billing, 71% on refunds, 88% on account changes.\" Most useful in practice.\n2. **Weight back to production proportions.** Multiply each stratum's rate by its real traffic share and sum.\n\n```python\noverall = sum(rate[s] * traffic_share[s] for s in strata)\n```\n\nThe weighted number has wider uncertainty than its item count suggests, because the rare strata carry heavy weights on few items. Say so when you quote it.\n\n## Deduplicate before you label, not after\n\nNear-duplicates are the silent inflater. Four hundred copies of the same automated message make a set that looks like 400 items and behaves like one. Normalise (lowercase, strip identifiers and timestamps), hash, and keep one, but record how often it occurred, because that frequency is exactly what you need for the weighting above.\n\nThe check that catches this: if any single normalised string is more than 2% of your set, it is not a set, it is a template.\n\n## Real user text is personal data\n\nLogs are people's words. Before any of it lands in an evaluation file:\n\n- Redact identifiers automatically first (emails, phone numbers, account numbers, addresses) and then have a human read the sample, because automatic redaction misses the sentence where somebody names their employer.\n- Check what your privacy notice actually promised, and keep the set inside whatever retention window applies.\n- Store it where the rest of your production data lives, not in a spreadsheet on a laptop. An eval set is a copy of your users' most sensitive text with the access controls removed.\n\nWhere policy forbids using real content at all, you fall back to team-written and synthetic items, and you say in the report that the set is not drawn from traffic. That is a real limitation, not a footnote.\n\n## When you have no logs at all\n\nA feature that has not shipped has no traffic. You then have three sources, in descending order of trustworthiness:\n\n1. **Adjacent real text**: support tickets, search queries, forum posts about the same job, even if they were never typed at your system.\n2. **Team-written items**, produced by people who have watched users work. Have two people write independently and compare; the overlap is the obvious case, the difference is the interesting one.\n3. **Synthetic items**, which are the next lesson and come with the largest caveats.\n\nWrite the provenance of every item into the file (`source: logs | adjacent | team | synthetic`) because six months later nobody remembers, and every argument about a surprising number starts with where the items came from.\n\n## The habit\n\nSample once a quarter, not once. Traffic moves: a marketing campaign, a new market, a competitor's outage, a festival. A set drawn in March and still quoted in November is measuring a product that no longer exists. Keep the March set frozen for comparability, and add a fresh sample beside it: the gap between the two is itself a measurement."},"check":{"question":"A team stratifies its eval set so that refunds, which are 1% of live traffic, make up 12% of the 300 items. They report a single set-wide pass rate of 86%. What has gone wrong?","options":[{"text":"Nothing: stratifying makes the set more representative, so the average is more trustworthy than a uniform sample would have been.","correct":false,"tempting":"Confuses coverage with representativeness: stratifying buys resolution on rare strata, and by construction destroys the traffic proportions the average depends on."},{"text":"The 86% describes a world where 12% of requests are refunds, so it overstates or understates live performance depending on whether refunds are harder.","correct":true},{"text":"Stratified samples cannot be used for pass rates at all, only for precision and recall.","correct":false,"tempting":"Invents a restriction; stratified samples support any metric, provided the strata are reported separately or weighted back."},{"text":"The set is too small for a headline number, and 3,000 items sampled the same way would fix it.","correct":false,"tempting":"Treats a bias as a sample-size problem; ten times the items sampled in the same proportions gives the same distorted average, just with tighter error bars around the wrong figure."}]},"prev":{"slug":"the-hundred-examples","title":"A hundred examples, made by hand","url":"https://addaly.com/learn/evaluating-ai/the-hundred-examples"},"next":{"slug":"labelling-without-lying","title":"Writing a labelling guide two people can agree on","url":"https://addaly.com/learn/evaluating-ai/labelling-without-lying"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Where the examples come from', in Knowing If It Works. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/evaluating-ai/where-examples-come-from (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Where the examples come from","courseTitle":"Knowing If It Works","module":{"index":1,"slug":"evidence","title":"What counts as evidence"},"answer":null,"keyIdea":"Sample your traffic deliberately (stratified, deduplicated, and weighted back to real proportions) because the easiest hundred items to grab are the least representative hundred you own.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"the-last-hundred-requests-are-not-a-sample","runs":[{"text":"The last hundred requests are not a sample"}]},{"t":"p","runs":[{"text":"The quickest way to build an evaluation set is to open the logs, copy the most recent hundred rows, and start labelling. It feels like real data because it is real data. It is still a bad sample, for four reasons you can name before you look."}]},{"t":"ul","items":[[{"text":"Recency.","b":true},{"text":" The last hundred rows cover roughly the last twenty minutes. If you pull them at 11am on a Tuesday you have a working-hours, weekday, one-time-zone sample."}],[{"text":"Volume bias.","b":true},{"text":" One integration customer running a nightly script can be forty of your hundred rows. You will tune the system for that script."}],[{"text":"Survivor bias.","b":true},{"text":" Logs contain what people asked a system that already exists. Questions your product visibly cannot answer stopped being asked months ago."}],[{"text":"Head dominance.","b":true},{"text":" Most traffic is a handful of intents. A uniform sample of 300 from 40,000 weekly queries might contain two refund requests, which is not enough to measure anything about refunds."}]]},{"t":"h2","id":"draw-the-sample-on-purpose","runs":[{"text":"Draw the sample on purpose"}]},{"t":"p","runs":[{"text":"Decide the strata first: the dimensions along which the system plausibly behaves differently. For a support assistant that is usually intent, language, and input length. Then sample within each stratum."}]},{"t":"code","lang":"python","label":"Python","text":"# 25 items per intent, reproducible, from a week of logs\nsample = (df\n          .drop_duplicates(subset=\"normalised_text\")\n          .groupby(\"intent\", group_keys=False)\n          .apply(lambda g: g.sample(n=min(25, len(g)), random_state=0)))"},{"t":"p","runs":[{"text":"This is stratified sampling, and it buys you the ability to say something about refunds. It also breaks your headline number, and you have to fix that deliberately."}]},{"t":"h2","id":"weight-back-or-do-not-quote-a-headline","runs":[{"text":"Weight back, or do not quote a headline"}]},{"t":"p","runs":[{"text":"If refunds are 1% of traffic and 12% of your set, then a set-wide pass rate is answering a question about a world that does not exist. Two honest options:"}]},{"t":"ol","start":1,"items":[[{"text":"Report per stratum and never average.","b":true},{"text":" “94% on billing, 71% on refunds, 88% on account changes.” Most useful in practice."}],[{"text":"Weight back to production proportions.","b":true},{"text":" Multiply each stratum’s rate by its real traffic share and sum."}]]},{"t":"code","lang":"python","label":"Python","text":"overall = sum(rate[s] * traffic_share[s] for s in strata)"},{"t":"p","runs":[{"text":"The weighted number has wider uncertainty than its item count suggests, because the rare strata carry heavy weights on few items. Say so when you quote it."}]},{"t":"h2","id":"deduplicate-before-you-label-not-after","runs":[{"text":"Deduplicate before you label, not after"}]},{"t":"p","runs":[{"text":"Near-duplicates are the silent inflater. Four hundred copies of the same automated message make a set that looks like 400 items and behaves like one. Normalise (lowercase, strip identifiers and timestamps), hash, and keep one, but record how often it occurred, because that frequency is exactly what you need for the weighting above."}]},{"t":"p","runs":[{"text":"The check that catches this: if any single normalised string is more than 2% of your set, it is not a set, it is a template."}]},{"t":"h2","id":"real-user-text-is-personal-data","runs":[{"text":"Real user text is personal data"}]},{"t":"p","runs":[{"text":"Logs are people’s words. Before any of it lands in an evaluation file:"}]},{"t":"ul","items":[[{"text":"Redact identifiers automatically first (emails, phone numbers, account numbers, addresses) and then have a human read the sample, because automatic redaction misses the sentence where somebody names their employer."}],[{"text":"Check what your privacy notice actually promised, and keep the set inside whatever retention window applies."}],[{"text":"Store it where the rest of your production data lives, not in a spreadsheet on a laptop. An eval set is a copy of your users’ most sensitive text with the access controls removed."}]]},{"t":"p","runs":[{"text":"Where policy forbids using real content at all, you fall back to team-written and synthetic items, and you say in the report that the set is not drawn from traffic. That is a real limitation, not a footnote."}]},{"t":"h2","id":"when-you-have-no-logs-at-all","runs":[{"text":"When you have no logs at all"}]},{"t":"p","runs":[{"text":"A feature that has not shipped has no traffic. You then have three sources, in descending order of trustworthiness:"}]},{"t":"ol","start":1,"items":[[{"text":"Adjacent real text","b":true},{"text":": support tickets, search queries, forum posts about the same job, even if they were never typed at your system."}],[{"text":"Team-written items","b":true},{"text":", produced by people who have watched users work. Have two people write independently and compare; the overlap is the obvious case, the difference is the interesting one."}],[{"text":"Synthetic items","b":true},{"text":", which are the next lesson and come with the largest caveats."}]]},{"t":"p","runs":[{"text":"Write the provenance of every item into the file ("},{"text":"source: logs | adjacent | team | synthetic","code":true},{"text":") because six months later nobody remembers, and every argument about a surprising number starts with where the items came from."}]},{"t":"h2","id":"the-habit","runs":[{"text":"The habit"}]},{"t":"p","runs":[{"text":"Sample once a quarter, not once. Traffic moves: a marketing campaign, a new market, a competitor’s outage, a festival. A set drawn in March and still quoted in November is measuring a product that no longer exists. Keep the March set frozen for comparability, and add a fresh sample beside it: the gap between the two is itself a measurement."}]}],"check":{"question":[{"text":"A team stratifies its eval set so that refunds, which are 1% of live traffic, make up 12% of the 300 items. They report a single set-wide pass rate of 86%. What has gone wrong?"}],"options":[{"text":[{"text":"Nothing: stratifying makes the set more representative, so the average is more trustworthy than a uniform sample would have been."}],"correct":false,"tempting":[{"text":"Confuses coverage with representativeness: stratifying buys resolution on rare strata, and by construction destroys the traffic proportions the average depends on."}]},{"text":[{"text":"The 86% describes a world where 12% of requests are refunds, so it overstates or understates live performance depending on whether refunds are harder."}],"correct":true,"tempting":null},{"text":[{"text":"Stratified samples cannot be used for pass rates at all, only for precision and recall."}],"correct":false,"tempting":[{"text":"Invents a restriction; stratified samples support any metric, provided the strata are reported separately or weighted back."}]},{"text":[{"text":"The set is too small for a headline number, and 3,000 items sampled the same way would fix it."}],"correct":false,"tempting":[{"text":"Treats a bias as a sample-size problem; ten times the items sampled in the same proportions gives the same distorted average, just with tighter error bars around the wrong figure."}]}],"order":[2,0,1,3],"why":null},"prev":{"slug":"the-hundred-examples","title":"A hundred examples, made by hand","minutes":8},"next":{"slug":"labelling-without-lying","title":"Writing a labelling guide two people can agree on","minutes":9}}}