{"course":{"slug":"context-engineering","title":"Context Engineering","url":"https://addaly.com/learn/context-engineering","level":"advanced","track":"build"},"lesson":{"slug":"the-golden-set","title":"The set of examples you are allowed to trust","url":"https://addaly.com/learn/context-engineering/the-golden-set","position":79,"of":87,"minutes":9,"answer":"A golden set must be sampled from real traffic, labelled with required behaviour rather than expected wording, stratified to include abstentions and known failures.","keyIdea":"A golden set must be sampled from real traffic, labelled with required behaviour rather than expected wording, stratified to include abstentions and known failures, and split so that the portion you tune against is not the portion you report from."},"body":{"format":"markdown","text":"## Nothing else in this module works without it\n\nEvery decision in this course (what k should be, whether the reranker earns its latency, whether the examples beat the instruction, whether a delimiter change was safe) reduces to comparing two variants. A comparison needs a set of cases. The quality of that set determines the quality of every conclusion you will draw for the next two years.\n\nIt is also the part teams skip, because writing twenty example questions takes an hour and building a real one takes a week.\n\n## Sample it, do not invent it\n\nImagined test cases are systematically wrong in the same direction. They are well-formed, single-intent, correctly spelled, in one language, and about the thing the product is for.\n\nReal traffic is none of those. It contains half-sentences, two questions in one message, a pasted error, a follow-up with an unresolved pronoun, a question about last year, transliterated Hindi in Latin script, and a substantial number of messages that are not questions at all.\n\nA system evaluated only on the first kind looks fine and behaves badly, and the gap is invisible until someone complains. So take the set from logs. If you have no traffic yet, take it from the support queue, from the questions colleagues actually ask, or from the first fifty real sessions, and replace it with sampled traffic as soon as you have any.\n\n## Label the requirement, not the answer\n\nThe instinct is to store the correct answer text and compare. This fails almost immediately, because there are many correct wordings and string comparison rejects all but one.\n\nStore the *requirement* instead: the property that makes an answer acceptable:\n\n```json\n{\n  \"id\": \"gs-0042\",\n  \"query\": \"can i return a sale item after 30 days\",\n  \"must_cite\": [\"returns-policy#sale-items\"],\n  \"must_contain\": [\"14 days\"],\n  \"must_not_contain\": [\"30 days\"],\n  \"must_abstain\": false,\n  \"notes\": \"Sale items are 14 days, not the standard 30. Common error.\"\n}\n```\n\nMost requirements are checkable by code, for free: a substring, a cited identifier, a refusal, the presence of a number. Only the open-ended ones need a judge, and pushing as much as possible into mechanical checks is what keeps an evaluation cheap enough to run on every change.\n\n## Stratify deliberately\n\nA random sample of traffic is dominated by easy, frequent questions, and a change that helps the hard tail will not move the number at all. Build the set in named groups and report each separately:\n\n- **Common.** The frequent, straightforward cases. This is the regression guard: the group you are protecting, not improving.\n- **Known failures.** Every case a user complained about, every bug you fixed. This group only grows, and it is the most valuable one you own.\n- **Abstentions.** Questions the corpus does not answer, where the only correct behaviour is to say so. If this group is missing, every change that makes the model more willing to guess will look like an improvement.\n- **Edge and adversarial.** Ambiguous phrasing, multi-part questions, questions about the wrong entity, injections from the security module.\n- **Languages and scripts you serve.** Separately, because an average across languages hides a collapse in one.\n\nReporting one aggregate number across these groups is how a change that improves the common case by two points and destroys abstention by twenty gets shipped.\n\n> Figure: A hundred and forty cases, in named groups. Bars (cases): Common, frequent questions 60; Known failures, each one reported by a user 30; Abstentions, where the corpus has no answer 20; Edge and adversarial phrasing 18; Each language and script you serve 12. Reported separately, always. One aggregate number across these groups is how a change that improves the common case by two points and destroys abstention by twenty gets shipped. Split the set again into a development portion you iterate against freely and a held-out portion you look at before a release rather than during an afternoon of tinkering.\n\n## Split it, and mean it\n\nIf you tune against the whole set, your score stops predicting production. You have not measured the system; you have measured how many times you adjusted the prompt until this particular set went up.\n\nSo split: a **development** portion you iterate against freely, and a **held-out** portion you look at rarely: before a release, not during an afternoon of tinkering. The held-out number is the one you report and the one you believe.\n\nThe discipline is harder than the mechanism. The temptation to peek is constant and every peek costs a little of the number's meaning. A workable compromise many teams use: the held-out set is run by CI on release candidates only, and nobody looks at individual failures in it until after the release decision.\n\n## How many\n\nThere is no universal answer, and the shape of it is: below about 50 cases you can only detect very large differences, a few hundred is where most teams land, and beyond a thousand the cost of maintaining labels starts to dominate.\n\nThe next lesson but two is about what a given size can actually resolve, which is the question underneath \"how many\". Start with 100 real, well-labelled cases rather than 500 invented ones. A small honest set beats a large fictional one, and it is also the one you will actually keep up to date.\n\n## It rots\n\nTraffic drifts, the corpus changes, the product adds features. A set built eighteen months ago is measuring a system that no longer exists, and the most dangerous version of this is a set whose *answers* have gone stale: where the policy changed and the labels did not, so the evaluation now punishes correct behaviour.\n\nPut a review in the calendar quarterly. Re-sample a slice of recent traffic, re-check the labels against current documents, and add everything that went wrong since. Record the date the set was last reviewed next to the score, because a number from an unreviewed set deserves less confidence and nothing else will tell you that."},"check":{"question":"A golden set is sampled randomly from traffic and reported as one accuracy figure. A change improves it by two points. What has the design failed to reveal?","options":[{"text":"Whether the improvement came from retrieval or from generation, since both stages contribute to the same figure.","correct":false,"tempting":"A genuine limitation of end-to-end scoring, but it is fixed by stage metrics rather than by how the set is sampled or grouped."},{"text":"Whether a rare group, such as abstention cases, collapsed while the frequent cases improved.","correct":true},{"text":"Whether the two-point difference is larger than the noise of the measurement itself.","correct":false,"tempting":"A real and separate problem about significance; stratification would not answer it, and grouping is the specific gap here."},{"text":"Whether the labels are still valid, given that policies referenced by the corpus may have changed since collection.","correct":false,"tempting":"Label rot is a maintenance failure that afflicts stratified and unstratified sets alike, so it is not what this design choice hides."}]},"prev":{"slug":"prompts-you-can-roll-back","title":"A prompt you can version, test and roll back","url":"https://addaly.com/learn/context-engineering/prompts-you-can-roll-back"},"next":{"slug":"offline-checks-on-the-assembler","title":"The tests that need no model at all","url":"https://addaly.com/learn/context-engineering/offline-checks-on-the-assembler"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'The set of examples you are allowed to trust', in Context Engineering. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/context-engineering/the-golden-set (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"The set of examples you are allowed to trust","courseTitle":"Context Engineering","module":{"index":9,"slug":"shipping-and-measuring","title":"Shipping context, and proving it got better"},"answer":null,"keyIdea":"A golden set must be sampled from real traffic, labelled with required behaviour rather than expected wording, stratified to include abstentions and known failures, and split so that the portion you tune against is not the portion you report from.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"type":"bars","unit":"cases","title":"A hundred and forty cases, in named groups","series":[{"label":"Common, frequent questions","value":60},{"label":"Known failures, each one reported by a user","value":30},{"label":"Abstentions, where the corpus has no answer","value":20,"highlight":true},{"label":"Edge and adversarial phrasing","value":18},{"label":"Each language and script you serve","value":12}],"caption":"Reported separately, always. One aggregate number across these groups is how a change that improves the common case by two points and destroys abstention by twenty gets shipped. Split the set again into a development portion you iterate against freely and a held-out portion you look at before a release rather than during an afternoon of tinkering."},"alt":"Figure: A hundred and forty cases, in named groups. Bars (cases): Common, frequent questions 60; Known failures, each one reported by a user 30; Abstentions, where the corpus has no answer 20; Edge and adversarial phrasing 18; Each language and script you serve 12. Reported separately, always. One aggregate number across these groups is how a change that improves the common case by two points and destroys abstention by twenty gets shipped. Split the set again into a development portion you iterate against freely and a held-out portion you look at before a release rather than during an afternoon of tinkering."}},"blocks":[{"t":"h2","id":"nothing-else-in-this-module-works-without-it","runs":[{"text":"Nothing else in this module works without it"}]},{"t":"p","runs":[{"text":"Every decision in this course (what k should be, whether the reranker earns its latency, whether the examples beat the instruction, whether a delimiter change was safe) reduces to comparing two variants. A comparison needs a set of cases. The quality of that set determines the quality of every conclusion you will draw for the next two years."}]},{"t":"p","runs":[{"text":"It is also the part teams skip, because writing twenty example questions takes an hour and building a real one takes a week."}]},{"t":"h2","id":"sample-it-do-not-invent-it","runs":[{"text":"Sample it, do not invent it"}]},{"t":"p","runs":[{"text":"Imagined test cases are systematically wrong in the same direction. They are well-formed, single-intent, correctly spelled, in one language, and about the thing the product is for."}]},{"t":"p","runs":[{"text":"Real traffic is none of those. It contains half-sentences, two questions in one message, a pasted error, a follow-up with an unresolved pronoun, a question about last year, transliterated Hindi in Latin script, and a substantial number of messages that are not questions at all."}]},{"t":"p","runs":[{"text":"A system evaluated only on the first kind looks fine and behaves badly, and the gap is invisible until someone complains. So take the set from logs. If you have no traffic yet, take it from the support queue, from the questions colleagues actually ask, or from the first fifty real sessions, and replace it with sampled traffic as soon as you have any."}]},{"t":"h2","id":"label-the-requirement-not-the-answer","runs":[{"text":"Label the requirement, not the answer"}]},{"t":"p","runs":[{"text":"The instinct is to store the correct answer text and compare. This fails almost immediately, because there are many correct wordings and string comparison rejects all but one."}]},{"t":"p","runs":[{"text":"Store the "},{"text":"requirement","i":true},{"text":" instead: the property that makes an answer acceptable:"}]},{"t":"code","lang":"json","label":"JSON","text":"{\n  \"id\": \"gs-0042\",\n  \"query\": \"can i return a sale item after 30 days\",\n  \"must_cite\": [\"returns-policy#sale-items\"],\n  \"must_contain\": [\"14 days\"],\n  \"must_not_contain\": [\"30 days\"],\n  \"must_abstain\": false,\n  \"notes\": \"Sale items are 14 days, not the standard 30. Common error.\"\n}"},{"t":"p","runs":[{"text":"Most requirements are checkable by code, for free: a substring, a cited identifier, a refusal, the presence of a number. Only the open-ended ones need a judge, and pushing as much as possible into mechanical checks is what keeps an evaluation cheap enough to run on every change."}]},{"t":"h2","id":"stratify-deliberately","runs":[{"text":"Stratify deliberately"}]},{"t":"p","runs":[{"text":"A random sample of traffic is dominated by easy, frequent questions, and a change that helps the hard tail will not move the number at all. Build the set in named groups and report each separately:"}]},{"t":"ul","items":[[{"text":"Common.","b":true},{"text":" The frequent, straightforward cases. This is the regression guard: the group you are protecting, not improving."}],[{"text":"Known failures.","b":true},{"text":" Every case a user complained about, every bug you fixed. This group only grows, and it is the most valuable one you own."}],[{"text":"Abstentions.","b":true},{"text":" Questions the corpus does not answer, where the only correct behaviour is to say so. If this group is missing, every change that makes the model more willing to guess will look like an improvement."}],[{"text":"Edge and adversarial.","b":true},{"text":" Ambiguous phrasing, multi-part questions, questions about the wrong entity, injections from the security module."}],[{"text":"Languages and scripts you serve.","b":true},{"text":" Separately, because an average across languages hides a collapse in one."}]]},{"t":"p","runs":[{"text":"Reporting one aggregate number across these groups is how a change that improves the common case by two points and destroys abstention by twenty gets shipped."}]},{"t":"h2","id":"split-it-and-mean-it","runs":[{"text":"Split it, and mean it"}]},{"t":"p","runs":[{"text":"If you tune against the whole set, your score stops predicting production. You have not measured the system; you have measured how many times you adjusted the prompt until this particular set went up."}]},{"t":"p","runs":[{"text":"So split: a "},{"text":"development","b":true},{"text":" portion you iterate against freely, and a "},{"text":"held-out","b":true},{"text":" portion you look at rarely: before a release, not during an afternoon of tinkering. The held-out number is the one you report and the one you believe."}]},{"t":"p","runs":[{"text":"The discipline is harder than the mechanism. The temptation to peek is constant and every peek costs a little of the number’s meaning. A workable compromise many teams use: the held-out set is run by CI on release candidates only, and nobody looks at individual failures in it until after the release decision."}]},{"t":"h2","id":"how-many","runs":[{"text":"How many"}]},{"t":"p","runs":[{"text":"There is no universal answer, and the shape of it is: below about 50 cases you can only detect very large differences, a few hundred is where most teams land, and beyond a thousand the cost of maintaining labels starts to dominate."}]},{"t":"p","runs":[{"text":"The next lesson but two is about what a given size can actually resolve, which is the question underneath “how many”. Start with 100 real, well-labelled cases rather than 500 invented ones. A small honest set beats a large fictional one, and it is also the one you will actually keep up to date."}]},{"t":"h2","id":"it-rots","runs":[{"text":"It rots"}]},{"t":"p","runs":[{"text":"Traffic drifts, the corpus changes, the product adds features. A set built eighteen months ago is measuring a system that no longer exists, and the most dangerous version of this is a set whose "},{"text":"answers","i":true},{"text":" have gone stale: where the policy changed and the labels did not, so the evaluation now punishes correct behaviour."}]},{"t":"p","runs":[{"text":"Put a review in the calendar quarterly. Re-sample a slice of recent traffic, re-check the labels against current documents, and add everything that went wrong since. Record the date the set was last reviewed next to the score, because a number from an unreviewed set deserves less confidence and nothing else will tell you that."}]}],"check":{"question":[{"text":"A golden set is sampled randomly from traffic and reported as one accuracy figure. A change improves it by two points. What has the design failed to reveal?"}],"options":[{"text":[{"text":"Whether the improvement came from retrieval or from generation, since both stages contribute to the same figure."}],"correct":false,"tempting":[{"text":"A genuine limitation of end-to-end scoring, but it is fixed by stage metrics rather than by how the set is sampled or grouped."}]},{"text":[{"text":"Whether a rare group, such as abstention cases, collapsed while the frequent cases improved."}],"correct":true,"tempting":null},{"text":[{"text":"Whether the two-point difference is larger than the noise of the measurement itself."}],"correct":false,"tempting":[{"text":"A real and separate problem about significance; stratification would not answer it, and grouping is the specific gap here."}]},{"text":[{"text":"Whether the labels are still valid, given that policies referenced by the corpus may have changed since collection."}],"correct":false,"tempting":[{"text":"Label rot is a maintenance failure that afflicts stratified and unstratified sets alike, so it is not what this design choice hides."}]}],"order":[0,3,2,1],"why":null},"prev":{"slug":"prompts-you-can-roll-back","title":"A prompt you can version, test and roll back","minutes":10},"next":{"slug":"offline-checks-on-the-assembler","title":"The tests that need no model at all","minutes":8}}}