{"course":{"slug":"fine-tuning","title":"Fine-Tuning, and When Not To","url":"https://addaly.com/learn/fine-tuning","level":"advanced","track":"deep"},"lesson":{"slug":"a-held-out-set-that-holds","title":"A held-out set that actually holds","url":"https://addaly.com/learn/fine-tuning/a-held-out-set-that-holds","position":69,"of":84,"minutes":9,"answer":"Random splits leak through near-duplicates, groups, time and templates, so split by group and by time and then measure cross-split near-duplicate overlap, and keep a locked set you open a handful of times.","keyIdea":"Random splits leak through near-duplicates, groups, time and templates, so split by group and by time and then measure cross-split near-duplicate overlap, and keep a locked set you open a handful of times, because an evaluation set you have consulted forty times has become a training set for your own decisions."},"body":{"format":"markdown","text":"## Random splitting is not enough\n\nA random 90/10 split feels safe and leaks constantly. Four distinct leaks, each with a fix.\n\n**Near-duplicate leakage.** Your corpus contains variations of the same ticket. A random split puts some in training and their near-twins in evaluation, and the model scores well by recall rather than by generalisation.\n\nThis is the big one, it is invisible, and almost nobody measures it. Run the same MinHash comparison you used for cleaning, but *across the split*:\n\n```python\nlsh = MinHashLSH(threshold=0.8, num_perm=128)\nfor i, t in enumerate(train_texts):\n    lsh.insert(f\"tr{i}\", minhash(t))\nleaked = sum(1 for t in eval_texts if lsh.query(minhash(t)))\nprint(f\"{leaked}/{len(eval_texts)} eval items have a near-duplicate in train\")\n```\n\nOn real assembled datasets this frequently comes back at 5 to 15%. Every one of those items is inflating your score.\n\n**Group leakage.** One customer generates forty tickets, split across both sides. The model learns that customer's vocabulary, their product, their way of describing things, and scores well on their evaluation items for reasons that do not generalise.\n\nSplit by group, not by row: by customer, document, session, author, or whatever unit actually repeats.\n\n```python\nfrom sklearn.model_selection import GroupShuffleSplit\ntrain_idx, eval_idx = next(GroupShuffleSplit(test_size=0.1, random_state=0)\n                           .split(X, groups=df[\"customer_id\"]))\n```\n\n**Temporal leakage.** Production always runs on the future. A random split lets the model train on September and test on August, which is a cheat: the pricing change, the product launch, the new phrasing are all in the training data.\n\nSplit by time: train on everything before a cutoff, evaluate on everything after. The number will be lower, and it is the number that predicts production.\n\n**Template leakage.** If your data was generated from templates, a random split puts the same template on both sides with different slot values. You are measuring slot-filling, not the task.\n\n## Use it sparingly\n\nAn evaluation set decays with use. Every time you look at it and change something in response, you fit a little to it, not through gradient descent but through your own decisions, which is slower and just as real. After forty runs, your held-out set is partly a training set.\n\nSo keep two:\n\n- **A development set**, used freely during iteration.\n- **A locked set**, opened a handful of times: before the project starts, at one or two milestones, and before shipping. Written down each time, so the count is visible.\n\nIf the locked set and the development set have drifted apart, the gap is how much you overfitted your own process.\n\n> Figure: Four ways a random split leaks. Layers, top to bottom: Near-duplicate leakage (Variations of the same ticket land on both sides, and the model scores well by recall rather than by generalising. Invisible, and almost nobody measures it.); Group leakage (One customer's forty tickets split across both sides. Split by customer, document, session or author: whatever unit actually repeats.); Temporal leakage (Training on September and testing on August. Production always runs on the future, so split by time and accept the lower number.); Template leakage (The same template on both sides with different slot values. You have measured slot-filling, not the task.). Measure the overlap rather than assuming it: run the same MinHash comparison you used for cleaning, across the split. On real assembled datasets it comes back at 5 to 15 per cent.\n\n## How large\n\nThe interval on a proportion tells you:\n\n- 100 items → ±6 to 10 points. Enough to spot a disaster, not enough to compare two candidates.\n- 300 items → ±3 to 5 points.\n- 1,000 items → ±2 to 3 points.\n\nMost teams want 300 to 500 for the development set and 300 for the locked one. Below 200 you cannot distinguish anything you would plausibly be arguing about.\n\nThe cost is in labelling, so the practical move is to stratify: 50 items per important category rather than 500 drawn at random, with the caveat from the sampling lesson that you must then report per stratum or weight back to real proportions.\n\n## The items to include on purpose\n\nA set drawn only from typical traffic misses everything you most need to know:\n\n- **Cases that should be declined.** Out of scope, missing information, ambiguous. If the model cannot decline, you want that in a number.\n- **The rare-but-expensive categories.** Refunds, safety-relevant requests, anything where a mistake is costly.\n- **Inputs from the tail.** Very short, very long, mixed-language, badly formatted.\n- **Adversarial items.** Prompt injection attempts if your inputs come from users, and the ordinary rudeness your support queue actually receives.\n- **Items the previous version got wrong.** Every production incident becomes a permanent evaluation item. This is the habit that compounds; after a year your set encodes everything the system has ever failed at, and no regression can pass unnoticed.\n\n## One rule\n\nBuild it before the training set. Not because of some discipline, but because the act of deciding what counts as a correct answer is the act of specifying the task, and doing that after you have seen the model's output means specifying it to fit what the model produced."},"check":{"question":"A team splits 10,000 support tickets randomly 90/10. Held-out accuracy is 0.94; production accuracy is 0.79. A check shows 12% of eval items have a near-duplicate in training and several customers appear on both sides. What is the mechanism?","options":[{"text":"The evaluation set is too small at 1,000 items to estimate accuracy within 15 points.","correct":false,"tempting":"A thousand items gives an interval of about 2 points, so sampling error cannot account for a 15-point gap."},{"text":"The evaluation measured recall of seen text and customer vocabulary, neither available on new tickets.","correct":true},{"text":"Production traffic has drifted since the data was collected, which a larger evaluation set would have revealed.","correct":false,"tempting":"Names a real cause of train-production gaps, but the diagnostics given point at leakage within the existing data rather than at drift."},{"text":"Randomly split evaluation sets always overstate performance because they share the training set's label distribution.","correct":false,"tempting":"Treats a matched label distribution as a defect, when a distribution matching production is what you want; the problem is shared content, not shared proportions."}]},"prev":{"slug":"the-baseline-you-must-beat","title":"The baseline you must beat","url":"https://addaly.com/learn/fine-tuning/the-baseline-you-must-beat"},"next":{"slug":"pairwise-human-comparison","title":"Pairwise human comparison, done properly","url":"https://addaly.com/learn/fine-tuning/pairwise-human-comparison"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'A held-out set that actually holds', in Fine-Tuning, and When Not To. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/fine-tuning/a-held-out-set-that-holds (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"A held-out set that actually holds","courseTitle":"Fine-Tuning, and When Not To","module":{"index":8,"slug":"did-it-actually-work","title":"Did it actually work"},"answer":null,"keyIdea":"Random splits leak through near-duplicates, groups, time and templates, so split by group and by time and then measure cross-split near-duplicate overlap, and keep a locked set you open a handful of times, because an evaluation set you have consulted forty times has become a training set for your own decisions.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"type":"stack","title":"Four ways a random split leaks","layers":[{"label":"Near-duplicate leakage","detail":"Variations of the same ticket land on both sides, and the model scores well by recall rather than by generalising. Invisible, and almost nobody measures it."},{"label":"Group leakage","detail":"One customer’s forty tickets split across both sides. Split by customer, document, session or author: whatever unit actually repeats."},{"label":"Temporal leakage","detail":"Training on September and testing on August. Production always runs on the future, so split by time and accept the lower number."},{"label":"Template leakage","detail":"The same template on both sides with different slot values. You have measured slot-filling, not the task."}],"caption":"Measure the overlap rather than assuming it: run the same MinHash comparison you used for cleaning, across the split. On real assembled datasets it comes back at 5 to 15 per cent."},"alt":"Figure: Four ways a random split leaks. Layers, top to bottom: Near-duplicate leakage (Variations of the same ticket land on both sides, and the model scores well by recall rather than by generalising. Invisible, and almost nobody measures it.); Group leakage (One customer’s forty tickets split across both sides. Split by customer, document, session or author: whatever unit actually repeats.); Temporal leakage (Training on September and testing on August. Production always runs on the future, so split by time and accept the lower number.); Template leakage (The same template on both sides with different slot values. You have measured slot-filling, not the task.). Measure the overlap rather than assuming it: run the same MinHash comparison you used for cleaning, across the split. On real assembled datasets it comes back at 5 to 15 per cent."}},"blocks":[{"t":"h2","id":"random-splitting-is-not-enough","runs":[{"text":"Random splitting is not enough"}]},{"t":"p","runs":[{"text":"A random 90/10 split feels safe and leaks constantly. Four distinct leaks, each with a fix."}]},{"t":"p","runs":[{"text":"Near-duplicate leakage.","b":true},{"text":" Your corpus contains variations of the same ticket. A random split puts some in training and their near-twins in evaluation, and the model scores well by recall rather than by generalisation."}]},{"t":"p","runs":[{"text":"This is the big one, it is invisible, and almost nobody measures it. Run the same MinHash comparison you used for cleaning, but "},{"text":"across the split","i":true},{"text":":"}]},{"t":"code","lang":"python","label":"Python","text":"lsh = MinHashLSH(threshold=0.8, num_perm=128)\nfor i, t in enumerate(train_texts):\n    lsh.insert(f\"tr{i}\", minhash(t))\nleaked = sum(1 for t in eval_texts if lsh.query(minhash(t)))\nprint(f\"{leaked}/{len(eval_texts)} eval items have a near-duplicate in train\")"},{"t":"p","runs":[{"text":"On real assembled datasets this frequently comes back at 5 to 15%. Every one of those items is inflating your score."}]},{"t":"p","runs":[{"text":"Group leakage.","b":true},{"text":" One customer generates forty tickets, split across both sides. The model learns that customer’s vocabulary, their product, their way of describing things, and scores well on their evaluation items for reasons that do not generalise."}]},{"t":"p","runs":[{"text":"Split by group, not by row: by customer, document, session, author, or whatever unit actually repeats."}]},{"t":"code","lang":"python","label":"Python","text":"from sklearn.model_selection import GroupShuffleSplit\ntrain_idx, eval_idx = next(GroupShuffleSplit(test_size=0.1, random_state=0)\n                           .split(X, groups=df[\"customer_id\"]))"},{"t":"p","runs":[{"text":"Temporal leakage.","b":true},{"text":" Production always runs on the future. A random split lets the model train on September and test on August, which is a cheat: the pricing change, the product launch, the new phrasing are all in the training data."}]},{"t":"p","runs":[{"text":"Split by time: train on everything before a cutoff, evaluate on everything after. The number will be lower, and it is the number that predicts production."}]},{"t":"p","runs":[{"text":"Template leakage.","b":true},{"text":" If your data was generated from templates, a random split puts the same template on both sides with different slot values. You are measuring slot-filling, not the task."}]},{"t":"h2","id":"use-it-sparingly","runs":[{"text":"Use it sparingly"}]},{"t":"p","runs":[{"text":"An evaluation set decays with use. Every time you look at it and change something in response, you fit a little to it, not through gradient descent but through your own decisions, which is slower and just as real. After forty runs, your held-out set is partly a training set."}]},{"t":"p","runs":[{"text":"So keep two:"}]},{"t":"ul","items":[[{"text":"A development set","b":true},{"text":", used freely during iteration."}],[{"text":"A locked set","b":true},{"text":", opened a handful of times: before the project starts, at one or two milestones, and before shipping. Written down each time, so the count is visible."}]]},{"t":"p","runs":[{"text":"If the locked set and the development set have drifted apart, the gap is how much you overfitted your own process."}]},{"t":"h2","id":"how-large","runs":[{"text":"How large"}]},{"t":"p","runs":[{"text":"The interval on a proportion tells you:"}]},{"t":"ul","items":[[{"text":"100 items → ±6 to 10 points. Enough to spot a disaster, not enough to compare two candidates."}],[{"text":"300 items → ±3 to 5 points."}],[{"text":"1,000 items → ±2 to 3 points."}]]},{"t":"p","runs":[{"text":"Most teams want 300 to 500 for the development set and 300 for the locked one. Below 200 you cannot distinguish anything you would plausibly be arguing about."}]},{"t":"p","runs":[{"text":"The cost is in labelling, so the practical move is to stratify: 50 items per important category rather than 500 drawn at random, with the caveat from the sampling lesson that you must then report per stratum or weight back to real proportions."}]},{"t":"h2","id":"the-items-to-include-on-purpose","runs":[{"text":"The items to include on purpose"}]},{"t":"p","runs":[{"text":"A set drawn only from typical traffic misses everything you most need to know:"}]},{"t":"ul","items":[[{"text":"Cases that should be declined.","b":true},{"text":" Out of scope, missing information, ambiguous. If the model cannot decline, you want that in a number."}],[{"text":"The rare-but-expensive categories.","b":true},{"text":" Refunds, safety-relevant requests, anything where a mistake is costly."}],[{"text":"Inputs from the tail.","b":true},{"text":" Very short, very long, mixed-language, badly formatted."}],[{"text":"Adversarial items.","b":true},{"text":" Prompt injection attempts if your inputs come from users, and the ordinary rudeness your support queue actually receives."}],[{"text":"Items the previous version got wrong.","b":true},{"text":" Every production incident becomes a permanent evaluation item. This is the habit that compounds; after a year your set encodes everything the system has ever failed at, and no regression can pass unnoticed."}]]},{"t":"h2","id":"one-rule","runs":[{"text":"One rule"}]},{"t":"p","runs":[{"text":"Build it before the training set. Not because of some discipline, but because the act of deciding what counts as a correct answer is the act of specifying the task, and doing that after you have seen the model’s output means specifying it to fit what the model produced."}]}],"check":{"question":[{"text":"A team splits 10,000 support tickets randomly 90/10. Held-out accuracy is 0.94; production accuracy is 0.79. A check shows 12% of eval items have a near-duplicate in training and several customers appear on both sides. What is the mechanism?"}],"options":[{"text":[{"text":"The evaluation set is too small at 1,000 items to estimate accuracy within 15 points."}],"correct":false,"tempting":[{"text":"A thousand items gives an interval of about 2 points, so sampling error cannot account for a 15-point gap."}]},{"text":[{"text":"The evaluation measured recall of seen text and customer vocabulary, neither available on new tickets."}],"correct":true,"tempting":null},{"text":[{"text":"Production traffic has drifted since the data was collected, which a larger evaluation set would have revealed."}],"correct":false,"tempting":[{"text":"Names a real cause of train-production gaps, but the diagnostics given point at leakage within the existing data rather than at drift."}]},{"text":[{"text":"Randomly split evaluation sets always overstate performance because they share the training set’s label distribution."}],"correct":false,"tempting":[{"text":"Treats a matched label distribution as a defect, when a distribution matching production is what you want; the problem is shared content, not shared proportions."}]}],"order":[3,0,2,1],"why":null},"prev":{"slug":"the-baseline-you-must-beat","title":"The baseline you must beat","minutes":9},"next":{"slug":"pairwise-human-comparison","title":"Pairwise human comparison, done properly","minutes":9}}}