A held-out set that actually holds

Random splitting is not enough

A random 90/10 split feels safe and leaks constantly. Four distinct leaks, each with a fix.

Near-duplicate leakage. Your corpus contains variations of the same ticket. A random split puts some in training and their near-twins in evaluation, and the model scores well by recall rather than by generalisation.

This is the big one, it is invisible, and almost nobody measures it. Run the same MinHash comparison you used for cleaning, but across the split:

lsh = MinHashLSH(threshold=0.8, num_perm=128)
for i, t in enumerate(train_texts):
    lsh.insert(f"tr{i}", minhash(t))
leaked = sum(1 for t in eval_texts if lsh.query(minhash(t)))
print(f"{leaked}/{len(eval_texts)} eval items have a near-duplicate in train")

On real assembled datasets this frequently comes back at 5 to 15%. Every one of those items is inflating your score.

Group leakage. One customer generates forty tickets, split across both sides. The model learns that customer’s vocabulary, their product, their way of describing things, and scores well on their evaluation items for reasons that do not generalise.

Split by group, not by row: by customer, document, session, author, or whatever unit actually repeats.

from sklearn.model_selection import GroupShuffleSplit
train_idx, eval_idx = next(GroupShuffleSplit(test_size=0.1, random_state=0)
                           .split(X, groups=df["customer_id"]))

Temporal leakage. Production always runs on the future. A random split lets the model train on September and test on August, which is a cheat: the pricing change, the product launch, the new phrasing are all in the training data.

Split by time: train on everything before a cutoff, evaluate on everything after. The number will be lower, and it is the number that predicts production.

Template leakage. If your data was generated from templates, a random split puts the same template on both sides with different slot values. You are measuring slot-filling, not the task.

Use it sparingly

An evaluation set decays with use. Every time you look at it and change something in response, you fit a little to it, not through gradient descent but through your own decisions, which is slower and just as real. After forty runs, your held-out set is partly a training set.

So keep two:

• A development set, used freely during iteration.
• A locked set, opened a handful of times: before the project starts, at one or two milestones, and before shipping. Written down each time, so the count is visible.

If the locked set and the development set have drifted apart, the gap is how much you overfitted your own process.

How large

The interval on a proportion tells you:

• 100 items → ±6 to 10 points. Enough to spot a disaster, not enough to compare two candidates.
• 300 items → ±3 to 5 points.
• 1,000 items → ±2 to 3 points.

Most teams want 300 to 500 for the development set and 300 for the locked one. Below 200 you cannot distinguish anything you would plausibly be arguing about.

The cost is in labelling, so the practical move is to stratify: 50 items per important category rather than 500 drawn at random, with the caveat from the sampling lesson that you must then report per stratum or weight back to real proportions.

The items to include on purpose

A set drawn only from typical traffic misses everything you most need to know:

• Cases that should be declined. Out of scope, missing information, ambiguous. If the model cannot decline, you want that in a number.
• The rare-but-expensive categories. Refunds, safety-relevant requests, anything where a mistake is costly.
• Inputs from the tail. Very short, very long, mixed-language, badly formatted.
• Adversarial items. Prompt injection attempts if your inputs come from users, and the ordinary rudeness your support queue actually receives.
• Items the previous version got wrong. Every production incident becomes a permanent evaluation item. This is the habit that compounds; after a year your set encodes everything the system has ever failed at, and no regression can pass unnoticed.

One rule

Build it before the training set. Not because of some discipline, but because the act of deciding what counts as a correct answer is the act of specifying the task, and doing that after you have seen the model’s output means specifying it to fit what the model produced.