Duplicates, leakage, balance, and label noise

Four checks, an hour, before any training

This is the least glamorous lesson in the course and the one with the highest expected value. Every check below has, in somebody’s project, been the difference between a result and an illusion.

1. Duplicates

Near-duplicates inflate everything. A dataset with 400 copies of one automated message looks like 400 rows and contains one row of information. Trained on, the model over-weights it; evaluated on, your test set is a repeated question.

Exact duplicates first, because they are free:

import hashlib

def norm(t):
    return " ".join(t.lower().split())

seen, keep = set(), []
for i, t in enumerate(ds["text"]):
    h = hashlib.md5(norm(t).encode()).hexdigest()
    if h not in seen:
        seen.add(h); keep.append(i)
ds = ds.select(keep)
print(f"removed {len(ds['text']) - len(keep)} exact duplicates")

Near-duplicates need MinHash or embeddings. The datatrove library implements MinHash deduplication at corpus scale; for a few hundred thousand rows, embedding everything and dropping pairs above a cosine similarity of about 0.95 works and uses the tools from module six.

The diagnostic that tells you whether to bother: if any single normalised string is more than 2% of your dataset, it is a template, not a sample.

2. Leakage between splits

The check that invalidates published numbers:

train_hashes = {hashlib.md5(norm(t).encode()).hexdigest() for t in ds["train"]["text"]}
overlap = sum(hashlib.md5(norm(t).encode()).hexdigest() in train_hashes
              for t in ds["test"]["text"])
print(f"{overlap} of {len(ds['test'])} test rows appear in train")

Exact overlap is the easy case. The subtler forms are the ones that catch experienced people:

• Grouped leakage. Three chunks from the same document split across train and test. The model has seen the neighbouring paragraph. Split by document, not by chunk.
• Temporal leakage. A random split of time-ordered data lets the model train on the future and predict the past. If deployment means predicting forward, split by date.
• Speaker or user leakage. The same person in both splits. You measure memorising them.
• Preprocessing leakage. Computing normalisation statistics, a vocabulary, or an imputation value over the whole dataset before splitting. The test set has influenced the pipeline.

The general rule: split along the axis your deployment will have to generalise across. If the model will see new customers, split by customer.

3. Class balance, and what it does to your metric

from collections import Counter
c = Counter(ds["label"])
for k, v in c.most_common():
    print(f"{k}: {v} ({v/len(ds):.1%})")

At 98% negative, a model that always says negative scores 98% accuracy. That number is worthless and it will be the first number anybody quotes. Report per-class precision and recall, or macro-F1, and say which you used.

What to do about imbalance, in the order usually worth trying:

1. Nothing. Choose a metric that is not fooled and a threshold set on a validation set. This is right more often than people expect.
2. Class weights in the loss, which costs one argument and no data.
3. Collect more of the rare class, which is the only approach that adds information.
4. Undersample the majority, which throws information away.
5. Oversample or synthesise the minority, which risks the model memorising a handful of examples many times.

4. Label noise

Read a hundred rows. Not a summary, the rows.

An error rate of 5 to 10% in labels is normal in crowd-sourced data, and it puts a ceiling on measured accuracy that no model can pass: a perfect model scores 92% against 8% wrong labels, and the missing 8% is the labels, not the model. If your card reports inter-annotator agreement, that figure is roughly your ceiling.

The cheap detector: train a quick model, look at the examples it is most confident about and wrong on. A large share of them are mislabelled rather than hard. This takes twenty minutes and routinely finds systematic errors: a whole category labelled by a rule that was wrong.

The other two things to look at

Language and script mix. langdetect or fasttext over a sample. A “multilingual” dataset that is 94% English trains a model that is 94% English.

Length distribution. The 95th percentile decides your max_length and therefore your compute. A median of 80 tokens with a maximum of 40,000 means one row will dominate a batch.

Write the results down

rows 184,203 → 171,908 after exact dedup (6.7%)
test/train exact overlap: 0 (after dedup)
split: by ticket_id, not by row
classes: refund 61%, delivery 22%, quality 12%, other 5%
langs: en 71%, hi 18%, hinglish 11% (sample of 2,000)
length p50 84 tokens, p95 612, max 31,004 → max_length 640
read 100 rows: 6 look mislabelled (4 refund→delivery)

Eight lines. It goes in the dataset card and in the repository. Every surprising result later starts by reading it, and a team that has it argues about causes instead of about facts.

Do this now

Run the duplicate check and the leakage check on the dataset you are currently using. Most people find something on the first attempt, and finding it before training is worth more than any model choice in this course.