{"course":{"slug":"hugging-face","title":"Hugging Face, End to End","url":"https://addaly.com/learn/hugging-face","level":"intermediate","track":"create"},"lesson":{"slug":"data-quality-before-the-model","title":"Duplicates, leakage, balance, and label noise","url":"https://addaly.com/learn/hugging-face/data-quality-before-the-model","position":46,"of":75,"minutes":10,"answer":"Before training, four checks decide whether a result means anything: exact and near duplicates, leakage along the axis deployment must generalise across, class balance against the metric you will quote.","keyIdea":"Before training, four checks decide whether a result means anything: exact and near duplicates, leakage along the axis deployment must generalise across, class balance against the metric you will quote, and label noise that sets a ceiling no model can pass."},"body":{"format":"markdown","text":"## Four checks, an hour, before any training\n\nThis is the least glamorous lesson in the course and the one with the highest expected value. Every check below has, in somebody's project, been the difference between a result and an illusion.\n\n## 1. Duplicates\n\nNear-duplicates inflate everything. A dataset with 400 copies of one automated message looks like 400 rows and contains one row of information. Trained on, the model over-weights it; evaluated on, your test set is a repeated question.\n\nExact duplicates first, because they are free:\n\n```python\nimport hashlib\n\ndef norm(t):\n    return \" \".join(t.lower().split())\n\n> Figure: A support queue's four intents, counted before training. Bars (% of 171,908 rows after deduplication): Refund 61; Delivery 22; Product quality 12; Everything else 5. A model that always answers refund scores 61 per cent accuracy on this set and has learned nothing at all. That number will be the first one anybody quotes, which is why macro-F1 and per-class recall belong in the same sentence as it.\n\nseen, keep = set(), []\nfor i, t in enumerate(ds[\"text\"]):\n    h = hashlib.md5(norm(t).encode()).hexdigest()\n    if h not in seen:\n        seen.add(h); keep.append(i)\nds = ds.select(keep)\nprint(f\"removed {len(ds['text']) - len(keep)} exact duplicates\")\n```\n\nNear-duplicates need MinHash or embeddings. The `datatrove` library implements MinHash deduplication at corpus scale; for a few hundred thousand rows, embedding everything and dropping pairs above a cosine similarity of about 0.95 works and uses the tools from module six.\n\nThe diagnostic that tells you whether to bother: **if any single normalised string is more than 2% of your dataset, it is a template, not a sample.**\n\n## 2. Leakage between splits\n\nThe check that invalidates published numbers:\n\n```python\ntrain_hashes = {hashlib.md5(norm(t).encode()).hexdigest() for t in ds[\"train\"][\"text\"]}\noverlap = sum(hashlib.md5(norm(t).encode()).hexdigest() in train_hashes\n              for t in ds[\"test\"][\"text\"])\nprint(f\"{overlap} of {len(ds['test'])} test rows appear in train\")\n```\n\nExact overlap is the easy case. The subtler forms are the ones that catch experienced people:\n\n- **Grouped leakage.** Three chunks from the same document split across train and test. The model has seen the neighbouring paragraph. Split by document, not by chunk.\n- **Temporal leakage.** A random split of time-ordered data lets the model train on the future and predict the past. If deployment means predicting forward, split by date.\n- **Speaker or user leakage.** The same person in both splits. You measure memorising them.\n- **Preprocessing leakage.** Computing normalisation statistics, a vocabulary, or an imputation value over the whole dataset before splitting. The test set has influenced the pipeline.\n\nThe general rule: **split along the axis your deployment will have to generalise across.** If the model will see new customers, split by customer.\n\n## 3. Class balance, and what it does to your metric\n\n```python\nfrom collections import Counter\nc = Counter(ds[\"label\"])\nfor k, v in c.most_common():\n    print(f\"{k}: {v} ({v/len(ds):.1%})\")\n```\n\nAt 98% negative, a model that always says negative scores 98% accuracy. That number is worthless and it will be the first number anybody quotes. Report per-class precision and recall, or macro-F1, and say which you used.\n\nWhat to do about imbalance, in the order usually worth trying:\n\n1. **Nothing.** Choose a metric that is not fooled and a threshold set on a validation set. This is right more often than people expect.\n2. **Class weights in the loss**, which costs one argument and no data.\n3. **Collect more of the rare class**, which is the only approach that adds information.\n4. **Undersample the majority**, which throws information away.\n5. **Oversample or synthesise the minority**, which risks the model memorising a handful of examples many times.\n\n## 4. Label noise\n\nRead a hundred rows. Not a summary, the rows.\n\nAn error rate of 5 to 10% in labels is normal in crowd-sourced data, and it puts a ceiling on measured accuracy that no model can pass: a perfect model scores 92% against 8% wrong labels, and the missing 8% is the labels, not the model. If your card reports inter-annotator agreement, that figure is roughly your ceiling.\n\nThe cheap detector: train a quick model, look at the examples it is most confident about and wrong on. A large share of them are mislabelled rather than hard. This takes twenty minutes and routinely finds systematic errors: a whole category labelled by a rule that was wrong.\n\n## The other two things to look at\n\n**Language and script mix.** `langdetect` or `fasttext` over a sample. A \"multilingual\" dataset that is 94% English trains a model that is 94% English.\n\n**Length distribution.** The 95th percentile decides your `max_length` and therefore your compute. A median of 80 tokens with a maximum of 40,000 means one row will dominate a batch.\n\n## Write the results down\n\n```\nrows 184,203 → 171,908 after exact dedup (6.7%)\ntest/train exact overlap: 0 (after dedup)\nsplit: by ticket_id, not by row\nclasses: refund 61%, delivery 22%, quality 12%, other 5%\nlangs: en 71%, hi 18%, hinglish 11% (sample of 2,000)\nlength p50 84 tokens, p95 612, max 31,004 → max_length 640\nread 100 rows: 6 look mislabelled (4 refund→delivery)\n```\n\nEight lines. It goes in the dataset card and in the repository. Every surprising result later starts by reading it, and a team that has it argues about causes instead of about facts.\n\n## Do this now\n\nRun the duplicate check and the leakage check on the dataset you are currently using. Most people find something on the first attempt, and finding it before training is worth more than any model choice in this course."},"check":{"question":"A document-chunking pipeline splits chunks randomly into train and test, and the model scores far better offline than in production. What is the most likely cause?","options":[{"text":"The chunks are too short, so the model memorises phrasing rather than learning the task.","correct":false,"tempting":"Chunk length affects difficulty but would degrade offline and online scores together."},{"text":"Random splitting failed to stratify by class, so the test set's class mix differs from production.","correct":false,"tempting":"A real hazard, but it would shift the score in either direction rather than systematically flatter it."},{"text":"Neighbouring chunks of one document land in both splits, so the test material was seen.","correct":true},{"text":"Production input is not chunked the same way, so the model receives inputs of a length it never saw.","correct":false,"tempting":"A genuine deployment mismatch, but it does not explain an offline score that is too high."}]},"prev":{"slug":"audio-and-image-datasets","title":"Media columns decode when you touch them","url":"https://addaly.com/learn/hugging-face/audio-and-image-datasets"},"next":{"slug":"preparing-a-dataset-for-training","title":"From a dataset to batches a trainer can eat","url":"https://addaly.com/learn/hugging-face/preparing-a-dataset-for-training"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Duplicates, leakage, balance, and label noise', in Hugging Face, End to End. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/hugging-face/data-quality-before-the-model (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Duplicates, leakage, balance, and label noise","courseTitle":"Hugging Face, End to End","module":{"index":5,"slug":"datasets-and-the-work-before-the-model","title":"Datasets, and the work that happens before any model"},"answer":null,"keyIdea":"Before training, four checks decide whether a result means anything: exact and near duplicates, leakage along the axis deployment must generalise across, class balance against the metric you will quote, and label noise that sets a ceiling no model can pass.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"type":"bars","unit":"% of 171,908 rows after deduplication","title":"A support queue’s four intents, counted before training","series":[{"label":"Refund","value":61,"highlight":true},{"label":"Delivery","value":22},{"label":"Product quality","value":12},{"label":"Everything else","value":5}],"caption":"A model that always answers refund scores 61 per cent accuracy on this set and has learned nothing at all. That number will be the first one anybody quotes, which is why macro-F1 and per-class recall belong in the same sentence as it."},"alt":"Figure: A support queue’s four intents, counted before training. Bars (% of 171,908 rows after deduplication): Refund 61; Delivery 22; Product quality 12; Everything else 5. A model that always answers refund scores 61 per cent accuracy on this set and has learned nothing at all. That number will be the first one anybody quotes, which is why macro-F1 and per-class recall belong in the same sentence as it."}},"blocks":[{"t":"h2","id":"four-checks-an-hour-before-any-training","runs":[{"text":"Four checks, an hour, before any training"}]},{"t":"p","runs":[{"text":"This is the least glamorous lesson in the course and the one with the highest expected value. Every check below has, in somebody’s project, been the difference between a result and an illusion."}]},{"t":"h2","id":"1-duplicates","runs":[{"text":"1. Duplicates"}]},{"t":"p","runs":[{"text":"Near-duplicates inflate everything. A dataset with 400 copies of one automated message looks like 400 rows and contains one row of information. Trained on, the model over-weights it; evaluated on, your test set is a repeated question."}]},{"t":"p","runs":[{"text":"Exact duplicates first, because they are free:"}]},{"t":"code","lang":"python","label":"Python","text":"import hashlib\n\ndef norm(t):\n    return \" \".join(t.lower().split())\n\nseen, keep = set(), []\nfor i, t in enumerate(ds[\"text\"]):\n    h = hashlib.md5(norm(t).encode()).hexdigest()\n    if h not in seen:\n        seen.add(h); keep.append(i)\nds = ds.select(keep)\nprint(f\"removed {len(ds['text']) - len(keep)} exact duplicates\")"},{"t":"p","runs":[{"text":"Near-duplicates need MinHash or embeddings. The "},{"text":"datatrove","code":true},{"text":" library implements MinHash deduplication at corpus scale; for a few hundred thousand rows, embedding everything and dropping pairs above a cosine similarity of about 0.95 works and uses the tools from module six."}]},{"t":"p","runs":[{"text":"The diagnostic that tells you whether to bother: "},{"text":"if any single normalised string is more than 2% of your dataset, it is a template, not a sample.","b":true}]},{"t":"h2","id":"2-leakage-between-splits","runs":[{"text":"2. Leakage between splits"}]},{"t":"p","runs":[{"text":"The check that invalidates published numbers:"}]},{"t":"code","lang":"python","label":"Python","text":"train_hashes = {hashlib.md5(norm(t).encode()).hexdigest() for t in ds[\"train\"][\"text\"]}\noverlap = sum(hashlib.md5(norm(t).encode()).hexdigest() in train_hashes\n              for t in ds[\"test\"][\"text\"])\nprint(f\"{overlap} of {len(ds['test'])} test rows appear in train\")"},{"t":"p","runs":[{"text":"Exact overlap is the easy case. The subtler forms are the ones that catch experienced people:"}]},{"t":"ul","items":[[{"text":"Grouped leakage.","b":true},{"text":" Three chunks from the same document split across train and test. The model has seen the neighbouring paragraph. Split by document, not by chunk."}],[{"text":"Temporal leakage.","b":true},{"text":" A random split of time-ordered data lets the model train on the future and predict the past. If deployment means predicting forward, split by date."}],[{"text":"Speaker or user leakage.","b":true},{"text":" The same person in both splits. You measure memorising them."}],[{"text":"Preprocessing leakage.","b":true},{"text":" Computing normalisation statistics, a vocabulary, or an imputation value over the whole dataset before splitting. The test set has influenced the pipeline."}]]},{"t":"p","runs":[{"text":"The general rule: "},{"text":"split along the axis your deployment will have to generalise across.","b":true},{"text":" If the model will see new customers, split by customer."}]},{"t":"h2","id":"3-class-balance-and-what-it-does-to-your-metric","runs":[{"text":"3. Class balance, and what it does to your metric"}]},{"t":"code","lang":"python","label":"Python","text":"from collections import Counter\nc = Counter(ds[\"label\"])\nfor k, v in c.most_common():\n    print(f\"{k}: {v} ({v/len(ds):.1%})\")"},{"t":"p","runs":[{"text":"At 98% negative, a model that always says negative scores 98% accuracy. That number is worthless and it will be the first number anybody quotes. Report per-class precision and recall, or macro-F1, and say which you used."}]},{"t":"p","runs":[{"text":"What to do about imbalance, in the order usually worth trying:"}]},{"t":"ol","start":1,"items":[[{"text":"Nothing.","b":true},{"text":" Choose a metric that is not fooled and a threshold set on a validation set. This is right more often than people expect."}],[{"text":"Class weights in the loss","b":true},{"text":", which costs one argument and no data."}],[{"text":"Collect more of the rare class","b":true},{"text":", which is the only approach that adds information."}],[{"text":"Undersample the majority","b":true},{"text":", which throws information away."}],[{"text":"Oversample or synthesise the minority","b":true},{"text":", which risks the model memorising a handful of examples many times."}]]},{"t":"h2","id":"4-label-noise","runs":[{"text":"4. Label noise"}]},{"t":"p","runs":[{"text":"Read a hundred rows. Not a summary, the rows."}]},{"t":"p","runs":[{"text":"An error rate of 5 to 10% in labels is normal in crowd-sourced data, and it puts a ceiling on measured accuracy that no model can pass: a perfect model scores 92% against 8% wrong labels, and the missing 8% is the labels, not the model. If your card reports inter-annotator agreement, that figure is roughly your ceiling."}]},{"t":"p","runs":[{"text":"The cheap detector: train a quick model, look at the examples it is most confident about and wrong on. A large share of them are mislabelled rather than hard. This takes twenty minutes and routinely finds systematic errors: a whole category labelled by a rule that was wrong."}]},{"t":"h2","id":"the-other-two-things-to-look-at","runs":[{"text":"The other two things to look at"}]},{"t":"p","runs":[{"text":"Language and script mix.","b":true},{"text":" "},{"text":"langdetect","code":true},{"text":" or "},{"text":"fasttext","code":true},{"text":" over a sample. A “multilingual” dataset that is 94% English trains a model that is 94% English."}]},{"t":"p","runs":[{"text":"Length distribution.","b":true},{"text":" The 95th percentile decides your "},{"text":"max_length","code":true},{"text":" and therefore your compute. A median of 80 tokens with a maximum of 40,000 means one row will dominate a batch."}]},{"t":"h2","id":"write-the-results-down","runs":[{"text":"Write the results down"}]},{"t":"code","lang":"","label":"Text","text":"rows 184,203 → 171,908 after exact dedup (6.7%)\ntest/train exact overlap: 0 (after dedup)\nsplit: by ticket_id, not by row\nclasses: refund 61%, delivery 22%, quality 12%, other 5%\nlangs: en 71%, hi 18%, hinglish 11% (sample of 2,000)\nlength p50 84 tokens, p95 612, max 31,004 → max_length 640\nread 100 rows: 6 look mislabelled (4 refund→delivery)"},{"t":"p","runs":[{"text":"Eight lines. It goes in the dataset card and in the repository. Every surprising result later starts by reading it, and a team that has it argues about causes instead of about facts."}]},{"t":"h2","id":"do-this-now","runs":[{"text":"Do this now"}],"role":"try"},{"t":"p","runs":[{"text":"Run the duplicate check and the leakage check on the dataset you are currently using. Most people find something on the first attempt, and finding it before training is worth more than any model choice in this course."}]}],"check":{"question":[{"text":"A document-chunking pipeline splits chunks randomly into train and test, and the model scores far better offline than in production. What is the most likely cause?"}],"options":[{"text":[{"text":"The chunks are too short, so the model memorises phrasing rather than learning the task."}],"correct":false,"tempting":[{"text":"Chunk length affects difficulty but would degrade offline and online scores together."}]},{"text":[{"text":"Random splitting failed to stratify by class, so the test set’s class mix differs from production."}],"correct":false,"tempting":[{"text":"A real hazard, but it would shift the score in either direction rather than systematically flatter it."}]},{"text":[{"text":"Neighbouring chunks of one document land in both splits, so the test material was seen."}],"correct":true,"tempting":null},{"text":[{"text":"Production input is not chunked the same way, so the model receives inputs of a length it never saw."}],"correct":false,"tempting":[{"text":"A genuine deployment mismatch, but it does not explain an offline score that is too high."}]}],"order":[3,0,1,2],"why":null},"prev":{"slug":"audio-and-image-datasets","title":"Media columns decode when you touch them","minutes":8},"next":{"slug":"preparing-a-dataset-for-training","title":"From a dataset to batches a trainer can eat","minutes":9}}}