{"course":{"slug":"fine-tuning","title":"Fine-Tuning, and When Not To","url":"https://addaly.com/learn/fine-tuning","level":"advanced","track":"deep"},"lesson":{"slug":"cleaning-at-scale","title":"Cleaning at scale","url":"https://addaly.com/learn/fine-tuning/cleaning-at-scale","position":31,"of":84,"minutes":9,"answer":"Filter in order of cost (exact hashes, then MinHash near-duplicates, then length, truncation, language and repetition heuristics) and then read a hundred examples by hand.","keyIdea":"Filter in order of cost (exact hashes, then MinHash near-duplicates, then length, truncation, language and repetition heuristics) and then read a hundred examples by hand, because the systematic annotation problems that decide a fine-tune's quality have no automated detector."},"body":{"format":"markdown","text":"## A bad example is worse than a missing one\n\nA missing example teaches nothing. A bad example teaches something wrong, with the same weight as a good one, and the model has no way to tell them apart. Every filter below exists because somebody trained on data they had not looked at.\n\nWork through these in order: cheap deterministic filters first, judgement last.\n\n## Exact duplicates\n\nHash the normalised text and keep one copy. On data assembled from several sources, exact duplicates are routinely 5 to 20% of the file.\n\n```python\nimport hashlib\nseen, kept = set(), []\nfor row in data:\n    key = hashlib.sha256(\n        \" \".join(m[\"content\"].lower().split()).encode()\n        for-each-message  # normalise whitespace and case before hashing\n    )\n```\n\nThe normalisation matters: without lowercasing and collapsing whitespace, two copies of the same example differing by a trailing space survive as distinct rows.\n\n## Near duplicates\n\nThe harder and more important case. Four hundred tickets that differ only in an order number are, for training purposes, one example repeated four hundred times, and the model will weight it accordingly.\n\nMinHash with locality-sensitive hashing finds these in near-linear time:\n\n```python\nfrom datasketch import MinHash, MinHashLSH\n\nlsh = MinHashLSH(threshold=0.85, num_perm=128)\nfor i, text in enumerate(texts):\n    m = MinHash(num_perm=128)\n    for token in set(text.lower().split()):\n        m.update(token.encode())\n    if not lsh.query(m):        # nothing similar already kept\n        lsh.insert(str(i), m)\n        keep.append(i)\n```\n\n`datasketch` is free and handles a few hundred thousand documents on a laptop. The `text-dedup` package wraps the same idea with more options if you have millions.\n\nA threshold of 0.85 Jaccard similarity is a reasonable start. Check the result by hand: print ten pairs the filter considered duplicates, and ten it kept as distinct. If the boundary looks wrong, move the threshold rather than trusting it.\n\n## Length and truncation\n\nTwo filters, both cheap, both catching real damage:\n\n- **Too short.** An assistant response under about 5 tokens is usually a logging artefact, not an answer. Unless one-word answers are your task, drop them.\n- **Truncated.** An answer that ends mid-word or mid-sentence came from a generation that hit a token limit. These teach the model to stop in the middle of things. Check whether the final character is sentence-terminating punctuation, or a closing brace for structured output.\n\nAlso count how many examples exceed your training `max_length`, because those will be truncated by the tokenizer, losing the end-of-turn token along with the last part of the answer.\n\n## Language\n\nIf your product is monolingual and 4% of your logs are another language, decide deliberately rather than by accident. `fasttext`'s language identification model is free, runs at tens of thousands of documents a second, and gives a confidence score.\n\nThe reverse case matters too: if you *are* multilingual, check the balance. A set that is 92% English will produce a model whose other languages have quietly got worse.\n\n## Degenerate text\n\nA handful of heuristics catch most machine-generated junk:\n\n- **Repetition ratio.** If the most common 3-gram accounts for more than about 10% of the 3-grams in a response, it is looping.\n- **Non-word character ratio.** A high proportion of punctuation or symbols usually means markup, base64 or a corrupted encoding.\n- **Model tics.** If your data came from a hosted model, search for its characteristic openings: the self-identification phrases, the hedging preamble, the stock apology. You do not want to train those in. A regex over the first sentence finds them.\n\n## The filter you cannot automate\n\nRead a hundred examples. Not skim: read, from the beginning to the end of the assistant turn.\n\nThis is the highest-value hour in the whole project and it is the one most reliably skipped. What it finds is not the things above; it is the systematic problems no filter names. Every answer beginning with the same three words. Annotators who disagreed about whether to include a greeting. Two hundred examples where the \"correct\" answer is subtly wrong because the person writing them misunderstood the task.\n\n## Keep the record\n\nLog what each filter removed and how much:\n\n```\nstart                    48,201\nexact duplicates         -6,340\nnear duplicates (0.85)   -9,118\ntoo short                  -412\ntruncated                  -887\nnon-target language        -204\nrepetition heuristic       -156\n                        --------\nfinal                    31,084\n```\n\nIf one filter removes 40% of your data, that is not a cleaning step, that is a discovery about your source, and it deserves a look before you accept it.\n\n> Figure: What each filter removed, from 48,201 rows. Bars (rows removed): Near duplicates, MinHash at 0.85 9118; Exact duplicates 6340; Truncated mid-sentence 887; Too short to be an answer 412; Not the target language 204; Repetition heuristic 156. Thirty-one thousand survived. Log the counts: a filter that removes forty per cent of your data is a discovery about your source, not a cleaning step."},"check":{"question":"A dataset of 40,000 support exchanges contains 600 automated dispatch notices that differ only by an order number. Exact-duplicate hashing removes none of them. What is the consequence for training, and what finds them?","options":[{"text":"They are harmless because each is genuinely a distinct example; only byte-identical rows distort training.","correct":false,"tempting":"Treats textual distinctness as informational distinctness, when 600 near-identical rows contribute 600 times the gradient signal of one pattern."},{"text":"They cause the tokenizer to build an unbalanced vocabulary around order numbers, which length filtering resolves.","correct":false,"tempting":"Assumes the tokenizer is retrained on your data, which it is not; the vocabulary is fixed by the base model."},{"text":"They act as one pattern repeated 600 times, and near-duplicate detection such as MinHash catches them.","correct":true},{"text":"They will be caught by the repetition-ratio heuristic, since the repeated template raises the 3-gram concentration within each document.","correct":false,"tempting":"Confuses repetition within one document, which the heuristic measures, with repetition across documents, which it cannot see."}]},"prev":{"slug":"instruction-data-formats","title":"The three formats, and the conversion trap","url":"https://addaly.com/learn/fine-tuning/instruction-data-formats"},"next":{"slug":"writing-and-rewriting-outputs","title":"Writing the answers, and keeping them consistent","url":"https://addaly.com/learn/fine-tuning/writing-and-rewriting-outputs"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Cleaning at scale', in Fine-Tuning, and When Not To. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/fine-tuning/cleaning-at-scale (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Cleaning at scale","courseTitle":"Fine-Tuning, and When Not To","module":{"index":4,"slug":"building-the-dataset","title":"Building the dataset"},"answer":null,"keyIdea":"Filter in order of cost (exact hashes, then MinHash near-duplicates, then length, truncation, language and repetition heuristics) and then read a hundred examples by hand, because the systematic annotation problems that decide a fine-tune’s quality have no automated detector.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"type":"bars","unit":"rows removed","title":"What each filter removed, from 48,201 rows","series":[{"label":"Near duplicates, MinHash at 0.85","value":9118,"highlight":true},{"label":"Exact duplicates","value":6340},{"label":"Truncated mid-sentence","value":887},{"label":"Too short to be an answer","value":412},{"label":"Not the target language","value":204},{"label":"Repetition heuristic","value":156}],"caption":"Thirty-one thousand survived. Log the counts: a filter that removes forty per cent of your data is a discovery about your source, not a cleaning step."},"alt":"Figure: What each filter removed, from 48,201 rows. Bars (rows removed): Near duplicates, MinHash at 0.85 9118; Exact duplicates 6340; Truncated mid-sentence 887; Too short to be an answer 412; Not the target language 204; Repetition heuristic 156. Thirty-one thousand survived. Log the counts: a filter that removes forty per cent of your data is a discovery about your source, not a cleaning step."}},"blocks":[{"t":"h2","id":"a-bad-example-is-worse-than-a-missing-one","runs":[{"text":"A bad example is worse than a missing one"}]},{"t":"p","runs":[{"text":"A missing example teaches nothing. A bad example teaches something wrong, with the same weight as a good one, and the model has no way to tell them apart. Every filter below exists because somebody trained on data they had not looked at."}]},{"t":"p","runs":[{"text":"Work through these in order: cheap deterministic filters first, judgement last."}]},{"t":"h2","id":"exact-duplicates","runs":[{"text":"Exact duplicates"}]},{"t":"p","runs":[{"text":"Hash the normalised text and keep one copy. On data assembled from several sources, exact duplicates are routinely 5 to 20% of the file."}]},{"t":"code","lang":"python","label":"Python","text":"import hashlib\nseen, kept = set(), []\nfor row in data:\n    key = hashlib.sha256(\n        \" \".join(m[\"content\"].lower().split()).encode()\n        for-each-message  # normalise whitespace and case before hashing\n    )"},{"t":"p","runs":[{"text":"The normalisation matters: without lowercasing and collapsing whitespace, two copies of the same example differing by a trailing space survive as distinct rows."}]},{"t":"h2","id":"near-duplicates","runs":[{"text":"Near duplicates"}]},{"t":"p","runs":[{"text":"The harder and more important case. Four hundred tickets that differ only in an order number are, for training purposes, one example repeated four hundred times, and the model will weight it accordingly."}]},{"t":"p","runs":[{"text":"MinHash with locality-sensitive hashing finds these in near-linear time:"}]},{"t":"code","lang":"python","label":"Python","text":"from datasketch import MinHash, MinHashLSH\n\nlsh = MinHashLSH(threshold=0.85, num_perm=128)\nfor i, text in enumerate(texts):\n    m = MinHash(num_perm=128)\n    for token in set(text.lower().split()):\n        m.update(token.encode())\n    if not lsh.query(m):        # nothing similar already kept\n        lsh.insert(str(i), m)\n        keep.append(i)"},{"t":"p","runs":[{"text":"datasketch","code":true},{"text":" is free and handles a few hundred thousand documents on a laptop. The "},{"text":"text-dedup","code":true},{"text":" package wraps the same idea with more options if you have millions."}]},{"t":"p","runs":[{"text":"A threshold of 0.85 Jaccard similarity is a reasonable start. Check the result by hand: print ten pairs the filter considered duplicates, and ten it kept as distinct. If the boundary looks wrong, move the threshold rather than trusting it."}]},{"t":"h2","id":"length-and-truncation","runs":[{"text":"Length and truncation"}]},{"t":"p","runs":[{"text":"Two filters, both cheap, both catching real damage:"}]},{"t":"ul","items":[[{"text":"Too short.","b":true},{"text":" An assistant response under about 5 tokens is usually a logging artefact, not an answer. Unless one-word answers are your task, drop them."}],[{"text":"Truncated.","b":true},{"text":" An answer that ends mid-word or mid-sentence came from a generation that hit a token limit. These teach the model to stop in the middle of things. Check whether the final character is sentence-terminating punctuation, or a closing brace for structured output."}]]},{"t":"p","runs":[{"text":"Also count how many examples exceed your training "},{"text":"max_length","code":true},{"text":", because those will be truncated by the tokenizer, losing the end-of-turn token along with the last part of the answer."}]},{"t":"h2","id":"language","runs":[{"text":"Language"}]},{"t":"p","runs":[{"text":"If your product is monolingual and 4% of your logs are another language, decide deliberately rather than by accident. "},{"text":"fasttext","code":true},{"text":"’s language identification model is free, runs at tens of thousands of documents a second, and gives a confidence score."}]},{"t":"p","runs":[{"text":"The reverse case matters too: if you "},{"text":"are","i":true},{"text":" multilingual, check the balance. A set that is 92% English will produce a model whose other languages have quietly got worse."}]},{"t":"h2","id":"degenerate-text","runs":[{"text":"Degenerate text"}]},{"t":"p","runs":[{"text":"A handful of heuristics catch most machine-generated junk:"}]},{"t":"ul","items":[[{"text":"Repetition ratio.","b":true},{"text":" If the most common 3-gram accounts for more than about 10% of the 3-grams in a response, it is looping."}],[{"text":"Non-word character ratio.","b":true},{"text":" A high proportion of punctuation or symbols usually means markup, base64 or a corrupted encoding."}],[{"text":"Model tics.","b":true},{"text":" If your data came from a hosted model, search for its characteristic openings: the self-identification phrases, the hedging preamble, the stock apology. You do not want to train those in. A regex over the first sentence finds them."}]]},{"t":"h2","id":"the-filter-you-cannot-automate","runs":[{"text":"The filter you cannot automate"}]},{"t":"p","runs":[{"text":"Read a hundred examples. Not skim: read, from the beginning to the end of the assistant turn."}]},{"t":"p","runs":[{"text":"This is the highest-value hour in the whole project and it is the one most reliably skipped. What it finds is not the things above; it is the systematic problems no filter names. Every answer beginning with the same three words. Annotators who disagreed about whether to include a greeting. Two hundred examples where the “correct” answer is subtly wrong because the person writing them misunderstood the task."}]},{"t":"h2","id":"keep-the-record","runs":[{"text":"Keep the record"}]},{"t":"p","runs":[{"text":"Log what each filter removed and how much:"}]},{"t":"code","lang":"","label":"Text","text":"start                    48,201\nexact duplicates         -6,340\nnear duplicates (0.85)   -9,118\ntoo short                  -412\ntruncated                  -887\nnon-target language        -204\nrepetition heuristic       -156\n                        --------\nfinal                    31,084"},{"t":"p","runs":[{"text":"If one filter removes 40% of your data, that is not a cleaning step, that is a discovery about your source, and it deserves a look before you accept it."}]}],"check":{"question":[{"text":"A dataset of 40,000 support exchanges contains 600 automated dispatch notices that differ only by an order number. Exact-duplicate hashing removes none of them. What is the consequence for training, and what finds them?"}],"options":[{"text":[{"text":"They are harmless because each is genuinely a distinct example; only byte-identical rows distort training."}],"correct":false,"tempting":[{"text":"Treats textual distinctness as informational distinctness, when 600 near-identical rows contribute 600 times the gradient signal of one pattern."}]},{"text":[{"text":"They cause the tokenizer to build an unbalanced vocabulary around order numbers, which length filtering resolves."}],"correct":false,"tempting":[{"text":"Assumes the tokenizer is retrained on your data, which it is not; the vocabulary is fixed by the base model."}]},{"text":[{"text":"They act as one pattern repeated 600 times, and near-duplicate detection such as MinHash catches them."}],"correct":true,"tempting":null},{"text":[{"text":"They will be caught by the repetition-ratio heuristic, since the repeated template raises the 3-gram concentration within each document."}],"correct":false,"tempting":[{"text":"Confuses repetition within one document, which the heuristic measures, with repetition across documents, which it cannot see."}]}],"order":[0,1,3,2],"why":null},"prev":{"slug":"instruction-data-formats","title":"The three formats, and the conversion trap","minutes":8},"next":{"slug":"writing-and-rewriting-outputs","title":"Writing the answers, and keeping them consistent","minutes":9}}}