{"course":{"slug":"context-engineering","title":"Context Engineering","url":"https://addaly.com/learn/context-engineering","level":"advanced","track":"build"},"lesson":{"slug":"near-duplicates-and-boilerplate","title":"The same passage three times","url":"https://addaly.com/learn/context-engineering/near-duplicates-and-boilerplate","position":15,"of":87,"minutes":8,"answer":"Duplicate and near-duplicate passages spend budget twice and manufacture false consensus in the window, so deduplicate at ingestion and again before packing, detect boilerplate statistically rather than by rule.","keyIdea":"Duplicate and near-duplicate passages spend budget twice and manufacture false consensus in the window, so deduplicate at ingestion and again before packing, detect boilerplate statistically rather than by rule, and resolve version conflicts before assembly rather than hoping the model will."},"body":{"format":"markdown","text":"## Repetition is not evidence, but it reads like it\n\nRetrieve ten passages from a real corpus and you will often find that four of them say the same thing. A policy page, its copy in the onboarding guide, the version in last quarter's FAQ, and a support macro that quotes it.\n\nThis costs you twice.\n\n**Budget.** Four slots spent on one fact. The other three facts the question needed did not fit.\n\n**Confidence.** A statement repeated four times in the window is more likely to appear in the answer than a statement made once, regardless of which is correct. Models weight consistency across their context, and your retriever has just manufactured consistency out of a copy-paste.\n\nThe second effect is worse when the duplicates are not identical. Three versions of a policy from 2021, 2023 and 2026, all retrieved, all plausible. The model will not tell you it saw three; it will produce one answer that is a blend, sometimes with a number from the old version and a condition from the new one. That is the single most damaging kind of context error, because the output is fluent, specific and untraceable.\n\n## Exact duplicates: normalise and hash\n\nCheap and worth doing at ingestion:\n\n```python\ndef norm(t):\n    t = unicodedata.normalize(\"NFKC\", t.lower())\n    t = re.sub(r\"\\s+\", \" \", t)\n    t = re.sub(r\"[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-\", \"\", t)  # ids\n    return t.strip()\n\nkey = hashlib.sha256(norm(chunk).encode()).hexdigest()\n```\n\nKeep one, record how many copies existed and where. The count is useful later: a chunk that appears in eleven documents is probably boilerplate, and you have just detected it for free.\n\n## Near-duplicates: MinHash or embeddings\n\nExact hashing misses the version that changed one sentence. Two standard tools, both free:\n\n**MinHash with locality-sensitive hashing** (`datasketch` in Python) finds pairs above a Jaccard-similarity threshold over word shingles, in roughly linear time. Good for near-identical text, cheap on millions of chunks, and it does not need embeddings.\n\n**Embedding similarity** catches paraphrase that MinHash misses, at higher cost. If you are embedding everything anyway, a similarity above roughly 0.95 within a corpus is a strong duplicate signal; calibrate the exact number on your own data, since as the earlier lesson said, the scale is model-specific.\n\nDeduplicate at ingestion where you can afford to, and again at retrieval time as a cheap pass over the ten to forty candidates you are about to pack. The second pass is nearly free and catches duplicates that entered from different sources.\n\n## Boilerplate: detect it, do not hand-write rules\n\nEvery corpus has chrome: navigation, cookie notices, licence headers, email signatures, confidentiality footers, \"was this page helpful?\". It is textually meaningless and it competes for attention and budget.\n\nThe robust detection is statistical rather than rule-based. Split documents into blocks, count how many distinct documents each normalised block appears in, and treat any block appearing in more than a few percent of the corpus as chrome. This finds the footer you did not know about and adapts when the site template changes, which a regex does not.\n\nAn instructive case: a legal team's corpus where every one of 4,000 documents carried an identical 140-token confidentiality notice. That is 560,000 tokens of the index, and at k=10 it was 1,400 tokens of every single request, saying nothing. Removing it is a pure win with no quality trade-off to measure.\n\n## Versions: pick one, or label all of them\n\nDeduplication cannot resolve genuine version conflicts, because both versions are real documents. Two workable policies:\n\n1. **Choose at retrieval.** Group candidates by a document-family key, keep the most recent per family, and drop the rest before packing. Correct for policies, prices, specifications: anything with a current value.\n2. **Keep them and label them.** When history matters, include both with explicit dates and an instruction about which governs. Expect the model to get this right less often than you would like; the date has to be adjacent to the text, not in a header 3,000 tokens away.\n\nWhat does not work is retrieving both unlabelled and hoping. That is the blended-answer failure, and it is invisible in every metric except a human reading the output against the source.\n\n## The audit\n\nFor twenty real queries, print the retrieved set and mark any pair that says substantially the same thing. Most first-time audits find a duplication rate between 20% and 40%. That is the same fraction of your retrieval budget, and recovering it is usually a bigger quality win than swapping the embedding model."},"check":{"question":"An assistant answers a pricing question with a figure from a 2021 document and a condition from the 2026 revision, in one fluent sentence. Both documents were retrieved. What is the mechanism?","options":[{"text":"Undeduplicated versions of the same document were both in the window, and the model merged them because nothing marked which one governs.","correct":true},{"text":"The 2021 document ranked higher, so its content dominated the answer.","correct":false,"tempting":"Explains why one document might win, but not why parts of both appear in a single blended statement."},{"text":"The model hallucinated the figure, since neither document contained that combination.","correct":false,"tempting":"Reaches for hallucination when both components are traceable to real retrieved text; the invention is the combination, not the content."},{"text":"The chunker split each document mid-policy, so neither version was complete in the window.","correct":false,"tempting":"Describes a chunking failure that would more likely produce a partial or hedged answer than a confident merge of two versions."}]},"prev":{"slug":"the-k-curve","title":"How many passages is the right number","url":"https://addaly.com/learn/context-engineering/the-k-curve"},"next":{"slug":"freshness-and-as-of-lines","title":"Telling the model when things were true","url":"https://addaly.com/learn/context-engineering/freshness-and-as-of-lines"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'The same passage three times', in Context Engineering. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/context-engineering/near-duplicates-and-boilerplate (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"The same passage three times","courseTitle":"Context Engineering","module":{"index":2,"slug":"what-earns-a-place","title":"Selection: what earns a place"},"answer":null,"keyIdea":"Duplicate and near-duplicate passages spend budget twice and manufacture false consensus in the window, so deduplicate at ingestion and again before packing, detect boilerplate statistically rather than by rule, and resolve version conflicts before assembly rather than hoping the model will.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"repetition-is-not-evidence-but-it-reads-like-it","runs":[{"text":"Repetition is not evidence, but it reads like it"}]},{"t":"p","runs":[{"text":"Retrieve ten passages from a real corpus and you will often find that four of them say the same thing. A policy page, its copy in the onboarding guide, the version in last quarter’s FAQ, and a support macro that quotes it."}]},{"t":"p","runs":[{"text":"This costs you twice."}]},{"t":"p","runs":[{"text":"Budget.","b":true},{"text":" Four slots spent on one fact. The other three facts the question needed did not fit."}]},{"t":"p","runs":[{"text":"Confidence.","b":true},{"text":" A statement repeated four times in the window is more likely to appear in the answer than a statement made once, regardless of which is correct. Models weight consistency across their context, and your retriever has just manufactured consistency out of a copy-paste."}]},{"t":"p","runs":[{"text":"The second effect is worse when the duplicates are not identical. Three versions of a policy from 2021, 2023 and 2026, all retrieved, all plausible. The model will not tell you it saw three; it will produce one answer that is a blend, sometimes with a number from the old version and a condition from the new one. That is the single most damaging kind of context error, because the output is fluent, specific and untraceable."}]},{"t":"h2","id":"exact-duplicates-normalise-and-hash","runs":[{"text":"Exact duplicates: normalise and hash"}]},{"t":"p","runs":[{"text":"Cheap and worth doing at ingestion:"}]},{"t":"code","lang":"python","label":"Python","text":"def norm(t):\n    t = unicodedata.normalize(\"NFKC\", t.lower())\n    t = re.sub(r\"\\s+\", \" \", t)\n    t = re.sub(r\"[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-\", \"\", t)  # ids\n    return t.strip()\n\nkey = hashlib.sha256(norm(chunk).encode()).hexdigest()"},{"t":"p","runs":[{"text":"Keep one, record how many copies existed and where. The count is useful later: a chunk that appears in eleven documents is probably boilerplate, and you have just detected it for free."}]},{"t":"h2","id":"near-duplicates-minhash-or-embeddings","runs":[{"text":"Near-duplicates: MinHash or embeddings"}]},{"t":"p","runs":[{"text":"Exact hashing misses the version that changed one sentence. Two standard tools, both free:"}]},{"t":"p","runs":[{"text":"MinHash with locality-sensitive hashing","b":true},{"text":" ("},{"text":"datasketch","code":true},{"text":" in Python) finds pairs above a Jaccard-similarity threshold over word shingles, in roughly linear time. Good for near-identical text, cheap on millions of chunks, and it does not need embeddings."}]},{"t":"p","runs":[{"text":"Embedding similarity","b":true},{"text":" catches paraphrase that MinHash misses, at higher cost. If you are embedding everything anyway, a similarity above roughly 0.95 within a corpus is a strong duplicate signal; calibrate the exact number on your own data, since as the earlier lesson said, the scale is model-specific."}]},{"t":"p","runs":[{"text":"Deduplicate at ingestion where you can afford to, and again at retrieval time as a cheap pass over the ten to forty candidates you are about to pack. The second pass is nearly free and catches duplicates that entered from different sources."}]},{"t":"h2","id":"boilerplate-detect-it-do-not-hand-write-rules","runs":[{"text":"Boilerplate: detect it, do not hand-write rules"}]},{"t":"p","runs":[{"text":"Every corpus has chrome: navigation, cookie notices, licence headers, email signatures, confidentiality footers, “was this page helpful?”. It is textually meaningless and it competes for attention and budget."}]},{"t":"p","runs":[{"text":"The robust detection is statistical rather than rule-based. Split documents into blocks, count how many distinct documents each normalised block appears in, and treat any block appearing in more than a few percent of the corpus as chrome. This finds the footer you did not know about and adapts when the site template changes, which a regex does not."}]},{"t":"p","runs":[{"text":"An instructive case: a legal team’s corpus where every one of 4,000 documents carried an identical 140-token confidentiality notice. That is 560,000 tokens of the index, and at k=10 it was 1,400 tokens of every single request, saying nothing. Removing it is a pure win with no quality trade-off to measure."}]},{"t":"h2","id":"versions-pick-one-or-label-all-of-them","runs":[{"text":"Versions: pick one, or label all of them"}]},{"t":"p","runs":[{"text":"Deduplication cannot resolve genuine version conflicts, because both versions are real documents. Two workable policies:"}]},{"t":"ol","start":1,"items":[[{"text":"Choose at retrieval.","b":true},{"text":" Group candidates by a document-family key, keep the most recent per family, and drop the rest before packing. Correct for policies, prices, specifications: anything with a current value."}],[{"text":"Keep them and label them.","b":true},{"text":" When history matters, include both with explicit dates and an instruction about which governs. Expect the model to get this right less often than you would like; the date has to be adjacent to the text, not in a header 3,000 tokens away."}]]},{"t":"p","runs":[{"text":"What does not work is retrieving both unlabelled and hoping. That is the blended-answer failure, and it is invisible in every metric except a human reading the output against the source."}]},{"t":"h2","id":"the-audit","runs":[{"text":"The audit"}]},{"t":"p","runs":[{"text":"For twenty real queries, print the retrieved set and mark any pair that says substantially the same thing. Most first-time audits find a duplication rate between 20% and 40%. That is the same fraction of your retrieval budget, and recovering it is usually a bigger quality win than swapping the embedding model."}]}],"check":{"question":[{"text":"An assistant answers a pricing question with a figure from a 2021 document and a condition from the 2026 revision, in one fluent sentence. Both documents were retrieved. What is the mechanism?"}],"options":[{"text":[{"text":"Undeduplicated versions of the same document were both in the window, and the model merged them because nothing marked which one governs."}],"correct":true,"tempting":null},{"text":[{"text":"The 2021 document ranked higher, so its content dominated the answer."}],"correct":false,"tempting":[{"text":"Explains why one document might win, but not why parts of both appear in a single blended statement."}]},{"text":[{"text":"The model hallucinated the figure, since neither document contained that combination."}],"correct":false,"tempting":[{"text":"Reaches for hallucination when both components are traceable to real retrieved text; the invention is the combination, not the content."}]},{"text":[{"text":"The chunker split each document mid-policy, so neither version was complete in the window."}],"correct":false,"tempting":[{"text":"Describes a chunking failure that would more likely produce a partial or hedged answer than a confident merge of two versions."}]}],"order":[3,0,1,2],"why":null},"prev":{"slug":"the-k-curve","title":"How many passages is the right number","minutes":9},"next":{"slug":"freshness-and-as-of-lines","title":"Telling the model when things were true","minutes":8}}}