The same passage three times

Repetition is not evidence, but it reads like it

Retrieve ten passages from a real corpus and you will often find that four of them say the same thing. A policy page, its copy in the onboarding guide, the version in last quarter’s FAQ, and a support macro that quotes it.

This costs you twice.

Budget. Four slots spent on one fact. The other three facts the question needed did not fit.

Confidence. A statement repeated four times in the window is more likely to appear in the answer than a statement made once, regardless of which is correct. Models weight consistency across their context, and your retriever has just manufactured consistency out of a copy-paste.

The second effect is worse when the duplicates are not identical. Three versions of a policy from 2021, 2023 and 2026, all retrieved, all plausible. The model will not tell you it saw three; it will produce one answer that is a blend, sometimes with a number from the old version and a condition from the new one. That is the single most damaging kind of context error, because the output is fluent, specific and untraceable.

Exact duplicates: normalise and hash

Cheap and worth doing at ingestion:

def norm(t):
    t = unicodedata.normalize("NFKC", t.lower())
    t = re.sub(r"\s+", " ", t)
    t = re.sub(r"[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-", "", t)  # ids
    return t.strip()

key = hashlib.sha256(norm(chunk).encode()).hexdigest()

Keep one, record how many copies existed and where. The count is useful later: a chunk that appears in eleven documents is probably boilerplate, and you have just detected it for free.

Near-duplicates: MinHash or embeddings

Exact hashing misses the version that changed one sentence. Two standard tools, both free:

MinHash with locality-sensitive hashing (datasketch in Python) finds pairs above a Jaccard-similarity threshold over word shingles, in roughly linear time. Good for near-identical text, cheap on millions of chunks, and it does not need embeddings.

Embedding similarity catches paraphrase that MinHash misses, at higher cost. If you are embedding everything anyway, a similarity above roughly 0.95 within a corpus is a strong duplicate signal; calibrate the exact number on your own data, since as the earlier lesson said, the scale is model-specific.

Deduplicate at ingestion where you can afford to, and again at retrieval time as a cheap pass over the ten to forty candidates you are about to pack. The second pass is nearly free and catches duplicates that entered from different sources.

Boilerplate: detect it, do not hand-write rules

Every corpus has chrome: navigation, cookie notices, licence headers, email signatures, confidentiality footers, “was this page helpful?”. It is textually meaningless and it competes for attention and budget.

The robust detection is statistical rather than rule-based. Split documents into blocks, count how many distinct documents each normalised block appears in, and treat any block appearing in more than a few percent of the corpus as chrome. This finds the footer you did not know about and adapts when the site template changes, which a regex does not.

An instructive case: a legal team’s corpus where every one of 4,000 documents carried an identical 140-token confidentiality notice. That is 560,000 tokens of the index, and at k=10 it was 1,400 tokens of every single request, saying nothing. Removing it is a pure win with no quality trade-off to measure.

Versions: pick one, or label all of them

Deduplication cannot resolve genuine version conflicts, because both versions are real documents. Two workable policies:

1. Choose at retrieval. Group candidates by a document-family key, keep the most recent per family, and drop the rest before packing. Correct for policies, prices, specifications: anything with a current value.
2. Keep them and label them. When history matters, include both with explicit dates and an instruction about which governs. Expect the model to get this right less often than you would like; the date has to be adjacent to the text, not in a header 3,000 tokens away.

What does not work is retrieving both unlabelled and hoping. That is the blended-answer failure, and it is invisible in every metric except a human reading the output against the source.

The audit

For twenty real queries, print the retrieved set and mark any pair that says substantially the same thing. Most first-time audits find a duplication rate between 20% and 40%. That is the same fraction of your retrieval budget, and recovering it is usually a bigger quality win than swapping the embedding model.