Cleaning at scale

A bad example is worse than a missing one

A missing example teaches nothing. A bad example teaches something wrong, with the same weight as a good one, and the model has no way to tell them apart. Every filter below exists because somebody trained on data they had not looked at.

Work through these in order: cheap deterministic filters first, judgement last.

Exact duplicates

Hash the normalised text and keep one copy. On data assembled from several sources, exact duplicates are routinely 5 to 20% of the file.

import hashlib
seen, kept = set(), []
for row in data:
    key = hashlib.sha256(
        " ".join(m["content"].lower().split()).encode()
        for-each-message  # normalise whitespace and case before hashing
    )

The normalisation matters: without lowercasing and collapsing whitespace, two copies of the same example differing by a trailing space survive as distinct rows.

Near duplicates

The harder and more important case. Four hundred tickets that differ only in an order number are, for training purposes, one example repeated four hundred times, and the model will weight it accordingly.

MinHash with locality-sensitive hashing finds these in near-linear time:

from datasketch import MinHash, MinHashLSH

lsh = MinHashLSH(threshold=0.85, num_perm=128)
for i, text in enumerate(texts):
    m = MinHash(num_perm=128)
    for token in set(text.lower().split()):
        m.update(token.encode())
    if not lsh.query(m):        # nothing similar already kept
        lsh.insert(str(i), m)
        keep.append(i)

datasketch is free and handles a few hundred thousand documents on a laptop. The text-dedup package wraps the same idea with more options if you have millions.

A threshold of 0.85 Jaccard similarity is a reasonable start. Check the result by hand: print ten pairs the filter considered duplicates, and ten it kept as distinct. If the boundary looks wrong, move the threshold rather than trusting it.

Length and truncation

Two filters, both cheap, both catching real damage:

• Too short. An assistant response under about 5 tokens is usually a logging artefact, not an answer. Unless one-word answers are your task, drop them.
• Truncated. An answer that ends mid-word or mid-sentence came from a generation that hit a token limit. These teach the model to stop in the middle of things. Check whether the final character is sentence-terminating punctuation, or a closing brace for structured output.

Also count how many examples exceed your training max_length, because those will be truncated by the tokenizer, losing the end-of-turn token along with the last part of the answer.

Language

If your product is monolingual and 4% of your logs are another language, decide deliberately rather than by accident. fasttext’s language identification model is free, runs at tens of thousands of documents a second, and gives a confidence score.

The reverse case matters too: if you are multilingual, check the balance. A set that is 92% English will produce a model whose other languages have quietly got worse.

Degenerate text

A handful of heuristics catch most machine-generated junk:

• Repetition ratio. If the most common 3-gram accounts for more than about 10% of the 3-grams in a response, it is looping.
• Non-word character ratio. A high proportion of punctuation or symbols usually means markup, base64 or a corrupted encoding.
• Model tics. If your data came from a hosted model, search for its characteristic openings: the self-identification phrases, the hedging preamble, the stock apology. You do not want to train those in. A regex over the first sentence finds them.

The filter you cannot automate

Read a hundred examples. Not skim: read, from the beginning to the end of the assistant turn.

This is the highest-value hour in the whole project and it is the one most reliably skipped. What it finds is not the things above; it is the systematic problems no filter names. Every answer beginning with the same three words. Annotators who disagreed about whether to include a greeting. Two hundred examples where the “correct” answer is subtly wrong because the person writing them misunderstood the task.

Keep the record

Log what each filter removed and how much:

start                    48,201
exact duplicates         -6,340
near duplicates (0.85)   -9,118
too short                  -412
truncated                  -887
non-target language        -204
repetition heuristic       -156
                        --------
final                    31,084

If one filter removes 40% of your data, that is not a cleaning step, that is a discovery about your source, and it deserves a look before you accept it.