Chunking decides what your search can find

Why a whole document is the wrong unit

Embed a 40-page manual into one 768-number vector and you get the average of everything in it. A query about one specific procedure matches weakly, because that procedure is one fortieth of what the vector describes. Meanwhile the model truncated at 512 tokens anyway, so thirty-nine of those pages were never read.

Chunking splits documents into pieces small enough to be about one thing. It is the step that decides what your system can find, and it is almost always the highest-leverage thing to change when retrieval is disappointing, more so than swapping the model.

Sizes, with actual numbers

• Under 100 tokens: precise matches, no context. The retrieved chunk often does not contain enough to answer.
• 200 to 400 tokens: the usual sweet spot for question answering over documents.
• 500 to 800 tokens: better for summarising and broad questions, at the edge of many models’ windows.
• Over 1,000 tokens: beyond most embedding models’ limits, and dilute.

Start at 300 tokens with 50 tokens of overlap and adjust once you have measured.

Overlap exists because a fact can straddle a boundary. A 50-token overlap means a sentence cut in half appears whole in one of the two chunks. It costs storage proportional to the overlap fraction and prevents a failure that is otherwise invisible.

Split on structure, not on character counts

The naive version cuts every 1,500 characters, mid-word, mid-sentence, mid-table. Better, in ascending order of effort and quality:

By separator, recursively. Try paragraph breaks first; if a piece is still too long, split on sentences; if still too long, on words. A dozen lines of Python, and it keeps paragraphs whole whenever it can:

def split(text, limit=1200, seps=("\n\n", "\n", ". ", " ")):
    if len(text) <= limit or not seps:
        return [text]
    sep, rest = seps[0], seps[1:]
    parts, out, cur = text.split(sep), [], ""
    for p in parts:
        if len(cur) + len(p) + len(sep) <= limit:
            cur += (sep if cur else "") + p
        else:
            if cur: out.append(cur)
            out.extend(split(p, limit, rest) if len(p) > limit else [p])
            cur = ""
    if cur: out.append(cur)
    return out

By document structure. Markdown headings, HTML sections, PDF pages, slide boundaries. A chunk that is one section of a manual is a chunk about one thing, by construction. This is the best option when the structure exists.

By semantics. Embed each sentence, cut where consecutive sentences become dissimilar. Elegant, slower, and in most measured comparisons only slightly better than splitting on headings. Try the cheap thing first.

Carry the context with the chunk

A chunk stripped of its origin is much harder to use. Store metadata alongside every vector, and prepend the important parts to the text you embed:

chunk_text = f"{doc_title} — {section_heading}\n\n{body}"

That prefix is a few tokens and it fixes a common failure: a chunk reading “Enter the code and press confirm” is unfindable and useless without “Resetting your PIN” attached to it.

Keep, at minimum: source document id, title, section, page or position, and the date. The date matters because retrieval over a corpus containing three versions of a policy will happily return the 2019 one.

Tokens, not characters

A 1,500-character limit is roughly 375 English tokens and roughly 900 Hindi tokens on an English-centric tokenizer. Chunk in characters and your Hindi chunks silently exceed the model’s window while your English ones sit comfortably inside it: a bug that makes a multilingual system worse in exactly the language it was added for. Count with the model’s own tokenizer:

len(model.tokenizer.encode(chunk))

Retrieve small, give large

A technique worth knowing because it resolves the size trade-off. Embed and search over small chunks, then give the language model the parent section that chunk came from. You get the precision of a small chunk for matching and the completeness of a large one for answering. It costs one extra column in your metadata (the parent id) and it is usually the single best change after fixing chunk size.

Do this now

Take ten questions you would ask of your own documents. For each, find by hand the passage that answers it, and measure its length in tokens. That distribution is your chunk size, derived from your actual content rather than from a default.