Chunking that does not destroy meaning

The default is wrong for a reason worth understanding

Split at every 512 tokens with 50 tokens of overlap. It is the default in every framework, and it is wrong in a specific way: it treats a document as a string when a document is a structure.

A fixed split cuts tables away from their headers, separates a clause from the section that qualifies it, and orphans every pronoun. The chunk reads fine to a human skimming, which is why the bug is hard to see, and it is missing the exact word that makes it correct.

Here is the failure that costs money. A refund policy opens with “The following applies to customers outside the European Union.” Three pages later, chunk 41 reads “Refunds are processed within 14 days of receipt.” Retrieved alone, that chunk is a confident, well-formed, wrong answer for a customer in Dublin. No overlap setting reaches three pages up.

Split on structure, then on size

The order matters. Find the boundaries the document already has (headings, sections, list items, table rows, code blocks, slide breaks) and only then merge or split to hit a token budget.

def chunk(doc, target=700, floor=200):
    blocks = parse_structure(doc)        # [{level, heading_path, text, kind}]
    out, buf, path = [], [], None
    for b in blocks:
        if b.kind == "table":            # never merge a table into prose
            out += table_rows(b)         # one chunk per row, header attached
            continue
        if path and b.heading_path[:2] != path[:2]:
            out.append(flush(buf, path)); buf, path = [], b.heading_path
        buf.append(b.text); path = path or b.heading_path
        if tokens(buf) >= target:
            out.append(flush(buf, path)); buf = []
    return [c for c in out if tokens(c) >= floor or c.kind == "table_row"]

def flush(buf, path):
    return " > ".join(path) + "\n\n" + "\n".join(buf)

The last line is the important one. Every chunk carries its heading path (Refunds > Outside the EU > Timelines) so the qualifier travels with the text. This is a two-line change that fixes the Dublin bug.

Add the context the chunk lost

Go further: before embedding, prepend one or two sentences that situate the chunk in its document. Anthropic published this as “contextual retrieval” in 2024 and reported, on their benchmark, that contextual embeddings cut top-20 retrieval failures by about 35%, and by about 49% when combined with contextual BM25. Treat the exact figures as theirs rather than yours, and treat the direction as reliable.

This passage is from the 2026 EMEA refund policy, in the section covering
non-EU customers. It follows the eligibility criteria.

Refunds are processed within 14 days of receipt...

You generate that preamble once per chunk with a cheap model, at ingestion. With prompt caching against the full document, the cost is small (well under a dollar per million tokens of corpus on current small models) and it is paid once, not per query.

Index small, return large

A short passage embeds precisely. A long passage answers completely. You do not have to choose: embed the small unit, return its parent.

Index each paragraph or each summary. Store a pointer to the enclosing section. At query time, match on the small thing, retrieve the big thing, deduplicate parents so you do not return the same section four times. Frameworks call this parent-document or small-to-big retrieval; it is thirty lines of code without one.

The cases that break naive splitters

• Tables. One chunk per row, with the header row re-attached and the table’s caption in the heading path. A table split at row 40 is unreadable to the model and to you.
• Code. Split on function and class boundaries. Attach the file path and the import block. A function without its imports is missing its type information.
• Boilerplate. Strip repeated headers, footers, page numbers and navigation before embedding. Otherwise every chunk in a 400-page manual shares 30 identical tokens and the embeddings drift towards each other.
• Scanned documents. OCR output has no structure to split on. Recover it (heading detection, layout parsing) or accept that this corpus will retrieve badly.
• Conversations. Split on speaker turns and topic shifts, never on token count. Half a question is worse than nothing.

How to check your work

Sample 30 chunks at random and read them cold, without the source document open. For each, ask: could a careful colleague answer a real question from this alone, and would they know what it applies to?

If the answer is no, no amount of reranking downstream will save you. Retrieval quality is bounded by the quality of the units you built.