Chunking: the decision that quietly sets your ceiling

The tension

Embed a whole 40-page document as one vector and it means everything and nothing: the vector is an average of forty pages of topics, and it will match weakly against every specific question. Embed each sentence and you get precision with no context: a retrieved sentence saying “This is not permitted for accounts opened before that date” is useless without knowing what “this” and “that” refer to.

Everything about chunking is managing that tension. There is no correct size, but there are clearly wrong ones and a set of decisions that reliably help.

Start here

For prose in a general knowledge base, chunks of roughly 300 to 500 tokens with 10 to 15 per cent overlap are a sound default. Overlap exists so that an answer straddling a boundary appears intact in at least one chunk.

Then adjust by document type, because the type matters more than the tuning:

• Reference and policy documents have natural units. Split on headings, and keep a section together even if it is long. A policy clause cut in half is worse than a long chunk.
• Question-and-answer content splits at the question. Never separate a question from its answer.
• Code splits at function or class boundaries. A syntactic splitter beats a character-count splitter decisively here, and free tools using tree-sitter do it properly.
• Transcripts split on speaker turns or topic shifts, not on length.
• Tables should not be split at all. Extract them separately and store each row with its header, or store the whole table as one chunk with a text description.

The three improvements that matter more than size

Add a header to every chunk. Prepend the document title, the section path, and the date before embedding:

[Refund Policy > International Orders > Updated March 2026]
Refunds for orders shipped outside India are processed within 21 working days...

This is the highest-value change available and it is nearly free. It fixes the orphaned-pronoun problem, it lets the embedding carry the document’s topic as well as the passage’s, and it gives the model something to cite.

Store metadata and filter on it. Date, department, document type, language, version. Filtering to the current version before ranking by similarity removes a whole class of confidently-wrong answers, and it is a database operation rather than a machine-learning one.

Retrieve small, feed large. Index short chunks so matching is precise, but when a chunk is retrieved, send its parent section to the model. You get retrieval precision and generation context at the same time. This pattern (small-to-big, or parent-document retrieval) is available in every retrieval framework and is worth the small extra machinery.

Extraction comes before chunking, and is usually where the damage is

Chunking a badly extracted document cannot succeed. PDF extraction in particular is much harder than it looks: multi-column layouts interleave, headers and footers appear mid-sentence, tables collapse into a stream of numbers with no structure, and scanned pages produce nothing at all without OCR.

Free tools worth knowing: pypdf for simple text, pdfplumber for layout and tables, unstructured for mixed documents, and Tesseract for OCR. Spend the time here. Read fifty of your own extracted chunks before building anything on top of them, and you will find problems that no amount of retrieval tuning would ever have fixed.

How to decide, rather than guess

Chunking is easy to test because you can hold everything else constant:

1. Fix 50 questions and their correct source passages.
2. Build the index three ways: 200, 400 and 800 tokens, with and without headers.
3. Measure how often the correct passage is in the top 5.
4. Pick the winner.

An hour, no model calls needed for the retrieval half, and it replaces an argument that otherwise recurs every few months. Re-run it when your corpus changes materially, because the right answer for a set of policy documents is not the right answer for a set of chat transcripts.