{"course":{"slug":"how-llms-work","title":"How a Language Model Actually Works","url":"https://addaly.com/learn/how-llms-work","level":"intermediate","track":"start"},"lesson":{"slug":"chunking-decisions","title":"Chunking: the decision that quietly sets your ceiling","url":"https://addaly.com/learn/how-llms-work/chunking-decisions","position":35,"of":78,"minutes":8,"answer":"A chunk must be small enough to be a precise retrieval unit and complete enough to be a usable answer, and those two demands pull in opposite directions.","keyIdea":"A chunk must be small enough to be a precise retrieval unit and complete enough to be a usable answer, and those two demands pull in opposite directions."},"body":{"format":"markdown","text":"## The tension\n\nEmbed a whole 40-page document as one vector and it means everything and nothing: the vector is an average of forty pages of topics, and it will match weakly against every specific question. Embed each sentence and you get precision with no context: a retrieved sentence saying \"This is not permitted for accounts opened before that date\" is useless without knowing what \"this\" and \"that\" refer to.\n\nEverything about chunking is managing that tension. There is no correct size, but there are clearly wrong ones and a set of decisions that reliably help.\n\n## Start here\n\nFor prose in a general knowledge base, chunks of roughly 300 to 500 tokens with 10 to 15 per cent overlap are a sound default. Overlap exists so that an answer straddling a boundary appears intact in at least one chunk.\n\nThen adjust by document type, because the type matters more than the tuning:\n\n- **Reference and policy documents** have natural units. Split on headings, and keep a section together even if it is long. A policy clause cut in half is worse than a long chunk.\n- **Question-and-answer content** splits at the question. Never separate a question from its answer.\n- **Code** splits at function or class boundaries. A syntactic splitter beats a character-count splitter decisively here, and free tools using tree-sitter do it properly.\n- **Transcripts** split on speaker turns or topic shifts, not on length.\n- **Tables** should not be split at all. Extract them separately and store each row with its header, or store the whole table as one chunk with a text description.\n\n## The three improvements that matter more than size\n\n**Add a header to every chunk.** Prepend the document title, the section path, and the date before embedding:\n\n```text\n[Refund Policy > International Orders > Updated March 2026]\nRefunds for orders shipped outside India are processed within 21 working days...\n```\n\nThis is the highest-value change available and it is nearly free. It fixes the orphaned-pronoun problem, it lets the embedding carry the document's topic as well as the passage's, and it gives the model something to cite.\n\n**Store metadata and filter on it.** Date, department, document type, language, version. Filtering to the current version before ranking by similarity removes a whole class of confidently-wrong answers, and it is a database operation rather than a machine-learning one.\n\n**Retrieve small, feed large.** Index short chunks so matching is precise, but when a chunk is retrieved, send its *parent* section to the model. You get retrieval precision and generation context at the same time. This pattern (small-to-big, or parent-document retrieval) is available in every retrieval framework and is worth the small extra machinery.\n\n## Extraction comes before chunking, and is usually where the damage is\n\nChunking a badly extracted document cannot succeed. PDF extraction in particular is much harder than it looks: multi-column layouts interleave, headers and footers appear mid-sentence, tables collapse into a stream of numbers with no structure, and scanned pages produce nothing at all without OCR.\n\nFree tools worth knowing: `pypdf` for simple text, `pdfplumber` for layout and tables, `unstructured` for mixed documents, and Tesseract for OCR. Spend the time here. Read fifty of your own extracted chunks before building anything on top of them, and you will find problems that no amount of retrieval tuning would ever have fixed.\n\n## How to decide, rather than guess\n\nChunking is easy to test because you can hold everything else constant:\n\n1. Fix 50 questions and their correct source passages.\n2. Build the index three ways: 200, 400 and 800 tokens, with and without headers.\n3. Measure how often the correct passage is in the top 5.\n4. Pick the winner.\n\nAn hour, no model calls needed for the retrieval half, and it replaces an argument that otherwise recurs every few months. Re-run it when your corpus changes materially, because the right answer for a set of policy documents is not the right answer for a set of chat transcripts."},"check":{"question":"Retrieval works well for questions about single paragraphs but fails on questions whose answers are in tables. What is the mechanism?","options":[{"text":"A table split by a length-based chunker loses the association between headers and rows, so no chunk contains a self-contained fact.","correct":true},{"text":"Numbers embed poorly, so tables have weak vectors regardless of chunking.","correct":false,"tempting":"Numeric content genuinely does embed less distinctively than prose, which makes this a plausible representational explanation."},{"text":"Tables are usually longer than the chunk size and are therefore skipped by the indexer.","correct":false,"tempting":"Length-based skipping does occur in some pipelines, so an indexing omission is a fair operational suspicion."},{"text":"The model cannot read tabular data presented as plain text in a prompt.","correct":false,"tempting":"Models do struggle with badly formatted tables, so blaming the generation stage matches a real difficulty."}]},"prev":{"slug":"retrieval-mechanically","title":"Retrieval, as a pipeline that can break in six places","url":"https://addaly.com/learn/how-llms-work/retrieval-mechanically"},"next":{"slug":"tools-and-function-calling","title":"Tool use, with the magic removed","url":"https://addaly.com/learn/how-llms-work/tools-and-function-calling"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Chunking: the decision that quietly sets your ceiling', in How a Language Model Actually Works. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/how-llms-work/chunking-decisions (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Chunking: the decision that quietly sets your ceiling","courseTitle":"How a Language Model Actually Works","module":{"index":4,"slug":"the-context-window","title":"What the model can see"},"answer":null,"keyIdea":"A chunk must be small enough to be a precise retrieval unit and complete enough to be a usable answer, and those two demands pull in opposite directions.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"the-tension","runs":[{"text":"The tension"}]},{"t":"p","runs":[{"text":"Embed a whole 40-page document as one vector and it means everything and nothing: the vector is an average of forty pages of topics, and it will match weakly against every specific question. Embed each sentence and you get precision with no context: a retrieved sentence saying “This is not permitted for accounts opened before that date” is useless without knowing what “this” and “that” refer to."}]},{"t":"p","runs":[{"text":"Everything about chunking is managing that tension. There is no correct size, but there are clearly wrong ones and a set of decisions that reliably help."}]},{"t":"h2","id":"start-here","runs":[{"text":"Start here"}]},{"t":"p","runs":[{"text":"For prose in a general knowledge base, chunks of roughly 300 to 500 tokens with 10 to 15 per cent overlap are a sound default. Overlap exists so that an answer straddling a boundary appears intact in at least one chunk."}]},{"t":"p","runs":[{"text":"Then adjust by document type, because the type matters more than the tuning:"}]},{"t":"ul","items":[[{"text":"Reference and policy documents","b":true},{"text":" have natural units. Split on headings, and keep a section together even if it is long. A policy clause cut in half is worse than a long chunk."}],[{"text":"Question-and-answer content","b":true},{"text":" splits at the question. Never separate a question from its answer."}],[{"text":"Code","b":true},{"text":" splits at function or class boundaries. A syntactic splitter beats a character-count splitter decisively here, and free tools using tree-sitter do it properly."}],[{"text":"Transcripts","b":true},{"text":" split on speaker turns or topic shifts, not on length."}],[{"text":"Tables","b":true},{"text":" should not be split at all. Extract them separately and store each row with its header, or store the whole table as one chunk with a text description."}]]},{"t":"h2","id":"the-three-improvements-that-matter-more-than-size","runs":[{"text":"The three improvements that matter more than size"}]},{"t":"p","runs":[{"text":"Add a header to every chunk.","b":true},{"text":" Prepend the document title, the section path, and the date before embedding:"}]},{"t":"code","lang":"text","label":"Text","text":"[Refund Policy > International Orders > Updated March 2026]\nRefunds for orders shipped outside India are processed within 21 working days..."},{"t":"p","runs":[{"text":"This is the highest-value change available and it is nearly free. It fixes the orphaned-pronoun problem, it lets the embedding carry the document’s topic as well as the passage’s, and it gives the model something to cite."}]},{"t":"p","runs":[{"text":"Store metadata and filter on it.","b":true},{"text":" Date, department, document type, language, version. Filtering to the current version before ranking by similarity removes a whole class of confidently-wrong answers, and it is a database operation rather than a machine-learning one."}]},{"t":"p","runs":[{"text":"Retrieve small, feed large.","b":true},{"text":" Index short chunks so matching is precise, but when a chunk is retrieved, send its "},{"text":"parent","i":true},{"text":" section to the model. You get retrieval precision and generation context at the same time. This pattern (small-to-big, or parent-document retrieval) is available in every retrieval framework and is worth the small extra machinery."}]},{"t":"h2","id":"extraction-comes-before-chunking-and-is-usually-where-the-da","runs":[{"text":"Extraction comes before chunking, and is usually where the damage is"}]},{"t":"p","runs":[{"text":"Chunking a badly extracted document cannot succeed. PDF extraction in particular is much harder than it looks: multi-column layouts interleave, headers and footers appear mid-sentence, tables collapse into a stream of numbers with no structure, and scanned pages produce nothing at all without OCR."}]},{"t":"p","runs":[{"text":"Free tools worth knowing: "},{"text":"pypdf","code":true},{"text":" for simple text, "},{"text":"pdfplumber","code":true},{"text":" for layout and tables, "},{"text":"unstructured","code":true},{"text":" for mixed documents, and Tesseract for OCR. Spend the time here. Read fifty of your own extracted chunks before building anything on top of them, and you will find problems that no amount of retrieval tuning would ever have fixed."}]},{"t":"h2","id":"how-to-decide-rather-than-guess","runs":[{"text":"How to decide, rather than guess"}]},{"t":"p","runs":[{"text":"Chunking is easy to test because you can hold everything else constant:"}]},{"t":"ol","start":1,"items":[[{"text":"Fix 50 questions and their correct source passages."}],[{"text":"Build the index three ways: 200, 400 and 800 tokens, with and without headers."}],[{"text":"Measure how often the correct passage is in the top 5."}],[{"text":"Pick the winner."}]]},{"t":"p","runs":[{"text":"An hour, no model calls needed for the retrieval half, and it replaces an argument that otherwise recurs every few months. Re-run it when your corpus changes materially, because the right answer for a set of policy documents is not the right answer for a set of chat transcripts."}]}],"check":{"question":[{"text":"Retrieval works well for questions about single paragraphs but fails on questions whose answers are in tables. What is the mechanism?"}],"options":[{"text":[{"text":"A table split by a length-based chunker loses the association between headers and rows, so no chunk contains a self-contained fact."}],"correct":true,"tempting":null},{"text":[{"text":"Numbers embed poorly, so tables have weak vectors regardless of chunking."}],"correct":false,"tempting":[{"text":"Numeric content genuinely does embed less distinctively than prose, which makes this a plausible representational explanation."}]},{"text":[{"text":"Tables are usually longer than the chunk size and are therefore skipped by the indexer."}],"correct":false,"tempting":[{"text":"Length-based skipping does occur in some pipelines, so an indexing omission is a fair operational suspicion."}]},{"text":[{"text":"The model cannot read tabular data presented as plain text in a prompt."}],"correct":false,"tempting":[{"text":"Models do struggle with badly formatted tables, so blaming the generation stage matches a real difficulty."}]}],"order":[1,0,2,3],"why":null},"prev":{"slug":"retrieval-mechanically","title":"Retrieval, as a pipeline that can break in six places","minutes":9},"next":{"slug":"tools-and-function-calling","title":"Tool use, with the magic removed","minutes":8}}}