{"course":{"slug":"hugging-face","title":"Hugging Face, End to End","url":"https://addaly.com/learn/hugging-face","level":"intermediate","track":"create"},"lesson":{"slug":"chunking-text-for-retrieval","title":"Chunking decides what your search can find","url":"https://addaly.com/learn/hugging-face/chunking-text-for-retrieval","position":52,"of":75,"minutes":9,"answer":"Chunk size and boundaries decide what retrieval can find at all, so split on structure rather than character counts, count in the model's own tokens because scripts differ.","keyIdea":"Chunk size and boundaries decide what retrieval can find at all, so split on structure rather than character counts, count in the model's own tokens because scripts differ, carry the document and section title into the embedded text, and consider retrieving small chunks while passing their parent sections to the model."},"body":{"format":"markdown","text":"## Why a whole document is the wrong unit\n\nEmbed a 40-page manual into one 768-number vector and you get the average of everything in it. A query about one specific procedure matches weakly, because that procedure is one fortieth of what the vector describes. Meanwhile the model truncated at 512 tokens anyway, so thirty-nine of those pages were never read.\n\nChunking splits documents into pieces small enough to be about one thing. It is the step that decides what your system can find, and it is almost always the highest-leverage thing to change when retrieval is disappointing, more so than swapping the model.\n\n## Sizes, with actual numbers\n\n- **Under 100 tokens**: precise matches, no context. The retrieved chunk often does not contain enough to answer.\n- **200 to 400 tokens**: the usual sweet spot for question answering over documents.\n- **500 to 800 tokens**: better for summarising and broad questions, at the edge of many models' windows.\n- **Over 1,000 tokens**: beyond most embedding models' limits, and dilute.\n\nStart at 300 tokens with 50 tokens of overlap and adjust once you have measured.\n\n**Overlap** exists because a fact can straddle a boundary. A 50-token overlap means a sentence cut in half appears whole in one of the two chunks. It costs storage proportional to the overlap fraction and prevents a failure that is otherwise invisible.\n\n## Split on structure, not on character counts\n\nThe naive version cuts every 1,500 characters, mid-word, mid-sentence, mid-table. Better, in ascending order of effort and quality:\n\n> Figure: Recall against chunk size, for two kinds of question. Line chart of recall at five, per cent against chunk size in tokens: One fact, inside one paragraph (100, 62) (200, 84) (300, 91) (500, 88) (800, 79) (1200, 66); An answer spanning two sections (100, 31) (200, 48) (300, 61) (500, 74) (800, 80) (1200, 72). One corpus, one embedding model, thirty questions, and chunk size the only thing changed. The peak is in a different place for the two kinds of question, which is the argument for retrieving small chunks and passing their parent sections to the model rather than picking one size for everything.\n\n**By separator, recursively.** Try paragraph breaks first; if a piece is still too long, split on sentences; if still too long, on words. A dozen lines of Python, and it keeps paragraphs whole whenever it can:\n\n```python\ndef split(text, limit=1200, seps=(\"\\n\\n\", \"\\n\", \". \", \" \")):\n    if len(text) <= limit or not seps:\n        return [text]\n    sep, rest = seps[0], seps[1:]\n    parts, out, cur = text.split(sep), [], \"\"\n    for p in parts:\n        if len(cur) + len(p) + len(sep) <= limit:\n            cur += (sep if cur else \"\") + p\n        else:\n            if cur: out.append(cur)\n            out.extend(split(p, limit, rest) if len(p) > limit else [p])\n            cur = \"\"\n    if cur: out.append(cur)\n    return out\n```\n\n**By document structure.** Markdown headings, HTML sections, PDF pages, slide boundaries. A chunk that is one section of a manual is a chunk about one thing, by construction. This is the best option when the structure exists.\n\n**By semantics.** Embed each sentence, cut where consecutive sentences become dissimilar. Elegant, slower, and in most measured comparisons only slightly better than splitting on headings. Try the cheap thing first.\n\n## Carry the context with the chunk\n\nA chunk stripped of its origin is much harder to use. Store metadata alongside every vector, and prepend the important parts to the text you embed:\n\n```python\nchunk_text = f\"{doc_title} — {section_heading}\\n\\n{body}\"\n```\n\nThat prefix is a few tokens and it fixes a common failure: a chunk reading \"Enter the code and press confirm\" is unfindable and useless without \"Resetting your PIN\" attached to it.\n\nKeep, at minimum: source document id, title, section, page or position, and the date. The date matters because retrieval over a corpus containing three versions of a policy will happily return the 2019 one.\n\n## Tokens, not characters\n\nA 1,500-character limit is roughly 375 English tokens and roughly 900 Hindi tokens on an English-centric tokenizer. Chunk in characters and your Hindi chunks silently exceed the model's window while your English ones sit comfortably inside it: a bug that makes a multilingual system worse in exactly the language it was added for. Count with the model's own tokenizer:\n\n```python\nlen(model.tokenizer.encode(chunk))\n```\n\n## Retrieve small, give large\n\nA technique worth knowing because it resolves the size trade-off. Embed and search over small chunks, then give the language model the **parent** section that chunk came from. You get the precision of a small chunk for matching and the completeness of a large one for answering. It costs one extra column in your metadata (the parent id) and it is usually the single best change after fixing chunk size.\n\n## Do this now\n\nTake ten questions you would ask of your own documents. For each, find by hand the passage that answers it, and measure its length in tokens. That distribution is your chunk size, derived from your actual content rather than from a default."},"check":{"question":"A bilingual system chunks at 1,500 characters. English retrieval is good and Hindi retrieval is poor, using the same multilingual model. What is the likely mechanism?","options":[{"text":"Hindi text produces far more tokens per character, so its chunks exceed the model's window and are truncated.","correct":true},{"text":"The multilingual model has weaker Hindi representations, so no chunking strategy would fix the gap.","correct":false,"tempting":"Plausible in general, but it does not explain why the same model performs well in one language under the same settings."},{"text":"Devanagari characters occupy multiple bytes, so character-based splitting cuts words mid-codepoint and corrupts the text.","correct":false,"tempting":"Python strings count characters rather than bytes, so splitting does not break codepoints."},{"text":"Hindi documents are longer on average, so each one produces more chunks and dilutes the ranking.","correct":false,"tempting":"More chunks per document changes recall slightly but does not degrade matching the way truncation does."}]},"prev":{"slug":"similarity-and-distance","title":"Cosine, dot product, and the normalisation that connects them","url":"https://addaly.com/learn/hugging-face/similarity-and-distance"},"next":{"slug":"a-search-index-without-a-database","title":"NumPy first, FAISS second, a database rarely","url":"https://addaly.com/learn/hugging-face/a-search-index-without-a-database"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Chunking decides what your search can find', in Hugging Face, End to End. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/hugging-face/chunking-text-for-retrieval (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Chunking decides what your search can find","courseTitle":"Hugging Face, End to End","module":{"index":6,"slug":"embeddings-and-retrieval","title":"Embeddings, and search that understands"},"answer":null,"keyIdea":"Chunk size and boundaries decide what retrieval can find at all, so split on structure rather than character counts, count in the model’s own tokens because scripts differ, carry the document and section title into the embedded text, and consider retrieving small chunks while passing their parent sections to the model.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"type":"line","title":"Recall against chunk size, for two kinds of question","series":[{"label":"One fact, inside one paragraph","points":[[100,62],[200,84],[300,91],[500,88],[800,79],[1200,66]]},{"label":"An answer spanning two sections","points":[[100,31],[200,48],[300,61],[500,74],[800,80],[1200,72]]}],"xLabel":"chunk size in tokens","yLabel":"recall at five, per cent","caption":"One corpus, one embedding model, thirty questions, and chunk size the only thing changed. The peak is in a different place for the two kinds of question, which is the argument for retrieving small chunks and passing their parent sections to the model rather than picking one size for everything."},"alt":"Figure: Recall against chunk size, for two kinds of question. Line chart of recall at five, per cent against chunk size in tokens: One fact, inside one paragraph (100, 62) (200, 84) (300, 91) (500, 88) (800, 79) (1200, 66); An answer spanning two sections (100, 31) (200, 48) (300, 61) (500, 74) (800, 80) (1200, 72). One corpus, one embedding model, thirty questions, and chunk size the only thing changed. The peak is in a different place for the two kinds of question, which is the argument for retrieving small chunks and passing their parent sections to the model rather than picking one size for everything."}},"blocks":[{"t":"h2","id":"why-a-whole-document-is-the-wrong-unit","runs":[{"text":"Why a whole document is the wrong unit"}]},{"t":"p","runs":[{"text":"Embed a 40-page manual into one 768-number vector and you get the average of everything in it. A query about one specific procedure matches weakly, because that procedure is one fortieth of what the vector describes. Meanwhile the model truncated at 512 tokens anyway, so thirty-nine of those pages were never read."}]},{"t":"p","runs":[{"text":"Chunking splits documents into pieces small enough to be about one thing. It is the step that decides what your system can find, and it is almost always the highest-leverage thing to change when retrieval is disappointing, more so than swapping the model."}]},{"t":"h2","id":"sizes-with-actual-numbers","runs":[{"text":"Sizes, with actual numbers"}]},{"t":"ul","items":[[{"text":"Under 100 tokens","b":true},{"text":": precise matches, no context. The retrieved chunk often does not contain enough to answer."}],[{"text":"200 to 400 tokens","b":true},{"text":": the usual sweet spot for question answering over documents."}],[{"text":"500 to 800 tokens","b":true},{"text":": better for summarising and broad questions, at the edge of many models’ windows."}],[{"text":"Over 1,000 tokens","b":true},{"text":": beyond most embedding models’ limits, and dilute."}]]},{"t":"p","runs":[{"text":"Start at 300 tokens with 50 tokens of overlap and adjust once you have measured."}]},{"t":"p","runs":[{"text":"Overlap","b":true},{"text":" exists because a fact can straddle a boundary. A 50-token overlap means a sentence cut in half appears whole in one of the two chunks. It costs storage proportional to the overlap fraction and prevents a failure that is otherwise invisible."}]},{"t":"h2","id":"split-on-structure-not-on-character-counts","runs":[{"text":"Split on structure, not on character counts"}]},{"t":"p","runs":[{"text":"The naive version cuts every 1,500 characters, mid-word, mid-sentence, mid-table. Better, in ascending order of effort and quality:"}]},{"t":"p","runs":[{"text":"By separator, recursively.","b":true},{"text":" Try paragraph breaks first; if a piece is still too long, split on sentences; if still too long, on words. A dozen lines of Python, and it keeps paragraphs whole whenever it can:"}]},{"t":"code","lang":"python","label":"Python","text":"def split(text, limit=1200, seps=(\"\\n\\n\", \"\\n\", \". \", \" \")):\n    if len(text) <= limit or not seps:\n        return [text]\n    sep, rest = seps[0], seps[1:]\n    parts, out, cur = text.split(sep), [], \"\"\n    for p in parts:\n        if len(cur) + len(p) + len(sep) <= limit:\n            cur += (sep if cur else \"\") + p\n        else:\n            if cur: out.append(cur)\n            out.extend(split(p, limit, rest) if len(p) > limit else [p])\n            cur = \"\"\n    if cur: out.append(cur)\n    return out"},{"t":"p","runs":[{"text":"By document structure.","b":true},{"text":" Markdown headings, HTML sections, PDF pages, slide boundaries. A chunk that is one section of a manual is a chunk about one thing, by construction. This is the best option when the structure exists."}]},{"t":"p","runs":[{"text":"By semantics.","b":true},{"text":" Embed each sentence, cut where consecutive sentences become dissimilar. Elegant, slower, and in most measured comparisons only slightly better than splitting on headings. Try the cheap thing first."}]},{"t":"h2","id":"carry-the-context-with-the-chunk","runs":[{"text":"Carry the context with the chunk"}]},{"t":"p","runs":[{"text":"A chunk stripped of its origin is much harder to use. Store metadata alongside every vector, and prepend the important parts to the text you embed:"}]},{"t":"code","lang":"python","label":"Python","text":"chunk_text = f\"{doc_title} — {section_heading}\\n\\n{body}\""},{"t":"p","runs":[{"text":"That prefix is a few tokens and it fixes a common failure: a chunk reading “Enter the code and press confirm” is unfindable and useless without “Resetting your PIN” attached to it."}]},{"t":"p","runs":[{"text":"Keep, at minimum: source document id, title, section, page or position, and the date. The date matters because retrieval over a corpus containing three versions of a policy will happily return the 2019 one."}]},{"t":"h2","id":"tokens-not-characters","runs":[{"text":"Tokens, not characters"}]},{"t":"p","runs":[{"text":"A 1,500-character limit is roughly 375 English tokens and roughly 900 Hindi tokens on an English-centric tokenizer. Chunk in characters and your Hindi chunks silently exceed the model’s window while your English ones sit comfortably inside it: a bug that makes a multilingual system worse in exactly the language it was added for. Count with the model’s own tokenizer:"}]},{"t":"code","lang":"python","label":"Python","text":"len(model.tokenizer.encode(chunk))"},{"t":"h2","id":"retrieve-small-give-large","runs":[{"text":"Retrieve small, give large"}]},{"t":"p","runs":[{"text":"A technique worth knowing because it resolves the size trade-off. Embed and search over small chunks, then give the language model the "},{"text":"parent","b":true},{"text":" section that chunk came from. You get the precision of a small chunk for matching and the completeness of a large one for answering. It costs one extra column in your metadata (the parent id) and it is usually the single best change after fixing chunk size."}]},{"t":"h2","id":"do-this-now","runs":[{"text":"Do this now"}],"role":"try"},{"t":"p","runs":[{"text":"Take ten questions you would ask of your own documents. For each, find by hand the passage that answers it, and measure its length in tokens. That distribution is your chunk size, derived from your actual content rather than from a default."}]}],"check":{"question":[{"text":"A bilingual system chunks at 1,500 characters. English retrieval is good and Hindi retrieval is poor, using the same multilingual model. What is the likely mechanism?"}],"options":[{"text":[{"text":"Hindi text produces far more tokens per character, so its chunks exceed the model’s window and are truncated."}],"correct":true,"tempting":null},{"text":[{"text":"The multilingual model has weaker Hindi representations, so no chunking strategy would fix the gap."}],"correct":false,"tempting":[{"text":"Plausible in general, but it does not explain why the same model performs well in one language under the same settings."}]},{"text":[{"text":"Devanagari characters occupy multiple bytes, so character-based splitting cuts words mid-codepoint and corrupts the text."}],"correct":false,"tempting":[{"text":"Python strings count characters rather than bytes, so splitting does not break codepoints."}]},{"text":[{"text":"Hindi documents are longer on average, so each one produces more chunks and dilutes the ranking."}],"correct":false,"tempting":[{"text":"More chunks per document changes recall slightly but does not degrade matching the way truncation does."}]}],"order":[3,0,1,2],"why":null},"prev":{"slug":"similarity-and-distance","title":"Cosine, dot product, and the normalisation that connects them","minutes":8},"next":{"slug":"a-search-index-without-a-database","title":"NumPy first, FAISS second, a database rarely","minutes":9}}}