{"course":{"slug":"python-for-ai","title":"Python, from Zero, for AI","url":"https://addaly.com/learn/python-for-ai","level":"beginner","track":"build"},"lesson":{"slug":"chunking-text","title":"Chunking: splitting a document that does not fit, without cutting a sentence in half","url":"https://addaly.com/learn/python-for-ai/chunking-text","position":83,"of":89,"minutes":8,"answer":"Split on paragraph boundaries first, then sentences, then characters as a last resort; measure chunks in tokens not characters, overlap adjacent chunks so a fact on the boundary survives.","keyIdea":"Split on paragraph boundaries first, then sentences, then characters as a last resort; measure chunks in tokens not characters, overlap adjacent chunks so a fact on the boundary survives, and keep each chunk's origin so a search result can point back to the page."},"body":{"format":"markdown","text":"## Why chunk at all\n\nA model has a context window, and a document is often larger than it. Even when the document fits, sending forty pages to answer a question about one paragraph costs forty pages of tokens per question. And an embedding (module 8's vector) represents one *piece* of text; a vector for an entire book averages away everything specific. So documents get split into chunks: small enough to embed meaningfully and to send cheaply, large enough to carry a complete thought.\n\nThe naive version, and its failure:\n\n```python\nchunks = [text[i:i + 1000] for i in range(0, len(text), 1000)]\n```\n\nThis cuts at character 1,000 wherever that falls: mid-word, mid-sentence, mid-number. A fact at the boundary is split between two chunks, and neither chunk contains it. The search finds nothing, or finds half, and the answer is confidently wrong. Almost every complaint that \"the search does not find things I know are there\" traces to this.\n\n## Split on structure first\n\nText has boundaries that mean something: paragraphs, then sentences, then words. Split at the largest boundary that keeps chunks under the limit, and only fall back to smaller ones when a single unit is too big:\n\n```python\nimport re\n\ndef split_paragraphs(text):\n    return [p.strip() for p in re.split(r\"\\n\\s*\\n\", text) if p.strip()]\n\ndef split_sentences(text):\n    return [s.strip() for s in re.split(r\"(?<=[.!?।])\\s+\", text) if s.strip()]\n```\n\nThe sentence pattern splits after `.`, `!`, `?` and the Devanagari full stop `।`, followed by whitespace. It is deliberately simple. It will split \"Dr. Sharma\" in the wrong place and it does not know that \"3.5\" is a number. For most documents that is acceptable; when it is not, the free `nltk` sentence tokeniser or `spacy` handle abbreviations and more languages, at the cost of a download.\n\n## Assemble to a token budget\n\n```python\ndef chunk(text, max_tokens, count):\n    chunks, current = [], []\n    for para in split_paragraphs(text):\n        units = [para] if count(para) <= max_tokens else split_sentences(para)\n        for unit in units:\n            if count(\" \".join(current + [unit])) > max_tokens and current:\n                chunks.append(\" \".join(current))\n                current = []\n            current.append(unit)\n    if current:\n        chunks.append(\" \".join(current))\n    return chunks\n```\n\nGreedy: add units until the next would overflow, then start a new chunk. `count` is a token counter from module 6, because a chunk limit in characters is off by a factor of two or three for Hindi or code, and the model's limit is in tokens. A paragraph that fits goes in whole; one that does not is split into sentences first. A sentence larger than the budget on its own (a pasted log, a long table row) still passes through oversized; cut it by characters as a last resort, and log that you did.\n\n## Overlap\n\nEven with sentence boundaries, a fact can span two chunks: the question is set up at the end of one and answered at the start of the next. Overlap repeats the tail of each chunk at the head of the next:\n\n```python\ndef with_overlap(chunks, overlap_units=1):\n    out = []\n    for i, c in enumerate(chunks):\n        if i > 0:\n            tail = split_sentences(chunks[i - 1])[-overlap_units:]\n            c = \" \".join(tail) + \" \" + c\n        out.append(c)\n    return out\n```\n\nOne or two sentences of overlap is typical. It costs a little storage and duplicates a little text in results, and it is the difference between finding boundary facts and not.\n\n## Keep the origin\n\nA chunk that cannot say where it came from is a search result nobody can check:\n\n```python\nfrom dataclasses import dataclass\n\n@dataclass(frozen=True)\nclass Chunk:\n    doc_id: str\n    index: int\n    text: str\n    start_char: int\n```\n\nWith `doc_id` and `start_char`, a result can link back to the page and highlight the passage. Store these alongside the vector; module 8's `argsort` gives you an index, and this is what the index points to.\n\n## Unicode edges\n\nSlicing a string by characters can split a grapheme (a Devanagari consonant from its vowel sign, an emoji from its skin-tone modifier) producing a chunk that ends in a broken character and renders as a box. Splitting on whitespace and punctuation, as above, avoids it. If you must cut by count, `regex` (the third-party module, free) offers `\\X` to step by grapheme, and module 5's normalisation should happen before any of this.\n\n## Choosing the size\n\nThere is no right number. Smaller chunks (100 to 300 tokens) embed precisely and return exact passages, but lose context; a chunk saying \"it rose 12 per cent\" without the sentence saying what \"it\" is. Larger chunks (500 to 1,000) carry context and cost more per result. Whatever size you choose, a chunk plus its overlap must also fit the embedding model's own input limit, counted by that model's tokeniser: `all-MiniLM-L6-v2`, used in the next lesson, ignores everything after 256 word pieces, and `paraphrase-multilingual-MiniLM-L12-v2` after 128, without any warning. Most people start around 300 to 500 with a sentence of overlap and adjust after looking at what the search returns for twenty real questions, which is the evaluation habit from that course applied here. Measure with your own questions before believing anyone's default, including this one.\n\n## Try this now\n\nChunk a long document (a README, a chapter, a policy) with `max_tokens=300` and print each chunk's first and last sentence. Find a fact near a boundary and confirm it appears whole in at least one chunk. Then do the same with the naive character splitter and find the fact it cut."},"check":{"question":"A chunker splits a 40,000-character report into pieces of exactly 1,000 characters. A question about a figure that sits at character 12,990 is answered wrongly, though the figure is in the text. What is the most likely cause?","options":[{"text":"The fixed cut fell inside the sentence holding the figure, so half is in one chunk and half in the next, and neither carries the whole fact.","correct":true},{"text":"1,000 characters exceeds the embedding model's input limit, so every chunk was silently truncated to its first few hundred characters and the later parts of each chunk were never embedded at all.","correct":false,"tempting":"Most embedding models accept several thousand characters; and truncation would lose the ends of all chunks, not specifically a fact near a boundary."},{"text":"Character counts include whitespace, so chunks near the end of the document contain less text than earlier ones and are weaker matches.","correct":false,"tempting":"Whitespace is distributed evenly enough that this does not produce systematically weak late chunks."},{"text":"The embedding model normalises each chunk to a fixed length before encoding, discarding numbers first.","correct":false,"tempting":"Embedding models do not discard numbers; the failure is in what the chunk contains, not in how it is encoded."}]},"prev":{"slug":"a-sqlite-response-cache","title":"A response cache on sqlite: the re-run that costs nothing","url":"https://addaly.com/learn/python-for-ai/a-sqlite-response-cache"},"next":{"slug":"search-over-your-own-notes","title":"Search over your own notes: chunks, embeddings, cosine, and sixty lines","url":"https://addaly.com/learn/python-for-ai/search-over-your-own-notes"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Source and assessment review continues. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Chunking: splitting a document that does not fit, without cutting a sentence in half', in Python, from Zero, for AI. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/python-for-ai/chunking-text (Accessed: 21 November 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Chunking: splitting a document that does not fit, without cutting a sentence in half","courseTitle":"Python, from Zero, for AI","module":{"index":9,"slug":"building-the-thing","title":"Building the thing: a small AI program, end to end"},"answer":null,"keyIdea":"Split on paragraph boundaries first, then sentences, then characters as a last resort; measure chunks in tokens not characters, overlap adjacent chunks so a fact on the boundary survives, and keep each chunk’s origin so a search result can point back to the page.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"why-chunk-at-all","runs":[{"text":"Why chunk at all"}]},{"t":"p","runs":[{"text":"A model has a context window, and a document is often larger than it. Even when the document fits, sending forty pages to answer a question about one paragraph costs forty pages of tokens per question. And an embedding (module 8’s vector) represents one "},{"text":"piece","i":true},{"text":" of text; a vector for an entire book averages away everything specific. So documents get split into chunks: small enough to embed meaningfully and to send cheaply, large enough to carry a complete thought."}]},{"t":"p","runs":[{"text":"The naive version, and its failure:"}]},{"t":"code","lang":"python","label":"Python","text":"chunks = [text[i:i + 1000] for i in range(0, len(text), 1000)]"},{"t":"p","runs":[{"text":"This cuts at character 1,000 wherever that falls: mid-word, mid-sentence, mid-number. A fact at the boundary is split between two chunks, and neither chunk contains it. The search finds nothing, or finds half, and the answer is confidently wrong. Almost every complaint that “the search does not find things I know are there” traces to this."}]},{"t":"h2","id":"split-on-structure-first","runs":[{"text":"Split on structure first"}]},{"t":"p","runs":[{"text":"Text has boundaries that mean something: paragraphs, then sentences, then words. Split at the largest boundary that keeps chunks under the limit, and only fall back to smaller ones when a single unit is too big:"}]},{"t":"code","lang":"python","label":"Python","text":"import re\n\ndef split_paragraphs(text):\n    return [p.strip() for p in re.split(r\"\\n\\s*\\n\", text) if p.strip()]\n\ndef split_sentences(text):\n    return [s.strip() for s in re.split(r\"(?<=[.!?।])\\s+\", text) if s.strip()]"},{"t":"p","runs":[{"text":"The sentence pattern splits after "},{"text":".","code":true},{"text":", "},{"text":"!","code":true},{"text":", "},{"text":"?","code":true},{"text":" and the Devanagari full stop "},{"text":"।","code":true},{"text":", followed by whitespace. It is deliberately simple. It will split “Dr. Sharma” in the wrong place and it does not know that “3.5” is a number. For most documents that is acceptable; when it is not, the free "},{"text":"nltk","code":true},{"text":" sentence tokeniser or "},{"text":"spacy","code":true},{"text":" handle abbreviations and more languages, at the cost of a download."}]},{"t":"h2","id":"assemble-to-a-token-budget","runs":[{"text":"Assemble to a token budget"}]},{"t":"code","lang":"python","label":"Python","text":"def chunk(text, max_tokens, count):\n    chunks, current = [], []\n    for para in split_paragraphs(text):\n        units = [para] if count(para) <= max_tokens else split_sentences(para)\n        for unit in units:\n            if count(\" \".join(current + [unit])) > max_tokens and current:\n                chunks.append(\" \".join(current))\n                current = []\n            current.append(unit)\n    if current:\n        chunks.append(\" \".join(current))\n    return chunks"},{"t":"p","runs":[{"text":"Greedy: add units until the next would overflow, then start a new chunk. "},{"text":"count","code":true},{"text":" is a token counter from module 6, because a chunk limit in characters is off by a factor of two or three for Hindi or code, and the model’s limit is in tokens. A paragraph that fits goes in whole; one that does not is split into sentences first. A sentence larger than the budget on its own (a pasted log, a long table row) still passes through oversized; cut it by characters as a last resort, and log that you did."}]},{"t":"h2","id":"overlap","runs":[{"text":"Overlap"}]},{"t":"p","runs":[{"text":"Even with sentence boundaries, a fact can span two chunks: the question is set up at the end of one and answered at the start of the next. Overlap repeats the tail of each chunk at the head of the next:"}]},{"t":"code","lang":"python","label":"Python","text":"def with_overlap(chunks, overlap_units=1):\n    out = []\n    for i, c in enumerate(chunks):\n        if i > 0:\n            tail = split_sentences(chunks[i - 1])[-overlap_units:]\n            c = \" \".join(tail) + \" \" + c\n        out.append(c)\n    return out"},{"t":"p","runs":[{"text":"One or two sentences of overlap is typical. It costs a little storage and duplicates a little text in results, and it is the difference between finding boundary facts and not."}]},{"t":"h2","id":"keep-the-origin","runs":[{"text":"Keep the origin"}]},{"t":"p","runs":[{"text":"A chunk that cannot say where it came from is a search result nobody can check:"}]},{"t":"code","lang":"python","label":"Python","text":"from dataclasses import dataclass\n\n@dataclass(frozen=True)\nclass Chunk:\n    doc_id: str\n    index: int\n    text: str\n    start_char: int"},{"t":"p","runs":[{"text":"With "},{"text":"doc_id","code":true},{"text":" and "},{"text":"start_char","code":true},{"text":", a result can link back to the page and highlight the passage. Store these alongside the vector; module 8’s "},{"text":"argsort","code":true},{"text":" gives you an index, and this is what the index points to."}]},{"t":"h2","id":"unicode-edges","runs":[{"text":"Unicode edges"}]},{"t":"p","runs":[{"text":"Slicing a string by characters can split a grapheme (a Devanagari consonant from its vowel sign, an emoji from its skin-tone modifier) producing a chunk that ends in a broken character and renders as a box. Splitting on whitespace and punctuation, as above, avoids it. If you must cut by count, "},{"text":"regex","code":true},{"text":" (the third-party module, free) offers "},{"text":"\\X","code":true},{"text":" to step by grapheme, and module 5’s normalisation should happen before any of this."}]},{"t":"h2","id":"choosing-the-size","runs":[{"text":"Choosing the size"}]},{"t":"p","runs":[{"text":"There is no right number. Smaller chunks (100 to 300 tokens) embed precisely and return exact passages, but lose context; a chunk saying “it rose 12 per cent” without the sentence saying what “it” is. Larger chunks (500 to 1,000) carry context and cost more per result. Whatever size you choose, a chunk plus its overlap must also fit the embedding model’s own input limit, counted by that model’s tokeniser: "},{"text":"all-MiniLM-L6-v2","code":true},{"text":", used in the next lesson, ignores everything after 256 word pieces, and "},{"text":"paraphrase-multilingual-MiniLM-L12-v2","code":true},{"text":" after 128, without any warning. Most people start around 300 to 500 with a sentence of overlap and adjust after looking at what the search returns for twenty real questions, which is the evaluation habit from that course applied here. Measure with your own questions before believing anyone’s default, including this one."}]},{"t":"h2","id":"try-this-now","runs":[{"text":"Try this now"}],"role":"try"},{"t":"p","runs":[{"text":"Chunk a long document (a README, a chapter, a policy) with "},{"text":"max_tokens=300","code":true},{"text":" and print each chunk’s first and last sentence. Find a fact near a boundary and confirm it appears whole in at least one chunk. Then do the same with the naive character splitter and find the fact it cut."}]}],"check":{"question":[{"text":"A chunker splits a 40,000-character report into pieces of exactly 1,000 characters. A question about a figure that sits at character 12,990 is answered wrongly, though the figure is in the text. What is the most likely cause?"}],"options":[{"text":[{"text":"The fixed cut fell inside the sentence holding the figure, so half is in one chunk and half in the next, and neither carries the whole fact."}],"correct":true,"tempting":null},{"text":[{"text":"1,000 characters exceeds the embedding model’s input limit, so every chunk was silently truncated to its first few hundred characters and the later parts of each chunk were never embedded at all."}],"correct":false,"tempting":[{"text":"Most embedding models accept several thousand characters; and truncation would lose the ends of all chunks, not specifically a fact near a boundary."}]},{"text":[{"text":"Character counts include whitespace, so chunks near the end of the document contain less text than earlier ones and are weaker matches."}],"correct":false,"tempting":[{"text":"Whitespace is distributed evenly enough that this does not produce systematically weak late chunks."}]},{"text":[{"text":"The embedding model normalises each chunk to a fixed length before encoding, discarding numbers first."}],"correct":false,"tempting":[{"text":"Embedding models do not discard numbers; the failure is in what the chunk contains, not in how it is encoded."}]}],"order":[3,2,1,0],"why":null},"prev":{"slug":"a-sqlite-response-cache","title":"A response cache on sqlite: the re-run that costs nothing","minutes":9},"next":{"slug":"search-over-your-own-notes","title":"Search over your own notes: chunks, embeddings, cosine, and sixty lines","minutes":10}}}