{"course":{"slug":"building-with-ai","title":"Building with AI","url":"https://addaly.com/learn/building-with-ai","level":"intermediate","track":"build"},"lesson":{"slug":"chunking-and-embeddings","title":"Chunking and embeddings in practice","url":"https://addaly.com/learn/building-with-ai/chunking-and-embeddings","position":34,"of":80,"minutes":8,"answer":"Embeddings rank what a chunk is about, not whether it answers you, so rerank before trusting the top hit.","keyIdea":"Embeddings rank what a chunk is about, not whether it answers you, so rerank before trusting the top hit."},"body":{"format":"markdown","text":"## What an embedding is\n\nAn embedding model turns a piece of text into a list of numbers: commonly 384, 768 or 1536 of them. Texts that are about similar things end up with vectors pointing in similar directions, so you can compare them with cosine similarity.\n\n```python\nimport numpy as np\n\ndef cosine(a, b):\n    return float(np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b)))\n```\n\nThat is the whole trick. Embed your documents once, store the vectors, embed the query at request time, return the nearest.\n\nBe precise about what \"similar\" means here: **topically similar**, in the sense the embedding model was trained on. Not true. Not recent. Not answering. A chunk that is about refunds scores highly against a question about refunds whether or not it contains the answer.\n\n## Chunking, with real numbers\n\nYou cannot embed a 60-page handbook as one vector: it would average into mush. You split it.\n\nStarting points that work, adjust from there:\n\n- **300 to 800 tokens** per chunk. Smaller retrieves precisely but loses context; larger drags in noise and costs tokens on every answer.\n- **10 to 15% overlap**, so a sentence straddling a boundary survives in one piece.\n- **Split on structure first.** Headings, then paragraphs, then sentences. A fixed 1,000-character split cuts through the middle of tables and code, and both are destroyed by it.\n\nThe single highest-value habit: **make every chunk self-describing.** Prepend the document title and heading path to the chunk text before embedding.\n\n```python\ntext = f\"{doc.title} > {heading_path}\\n\\n{chunk_body}\"\n```\n\nWithout it you get chunks that begin \"This does not apply to part-time staff\" with no way to know what \"this\" is, neither for the embedding model nor for the model that eventually reads it.\n\nStore metadata alongside every chunk: document id, section, updated date, language, access level. You will need it for filtering, and filtering before search is far more effective than hoping similarity sorts it out.\n\n## Three rules people learn the hard way\n\n**Index and query with the same embedding model.** Vectors from different models are not comparable. Results will not error; they will just be quietly meaningless.\n\n**Changing the embedding model means re-embedding everything.** Budget for it. One million chunks at 1536 dimensions and 4 bytes per number is about 6 GB of vectors before any index overhead, plus the cost of a full re-embed.\n\n**Queries and documents are different shapes.** A five-word question and a 600-token passage are asymmetric. Several embedding models have separate query and document modes or prefixes. If yours does, use them; it is a free accuracy gain.\n\n## Rerank, and stop tuning chunk size\n\nHere is the change that moves the number most, and it is not chunking.\n\nRetrieve broadly, then rerank narrowly. Pull the top 50 candidates from keyword search and vector search combined, then score those 50 against the question with a cross-encoder reranker, and pass the top 5 to the model.\n\n```python\ncands = dedupe(bm25(query, k=30) + vector_search(query, k=30))\nranked = reranker.score(query, [c.text for c in cands])   # one small model, ~50 pairs\ntop = [c for _, c in sorted(zip(ranked, cands), reverse=True)[:5]]\n```\n\nA reranker reads the question and the passage together, so it can tell the difference between a page about refunds and a page that states the refund window. Embeddings cannot: they compressed the passage into a vector before ever seeing your question.\n\nMost teams spend weeks on chunk sizes and get a few points. Adding a reranker usually gets more, in an afternoon. Do the reranker first, then tune chunking if you still need to."},"check":{"question":"A user asks \"what is the refund window for damaged goods?\". The top-scoring chunk is a page headed \"Refunds\" that lists common questions but states no rules. The chunk that actually contains the 14-day rule sits under a heading called \"Damaged and missing items\" and ranks eighth. Why did the wrong chunk win?","options":[{"text":"Embeddings score topical similarity to the question, and a chunk that is about refunds looks more like the question than a chunk that answers it.","correct":true},{"text":"The chunks are the wrong size; splitting the handbook more finely would have surfaced the rule.","correct":false,"tempting":"Chunk size is the usual scapegoat and the easiest knob to reach for, and resizing does shuffle rankings enough to look like progress."},{"text":"The embedding model is too weak; a newer, higher-dimensional one would rank the answer first.","correct":false,"tempting":"Leaderboard thinking is everywhere, and a better model does help a little, which hides the fact that the problem is structural."},{"text":"Vector search matches on words, and the answer chunk lost because it never uses the word \"refund\".","correct":false,"tempting":"People often picture vector search as fuzzy keyword matching, so a missing keyword feels like an obvious cause."}]},"prev":{"slug":"embeddings-what-the-numbers-mean","title":"What an embedding is, and what \"close\" actually means","url":"https://addaly.com/learn/building-with-ai/embeddings-what-the-numbers-mean"},"next":{"slug":"choosing-an-embedding-model","title":"Choosing an embedding model: dimensions, languages and the leaderboard trap","url":"https://addaly.com/learn/building-with-ai/choosing-an-embedding-model"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Chunking and embeddings in practice', in Building with AI. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/building-with-ai/chunking-and-embeddings (Accessed: 31 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Chunking and embeddings in practice","courseTitle":"Building with AI","module":{"index":4,"slug":"retrieval-and-the-knowledge-problem","title":"Retrieval and the knowledge problem"},"answer":null,"keyIdea":"Embeddings rank what a chunk is about, not whether it answers you, so rerank before trusting the top hit.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"what-an-embedding-is","runs":[{"text":"What an embedding is"}]},{"t":"p","runs":[{"text":"An embedding model turns a piece of text into a list of numbers: commonly 384, 768 or 1536 of them. Texts that are about similar things end up with vectors pointing in similar directions, so you can compare them with cosine similarity."}]},{"t":"code","lang":"python","label":"Python","text":"import numpy as np\n\ndef cosine(a, b):\n    return float(np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b)))"},{"t":"p","runs":[{"text":"That is the whole trick. Embed your documents once, store the vectors, embed the query at request time, return the nearest."}]},{"t":"p","runs":[{"text":"Be precise about what “similar” means here: "},{"text":"topically similar","b":true},{"text":", in the sense the embedding model was trained on. Not true. Not recent. Not answering. A chunk that is about refunds scores highly against a question about refunds whether or not it contains the answer."}]},{"t":"h2","id":"chunking-with-real-numbers","runs":[{"text":"Chunking, with real numbers"}]},{"t":"p","runs":[{"text":"You cannot embed a 60-page handbook as one vector: it would average into mush. You split it."}]},{"t":"p","runs":[{"text":"Starting points that work, adjust from there:"}]},{"t":"ul","items":[[{"text":"300 to 800 tokens","b":true},{"text":" per chunk. Smaller retrieves precisely but loses context; larger drags in noise and costs tokens on every answer."}],[{"text":"10 to 15% overlap","b":true},{"text":", so a sentence straddling a boundary survives in one piece."}],[{"text":"Split on structure first.","b":true},{"text":" Headings, then paragraphs, then sentences. A fixed 1,000-character split cuts through the middle of tables and code, and both are destroyed by it."}]]},{"t":"p","runs":[{"text":"The single highest-value habit: "},{"text":"make every chunk self-describing.","b":true},{"text":" Prepend the document title and heading path to the chunk text before embedding."}]},{"t":"code","lang":"python","label":"Python","text":"text = f\"{doc.title} > {heading_path}\\n\\n{chunk_body}\""},{"t":"p","runs":[{"text":"Without it you get chunks that begin “This does not apply to part-time staff” with no way to know what “this” is, neither for the embedding model nor for the model that eventually reads it."}]},{"t":"p","runs":[{"text":"Store metadata alongside every chunk: document id, section, updated date, language, access level. You will need it for filtering, and filtering before search is far more effective than hoping similarity sorts it out."}]},{"t":"h2","id":"three-rules-people-learn-the-hard-way","runs":[{"text":"Three rules people learn the hard way"}]},{"t":"p","runs":[{"text":"Index and query with the same embedding model.","b":true},{"text":" Vectors from different models are not comparable. Results will not error; they will just be quietly meaningless."}]},{"t":"p","runs":[{"text":"Changing the embedding model means re-embedding everything.","b":true},{"text":" Budget for it. One million chunks at 1536 dimensions and 4 bytes per number is about 6 GB of vectors before any index overhead, plus the cost of a full re-embed."}]},{"t":"p","runs":[{"text":"Queries and documents are different shapes.","b":true},{"text":" A five-word question and a 600-token passage are asymmetric. Several embedding models have separate query and document modes or prefixes. If yours does, use them; it is a free accuracy gain."}]},{"t":"h2","id":"rerank-and-stop-tuning-chunk-size","runs":[{"text":"Rerank, and stop tuning chunk size"}]},{"t":"p","runs":[{"text":"Here is the change that moves the number most, and it is not chunking."}]},{"t":"p","runs":[{"text":"Retrieve broadly, then rerank narrowly. Pull the top 50 candidates from keyword search and vector search combined, then score those 50 against the question with a cross-encoder reranker, and pass the top 5 to the model."}]},{"t":"code","lang":"python","label":"Python","text":"cands = dedupe(bm25(query, k=30) + vector_search(query, k=30))\nranked = reranker.score(query, [c.text for c in cands])   # one small model, ~50 pairs\ntop = [c for _, c in sorted(zip(ranked, cands), reverse=True)[:5]]"},{"t":"p","runs":[{"text":"A reranker reads the question and the passage together, so it can tell the difference between a page about refunds and a page that states the refund window. Embeddings cannot: they compressed the passage into a vector before ever seeing your question."}]},{"t":"p","runs":[{"text":"Most teams spend weeks on chunk sizes and get a few points. Adding a reranker usually gets more, in an afternoon. Do the reranker first, then tune chunking if you still need to."}]}],"check":{"question":[{"text":"A user asks “what is the refund window for damaged goods?”. The top-scoring chunk is a page headed “Refunds” that lists common questions but states no rules. The chunk that actually contains the 14-day rule sits under a heading called “Damaged and missing items” and ranks eighth. Why did the wrong chunk win?"}],"options":[{"text":[{"text":"Embeddings score topical similarity to the question, and a chunk that is about refunds looks more like the question than a chunk that answers it."}],"correct":true,"tempting":null},{"text":[{"text":"The chunks are the wrong size; splitting the handbook more finely would have surfaced the rule."}],"correct":false,"tempting":[{"text":"Chunk size is the usual scapegoat and the easiest knob to reach for, and resizing does shuffle rankings enough to look like progress."}]},{"text":[{"text":"The embedding model is too weak; a newer, higher-dimensional one would rank the answer first."}],"correct":false,"tempting":[{"text":"Leaderboard thinking is everywhere, and a better model does help a little, which hides the fact that the problem is structural."}]},{"text":[{"text":"Vector search matches on words, and the answer chunk lost because it never uses the word “refund”."}],"correct":false,"tempting":[{"text":"People often picture vector search as fuzzy keyword matching, so a missing keyword feels like an obvious cause."}]}],"order":[1,3,0,2],"why":null},"prev":{"slug":"embeddings-what-the-numbers-mean","title":"What an embedding is, and what “close” actually means","minutes":10},"next":{"slug":"choosing-an-embedding-model","title":"Choosing an embedding model: dimensions, languages and the leaderboard trap","minutes":9}}}