Chunking and embeddings in practice

What an embedding is

An embedding model turns a piece of text into a list of numbers — commonly 384, 768 or 1536 of them. Texts that are about similar things end up with vectors pointing in similar directions, so you can compare them with cosine similarity.

import numpy as np

def cosine(a, b):
    return float(np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b)))

That is the whole trick. Embed your documents once, store the vectors, embed the query at request time, return the nearest.

Be precise about what "similar" means here: topically similar, in the sense the embedding model was trained on. Not true. Not recent. Not answering. A chunk that is about refunds scores highly against a question about refunds whether or not it contains the answer.

Chunking, with real numbers

You cannot embed a 60-page handbook as one vector — it would average into mush. You split it.

Starting points that work, adjust from there:

• 300 to 800 tokens per chunk. Smaller retrieves precisely but loses context; larger drags in noise and costs tokens on every answer.
• 10 to 15% overlap, so a sentence straddling a boundary survives in one piece.
• Split on structure first. Headings, then paragraphs, then sentences. A fixed 1,000-character split cuts through the middle of tables and code, and both are destroyed by it.

The single highest-value habit: make every chunk self-describing. Prepend the document title and heading path to the chunk text before embedding.

text = f"{doc.title} > {heading_path}\n\n{chunk_body}"

Without it you get chunks that begin "This does not apply to part-time staff" with no way to know what "this" is — neither for the embedding model nor for the model that eventually reads it.

Store metadata alongside every chunk: document id, section, updated date, language, access level. You will need it for filtering, and filtering before search is far more effective than hoping similarity sorts it out.

Three rules people learn the hard way

Index and query with the same embedding model. Vectors from different models are not comparable. Results will not error; they will just be quietly meaningless.

Changing the embedding model means re-embedding everything. Budget for it. One million chunks at 1536 dimensions and 4 bytes per number is about 6 GB of vectors before any index overhead, plus the cost of a full re-embed.

Queries and documents are different shapes. A five-word question and a 600-token passage are asymmetric. Several embedding models have separate query and document modes or prefixes. If yours does, use them; it is a free accuracy gain.

Rerank, and stop tuning chunk size

Here is the change that moves the number most, and it is not chunking.

Retrieve broadly, then rerank narrowly. Pull the top 50 candidates from keyword search and vector search combined, then score those 50 against the question with a cross-encoder reranker, and pass the top 5 to the model.

cands = dedupe(bm25(query, k=30) + vector_search(query, k=30))
ranked = reranker.score(query, [c.text for c in cands])   # one small model, ~50 pairs
top = [c for _, c in sorted(zip(ranked, cands), reverse=True)[:5]]

A reranker reads the question and the passage together, so it can tell the difference between a page about refunds and a page that states the refund window. Embeddings cannot: they compressed the passage into a vector before ever seeing your question.

Most teams spend weeks on chunk sizes and get a few points. Adding a reranker usually gets more, in an afternoon. Do the reranker first, then tune chunking if you still need to.