Hugging Face, End to End · Chapter 06
Chunking decides what your search can find
2:08 · Narrated with a synthetic voice. Captions are available in the player.
Read the transcript
Search that understands meaning is a few lines of code and one decision that matters more than the model does. An embedding turns a text into a fixed list of numbers where near means similar, and one matrix multiply searches a hundred thousand of them in milliseconds. The decision is chunking. Embed a forty page manual as one vector and you get the average of everything in it, and the model truncated thirty nine of those pages anyway. So split on structure rather than character counts, count in the model’s own tokens because scripts differ, and carry the document and section titles into the text you embed. Here is recall against chunk size for two kinds of question. The best size differs between the two, which is the argument for retrieving small chunks and passing their parent sections. Two more things, and both are cheap. Similarity is not relevance. How do I cancel my subscription sits very close to how do I renew it, because the two are nearly identical in form and topic, and one is a wrong answer to the other. So retrieve fifty candidates cheaply, then rescore them with a cross encoder, which reads the query and the document together and judges whether one actually answers the other. Here is where the time goes: about five milliseconds to embed the query, ten to search a million vectors, and a hundred to rescore fifty candidates on an ordinary processor. That last step is most of the latency and most of the accuracy. And add plain keyword search alongside, because embeddings are poor at part numbers, error codes and surnames.