{"course":{"slug":"machine-learning-foundations","title":"Machine Learning, Foundations","url":"https://addaly.com/learn/machine-learning-foundations","level":"intermediate","track":"start"},"lesson":{"slug":"embeddings-in-practice","title":"Building a working semantic search with no GPU","url":"https://addaly.com/learn/machine-learning-foundations/embeddings-in-practice","position":69,"of":92,"minutes":9,"answer":"Free CPU-sized embedding models plus an approximate nearest-neighbour index give working semantic search, but chunking affects quality more than model choice and pure embedding search fails on identifiers and negation.","keyIdea":"Free CPU-sized embedding models plus an approximate nearest-neighbour index give working semantic search, but chunking affects quality more than model choice and pure embedding search fails on identifiers and negation, which is why production systems combine it with keyword search."},"body":{"format":"markdown","text":"## Getting embeddings for free\n\nYou do not train these yourself. Pretrained models are free, small enough to run on a CPU, and good enough for most applications.\n\n```python\nfrom sentence_transformers import SentenceTransformer\nm = SentenceTransformer(\"all-MiniLM-L6-v2\")\nvecs = m.encode(list_of_texts, normalize_embeddings=True)\nvecs.shape        # (n, 384)\n```\n\n`all-MiniLM-L6-v2` is about 80 MB, produces 384-dimensional vectors, and encodes a few hundred short texts per second on an ordinary laptop CPU. It is the sensible default.\n\nWhen you need better quality, `all-mpnet-base-v2` is larger and slower. For non-English or mixed-language text, the multilingual models (`paraphrase-multilingual-MiniLM-L12-v2` and its relatives) cover 50 or more languages in one shared space, which means a Hindi query can retrieve an English document. For Indian languages specifically, `MuRIL` from Google Research is free on Hugging Face and trained on 17 Indian languages including transliterated text.\n\nAll of these run locally. No API key, no per-call cost, no data leaving the machine, which matters when the documents are confidential.\n\n## Searching a million vectors\n\nComparing a query against a million vectors by brute force is a million dot products of 384 numbers each. In NumPy that is one matrix multiplication and takes a few hundred milliseconds, fine for many applications, and you should measure before assuming otherwise.\n\n```python\nsims = corpus_vecs @ query_vec        # both normalised\ntop = np.argpartition(-sims, 10)[:10]\n```\n\nAbove a few million, use an approximate index. These trade a small amount of recall for very large speedups.\n\n- **FAISS**: free, from Meta, the standard. Runs on CPU. `IndexFlatIP` is exact; `IndexHNSWFlat` is the usual approximate choice.\n- **hnswlib**: a small, fast standalone HNSW implementation.\n- **Annoy**: from Spotify, memory-mapped so an index larger than RAM can be searched.\n\nHNSW builds a layered graph of neighbours and walks it from a coarse layer down to a fine one. It typically achieves 95 to 99% recall of the true nearest neighbours at a hundred times the speed of brute force. That recall figure is a parameter you set, and it is worth measuring on your own data rather than accepting the default, because \"the tenth result is sometimes wrong\" matters more for some products than others.\n\n## Chunking, which decides more than the model\n\nFor documents longer than the model's input limit (often 256 or 512 tokens) you must split before embedding. This choice affects retrieval quality more than the choice of model, and it is usually made carelessly.\n\n- **Too large**, and one vector averages several topics, so it matches everything weakly and nothing strongly.\n- **Too small**, and the chunk lacks the context needed to be interpretable. A sentence beginning \"This is not recommended\" is useless without its antecedent.\n- **Overlap** of 10 to 20% keeps sentences that straddle a boundary retrievable from both sides.\n- **Split on structure where you can** (paragraphs, sections, headings) rather than on a fixed character count. A chunk that respects the document's own boundaries is more coherent than one that ends mid-sentence.\n\nA reasonable starting point for prose is 300 to 500 tokens with 50 tokens of overlap, split at paragraph boundaries, and then measured.\n\n## Measuring retrieval\n\nDo not deploy on the impression that results look reasonable. Build a small evaluation set: 50 real queries with the document that should be returned for each, written by hand. Then compute recall at 5 and mean reciprocal rank.\n\nFifty queries takes an afternoon and turns every subsequent decision (model, chunk size, index type) into a measurement rather than an argument. This is the same discipline as the rest of this course applied to a system that superficially seems not to need it.\n\n## Where pure embedding search fails\n\nThree failure modes worth knowing before you commit.\n\n**Exact identifiers.** Searching for an order number, a part code or an error string. Embeddings represent meaning, and a part number has no meaning to represent. Keyword search wins outright.\n\n**Negation and specificity.** \"Restaurants without parking\" embeds close to \"restaurants with parking\". The vectors capture topic much better than logical structure.\n\n**Rare exact terms.** A drug name appearing three times in the corpus may be poorly represented, where an inverted index finds it perfectly.\n\nThe standard answer is **hybrid search**: run keyword search (BM25) and embedding search in parallel and combine the rankings, usually with reciprocal rank fusion, which needs no score calibration between the two systems. Hybrid reliably beats either alone, it is not much more code, and it is what production systems do.\n\n> Figure: Why production search runs both. Embedding search wins: Paraphrase: 'refund not received' finds 'the money has not come back'; A Hindi query retrieving an English document, in one multilingual space; Questions phrased nothing like the passage that answers them; The long tail of wordings for one meaning. Keyword search (BM25) wins: Exact identifiers: an order number, a part code, an error string; Negation: 'restaurants without parking' embeds beside 'with parking'; Rare exact terms: a drug name that appears three times in the corpus; Anything a user expects to match letter for letter. Hybrid search runs BM25 and embedding search in parallel and fuses the two rankings with reciprocal rank fusion, which needs no score calibration between the systems. It reliably beats either alone and is not much more code."},"check":{"question":"A support search built on embeddings fails to find the article for the query 'error code E4302' even though the article contains that exact string. What is the mechanism, and what fixes it?","options":[{"text":"The embedding model represents meaning, and an arbitrary code carries none, so the query vector lands nowhere near the article's; adding keyword search and fusing the rankings recovers it.","correct":true},{"text":"The article chunk containing the code was too long, so the code was averaged away; smaller chunks would fix it.","correct":false,"tempting":"Smaller chunks do sharpen matching, but a code with no semantic content remains poorly represented at any chunk size."},{"text":"The vectors were not normalised, so cosine similarity was computed incorrectly for short queries.","correct":false,"tempting":"Missing normalisation distorts scores generally rather than failing specifically on identifier queries."},{"text":"The index is approximate and dropped the correct neighbour; switching to exact search would find it.","correct":false,"tempting":"Approximate indexes do miss occasionally, but a systematic failure on identifiers points at the representation rather than the index's recall."}]},"prev":{"slug":"what-an-embedding-is","title":"What an embedding actually is","url":"https://addaly.com/learn/machine-learning-foundations/what-an-embedding-is"},"next":{"slug":"recommendation-by-factorisation","title":"Recommendation: learning a vector per user and per item","url":"https://addaly.com/learn/machine-learning-foundations/recommendation-by-factorisation"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Building a working semantic search with no GPU', in Machine Learning, Foundations. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/machine-learning-foundations/embeddings-in-practice (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Building a working semantic search with no GPU","courseTitle":"Machine Learning, Foundations","module":{"index":7,"slug":"learning-without-labels","title":"Learning without labels, and learning a representation"},"answer":null,"keyIdea":"Free CPU-sized embedding models plus an approximate nearest-neighbour index give working semantic search, but chunking affects quality more than model choice and pure embedding search fails on identifiers and negation, which is why production systems combine it with keyword search.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"left":{"title":"Embedding search wins","items":["Paraphrase: ‘refund not received’ finds ‘the money has not come back’","A Hindi query retrieving an English document, in one multilingual space","Questions phrased nothing like the passage that answers them","The long tail of wordings for one meaning"]},"type":"compare","right":{"title":"Keyword search (BM25) wins","items":["Exact identifiers: an order number, a part code, an error string","Negation: ‘restaurants without parking’ embeds beside ‘with parking’","Rare exact terms: a drug name that appears three times in the corpus","Anything a user expects to match letter for letter"]},"title":"Why production search runs both","caption":"Hybrid search runs BM25 and embedding search in parallel and fuses the two rankings with reciprocal rank fusion, which needs no score calibration between the systems. It reliably beats either alone and is not much more code."},"alt":"Figure: Why production search runs both. Embedding search wins: Paraphrase: ‘refund not received’ finds ‘the money has not come back’; A Hindi query retrieving an English document, in one multilingual space; Questions phrased nothing like the passage that answers them; The long tail of wordings for one meaning. Keyword search (BM25) wins: Exact identifiers: an order number, a part code, an error string; Negation: ‘restaurants without parking’ embeds beside ‘with parking’; Rare exact terms: a drug name that appears three times in the corpus; Anything a user expects to match letter for letter. Hybrid search runs BM25 and embedding search in parallel and fuses the two rankings with reciprocal rank fusion, which needs no score calibration between the systems. It reliably beats either alone and is not much more code."}},"blocks":[{"t":"h2","id":"getting-embeddings-for-free","runs":[{"text":"Getting embeddings for free"}]},{"t":"p","runs":[{"text":"You do not train these yourself. Pretrained models are free, small enough to run on a CPU, and good enough for most applications."}]},{"t":"code","lang":"python","label":"Python","text":"from sentence_transformers import SentenceTransformer\nm = SentenceTransformer(\"all-MiniLM-L6-v2\")\nvecs = m.encode(list_of_texts, normalize_embeddings=True)\nvecs.shape        # (n, 384)"},{"t":"p","runs":[{"text":"all-MiniLM-L6-v2","code":true},{"text":" is about 80 MB, produces 384-dimensional vectors, and encodes a few hundred short texts per second on an ordinary laptop CPU. It is the sensible default."}]},{"t":"p","runs":[{"text":"When you need better quality, "},{"text":"all-mpnet-base-v2","code":true},{"text":" is larger and slower. For non-English or mixed-language text, the multilingual models ("},{"text":"paraphrase-multilingual-MiniLM-L12-v2","code":true},{"text":" and its relatives) cover 50 or more languages in one shared space, which means a Hindi query can retrieve an English document. For Indian languages specifically, "},{"text":"MuRIL","code":true},{"text":" from Google Research is free on Hugging Face and trained on 17 Indian languages including transliterated text."}]},{"t":"p","runs":[{"text":"All of these run locally. No API key, no per-call cost, no data leaving the machine, which matters when the documents are confidential."}]},{"t":"h2","id":"searching-a-million-vectors","runs":[{"text":"Searching a million vectors"}]},{"t":"p","runs":[{"text":"Comparing a query against a million vectors by brute force is a million dot products of 384 numbers each. In NumPy that is one matrix multiplication and takes a few hundred milliseconds, fine for many applications, and you should measure before assuming otherwise."}]},{"t":"code","lang":"python","label":"Python","text":"sims = corpus_vecs @ query_vec        # both normalised\ntop = np.argpartition(-sims, 10)[:10]"},{"t":"p","runs":[{"text":"Above a few million, use an approximate index. These trade a small amount of recall for very large speedups."}]},{"t":"ul","items":[[{"text":"FAISS","b":true},{"text":": free, from Meta, the standard. Runs on CPU. "},{"text":"IndexFlatIP","code":true},{"text":" is exact; "},{"text":"IndexHNSWFlat","code":true},{"text":" is the usual approximate choice."}],[{"text":"hnswlib","b":true},{"text":": a small, fast standalone HNSW implementation."}],[{"text":"Annoy","b":true},{"text":": from Spotify, memory-mapped so an index larger than RAM can be searched."}]]},{"t":"p","runs":[{"text":"HNSW builds a layered graph of neighbours and walks it from a coarse layer down to a fine one. It typically achieves 95 to 99% recall of the true nearest neighbours at a hundred times the speed of brute force. That recall figure is a parameter you set, and it is worth measuring on your own data rather than accepting the default, because “the tenth result is sometimes wrong” matters more for some products than others."}]},{"t":"h2","id":"chunking-which-decides-more-than-the-model","runs":[{"text":"Chunking, which decides more than the model"}]},{"t":"p","runs":[{"text":"For documents longer than the model’s input limit (often 256 or 512 tokens) you must split before embedding. This choice affects retrieval quality more than the choice of model, and it is usually made carelessly."}]},{"t":"ul","items":[[{"text":"Too large","b":true},{"text":", and one vector averages several topics, so it matches everything weakly and nothing strongly."}],[{"text":"Too small","b":true},{"text":", and the chunk lacks the context needed to be interpretable. A sentence beginning “This is not recommended” is useless without its antecedent."}],[{"text":"Overlap","b":true},{"text":" of 10 to 20% keeps sentences that straddle a boundary retrievable from both sides."}],[{"text":"Split on structure where you can","b":true},{"text":" (paragraphs, sections, headings) rather than on a fixed character count. A chunk that respects the document’s own boundaries is more coherent than one that ends mid-sentence."}]]},{"t":"p","runs":[{"text":"A reasonable starting point for prose is 300 to 500 tokens with 50 tokens of overlap, split at paragraph boundaries, and then measured."}]},{"t":"h2","id":"measuring-retrieval","runs":[{"text":"Measuring retrieval"}]},{"t":"p","runs":[{"text":"Do not deploy on the impression that results look reasonable. Build a small evaluation set: 50 real queries with the document that should be returned for each, written by hand. Then compute recall at 5 and mean reciprocal rank."}]},{"t":"p","runs":[{"text":"Fifty queries takes an afternoon and turns every subsequent decision (model, chunk size, index type) into a measurement rather than an argument. This is the same discipline as the rest of this course applied to a system that superficially seems not to need it."}]},{"t":"h2","id":"where-pure-embedding-search-fails","runs":[{"text":"Where pure embedding search fails"}]},{"t":"p","runs":[{"text":"Three failure modes worth knowing before you commit."}]},{"t":"p","runs":[{"text":"Exact identifiers.","b":true},{"text":" Searching for an order number, a part code or an error string. Embeddings represent meaning, and a part number has no meaning to represent. Keyword search wins outright."}]},{"t":"p","runs":[{"text":"Negation and specificity.","b":true},{"text":" “Restaurants without parking” embeds close to “restaurants with parking”. The vectors capture topic much better than logical structure."}]},{"t":"p","runs":[{"text":"Rare exact terms.","b":true},{"text":" A drug name appearing three times in the corpus may be poorly represented, where an inverted index finds it perfectly."}]},{"t":"p","runs":[{"text":"The standard answer is "},{"text":"hybrid search","b":true},{"text":": run keyword search (BM25) and embedding search in parallel and combine the rankings, usually with reciprocal rank fusion, which needs no score calibration between the two systems. Hybrid reliably beats either alone, it is not much more code, and it is what production systems do."}]}],"check":{"question":[{"text":"A support search built on embeddings fails to find the article for the query ‘error code E4302’ even though the article contains that exact string. What is the mechanism, and what fixes it?"}],"options":[{"text":[{"text":"The embedding model represents meaning, and an arbitrary code carries none, so the query vector lands nowhere near the article’s; adding keyword search and fusing the rankings recovers it."}],"correct":true,"tempting":null},{"text":[{"text":"The article chunk containing the code was too long, so the code was averaged away; smaller chunks would fix it."}],"correct":false,"tempting":[{"text":"Smaller chunks do sharpen matching, but a code with no semantic content remains poorly represented at any chunk size."}]},{"text":[{"text":"The vectors were not normalised, so cosine similarity was computed incorrectly for short queries."}],"correct":false,"tempting":[{"text":"Missing normalisation distorts scores generally rather than failing specifically on identifier queries."}]},{"text":[{"text":"The index is approximate and dropped the correct neighbour; switching to exact search would find it."}],"correct":false,"tempting":[{"text":"Approximate indexes do miss occasionally, but a systematic failure on identifiers points at the representation rather than the index’s recall."}]}],"order":[2,3,1,0],"why":null},"prev":{"slug":"what-an-embedding-is","title":"What an embedding actually is","minutes":10},"next":{"slug":"recommendation-by-factorisation","title":"Recommendation: learning a vector per user and per item","minutes":9}}}