{"course":{"slug":"running-models-yourself","title":"Running Models Yourself","url":"https://addaly.com/learn/running-models-yourself","level":"intermediate","track":"deep"},"lesson":{"slug":"embeddings-and-rerankers-locally","title":"Embeddings and rerankers on your own machine","url":"https://addaly.com/learn/running-models-yourself/embeddings-and-rerankers-locally","position":36,"of":82,"minutes":10,"answer":"Embedding models are small enough to run on any CPU and are the foundation of local retrieval, but similarity is not relevance: pair them with a cross-encoder reranker and exact matching.","keyIdea":"Embedding models are small enough to run on any CPU and are the foundation of local retrieval, but similarity is not relevance: pair them with a cross-encoder reranker and exact matching, and expect negation to be invisible."},"body":{"format":"markdown","text":"## A different kind of model, and a much smaller one\n\nAn embedding model turns a piece of text into a fixed-length vector of numbers (commonly 384, 768 or 1,024 of them) such that texts with similar meaning land near each other. Nothing is generated. There is one forward pass and the output is the vector.\n\nThat makes them cheap in a way generative models are not. A 100M-parameter embedding model is 200 MB in fp16, runs on any CPU, and embeds thousands of short texts per second. If you have been told local AI needs a GPU, this is the large exception, and it is the foundation of search, retrieval, clustering, deduplication and recommendation.\n\n## Running one\n\n```bash\n# Ollama\nollama pull nomic-embed-text\ncurl http://localhost:11434/api/embed \\\n  -d '{\"model\":\"nomic-embed-text\",\"input\":\"the cat sat on the mat\"}'\n\n# llama.cpp\nllama-server -m bge-m3-Q8_0.gguf --embedding --pooling mean -c 8192\n```\n\nOr in Python, which is where most of this work happens:\n\n```python\nfrom sentence_transformers import SentenceTransformer\nm = SentenceTransformer(\"BAAI/bge-m3\")\nvecs = m.encode([\"the cat sat\", \"a feline was seated\"],\n                normalize_embeddings=True)\nprint(vecs @ vecs.T)   # cosine similarity, since they are normalised\n```\n\nNormalise, then a dot product is cosine similarity. Two sentences meaning the same thing typically score above 0.8; unrelated ones sit near 0.1 to 0.3. Those numbers are model-specific, which is the first thing people get wrong: a similarity threshold tuned for one model is meaningless for another.\n\n## Four details that decide whether it works\n\n**Query and document may need different prefixes.** Several strong models were trained asymmetrically: documents embedded plainly, queries prefixed with an instruction such as \"Represent this sentence for searching relevant passages:\". Omitting the prefix degrades retrieval substantially and produces no error. The model card states it; almost nobody reads that part.\n\n**Sequence length is short.** Many embedding models take 512 tokens and truncate the rest silently. A 2,000-word document embedded by such a model is an embedding of its first few paragraphs. Chunk deliberately rather than discovering this later.\n\n**Multilingual is a property of the model, not of embeddings.** An English-trained model maps Hindi text to essentially arbitrary positions. If your corpus is not English, pick a model trained multilingually and test it on your own pairs.\n\n**Dimension is a storage and speed decision.** 1,024 dimensions at fp32 is 4 KB per vector; a million documents is 4 GB. Some recent models support Matryoshka truncation (the vector is trained so that its first 256 dimensions are a usable embedding on their own) which lets you trade a little accuracy for a quarter of the storage by cutting the vector short.\n\n## Rerankers: the step most people skip\n\nAn embedding search is fast because every document was embedded in advance and the query only has to be compared against stored vectors. The cost is that the query and the document never meet inside the model.\n\n> Figure: The two-stage retrieval most people stop halfway through. Steps: 1. Embed and search (Fifty candidates from stored vectors in milliseconds); 2. Rerank the fifty (A cross-encoder reads query and document together); 3. Match identifiers (An invoice number does not embed. Keywords find it); 4. Send five passages (The model sees only what survived all three stages). Embedding search is fast because the query and the document never meet inside the model, which is also why it is imprecise. The reranker is a 500 MB model on the CPU, and on most real corpora adding it improves answers more than upgrading the generation model does.\n\nA **cross-encoder reranker** takes the query and one document together, runs both through a model, and outputs a relevance score. Far more accurate, and far too slow to run over a whole corpus.\n\nThe standard arrangement uses both. Retrieve fifty candidates with embeddings in a few milliseconds, then rerank those fifty with a cross-encoder in a few hundred milliseconds, and pass the top five to the language model.\n\n```python\nfrom sentence_transformers import CrossEncoder\nce = CrossEncoder(\"BAAI/bge-reranker-v2-m3\")\nscores = ce.predict([(query, d) for d in candidates])\n```\n\nOn most real corpora this two-stage design improves answer quality more than upgrading the generation model does, and it costs a 500 MB model on the CPU. It is the highest-value under-used component in local retrieval.\n\n## Where embeddings fail\n\nThey are similarity machines, and similarity is not relevance.\n\n- **Negation is nearly invisible.** \"The drug is safe for children\" and \"the drug is not safe for children\" embed close together. If a wrong answer here matters, embeddings alone must not be your filter.\n- **Numbers and identifiers do not embed usefully.** Searching for invoice 88213 is a job for exact matching, not vectors. Hybrid search (keyword scoring combined with vector similarity) exists precisely for this, and it beats either alone on most real corpora.\n- **Long documents lose their specifics.** Averaging a whole page into one vector produces something generically about the topic and specifically about nothing.\n\nThe practical rule: use embeddings to find candidates, a reranker to order them, exact matching for anything with an identifier, and a language model only at the end, on a handful of passages you have good reason to trust."},"check":{"question":"A local search system embeds documents and queries with a strong retrieval model but returns poor results. Investigation shows documents were embedded plainly while queries were also embedded plainly, as the model card's example for documents showed. What is most likely wrong?","options":[{"text":"The documents exceed the model's sequence limit and are being truncated, so most of their content is unrepresented.","correct":false,"tempting":"A real and common failure, but it would not be revealed by the query-versus-document detail the investigation actually surfaced."},{"text":"Queries and documents should be embedded by separate models trained as a pair, and using one model for both collapses the space.","correct":false,"tempting":"Describes an architecture that dual-encoder retrieval does not require; one model with role-appropriate inputs is the standard design."},{"text":"The model was trained asymmetrically and expects an instruction prefix on queries, so query vectors sit in a different region from the document vectors they should match.","correct":true},{"text":"The embeddings were not normalised, so dot products reflect text length rather than similarity.","correct":false,"tempting":"Names a genuine bug with a different signature: it skews ranking towards long or short texts rather than breaking query-document alignment."}]},"prev":{"slug":"structured-output-locally","title":"Output that parses every time","url":"https://addaly.com/learn/running-models-yourself/structured-output-locally"},"next":{"slug":"speech-and-vision-locally","title":"Speech, vision and the rest of the local stack","url":"https://addaly.com/learn/running-models-yourself/speech-and-vision-locally"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Embeddings and rerankers on your own machine', in Running Models Yourself. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/running-models-yourself/embeddings-and-rerankers-locally (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Embeddings and rerankers on your own machine","courseTitle":"Running Models Yourself","module":{"index":4,"slug":"getting-it-running","title":"Getting it running"},"answer":null,"keyIdea":"Embedding models are small enough to run on any CPU and are the foundation of local retrieval, but similarity is not relevance: pair them with a cross-encoder reranker and exact matching, and expect negation to be invisible.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"type":"flow","steps":[{"label":"Embed and search","detail":"Fifty candidates from stored vectors in milliseconds"},{"label":"Rerank the fifty","detail":"A cross-encoder reads query and document together"},{"label":"Match identifiers","detail":"An invoice number does not embed. Keywords find it"},{"label":"Send five passages","detail":"The model sees only what survived all three stages"}],"title":"The two-stage retrieval most people stop halfway through","caption":"Embedding search is fast because the query and the document never meet inside the model, which is also why it is imprecise. The reranker is a 500 MB model on the CPU, and on most real corpora adding it improves answers more than upgrading the generation model does."},"alt":"Figure: The two-stage retrieval most people stop halfway through. Steps: 1. Embed and search (Fifty candidates from stored vectors in milliseconds); 2. Rerank the fifty (A cross-encoder reads query and document together); 3. Match identifiers (An invoice number does not embed. Keywords find it); 4. Send five passages (The model sees only what survived all three stages). Embedding search is fast because the query and the document never meet inside the model, which is also why it is imprecise. The reranker is a 500 MB model on the CPU, and on most real corpora adding it improves answers more than upgrading the generation model does."}},"blocks":[{"t":"h2","id":"a-different-kind-of-model-and-a-much-smaller-one","runs":[{"text":"A different kind of model, and a much smaller one"}]},{"t":"p","runs":[{"text":"An embedding model turns a piece of text into a fixed-length vector of numbers (commonly 384, 768 or 1,024 of them) such that texts with similar meaning land near each other. Nothing is generated. There is one forward pass and the output is the vector."}]},{"t":"p","runs":[{"text":"That makes them cheap in a way generative models are not. A 100M-parameter embedding model is 200 MB in fp16, runs on any CPU, and embeds thousands of short texts per second. If you have been told local AI needs a GPU, this is the large exception, and it is the foundation of search, retrieval, clustering, deduplication and recommendation."}]},{"t":"h2","id":"running-one","runs":[{"text":"Running one"}]},{"t":"code","lang":"bash","label":"Bash","text":"# Ollama\nollama pull nomic-embed-text\ncurl http://localhost:11434/api/embed \\\n  -d '{\"model\":\"nomic-embed-text\",\"input\":\"the cat sat on the mat\"}'\n\n# llama.cpp\nllama-server -m bge-m3-Q8_0.gguf --embedding --pooling mean -c 8192"},{"t":"p","runs":[{"text":"Or in Python, which is where most of this work happens:"}]},{"t":"code","lang":"python","label":"Python","text":"from sentence_transformers import SentenceTransformer\nm = SentenceTransformer(\"BAAI/bge-m3\")\nvecs = m.encode([\"the cat sat\", \"a feline was seated\"],\n                normalize_embeddings=True)\nprint(vecs @ vecs.T)   # cosine similarity, since they are normalised"},{"t":"p","runs":[{"text":"Normalise, then a dot product is cosine similarity. Two sentences meaning the same thing typically score above 0.8; unrelated ones sit near 0.1 to 0.3. Those numbers are model-specific, which is the first thing people get wrong: a similarity threshold tuned for one model is meaningless for another."}]},{"t":"h2","id":"four-details-that-decide-whether-it-works","runs":[{"text":"Four details that decide whether it works"}]},{"t":"p","runs":[{"text":"Query and document may need different prefixes.","b":true},{"text":" Several strong models were trained asymmetrically: documents embedded plainly, queries prefixed with an instruction such as “Represent this sentence for searching relevant passages:”. Omitting the prefix degrades retrieval substantially and produces no error. The model card states it; almost nobody reads that part."}]},{"t":"p","runs":[{"text":"Sequence length is short.","b":true},{"text":" Many embedding models take 512 tokens and truncate the rest silently. A 2,000-word document embedded by such a model is an embedding of its first few paragraphs. Chunk deliberately rather than discovering this later."}]},{"t":"p","runs":[{"text":"Multilingual is a property of the model, not of embeddings.","b":true},{"text":" An English-trained model maps Hindi text to essentially arbitrary positions. If your corpus is not English, pick a model trained multilingually and test it on your own pairs."}]},{"t":"p","runs":[{"text":"Dimension is a storage and speed decision.","b":true},{"text":" 1,024 dimensions at fp32 is 4 KB per vector; a million documents is 4 GB. Some recent models support Matryoshka truncation (the vector is trained so that its first 256 dimensions are a usable embedding on their own) which lets you trade a little accuracy for a quarter of the storage by cutting the vector short."}]},{"t":"h2","id":"rerankers-the-step-most-people-skip","runs":[{"text":"Rerankers: the step most people skip"}]},{"t":"p","runs":[{"text":"An embedding search is fast because every document was embedded in advance and the query only has to be compared against stored vectors. The cost is that the query and the document never meet inside the model."}]},{"t":"p","runs":[{"text":"A "},{"text":"cross-encoder reranker","b":true},{"text":" takes the query and one document together, runs both through a model, and outputs a relevance score. Far more accurate, and far too slow to run over a whole corpus."}]},{"t":"p","runs":[{"text":"The standard arrangement uses both. Retrieve fifty candidates with embeddings in a few milliseconds, then rerank those fifty with a cross-encoder in a few hundred milliseconds, and pass the top five to the language model."}]},{"t":"code","lang":"python","label":"Python","text":"from sentence_transformers import CrossEncoder\nce = CrossEncoder(\"BAAI/bge-reranker-v2-m3\")\nscores = ce.predict([(query, d) for d in candidates])"},{"t":"p","runs":[{"text":"On most real corpora this two-stage design improves answer quality more than upgrading the generation model does, and it costs a 500 MB model on the CPU. It is the highest-value under-used component in local retrieval."}]},{"t":"h2","id":"where-embeddings-fail","runs":[{"text":"Where embeddings fail"}]},{"t":"p","runs":[{"text":"They are similarity machines, and similarity is not relevance."}]},{"t":"ul","items":[[{"text":"Negation is nearly invisible.","b":true},{"text":" “The drug is safe for children” and “the drug is not safe for children” embed close together. If a wrong answer here matters, embeddings alone must not be your filter."}],[{"text":"Numbers and identifiers do not embed usefully.","b":true},{"text":" Searching for invoice 88213 is a job for exact matching, not vectors. Hybrid search (keyword scoring combined with vector similarity) exists precisely for this, and it beats either alone on most real corpora."}],[{"text":"Long documents lose their specifics.","b":true},{"text":" Averaging a whole page into one vector produces something generically about the topic and specifically about nothing."}]]},{"t":"p","runs":[{"text":"The practical rule: use embeddings to find candidates, a reranker to order them, exact matching for anything with an identifier, and a language model only at the end, on a handful of passages you have good reason to trust."}]}],"check":{"question":[{"text":"A local search system embeds documents and queries with a strong retrieval model but returns poor results. Investigation shows documents were embedded plainly while queries were also embedded plainly, as the model card’s example for documents showed. What is most likely wrong?"}],"options":[{"text":[{"text":"The documents exceed the model’s sequence limit and are being truncated, so most of their content is unrepresented."}],"correct":false,"tempting":[{"text":"A real and common failure, but it would not be revealed by the query-versus-document detail the investigation actually surfaced."}]},{"text":[{"text":"Queries and documents should be embedded by separate models trained as a pair, and using one model for both collapses the space."}],"correct":false,"tempting":[{"text":"Describes an architecture that dual-encoder retrieval does not require; one model with role-appropriate inputs is the standard design."}]},{"text":[{"text":"The model was trained asymmetrically and expects an instruction prefix on queries, so query vectors sit in a different region from the document vectors they should match."}],"correct":true,"tempting":null},{"text":[{"text":"The embeddings were not normalised, so dot products reflect text length rather than similarity."}],"correct":false,"tempting":[{"text":"Names a genuine bug with a different signature: it skews ranking towards long or short texts rather than breaking query-document alignment."}]}],"order":[1,2,0,3],"why":null},"prev":{"slug":"structured-output-locally","title":"Output that parses every time","minutes":9},"next":{"slug":"speech-and-vision-locally","title":"Speech, vision and the rest of the local stack","minutes":9}}}