Embeddings and rerankers on your own machine

A different kind of model, and a much smaller one

An embedding model turns a piece of text into a fixed-length vector of numbers (commonly 384, 768 or 1,024 of them) such that texts with similar meaning land near each other. Nothing is generated. There is one forward pass and the output is the vector.

That makes them cheap in a way generative models are not. A 100M-parameter embedding model is 200 MB in fp16, runs on any CPU, and embeds thousands of short texts per second. If you have been told local AI needs a GPU, this is the large exception, and it is the foundation of search, retrieval, clustering, deduplication and recommendation.

Running one

# Ollama
ollama pull nomic-embed-text
curl http://localhost:11434/api/embed \
  -d '{"model":"nomic-embed-text","input":"the cat sat on the mat"}'

# llama.cpp
llama-server -m bge-m3-Q8_0.gguf --embedding --pooling mean -c 8192

Or in Python, which is where most of this work happens:

from sentence_transformers import SentenceTransformer
m = SentenceTransformer("BAAI/bge-m3")
vecs = m.encode(["the cat sat", "a feline was seated"],
                normalize_embeddings=True)
print(vecs @ vecs.T)   # cosine similarity, since they are normalised

Normalise, then a dot product is cosine similarity. Two sentences meaning the same thing typically score above 0.8; unrelated ones sit near 0.1 to 0.3. Those numbers are model-specific, which is the first thing people get wrong: a similarity threshold tuned for one model is meaningless for another.

Four details that decide whether it works

Query and document may need different prefixes. Several strong models were trained asymmetrically: documents embedded plainly, queries prefixed with an instruction such as “Represent this sentence for searching relevant passages:”. Omitting the prefix degrades retrieval substantially and produces no error. The model card states it; almost nobody reads that part.

Sequence length is short. Many embedding models take 512 tokens and truncate the rest silently. A 2,000-word document embedded by such a model is an embedding of its first few paragraphs. Chunk deliberately rather than discovering this later.

Multilingual is a property of the model, not of embeddings. An English-trained model maps Hindi text to essentially arbitrary positions. If your corpus is not English, pick a model trained multilingually and test it on your own pairs.

Dimension is a storage and speed decision. 1,024 dimensions at fp32 is 4 KB per vector; a million documents is 4 GB. Some recent models support Matryoshka truncation (the vector is trained so that its first 256 dimensions are a usable embedding on their own) which lets you trade a little accuracy for a quarter of the storage by cutting the vector short.

Rerankers: the step most people skip

An embedding search is fast because every document was embedded in advance and the query only has to be compared against stored vectors. The cost is that the query and the document never meet inside the model.

A cross-encoder reranker takes the query and one document together, runs both through a model, and outputs a relevance score. Far more accurate, and far too slow to run over a whole corpus.

The standard arrangement uses both. Retrieve fifty candidates with embeddings in a few milliseconds, then rerank those fifty with a cross-encoder in a few hundred milliseconds, and pass the top five to the language model.

from sentence_transformers import CrossEncoder
ce = CrossEncoder("BAAI/bge-reranker-v2-m3")
scores = ce.predict([(query, d) for d in candidates])

On most real corpora this two-stage design improves answer quality more than upgrading the generation model does, and it costs a 500 MB model on the CPU. It is the highest-value under-used component in local retrieval.

Where embeddings fail

They are similarity machines, and similarity is not relevance.

• Negation is nearly invisible. “The drug is safe for children” and “the drug is not safe for children” embed close together. If a wrong answer here matters, embeddings alone must not be your filter.
• Numbers and identifiers do not embed usefully. Searching for invoice 88213 is a job for exact matching, not vectors. Hybrid search (keyword scoring combined with vector similarity) exists precisely for this, and it beats either alone on most real corpora.
• Long documents lose their specifics. Averaging a whole page into one vector produces something generically about the topic and specifically about nothing.

The practical rule: use embeddings to find candidates, a reranker to order them, exact matching for anything with an identifier, and a language model only at the end, on a handful of passages you have good reason to trust.