Embeddings you can host yourself

What the model gives you

An embedding model turns a piece of text into a fixed-length list of numbers (384, 768 or 1024 of them, typically) arranged so that texts with similar meaning sit close together. Closeness is measured with cosine similarity, which is the dot product of two vectors after both have been scaled to length one.

That is the whole idea, and it is the machinery under semantic search, retrieval-augmented generation, deduplication, clustering, recommendation and topic discovery. It is also the cheapest useful thing in this entire course: an embedding model is 20 to 600 million parameters, runs happily on a CPU, and embedding a hundred thousand paragraphs is minutes rather than hours.

The families

• all-MiniLM-L6-v2. 22 million parameters, 384 dimensions, Apache 2.0, and still the right default for a first system. It embeds thousands of sentences a second on a laptop CPU.
• BGE (BAAI), E5 (Microsoft) and GTE (Alibaba) are the three families that trade the top of the retrieval charts. Small, base and large sizes; multilingual variants; permissive licences.
• nomic-embed-text. Apache 2.0, 8192-token context, which matters when your documents do not chunk neatly.
• jina-embeddings. Long context, multilingual, strong on code.
• Qwen3-Embedding. A recent family in several sizes with strong multilingual results.

Bigger is not automatically better here. The jump from MiniLM to a base-sized model is usually worth it; the jump from base to large often buys two points of recall for four times the compute and twice the storage.

The four details that decide whether it works

Prefixes. This is the mistake that silently halves quality. Several families are trained asymmetrically: queries and documents get different prefixes. E5 wants query: and passage: . BGE wants an instruction prefix on the query only. If you embed both sides identically, nothing errors and retrieval quietly gets worse.

from sentence_transformers import SentenceTransformer
m = SentenceTransformer("intfloat/multilingual-e5-base")
q = m.encode(["query: how do I cancel my order"], normalize_embeddings=True)
d = m.encode(["passage: To cancel an order, open ..."], normalize_embeddings=True)

Read the model card for the exact strings. They differ per family and getting them wrong is undetectable without a measurement.

Normalisation. Cosine similarity assumes unit-length vectors. Most libraries offer normalize_embeddings=True; many vector databases assume you have already done it. Do it once, at write time, and be consistent.

Sequence length. Most embedding models truncate at 512 tokens, and truncation is silent. A 900-token chunk becomes an embedding of its first half, and the second half is unsearchable. Either chunk to fit the model or pick a long-context model deliberately.

Pooling. Whether the vector comes from the CLS token or the mean of all token vectors is decided by the model, not by you. Use the library’s own wrapper rather than calling the transformer directly, or you will pick the wrong one.

Matryoshka, and cheap storage

Newer models are trained so that the first N dimensions of the vector are usable on their own. Truncating a 1024-dimension vector to 256 keeps most of the retrieval quality and cuts index size and search cost by four. If your card mentions Matryoshka representation learning, this is available; test the truncation on your own data rather than assuming the published loss figure transfers.

The arithmetic matters at scale. A million chunks at 1024 dimensions in float32 is 4 GB of index. The same million at 256 dimensions is 1 GB, and quantised to 8-bit integers it is 256 MB, which fits in memory on an ordinary server.

MTEB, and how to read it

MTEB is the standard embedding leaderboard, covering retrieval, classification, clustering and reranking across many datasets. Two cautions. Models are routinely trained on the training splits of MTEB tasks, so the leaderboard increasingly measures familiarity with those datasets. And the tasks are mostly English or translated English; performance in your language is a separate question the aggregate score hides.

The evaluation that decides your choice takes an hour: fifty real queries from your own logs, the correct document for each marked by hand, and recall at 10 measured for three candidate models. Whichever wins on your fifty queries wins, whatever the leaderboard says.

Where it goes wrong

Embeddings capture topical similarity, not logical relation. The sentence the payment succeeded and the payment failed sit very close together because they are about the same thing. Any system where the difference between two near-identical sentences matters needs something after retrieval to tell them apart, which is the next lesson.