A list of numbers where near means similar

The smallest true description

An embedding model takes a piece of text and returns a fixed-length list of floating-point numbers: 384, 768 or 1,024 of them, usually. The same model always produces the same length, whatever the input length. That is the whole interface.

What makes it useful is a property of how the model was trained: texts with similar meaning land close together in that space, and texts with different meaning land far apart. “Where is my parcel?” and “My delivery has not arrived” share almost no words and sit close together. “Bank of the river” and “bank account” share a word and sit far apart.

from sentence_transformers import SentenceTransformer

m = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")
v = m.encode("Where is my parcel?")
print(v.shape, v[:5])
# (384,) [ 0.041 -0.113  0.028  0.067 -0.019]

Those 384 numbers mean nothing individually. No dimension is “formality” or “topic”. Only the geometry is meaningful: distances and angles between vectors. People waste time trying to interpret single dimensions; there is nothing there to find.

Where the vectors come from

A transformer produces one vector per token. An embedding model reduces that to one vector per text, usually by mean pooling (averaging the token vectors, weighted by the attention mask so padding contributes nothing) or by taking the [CLS] position, or by a small trained head.

That pooling step is why you should not roll your own from a base model. AutoModel gives you token vectors; getting from there to a good sentence vector requires the pooling the model was trained with, and guessing gives you something that works badly enough to be misleading. sentence-transformers stores the right pooling alongside the weights and applies it for you.

The deeper point: an embedding model is trained with a contrastive objective: shown pairs that should be close and pairs that should be far, and adjusted until they are. A base language model was never asked to do this, which is why bert-base-uncased produces famously poor sentence embeddings despite being a perfectly good language model. The capability is trained in, not inherent.

Why this is worth your attention

Once text is a vector, several different-looking problems become one problem:

• Search: embed the query, find the nearest documents. Works across paraphrase and, with the right model, across languages.
• Retrieval-augmented generation: the same search, with the results pasted into a prompt.
• Clustering: group vectors, get topics nobody labelled.
• Deduplication: pairs above a similarity threshold are near-duplicates, which is the check the data-quality lesson needed.
• Classification with no training data: embed a handful of labelled examples per class, classify by nearest centroid. This works surprisingly well at twenty examples per class.
• Recommendation: items near the ones somebody liked.

All six are the same two lines of code with different post-processing.

What it costs

This is the part that decides whether you can use it, and the numbers are friendly. all-MiniLM-L6-v2 is 90 MB, six transformer layers, and encodes several hundred short texts per second on an ordinary CPU. Embedding 100,000 documents on a laptop is minutes, not hours, and it is a one-off: store the vectors and searching them afterwards is arithmetic.

A million 384-dimensional float32 vectors is 1.5 GB: fits in RAM. At float16 it is 750 MB. This is why semantic search over a substantial corpus is free, and why the module-three advice about small machines holds here.

The limitations, by mechanism

Similarity is not relevance. Two texts about the same topic are close whether one answers the other or not. “How do I cancel my subscription?” is close to “How do I renew my subscription?” because they are near-identical in form and topic, and one is a wrong answer to the other. Embeddings retrieve the neighbourhood; deciding within it is what the reranker lesson is for.

Negation is weak. “The battery lasts all day” and “The battery does not last all day” are close in most embedding spaces, because the words are nearly identical and one token is doing all the semantic work. For any task where negation matters, this is a real failure and not a tuning problem.

Long text averages away. Pool a 3,000-word document into 384 numbers and the specific fact you wanted is diluted by everything else in it. This is the whole reason chunking exists.

Out-of-domain is out-of-domain. A model trained on web English places medical abbreviations or Hinglish code-switching roughly at random. Check on your own text.

Do this now

Encode five sentences: two paraphrases, two on the same topic with opposite meaning, and one unrelated. Print all ten pairwise similarities. The paraphrase pair will be high, the unrelated pair low, and the opposite-meaning pair will be uncomfortably high, which is the most useful thing this lesson has to show you.