Meaning as a position in space

The idea under search, recommendation and memory

One idea sits beneath semantic search, recommendation, clustering, deduplication and every “chat with your documents” tool. It is called an embedding, and it is more approachable than its reputation.

An embedding turns a piece of text into a list of numbers (say 384 of them, or 768, or 1,536) that acts as a set of coordinates. The text is now a point in a space with that many dimensions. And the arrangement has a property that makes it useful: texts used in similar ways end up near each other.

“Doctor” and “physician” land close together. “Doctor” and “bicycle” land far apart. “My car will not start” and “vehicle trouble in the morning” land close, despite sharing not one word.

That last example is the whole point. Keyword search fails there. Embedding search does not.

Why closeness means anything at all

Nobody wrote the coordinates. They come out of training, on a principle stated by the linguist J. R. Firth in 1957 and now doing industrial work: you shall know a word by the company it keeps.

If two words appear in the same sorts of sentences, surrounded by the same sorts of words, the training pushes them to similar positions. Since “physician” and “doctor” are used almost interchangeably, they end up almost interchangeably placed. The geometry is a record of usage.

Distance is usually measured as the angle between the two arrows from the origin, which has the name cosine similarity. A value near 1 means nearly the same direction; near 0 means unrelated. You do not need the formula, only the picture: same direction, same meaning.

The famous trick, and what is exaggerated about it

The result that made embeddings famous came from word2vec in 2013: take the vector for “king”, subtract “man”, add “woman”, and the nearest point is “queen”. Meaning as arithmetic.

It is real, and it is oversold. The original evaluation excluded the three input words from the candidate answers, and without that exclusion the nearest point is often “king” itself. The analogies work well for a few relationships (capital cities, verb tenses, gender pairs) and poorly for most others.

This is worth knowing not as trivia but as an example of a pattern: a striking result becomes a slogan, the slogan drops the conditions, and the conditions were where the honesty lived.

What you can build with it

The practical recipe is short, and the free tools are excellent.

1. Split your material into chunks: a few hundred words each, roughly a paragraph or a section.
2. Embed every chunk. sentence-transformers with the model all-MiniLM-L6-v2 is free, open source, produces 384 numbers per chunk, and runs on an ordinary laptop CPU with no graphics card. It will do thousands of chunks a minute.
3. Store the vectors.
4. When a question arrives, embed the question the same way and find the chunks whose vectors are closest.

That is semantic search, and it is what “chat with your PDF” is doing underneath before any language model is involved.

One honest note about storage: for under about a hundred thousand chunks you do not need a vector database. A NumPy array and one matrix multiplication will find your nearest neighbours in milliseconds. Vector databases such as Chroma, Qdrant or FAISS earn their place at larger scale or when you need filtering and persistence. A great deal of unnecessary infrastructure gets deployed for collections of four hundred documents.

The trap that catches everybody once

Embeddings capture used in similar contexts, which is not the same as means the same thing. Antonyms are used in identical contexts.

“This film was excellent” and “this film was terrible” sit close together in embedding space, because both are film opinions with the same grammar. If you build a complaints classifier on embedding similarity alone, it will cheerfully group praise with abuse.

The same property carries bias straight through. Embeddings trained on ordinary text place occupation words near gender words in the pattern of the text they read, which is why embedding-based CV matching can reproduce hiring bias without anyone writing a rule.

So: embeddings are a superb tool for finding related material and a poor tool for judging polarity or truth. Use them to retrieve, then use something else to decide.

Why this lesson matters more than it looks

Every time you hear that an assistant “remembers your documents”, “searches your notes” or “finds similar cases”, this is the machinery. Grasp it and three later subjects (retrieval-augmented generation, recommendation, and why an assistant sometimes retrieves the wrong file with complete confidence) stop being separate topics and become one topic seen from three sides.