Embeddings: meaning as a direction

A token id means nothing on its own

Token 24379 is just an index. The first thing the model does is look that index up in a table with one row per vocabulary entry and, in a mid-sized model, a few thousand numbers per row. That row is the token's embedding: a point in a space with a few thousand dimensions.

Nobody designed those numbers. They started as random noise and were adjusted during training, alongside every other parameter, to make next-token prediction less wrong. Any structure they have is structure that turned out useful for prediction.

What "direction" means here

Similarity in this space is usually measured by angle rather than distance. Two vectors pointing the same way score 1, perpendicular scores 0, opposite scores -1. That is cosine similarity, and it is three lines of code:

import numpy as np

def cos(a, b):
    return float(a @ b / (np.linalg.norm(a) * np.linalg.norm(b)))

Under that measure "hospital" sits near "clinic" and "doctor" and far from "sonnet". No rule produced that. Words used in similar company drift together, because a model that places them together predicts text better.

The famous demonstration is arithmetic on directions: king minus man plus woman lands near queen. Treat it carefully. The published examples were selected from curated lists, and the trick usually requires excluding the input words from the answer. The honest version is weaker and still worth knowing: some relations — country to capital, singular to plural, present to past — correspond roughly to a consistent direction of travel through the space.

No single number is "the gender dimension"

The tempting picture is a spreadsheet: column 12 is formality, column 900 is animacy. It is not like that. Features are spread across many dimensions at once, and interpretability research suggests models represent far more distinguishable features than they have dimensions, by using directions that are nearly, but not quite, independent of one another. The term for this is superposition. Which features exist and how cleanly they separate is an open research question, not a settled result.

The part you can rely on: meaning lives in directions, and a direction is a combination of many numbers, not one of them.

Static versus contextual

Older systems such as word2vec stopped at the lookup table. One vector per word, permanently. So "bank" got a single vector — an unhappy average of finance and riverside.

A transformer also starts with one row per token, but that row is only the starting position. As the vector travels through the layers, each block adds information gathered from the surrounding tokens. By the upper layers, "bank" in "I sat by the river bank" and "bank" in "the bank refused my loan" occupy different regions. Same starting row, different destinations.

Hold on to that. An embedding inside a running model is not a label attached to a word. It is a position that gets rewritten as context arrives. The mechanism that does the rewriting is attention, which is the next lesson.

Where you meet embeddings directly

The embedding endpoints providers sell give you one vector per chunk of text, typically 768 to 3,072 numbers. That is what powers semantic search: embed your documents once, embed the incoming query, return the nearest neighbours by cosine similarity. A shop in Jakarta can match "my order never arrived" to a help article titled "Delayed deliveries" with no shared words at all.

Two limits worth stating plainly. First, embedding similarity captures topical relatedness, not truth or logical relation — "the drug is safe" and "the drug is not safe" sit very close together, which is why naive retrieval can hand a model the exact opposite of what it needed. Second, spaces from different models are not comparable. Vectors produced by one provider's model are meaningless to another's, so changing embedding models means re-embedding everything you stored.