{"course":{"slug":"how-llms-work","title":"How a Language Model Actually Works","url":"https://addaly.com/learn/how-llms-work","level":"intermediate","track":"start"},"lesson":{"slug":"meaning-as-direction","title":"Embeddings: meaning as a direction","url":"https://addaly.com/learn/how-llms-work/meaning-as-direction","position":5,"of":78,"minutes":7,"answer":"An embedding is not a fixed label on a word; it is a position in space that context rewrites.","keyIdea":"An embedding is not a fixed label on a word; it is a position in space that context rewrites."},"body":{"format":"markdown","text":"## A token id means nothing on its own\n\nToken 24379 is just an index. The first thing the model does is look that index up in a table with one row per vocabulary entry and, in a mid-sized model, a few thousand numbers per row. That row is the token's embedding: a point in a space with a few thousand dimensions.\n\nNobody designed those numbers. They started as random noise and were adjusted during training, alongside every other parameter, to make next-token prediction less wrong. Any structure they have is structure that turned out useful for prediction.\n\n## What \"direction\" means here\n\nSimilarity in this space is usually measured by angle rather than distance. Two vectors pointing the same way score 1, perpendicular scores 0, opposite scores -1. That is cosine similarity, and it is three lines of code:\n\n```python\nimport numpy as np\n\ndef cos(a, b):\n    return float(a @ b / (np.linalg.norm(a) * np.linalg.norm(b)))\n```\n\nUnder that measure \"hospital\" sits near \"clinic\" and \"doctor\" and far from \"sonnet\". No rule produced that. Words used in similar company drift together, because a model that places them together predicts text better.\n\nThe famous demonstration is arithmetic on directions: king minus man plus woman lands near queen. Treat it carefully. The published examples were selected from curated lists, and the trick usually requires excluding the input words from the answer. The honest version is weaker and still worth knowing: some relations (country to capital, singular to plural, present to past) correspond roughly to a consistent direction of travel through the space.\n\n## No single number is \"the gender dimension\"\n\nThe tempting picture is a spreadsheet: column 12 is formality, column 900 is animacy. It is not like that. Features are spread across many dimensions at once, and interpretability research suggests models represent far more distinguishable features than they have dimensions, by using directions that are nearly, but not quite, independent of one another. The term for this is superposition. Which features exist and how cleanly they separate is an open research question, not a settled result.\n\nThe part you can rely on: meaning lives in directions, and a direction is a combination of many numbers, not one of them.\n\n## Static versus contextual\n\nOlder systems such as word2vec stopped at the lookup table. One vector per word, permanently. So \"bank\" got a single vector: an unhappy average of finance and riverside.\n\nA transformer also starts with one row per token, but that row is only the starting position. As the vector travels through the layers, each block adds information gathered from the surrounding tokens. By the upper layers, \"bank\" in \"I sat by the river bank\" and \"bank\" in \"the bank refused my loan\" occupy different regions. Same starting row, different destinations.\n\n> Figure: Two things both called an embedding. Static: one vector per word: Looked up once and never changed; \"bank\" is one unhappy average of finance and riverside; Enough for a keyword-like similarity list; word2vec and its descendants stop here. Contextual: a position that gets rewritten: Starts as exactly the same lookup row; Each block adds what it gathered from neighbours; River bank and loan bank end far apart by the upper layers; What a running transformer actually works with. The lookup row is only the starting position. Everything the model does afterwards moves it, which is why a transformer has no fixed vector for a word at all.\n\nHold on to that. An embedding inside a running model is not a label attached to a word. It is a position that gets rewritten as context arrives. The mechanism that does the rewriting is attention, which is the next lesson.\n\n## Where you meet embeddings directly\n\nThe embedding endpoints providers sell give you one vector per chunk of text, typically 768 to 3,072 numbers. That is what powers semantic search: embed your documents once, embed the incoming query, return the nearest neighbours by cosine similarity. A shop in Jakarta can match \"my order never arrived\" to a help article titled \"Delayed deliveries\" with no shared words at all.\n\nTwo limits worth stating plainly. First, embedding similarity captures topical relatedness, not truth or logical relation: \"the drug is safe\" and \"the drug is not safe\" sit very close together, which is why naive retrieval can hand a model the exact opposite of what it needed. Second, spaces from different models are not comparable. Vectors produced by one provider's model are meaningless to another's, so changing embedding models means re-embedding everything you stored."},"check":{"question":"You run a modern transformer on \"I sat by the river bank\" and on \"The bank refused my loan\". You inspect the vector at the position of \"bank\" in the final layer of each. What do you find?","options":[{"text":"They differ, because the vector at that position is rewritten by its context as it passes through the blocks.","correct":true},{"text":"They are identical, because a token maps to one fixed row in the embedding table.","correct":false,"tempting":"The input embedding really is a fixed lookup, and most short explanations stop there without mentioning what the layers do to it."},{"text":"They differ because the tokenizer assigns different ids to the two senses of \"bank\".","correct":false,"tempting":"It seems natural that disambiguation happens early, at the point where text is turned into symbols."},{"text":"They point in the same direction but differ in length, since magnitude is what carries context.","correct":false,"tempting":"There is a half-remembered fact that vector magnitude encodes things like word frequency, which makes this sound technically informed."}]},"prev":{"slug":"tokenizer-failures-in-the-wild","title":"Tokenizer failures you will actually hit","url":"https://addaly.com/learn/how-llms-work/tokenizer-failures-in-the-wild"},"next":{"slug":"embedding-space-in-practice","title":"Working in embedding space, with free tools","url":"https://addaly.com/learn/how-llms-work/embedding-space-in-practice"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Embeddings: meaning as a direction', in How a Language Model Actually Works. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/how-llms-work/meaning-as-direction (Accessed: 31 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Embeddings: meaning as a direction","courseTitle":"How a Language Model Actually Works","module":{"index":1,"slug":"from-characters-to-vectors","title":"From characters to vectors"},"answer":null,"keyIdea":"An embedding is not a fixed label on a word; it is a position in space that context rewrites.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"left":{"title":"Static: one vector per word","items":["Looked up once and never changed","“bank” is one unhappy average of finance and riverside","Enough for a keyword-like similarity list","word2vec and its descendants stop here"]},"type":"compare","right":{"title":"Contextual: a position that gets rewritten","items":["Starts as exactly the same lookup row","Each block adds what it gathered from neighbours","River bank and loan bank end far apart by the upper layers","What a running transformer actually works with"]},"title":"Two things both called an embedding","caption":"The lookup row is only the starting position. Everything the model does afterwards moves it, which is why a transformer has no fixed vector for a word at all."},"alt":"Figure: Two things both called an embedding. Static: one vector per word: Looked up once and never changed; “bank” is one unhappy average of finance and riverside; Enough for a keyword-like similarity list; word2vec and its descendants stop here. Contextual: a position that gets rewritten: Starts as exactly the same lookup row; Each block adds what it gathered from neighbours; River bank and loan bank end far apart by the upper layers; What a running transformer actually works with. The lookup row is only the starting position. Everything the model does afterwards moves it, which is why a transformer has no fixed vector for a word at all."}},"blocks":[{"t":"h2","id":"a-token-id-means-nothing-on-its-own","runs":[{"text":"A token id means nothing on its own"}]},{"t":"p","runs":[{"text":"Token 24379 is just an index. The first thing the model does is look that index up in a table with one row per vocabulary entry and, in a mid-sized model, a few thousand numbers per row. That row is the token’s embedding: a point in a space with a few thousand dimensions."}]},{"t":"p","runs":[{"text":"Nobody designed those numbers. They started as random noise and were adjusted during training, alongside every other parameter, to make next-token prediction less wrong. Any structure they have is structure that turned out useful for prediction."}]},{"t":"h2","id":"what-direction-means-here","runs":[{"text":"What “direction” means here"}]},{"t":"p","runs":[{"text":"Similarity in this space is usually measured by angle rather than distance. Two vectors pointing the same way score 1, perpendicular scores 0, opposite scores -1. That is cosine similarity, and it is three lines of code:"}]},{"t":"code","lang":"python","label":"Python","text":"import numpy as np\n\ndef cos(a, b):\n    return float(a @ b / (np.linalg.norm(a) * np.linalg.norm(b)))"},{"t":"p","runs":[{"text":"Under that measure “hospital” sits near “clinic” and “doctor” and far from “sonnet”. No rule produced that. Words used in similar company drift together, because a model that places them together predicts text better."}]},{"t":"p","runs":[{"text":"The famous demonstration is arithmetic on directions: king minus man plus woman lands near queen. Treat it carefully. The published examples were selected from curated lists, and the trick usually requires excluding the input words from the answer. The honest version is weaker and still worth knowing: some relations (country to capital, singular to plural, present to past) correspond roughly to a consistent direction of travel through the space."}]},{"t":"h2","id":"no-single-number-is-the-gender-dimension","runs":[{"text":"No single number is “the gender dimension”"}]},{"t":"p","runs":[{"text":"The tempting picture is a spreadsheet: column 12 is formality, column 900 is animacy. It is not like that. Features are spread across many dimensions at once, and interpretability research suggests models represent far more distinguishable features than they have dimensions, by using directions that are nearly, but not quite, independent of one another. The term for this is superposition. Which features exist and how cleanly they separate is an open research question, not a settled result."}]},{"t":"p","runs":[{"text":"The part you can rely on: meaning lives in directions, and a direction is a combination of many numbers, not one of them."}]},{"t":"h2","id":"static-versus-contextual","runs":[{"text":"Static versus contextual"}]},{"t":"p","runs":[{"text":"Older systems such as word2vec stopped at the lookup table. One vector per word, permanently. So “bank” got a single vector: an unhappy average of finance and riverside."}]},{"t":"p","runs":[{"text":"A transformer also starts with one row per token, but that row is only the starting position. As the vector travels through the layers, each block adds information gathered from the surrounding tokens. By the upper layers, “bank” in “I sat by the river bank” and “bank” in “the bank refused my loan” occupy different regions. Same starting row, different destinations."}]},{"t":"p","runs":[{"text":"Hold on to that. An embedding inside a running model is not a label attached to a word. It is a position that gets rewritten as context arrives. The mechanism that does the rewriting is attention, which is the next lesson."}]},{"t":"h2","id":"where-you-meet-embeddings-directly","runs":[{"text":"Where you meet embeddings directly"}]},{"t":"p","runs":[{"text":"The embedding endpoints providers sell give you one vector per chunk of text, typically 768 to 3,072 numbers. That is what powers semantic search: embed your documents once, embed the incoming query, return the nearest neighbours by cosine similarity. A shop in Jakarta can match “my order never arrived” to a help article titled “Delayed deliveries” with no shared words at all."}]},{"t":"p","runs":[{"text":"Two limits worth stating plainly. First, embedding similarity captures topical relatedness, not truth or logical relation: “the drug is safe” and “the drug is not safe” sit very close together, which is why naive retrieval can hand a model the exact opposite of what it needed. Second, spaces from different models are not comparable. Vectors produced by one provider’s model are meaningless to another’s, so changing embedding models means re-embedding everything you stored."}]}],"check":{"question":[{"text":"You run a modern transformer on “I sat by the river bank” and on “The bank refused my loan”. You inspect the vector at the position of “bank” in the final layer of each. What do you find?"}],"options":[{"text":[{"text":"They differ, because the vector at that position is rewritten by its context as it passes through the blocks."}],"correct":true,"tempting":null},{"text":[{"text":"They are identical, because a token maps to one fixed row in the embedding table."}],"correct":false,"tempting":[{"text":"The input embedding really is a fixed lookup, and most short explanations stop there without mentioning what the layers do to it."}]},{"text":[{"text":"They differ because the tokenizer assigns different ids to the two senses of “bank”."}],"correct":false,"tempting":[{"text":"It seems natural that disambiguation happens early, at the point where text is turned into symbols."}]},{"text":[{"text":"They point in the same direction but differ in length, since magnitude is what carries context."}],"correct":false,"tempting":[{"text":"There is a half-remembered fact that vector magnitude encodes things like word frequency, which makes this sound technically informed."}]}],"order":[1,3,0,2],"why":null},"prev":{"slug":"tokenizer-failures-in-the-wild","title":"Tokenizer failures you will actually hit","minutes":8},"next":{"slug":"embedding-space-in-practice","title":"Working in embedding space, with free tools","minutes":10}}}