{"course":{"slug":"ai-actually-explained","title":"AI, Actually Explained","url":"https://addaly.com/learn/ai-actually-explained","level":"beginner","track":"start"},"lesson":{"slug":"embeddings-meaning-as-position","title":"Meaning as a position in space","url":"https://addaly.com/learn/ai-actually-explained/embeddings-meaning-as-position","position":24,"of":92,"minutes":10,"answer":"An embedding places text as a point in space so that things used similarly sit close together, which makes it excellent for retrieval and unreliable for polarity, since praise and abuse occupy the same neighbourhood.","keyIdea":"An embedding places text as a point in space so that things used similarly sit close together, which makes it excellent for retrieval and unreliable for polarity, since praise and abuse occupy the same neighbourhood."},"body":{"format":"markdown","text":"## The idea under search, recommendation and memory\n\nOne idea sits beneath semantic search, recommendation, clustering, deduplication and every \"chat with your documents\" tool. It is called an **embedding**, and it is more approachable than its reputation.\n\nAn embedding turns a piece of text into a list of numbers (say 384 of them, or 768, or 1,536) that acts as a set of coordinates. The text is now a point in a space with that many dimensions. And the arrangement has a property that makes it useful: **texts used in similar ways end up near each other.**\n\n\"Doctor\" and \"physician\" land close together. \"Doctor\" and \"bicycle\" land far apart. \"My car will not start\" and \"vehicle trouble in the morning\" land close, despite sharing not one word.\n\nThat last example is the whole point. Keyword search fails there. Embedding search does not.\n\n## Why closeness means anything at all\n\nNobody wrote the coordinates. They come out of training, on a principle stated by the linguist J. R. Firth in 1957 and now doing industrial work: you shall know a word by the company it keeps.\n\nIf two words appear in the same sorts of sentences, surrounded by the same sorts of words, the training pushes them to similar positions. Since \"physician\" and \"doctor\" are used almost interchangeably, they end up almost interchangeably placed. The geometry is a record of usage.\n\nDistance is usually measured as the angle between the two arrows from the origin, which has the name **cosine similarity**. A value near 1 means nearly the same direction; near 0 means unrelated. You do not need the formula, only the picture: same direction, same meaning.\n\n## The famous trick, and what is exaggerated about it\n\nThe result that made embeddings famous came from word2vec in 2013: take the vector for \"king\", subtract \"man\", add \"woman\", and the nearest point is \"queen\". Meaning as arithmetic.\n\nIt is real, and it is oversold. The original evaluation excluded the three input words from the candidate answers, and without that exclusion the nearest point is often \"king\" itself. The analogies work well for a few relationships (capital cities, verb tenses, gender pairs) and poorly for most others.\n\nThis is worth knowing not as trivia but as an example of a pattern: a striking result becomes a slogan, the slogan drops the conditions, and the conditions were where the honesty lived.\n\n## What you can build with it\n\nThe practical recipe is short, and the free tools are excellent.\n\n1. Split your material into chunks: a few hundred words each, roughly a paragraph or a section.\n2. Embed every chunk. `sentence-transformers` with the model `all-MiniLM-L6-v2` is free, open source, produces 384 numbers per chunk, and runs on an ordinary laptop CPU with no graphics card. It will do thousands of chunks a minute.\n3. Store the vectors.\n4. When a question arrives, embed the question the same way and find the chunks whose vectors are closest.\n\nThat is semantic search, and it is what \"chat with your PDF\" is doing underneath before any language model is involved.\n\nOne honest note about storage: for under about a hundred thousand chunks you do not need a vector database. A NumPy array and one matrix multiplication will find your nearest neighbours in milliseconds. Vector databases such as Chroma, Qdrant or FAISS earn their place at larger scale or when you need filtering and persistence. A great deal of unnecessary infrastructure gets deployed for collections of four hundred documents.\n\n## The trap that catches everybody once\n\nEmbeddings capture *used in similar contexts*, which is not the same as *means the same thing*. Antonyms are used in identical contexts.\n\n\"This film was excellent\" and \"this film was terrible\" sit close together in embedding space, because both are film opinions with the same grammar. If you build a complaints classifier on embedding similarity alone, it will cheerfully group praise with abuse.\n\nThe same property carries bias straight through. Embeddings trained on ordinary text place occupation words near gender words in the pattern of the text they read, which is why embedding-based CV matching can reproduce hiring bias without anyone writing a rule.\n\nSo: embeddings are a superb tool for **finding related material** and a poor tool for **judging polarity or truth**. Use them to retrieve, then use something else to decide.\n\n## Why this lesson matters more than it looks\n\nEvery time you hear that an assistant \"remembers your documents\", \"searches your notes\" or \"finds similar cases\", this is the machinery. Grasp it and three later subjects (retrieval-augmented generation, recommendation, and why an assistant sometimes retrieves the wrong file with complete confidence) stop being separate topics and become one topic seen from three sides."},"check":{"question":"A team builds a complaints triage tool by embedding each incoming message and matching it to the nearest example in a labelled set. It routes many enthusiastic thank-you messages into the complaints queue. Why?","options":[{"text":"The embedding model was too small at 384 dimensions to separate the two classes, and a larger model would fix it.","correct":false,"tempting":"Treats a structural property as a capacity shortfall; larger embedding models place antonyms close together too."},{"text":"Embeddings encode contexts of use, and praise and complaint about the same product appear in near-identical contexts, so they sit close together.","correct":true},{"text":"The labelled examples were imbalanced, with too many complaints, biasing every match towards that class.","correct":false,"tempting":"Names a real modelling problem, but class imbalance would not specifically confuse positive and negative messages about the same subject."},{"text":"Cosine similarity ignores the direction of the vectors and measures only their length.","correct":false,"tempting":"Inverts what cosine similarity does; it measures angle precisely and ignores length."}]},"prev":{"slug":"attention-what-it-does","title":"Attention, without a single equation","url":"https://addaly.com/learn/ai-actually-explained/attention-what-it-does"},"next":{"slug":"the-context-window","title":"The desk it works on, and what falls off it","url":"https://addaly.com/learn/ai-actually-explained/the-context-window"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Meaning as a position in space', in AI, Actually Explained. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/ai-actually-explained/embeddings-meaning-as-position (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Meaning as a position in space","courseTitle":"AI, Actually Explained","module":{"index":3,"slug":"how-a-chatbot-works","title":"What a chatbot is actually doing"},"answer":null,"keyIdea":"An embedding places text as a point in space so that things used similarly sit close together, which makes it excellent for retrieval and unreliable for polarity, since praise and abuse occupy the same neighbourhood.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"the-idea-under-search-recommendation-and-memory","runs":[{"text":"The idea under search, recommendation and memory"}]},{"t":"p","runs":[{"text":"One idea sits beneath semantic search, recommendation, clustering, deduplication and every “chat with your documents” tool. It is called an "},{"text":"embedding","b":true},{"text":", and it is more approachable than its reputation."}]},{"t":"p","runs":[{"text":"An embedding turns a piece of text into a list of numbers (say 384 of them, or 768, or 1,536) that acts as a set of coordinates. The text is now a point in a space with that many dimensions. And the arrangement has a property that makes it useful: "},{"text":"texts used in similar ways end up near each other.","b":true}]},{"t":"p","runs":[{"text":"“Doctor” and “physician” land close together. “Doctor” and “bicycle” land far apart. “My car will not start” and “vehicle trouble in the morning” land close, despite sharing not one word."}]},{"t":"p","runs":[{"text":"That last example is the whole point. Keyword search fails there. Embedding search does not."}]},{"t":"h2","id":"why-closeness-means-anything-at-all","runs":[{"text":"Why closeness means anything at all"}]},{"t":"p","runs":[{"text":"Nobody wrote the coordinates. They come out of training, on a principle stated by the linguist J. R. Firth in 1957 and now doing industrial work: you shall know a word by the company it keeps."}]},{"t":"p","runs":[{"text":"If two words appear in the same sorts of sentences, surrounded by the same sorts of words, the training pushes them to similar positions. Since “physician” and “doctor” are used almost interchangeably, they end up almost interchangeably placed. The geometry is a record of usage."}]},{"t":"p","runs":[{"text":"Distance is usually measured as the angle between the two arrows from the origin, which has the name "},{"text":"cosine similarity","b":true},{"text":". A value near 1 means nearly the same direction; near 0 means unrelated. You do not need the formula, only the picture: same direction, same meaning."}]},{"t":"h2","id":"the-famous-trick-and-what-is-exaggerated-about-it","runs":[{"text":"The famous trick, and what is exaggerated about it"}]},{"t":"p","runs":[{"text":"The result that made embeddings famous came from word2vec in 2013: take the vector for “king”, subtract “man”, add “woman”, and the nearest point is “queen”. Meaning as arithmetic."}]},{"t":"p","runs":[{"text":"It is real, and it is oversold. The original evaluation excluded the three input words from the candidate answers, and without that exclusion the nearest point is often “king” itself. The analogies work well for a few relationships (capital cities, verb tenses, gender pairs) and poorly for most others."}]},{"t":"p","runs":[{"text":"This is worth knowing not as trivia but as an example of a pattern: a striking result becomes a slogan, the slogan drops the conditions, and the conditions were where the honesty lived."}]},{"t":"h2","id":"what-you-can-build-with-it","runs":[{"text":"What you can build with it"}]},{"t":"p","runs":[{"text":"The practical recipe is short, and the free tools are excellent."}]},{"t":"ol","start":1,"items":[[{"text":"Split your material into chunks: a few hundred words each, roughly a paragraph or a section."}],[{"text":"Embed every chunk. "},{"text":"sentence-transformers","code":true},{"text":" with the model "},{"text":"all-MiniLM-L6-v2","code":true},{"text":" is free, open source, produces 384 numbers per chunk, and runs on an ordinary laptop CPU with no graphics card. It will do thousands of chunks a minute."}],[{"text":"Store the vectors."}],[{"text":"When a question arrives, embed the question the same way and find the chunks whose vectors are closest."}]]},{"t":"p","runs":[{"text":"That is semantic search, and it is what “chat with your PDF” is doing underneath before any language model is involved."}]},{"t":"p","runs":[{"text":"One honest note about storage: for under about a hundred thousand chunks you do not need a vector database. A NumPy array and one matrix multiplication will find your nearest neighbours in milliseconds. Vector databases such as Chroma, Qdrant or FAISS earn their place at larger scale or when you need filtering and persistence. A great deal of unnecessary infrastructure gets deployed for collections of four hundred documents."}]},{"t":"h2","id":"the-trap-that-catches-everybody-once","runs":[{"text":"The trap that catches everybody once"}]},{"t":"p","runs":[{"text":"Embeddings capture "},{"text":"used in similar contexts","i":true},{"text":", which is not the same as "},{"text":"means the same thing","i":true},{"text":". Antonyms are used in identical contexts."}]},{"t":"p","runs":[{"text":"“This film was excellent” and “this film was terrible” sit close together in embedding space, because both are film opinions with the same grammar. If you build a complaints classifier on embedding similarity alone, it will cheerfully group praise with abuse."}]},{"t":"p","runs":[{"text":"The same property carries bias straight through. Embeddings trained on ordinary text place occupation words near gender words in the pattern of the text they read, which is why embedding-based CV matching can reproduce hiring bias without anyone writing a rule."}]},{"t":"p","runs":[{"text":"So: embeddings are a superb tool for "},{"text":"finding related material","b":true},{"text":" and a poor tool for "},{"text":"judging polarity or truth","b":true},{"text":". Use them to retrieve, then use something else to decide."}]},{"t":"h2","id":"why-this-lesson-matters-more-than-it-looks","runs":[{"text":"Why this lesson matters more than it looks"}]},{"t":"p","runs":[{"text":"Every time you hear that an assistant “remembers your documents”, “searches your notes” or “finds similar cases”, this is the machinery. Grasp it and three later subjects (retrieval-augmented generation, recommendation, and why an assistant sometimes retrieves the wrong file with complete confidence) stop being separate topics and become one topic seen from three sides."}]}],"check":{"question":[{"text":"A team builds a complaints triage tool by embedding each incoming message and matching it to the nearest example in a labelled set. It routes many enthusiastic thank-you messages into the complaints queue. Why?"}],"options":[{"text":[{"text":"The embedding model was too small at 384 dimensions to separate the two classes, and a larger model would fix it."}],"correct":false,"tempting":[{"text":"Treats a structural property as a capacity shortfall; larger embedding models place antonyms close together too."}]},{"text":[{"text":"Embeddings encode contexts of use, and praise and complaint about the same product appear in near-identical contexts, so they sit close together."}],"correct":true,"tempting":null},{"text":[{"text":"The labelled examples were imbalanced, with too many complaints, biasing every match towards that class."}],"correct":false,"tempting":[{"text":"Names a real modelling problem, but class imbalance would not specifically confuse positive and negative messages about the same subject."}]},{"text":[{"text":"Cosine similarity ignores the direction of the vectors and measures only their length."}],"correct":false,"tempting":[{"text":"Inverts what cosine similarity does; it measures angle precisely and ignores length."}]}],"order":[0,2,3,1],"why":null},"prev":{"slug":"attention-what-it-does","title":"Attention, without a single equation","minutes":9},"next":{"slug":"the-context-window","title":"The desk it works on, and what falls off it","minutes":9}}}