{"course":{"slug":"machine-learning-foundations","title":"Machine Learning, Foundations","url":"https://addaly.com/learn/machine-learning-foundations","level":"intermediate","track":"start"},"lesson":{"slug":"what-an-embedding-is","title":"What an embedding actually is","url":"https://addaly.com/learn/machine-learning-foundations/what-an-embedding-is","position":68,"of":92,"minutes":10,"answer":"An embedding maps things to vectors so that geometric closeness means similarity in the sense the training task defined.","keyIdea":"An embedding maps things to vectors so that geometric closeness means similarity in the sense the training task defined, which is why the vectors are compared by angle rather than length and why they carry the training corpus's biases in their geometry."},"body":{"format":"markdown","text":"## A definition worth being precise about\n\nAn embedding is a mapping from things (words, images, products, users, sentences) to vectors of real numbers, arranged so that **geometric closeness corresponds to similarity in whatever sense the training set defined.**\n\nThat last clause carries all the weight. There is no universal notion of similarity. An embedding is similar-in-a-particular-sense, and which sense depends entirely on what it was trained to do.\n\n## Why one-hot is not enough\n\nOne-hot encoding gives every category its own dimension. \"Cat\" is `[1, 0, 0, ...]` and \"dog\" is `[0, 1, 0, ...]`. The distance between cat and dog is exactly the distance between cat and \"quantitative easing\". Every pair of distinct categories is equidistant, so the representation cannot express any relationship at all.\n\nAn embedding gives \"cat\" something like `[0.24, -0.81, 0.15, ...]` in 300 dimensions, with \"dog\" nearby and \"quantitative easing\" far away. It has traded 50,000 sparse dimensions for 300 dense ones and gained the ability to represent degrees of similarity.\n\n## Where the numbers come from\n\nNobody chooses them. They fall out of training a model on a task where similarity is useful.\n\n**word2vec** (2013) trains a small network to predict a word from its neighbours, or the neighbours from the word. The training signal is only \"which words appear near which\", and the vectors are a side effect: the hidden layer weights. Words appearing in similar contexts end up with similar vectors, which is the distributional hypothesis from linguistics: a word is characterised by the company it keeps.\n\n**Matrix factorisation** for recommendations decomposes a users-by-items matrix into user vectors and item vectors whose dot product reconstructs the ratings. The vectors are what makes the reconstruction work.\n\n**Modern sentence embeddings** train on pairs known to be related (a question and its answer, a sentence and its translation) with a contrastive loss that pulls related pairs together and pushes unrelated ones apart.\n\nIn every case the embedding is not the goal of training. It is the internal representation a model built because that representation made the task easier, and it turns out to be useful elsewhere.\n\n## The arithmetic, and how much to believe\n\nThe famous demonstration: `king - man + woman` lands near `queen`. Directions in the space appear to encode relationships: a gender direction, a plural direction, a capital-city direction.\n\nIt is real and it is oversold. Later analyses showed the result depends on excluding the input words from the candidate answers, that many analogies fail, and that some apparent successes come from the query word being near the answer already. Treat it as a striking property of these spaces rather than a reliable operation.\n\nWhat is robust is that **similarity works.** Nearest neighbours in a good embedding space are related, and that is what everything downstream is built on.\n\n## Cosine, not Euclidean\n\nEmbeddings are almost always compared by cosine similarity (the cosine of the angle between two vectors) rather than by Euclidean distance.\n\n```python\nimport numpy as np\ndef cosine(a, b):\n    return a @ b / (np.linalg.norm(a) * np.linalg.norm(b))\n```\n\nThe reason is that vector *length* often encodes something other than meaning: word frequency, document length, how confident the model was. Two documents about the same topic at different lengths point in the same direction with different magnitudes. The angle captures the topic; the length captures something else.\n\nA useful practical note: if you normalise every vector to unit length first, cosine similarity and Euclidean distance give the same ranking, and the dot product becomes the cosine directly. Most vector databases normalise on the way in for exactly this reason.\n\n## What an embedding inherits\n\nThe geometry reflects the training data, including its biases. Word embeddings trained on web text place occupations near gendered words in patterns that match the corpus rather than any ideal: the Bolukbasi and Caliskan papers documented this in 2016 and 2017, and the finding has been reproduced many times since.\n\nDebiasing methods exist and are contested: some remove the measured association without removing the underlying structure, so the bias remains detectable by other probes. This is unresolved. What is not unresolved is that an embedding is a compressed summary of its training corpus, and anything systematic in that corpus is in the geometry.\n\n## Where they are used\n\n- **Semantic search.** Embed the query and the documents, return the nearest. Finds \"how do I cancel my subscription\" for a document titled \"ending your plan\", which keyword search cannot.\n- **Recommendation.** Items near the ones a user liked.\n- **Retrieval-augmented generation.** Fetch relevant passages by embedding similarity, then hand them to a language model.\n- **Deduplication and clustering.** Near-duplicates are near neighbours.\n- **Features for a classifier.** Embed the text, feed the vector to logistic regression. Often better than TF-IDF, and one line either way, so measure both.\n\nAnd inside a transformer, the first thing that happens to a token is that it becomes an embedding. The whole architecture operates on vectors of this kind, which is why understanding what one is matters well beyond the classical setting."},"check":{"question":"A team embeds product descriptions and finds that a 200-word description and a 20-word description of the same product have low cosine similarity, though the words overlap heavily. What is the most likely cause?","options":[{"text":"Cosine similarity is length-sensitive, so the differing document lengths reduce the score directly.","correct":false,"tempting":"Inverts the property: cosine normalises out magnitude, which is exactly why it is preferred for documents of different lengths."},{"text":"The embedding model was applied beyond its maximum input length, so the 200-word description was truncated and the vector represents only its opening.","correct":true},{"text":"Embeddings require standardised inputs, and the descriptions were not normalised before encoding.","correct":false,"tempting":"Text encoders handle raw text directly; there is no scaling step of the kind numeric features need."},{"text":"Cosine similarity is unsuitable for text and Euclidean distance should be used instead.","correct":false,"tempting":"On normalised vectors the two produce the same ranking, so switching metrics would not resolve a genuine difference in content."}]},"prev":{"slug":"seeing-high-dimensional-data","title":"t-SNE and UMAP, and how they mislead","url":"https://addaly.com/learn/machine-learning-foundations/seeing-high-dimensional-data"},"next":{"slug":"embeddings-in-practice","title":"Building a working semantic search with no GPU","url":"https://addaly.com/learn/machine-learning-foundations/embeddings-in-practice"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'What an embedding actually is', in Machine Learning, Foundations. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/machine-learning-foundations/what-an-embedding-is (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"What an embedding actually is","courseTitle":"Machine Learning, Foundations","module":{"index":7,"slug":"learning-without-labels","title":"Learning without labels, and learning a representation"},"answer":null,"keyIdea":"An embedding maps things to vectors so that geometric closeness means similarity in the sense the training task defined, which is why the vectors are compared by angle rather than length and why they carry the training corpus’s biases in their geometry.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"a-definition-worth-being-precise-about","runs":[{"text":"A definition worth being precise about"}]},{"t":"p","runs":[{"text":"An embedding is a mapping from things (words, images, products, users, sentences) to vectors of real numbers, arranged so that "},{"text":"geometric closeness corresponds to similarity in whatever sense the training set defined.","b":true}]},{"t":"p","runs":[{"text":"That last clause carries all the weight. There is no universal notion of similarity. An embedding is similar-in-a-particular-sense, and which sense depends entirely on what it was trained to do."}]},{"t":"h2","id":"why-one-hot-is-not-enough","runs":[{"text":"Why one-hot is not enough"}]},{"t":"p","runs":[{"text":"One-hot encoding gives every category its own dimension. “Cat” is "},{"text":"[1, 0, 0, ...]","code":true},{"text":" and “dog” is "},{"text":"[0, 1, 0, ...]","code":true},{"text":". The distance between cat and dog is exactly the distance between cat and “quantitative easing”. Every pair of distinct categories is equidistant, so the representation cannot express any relationship at all."}]},{"t":"p","runs":[{"text":"An embedding gives “cat” something like "},{"text":"[0.24, -0.81, 0.15, ...]","code":true},{"text":" in 300 dimensions, with “dog” nearby and “quantitative easing” far away. It has traded 50,000 sparse dimensions for 300 dense ones and gained the ability to represent degrees of similarity."}]},{"t":"h2","id":"where-the-numbers-come-from","runs":[{"text":"Where the numbers come from"}]},{"t":"p","runs":[{"text":"Nobody chooses them. They fall out of training a model on a task where similarity is useful."}]},{"t":"p","runs":[{"text":"word2vec","b":true},{"text":" (2013) trains a small network to predict a word from its neighbours, or the neighbours from the word. The training signal is only “which words appear near which”, and the vectors are a side effect: the hidden layer weights. Words appearing in similar contexts end up with similar vectors, which is the distributional hypothesis from linguistics: a word is characterised by the company it keeps."}]},{"t":"p","runs":[{"text":"Matrix factorisation","b":true},{"text":" for recommendations decomposes a users-by-items matrix into user vectors and item vectors whose dot product reconstructs the ratings. The vectors are what makes the reconstruction work."}]},{"t":"p","runs":[{"text":"Modern sentence embeddings","b":true},{"text":" train on pairs known to be related (a question and its answer, a sentence and its translation) with a contrastive loss that pulls related pairs together and pushes unrelated ones apart."}]},{"t":"p","runs":[{"text":"In every case the embedding is not the goal of training. It is the internal representation a model built because that representation made the task easier, and it turns out to be useful elsewhere."}]},{"t":"h2","id":"the-arithmetic-and-how-much-to-believe","runs":[{"text":"The arithmetic, and how much to believe"}]},{"t":"p","runs":[{"text":"The famous demonstration: "},{"text":"king - man + woman","code":true},{"text":" lands near "},{"text":"queen","code":true},{"text":". Directions in the space appear to encode relationships: a gender direction, a plural direction, a capital-city direction."}]},{"t":"p","runs":[{"text":"It is real and it is oversold. Later analyses showed the result depends on excluding the input words from the candidate answers, that many analogies fail, and that some apparent successes come from the query word being near the answer already. Treat it as a striking property of these spaces rather than a reliable operation."}]},{"t":"p","runs":[{"text":"What is robust is that "},{"text":"similarity works.","b":true},{"text":" Nearest neighbours in a good embedding space are related, and that is what everything downstream is built on."}]},{"t":"h2","id":"cosine-not-euclidean","runs":[{"text":"Cosine, not Euclidean"}]},{"t":"p","runs":[{"text":"Embeddings are almost always compared by cosine similarity (the cosine of the angle between two vectors) rather than by Euclidean distance."}]},{"t":"code","lang":"python","label":"Python","text":"import numpy as np\ndef cosine(a, b):\n    return a @ b / (np.linalg.norm(a) * np.linalg.norm(b))"},{"t":"p","runs":[{"text":"The reason is that vector "},{"text":"length","i":true},{"text":" often encodes something other than meaning: word frequency, document length, how confident the model was. Two documents about the same topic at different lengths point in the same direction with different magnitudes. The angle captures the topic; the length captures something else."}]},{"t":"p","runs":[{"text":"A useful practical note: if you normalise every vector to unit length first, cosine similarity and Euclidean distance give the same ranking, and the dot product becomes the cosine directly. Most vector databases normalise on the way in for exactly this reason."}]},{"t":"h2","id":"what-an-embedding-inherits","runs":[{"text":"What an embedding inherits"}]},{"t":"p","runs":[{"text":"The geometry reflects the training data, including its biases. Word embeddings trained on web text place occupations near gendered words in patterns that match the corpus rather than any ideal: the Bolukbasi and Caliskan papers documented this in 2016 and 2017, and the finding has been reproduced many times since."}]},{"t":"p","runs":[{"text":"Debiasing methods exist and are contested: some remove the measured association without removing the underlying structure, so the bias remains detectable by other probes. This is unresolved. What is not unresolved is that an embedding is a compressed summary of its training corpus, and anything systematic in that corpus is in the geometry."}]},{"t":"h2","id":"where-they-are-used","runs":[{"text":"Where they are used"}]},{"t":"ul","items":[[{"text":"Semantic search.","b":true},{"text":" Embed the query and the documents, return the nearest. Finds “how do I cancel my subscription” for a document titled “ending your plan”, which keyword search cannot."}],[{"text":"Recommendation.","b":true},{"text":" Items near the ones a user liked."}],[{"text":"Retrieval-augmented generation.","b":true},{"text":" Fetch relevant passages by embedding similarity, then hand them to a language model."}],[{"text":"Deduplication and clustering.","b":true},{"text":" Near-duplicates are near neighbours."}],[{"text":"Features for a classifier.","b":true},{"text":" Embed the text, feed the vector to logistic regression. Often better than TF-IDF, and one line either way, so measure both."}]]},{"t":"p","runs":[{"text":"And inside a transformer, the first thing that happens to a token is that it becomes an embedding. The whole architecture operates on vectors of this kind, which is why understanding what one is matters well beyond the classical setting."}]}],"check":{"question":[{"text":"A team embeds product descriptions and finds that a 200-word description and a 20-word description of the same product have low cosine similarity, though the words overlap heavily. What is the most likely cause?"}],"options":[{"text":[{"text":"Cosine similarity is length-sensitive, so the differing document lengths reduce the score directly."}],"correct":false,"tempting":[{"text":"Inverts the property: cosine normalises out magnitude, which is exactly why it is preferred for documents of different lengths."}]},{"text":[{"text":"The embedding model was applied beyond its maximum input length, so the 200-word description was truncated and the vector represents only its opening."}],"correct":true,"tempting":null},{"text":[{"text":"Embeddings require standardised inputs, and the descriptions were not normalised before encoding."}],"correct":false,"tempting":[{"text":"Text encoders handle raw text directly; there is no scaling step of the kind numeric features need."}]},{"text":[{"text":"Cosine similarity is unsuitable for text and Euclidean distance should be used instead."}],"correct":false,"tempting":[{"text":"On normalised vectors the two produce the same ranking, so switching metrics would not resolve a genuine difference in content."}]}],"order":[0,1,3,2],"why":null},"prev":{"slug":"seeing-high-dimensional-data","title":"t-SNE and UMAP, and how they mislead","minutes":8},"next":{"slug":"embeddings-in-practice","title":"Building a working semantic search with no GPU","minutes":9}}}