{"course":{"slug":"hugging-face","title":"Hugging Face, End to End","url":"https://addaly.com/learn/hugging-face","level":"intermediate","track":"create"},"lesson":{"slug":"what-an-embedding-is","title":"A list of numbers where near means similar","url":"https://addaly.com/learn/hugging-face/what-an-embedding-is","position":48,"of":75,"minutes":9,"answer":"An embedding is a fixed-length vector whose geometry encodes meaning, produced by a model trained contrastively to place similar texts close together, so search, clustering, deduplication and few-shot classification are all the same operation.","keyIdea":"An embedding is a fixed-length vector whose geometry encodes meaning, produced by a model trained contrastively to place similar texts close together, so search, clustering, deduplication and few-shot classification are all the same operation, and its central weakness is that similarity is not relevance and negation barely moves the vector."},"body":{"format":"markdown","text":"## The smallest true description\n\nAn embedding model takes a piece of text and returns a fixed-length list of floating-point numbers: 384, 768 or 1,024 of them, usually. The same model always produces the same length, whatever the input length. That is the whole interface.\n\nWhat makes it useful is a property of how the model was trained: texts with similar meaning land close together in that space, and texts with different meaning land far apart. \"Where is my parcel?\" and \"My delivery has not arrived\" share almost no words and sit close together. \"Bank of the river\" and \"bank account\" share a word and sit far apart.\n\n```python\nfrom sentence_transformers import SentenceTransformer\n\nm = SentenceTransformer(\"sentence-transformers/all-MiniLM-L6-v2\")\nv = m.encode(\"Where is my parcel?\")\nprint(v.shape, v[:5])\n# (384,) [ 0.041 -0.113  0.028  0.067 -0.019]\n```\n\nThose 384 numbers mean nothing individually. No dimension is \"formality\" or \"topic\". Only the geometry is meaningful: distances and angles between vectors. People waste time trying to interpret single dimensions; there is nothing there to find.\n\n## Where the vectors come from\n\nA transformer produces one vector per token. An embedding model reduces that to one vector per text, usually by **mean pooling** (averaging the token vectors, weighted by the attention mask so padding contributes nothing) or by taking the `[CLS]` position, or by a small trained head.\n\nThat pooling step is why you should not roll your own from a base model. `AutoModel` gives you token vectors; getting from there to a good sentence vector requires the pooling the model was trained with, and guessing gives you something that works badly enough to be misleading. `sentence-transformers` stores the right pooling alongside the weights and applies it for you.\n\nThe deeper point: an embedding model is trained with a **contrastive objective**: shown pairs that should be close and pairs that should be far, and adjusted until they are. A base language model was never asked to do this, which is why `bert-base-uncased` produces famously poor sentence embeddings despite being a perfectly good language model. The capability is trained in, not inherent.\n\n## Why this is worth your attention\n\nOnce text is a vector, several different-looking problems become one problem:\n\n- **Search**: embed the query, find the nearest documents. Works across paraphrase and, with the right model, across languages.\n- **Retrieval-augmented generation**: the same search, with the results pasted into a prompt.\n- **Clustering**: group vectors, get topics nobody labelled.\n- **Deduplication**: pairs above a similarity threshold are near-duplicates, which is the check the data-quality lesson needed.\n- **Classification with no training data**: embed a handful of labelled examples per class, classify by nearest centroid. This works surprisingly well at twenty examples per class.\n- **Recommendation**: items near the ones somebody liked.\n\nAll six are the same two lines of code with different post-processing.\n\n## What it costs\n\nThis is the part that decides whether you can use it, and the numbers are friendly. `all-MiniLM-L6-v2` is 90 MB, six transformer layers, and encodes several hundred short texts per second on an ordinary CPU. Embedding 100,000 documents on a laptop is minutes, not hours, and it is a **one-off**: store the vectors and searching them afterwards is arithmetic.\n\nA million 384-dimensional float32 vectors is 1.5 GB: fits in RAM. At float16 it is 750 MB. This is why semantic search over a substantial corpus is free, and why the module-three advice about small machines holds here.\n\n## The limitations, by mechanism\n\n**Similarity is not relevance.** Two texts about the same topic are close whether one answers the other or not. \"How do I cancel my subscription?\" is close to \"How do I renew my subscription?\" because they are near-identical in form and topic, and one is a wrong answer to the other. Embeddings retrieve the neighbourhood; deciding within it is what the reranker lesson is for.\n\n**Negation is weak.** \"The battery lasts all day\" and \"The battery does not last all day\" are close in most embedding spaces, because the words are nearly identical and one token is doing all the semantic work. For any task where negation matters, this is a real failure and not a tuning problem.\n\n**Long text averages away.** Pool a 3,000-word document into 384 numbers and the specific fact you wanted is diluted by everything else in it. This is the whole reason chunking exists.\n\n**Out-of-domain is out-of-domain.** A model trained on web English places medical abbreviations or Hinglish code-switching roughly at random. Check on your own text.\n\n## Do this now\n\nEncode five sentences: two paraphrases, two on the same topic with opposite meaning, and one unrelated. Print all ten pairwise similarities. The paraphrase pair will be high, the unrelated pair low, and the opposite-meaning pair will be uncomfortably high, which is the most useful thing this lesson has to show you."},"check":{"question":"A support search built on embeddings keeps returning \"How do I renew my subscription?\" for the query \"How do I cancel my subscription?\". What does this reveal?","options":[{"text":"Embeddings measure topical and structural similarity, which is high for two questions that differ by one decisive word.","correct":true},{"text":"The vectors were not normalised, so cosine similarity is being distorted by differences in text length.","correct":false,"tempting":"Normalisation matters for the arithmetic but would not systematically confuse two near-identical sentences."},{"text":"The model's context window truncated the query, removing the word that distinguishes the two intents.","correct":false,"tempting":"Truncation affects long inputs; both of these are short enough to encode fully."},{"text":"The index holds too few documents, so the nearest neighbour is the closest available rather than a genuinely relevant one.","correct":false,"tempting":"A small index does produce poor nearest neighbours, but here a semantically opposite document is winning on genuine similarity."}]},"prev":{"slug":"preparing-a-dataset-for-training","title":"From a dataset to batches a trainer can eat","url":"https://addaly.com/learn/hugging-face/preparing-a-dataset-for-training"},"next":{"slug":"sentence-transformers-in-practice","title":"The library, and the four calls you need","url":"https://addaly.com/learn/hugging-face/sentence-transformers-in-practice"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'A list of numbers where near means similar', in Hugging Face, End to End. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/hugging-face/what-an-embedding-is (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"A list of numbers where near means similar","courseTitle":"Hugging Face, End to End","module":{"index":6,"slug":"embeddings-and-retrieval","title":"Embeddings, and search that understands"},"answer":null,"keyIdea":"An embedding is a fixed-length vector whose geometry encodes meaning, produced by a model trained contrastively to place similar texts close together, so search, clustering, deduplication and few-shot classification are all the same operation, and its central weakness is that similarity is not relevance and negation barely moves the vector.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"the-smallest-true-description","runs":[{"text":"The smallest true description"}]},{"t":"p","runs":[{"text":"An embedding model takes a piece of text and returns a fixed-length list of floating-point numbers: 384, 768 or 1,024 of them, usually. The same model always produces the same length, whatever the input length. That is the whole interface."}]},{"t":"p","runs":[{"text":"What makes it useful is a property of how the model was trained: texts with similar meaning land close together in that space, and texts with different meaning land far apart. “Where is my parcel?” and “My delivery has not arrived” share almost no words and sit close together. “Bank of the river” and “bank account” share a word and sit far apart."}]},{"t":"code","lang":"python","label":"Python","text":"from sentence_transformers import SentenceTransformer\n\nm = SentenceTransformer(\"sentence-transformers/all-MiniLM-L6-v2\")\nv = m.encode(\"Where is my parcel?\")\nprint(v.shape, v[:5])\n# (384,) [ 0.041 -0.113  0.028  0.067 -0.019]"},{"t":"p","runs":[{"text":"Those 384 numbers mean nothing individually. No dimension is “formality” or “topic”. Only the geometry is meaningful: distances and angles between vectors. People waste time trying to interpret single dimensions; there is nothing there to find."}]},{"t":"h2","id":"where-the-vectors-come-from","runs":[{"text":"Where the vectors come from"}]},{"t":"p","runs":[{"text":"A transformer produces one vector per token. An embedding model reduces that to one vector per text, usually by "},{"text":"mean pooling","b":true},{"text":" (averaging the token vectors, weighted by the attention mask so padding contributes nothing) or by taking the "},{"text":"[CLS]","code":true},{"text":" position, or by a small trained head."}]},{"t":"p","runs":[{"text":"That pooling step is why you should not roll your own from a base model. "},{"text":"AutoModel","code":true},{"text":" gives you token vectors; getting from there to a good sentence vector requires the pooling the model was trained with, and guessing gives you something that works badly enough to be misleading. "},{"text":"sentence-transformers","code":true},{"text":" stores the right pooling alongside the weights and applies it for you."}]},{"t":"p","runs":[{"text":"The deeper point: an embedding model is trained with a "},{"text":"contrastive objective","b":true},{"text":": shown pairs that should be close and pairs that should be far, and adjusted until they are. A base language model was never asked to do this, which is why "},{"text":"bert-base-uncased","code":true},{"text":" produces famously poor sentence embeddings despite being a perfectly good language model. The capability is trained in, not inherent."}]},{"t":"h2","id":"why-this-is-worth-your-attention","runs":[{"text":"Why this is worth your attention"}]},{"t":"p","runs":[{"text":"Once text is a vector, several different-looking problems become one problem:"}]},{"t":"ul","items":[[{"text":"Search","b":true},{"text":": embed the query, find the nearest documents. Works across paraphrase and, with the right model, across languages."}],[{"text":"Retrieval-augmented generation","b":true},{"text":": the same search, with the results pasted into a prompt."}],[{"text":"Clustering","b":true},{"text":": group vectors, get topics nobody labelled."}],[{"text":"Deduplication","b":true},{"text":": pairs above a similarity threshold are near-duplicates, which is the check the data-quality lesson needed."}],[{"text":"Classification with no training data","b":true},{"text":": embed a handful of labelled examples per class, classify by nearest centroid. This works surprisingly well at twenty examples per class."}],[{"text":"Recommendation","b":true},{"text":": items near the ones somebody liked."}]]},{"t":"p","runs":[{"text":"All six are the same two lines of code with different post-processing."}]},{"t":"h2","id":"what-it-costs","runs":[{"text":"What it costs"}]},{"t":"p","runs":[{"text":"This is the part that decides whether you can use it, and the numbers are friendly. "},{"text":"all-MiniLM-L6-v2","code":true},{"text":" is 90 MB, six transformer layers, and encodes several hundred short texts per second on an ordinary CPU. Embedding 100,000 documents on a laptop is minutes, not hours, and it is a "},{"text":"one-off","b":true},{"text":": store the vectors and searching them afterwards is arithmetic."}]},{"t":"p","runs":[{"text":"A million 384-dimensional float32 vectors is 1.5 GB: fits in RAM. At float16 it is 750 MB. This is why semantic search over a substantial corpus is free, and why the module-three advice about small machines holds here."}]},{"t":"h2","id":"the-limitations-by-mechanism","runs":[{"text":"The limitations, by mechanism"}],"role":"limit"},{"t":"p","runs":[{"text":"Similarity is not relevance.","b":true},{"text":" Two texts about the same topic are close whether one answers the other or not. “How do I cancel my subscription?” is close to “How do I renew my subscription?” because they are near-identical in form and topic, and one is a wrong answer to the other. Embeddings retrieve the neighbourhood; deciding within it is what the reranker lesson is for."}]},{"t":"p","runs":[{"text":"Negation is weak.","b":true},{"text":" “The battery lasts all day” and “The battery does not last all day” are close in most embedding spaces, because the words are nearly identical and one token is doing all the semantic work. For any task where negation matters, this is a real failure and not a tuning problem."}]},{"t":"p","runs":[{"text":"Long text averages away.","b":true},{"text":" Pool a 3,000-word document into 384 numbers and the specific fact you wanted is diluted by everything else in it. This is the whole reason chunking exists."}]},{"t":"p","runs":[{"text":"Out-of-domain is out-of-domain.","b":true},{"text":" A model trained on web English places medical abbreviations or Hinglish code-switching roughly at random. Check on your own text."}]},{"t":"h2","id":"do-this-now","runs":[{"text":"Do this now"}],"role":"try"},{"t":"p","runs":[{"text":"Encode five sentences: two paraphrases, two on the same topic with opposite meaning, and one unrelated. Print all ten pairwise similarities. The paraphrase pair will be high, the unrelated pair low, and the opposite-meaning pair will be uncomfortably high, which is the most useful thing this lesson has to show you."}]}],"check":{"question":[{"text":"A support search built on embeddings keeps returning “How do I renew my subscription?” for the query “How do I cancel my subscription?”. What does this reveal?"}],"options":[{"text":[{"text":"Embeddings measure topical and structural similarity, which is high for two questions that differ by one decisive word."}],"correct":true,"tempting":null},{"text":[{"text":"The vectors were not normalised, so cosine similarity is being distorted by differences in text length."}],"correct":false,"tempting":[{"text":"Normalisation matters for the arithmetic but would not systematically confuse two near-identical sentences."}]},{"text":[{"text":"The model’s context window truncated the query, removing the word that distinguishes the two intents."}],"correct":false,"tempting":[{"text":"Truncation affects long inputs; both of these are short enough to encode fully."}]},{"text":[{"text":"The index holds too few documents, so the nearest neighbour is the closest available rather than a genuinely relevant one."}],"correct":false,"tempting":[{"text":"A small index does produce poor nearest neighbours, but here a semantically opposite document is winning on genuine similarity."}]}],"order":[3,2,1,0],"why":null},"prev":{"slug":"preparing-a-dataset-for-training","title":"From a dataset to batches a trainer can eat","minutes":9},"next":{"slug":"sentence-transformers-in-practice","title":"The library, and the four calls you need","minutes":8}}}