{"course":{"slug":"building-with-ai","title":"Building with AI","url":"https://addaly.com/learn/building-with-ai","level":"intermediate","track":"build"},"lesson":{"slug":"embeddings-what-the-numbers-mean","title":"What an embedding is, and what \"close\" actually means","url":"https://addaly.com/learn/building-with-ai/embeddings-what-the-numbers-mean","position":33,"of":80,"minutes":10,"answer":"An embedding's closeness means only what the model's training pairs meant by closeness, so similarity thresholds are model-specific, exact identifiers blur, negation nearly vanishes.","keyIdea":"An embedding's closeness means only what the model's training pairs meant by closeness, so similarity thresholds are model-specific, exact identifiers blur, negation nearly vanishes, and calibrating on your own data is the step that cannot be skipped."},"body":{"format":"markdown","text":"## A list of numbers that stands for meaning\n\nAn embedding model takes a piece of text and returns a fixed-length list of numbers. For `all-MiniLM-L6-v2` that list is 384 numbers long. For `bge-base` it is 768. For OpenAI's `text-embedding-3-small` it is 1,536, and for `text-embedding-3-large` 3,072. No individual number means anything you can name. What matters is the whole list, treated as a point in a space with that many dimensions, and where that point sits relative to other points.\n\nThe model was trained by contrastive learning. It was shown pairs that belong together (a question and its answer, a title and its body, two phrasings of the same query) and pairs that do not, and it was adjusted until the belonging pairs landed near each other and the others landed far apart. That sentence carries the whole caveat of this lesson. \"Close\" means \"the training data treated these as belonging together\". It does not mean synonymous, and it does not mean true.\n\n## Measuring closeness\n\nThe standard measure is cosine similarity: the cosine of the angle between two vectors. Most embedding APIs return vectors already normalised to length one, and then cosine similarity is just a dot product.\n\n```python\nimport numpy as np\n\ndef cosine(a, b):\n    a, b = np.asarray(a), np.asarray(b)\n    return float(a @ b / (np.linalg.norm(a) * np.linalg.norm(b)))\n```\n\nHere is the thing nobody tells you on a marketing page: the number you get back lives on a scale that belongs to the model, not to the idea of similarity. With some models, unrelated text scores around 0.2 to 0.4 and a relevant passage scores 0.6 to 0.8. With others, everything sits between 0.7 and 0.95, and the gap between relevant and irrelevant is a few hundredths. A threshold of 0.8 copied from a tutorial written against one model will, against another, either return everything or nothing. The mechanism is that the geometry of the space is an artefact of training; nothing forces two models to spread their points over the same range.\n\nThe consequence is a rule: never set a threshold from a blog post. Set it from your own data, using the check at the end of this lesson.\n\n## A question is not a passage\n\nQueries are short and passages are long, and several of the best open models were trained to treat them differently. The `e5` family expects the literal prefix `query: ` on searches and `passage: ` on documents. The `bge` models take an instruction on the query side. Leave the prefix off and the model still returns vectors, but they land in a slightly different region of the space than the ones it was trained to match, and recall drops by a measurable margin: several points on standard benchmarks, more on short queries. This is the most common silent mistake in a first retrieval system, because nothing errors. Read the model card before you embed a single document.\n\n## What an embedding cannot see\n\nThree blind spots, each explained by how the vector is made.\n\n**Exact identifiers blur.** `Order 48213` and `Order 48219` embed almost identically. The model breaks rare strings into subword pieces that carry no consistent meaning, so two order numbers that differ in one digit contribute nearly the same signal. If your users search for invoice numbers, part codes or people's names, an embedding alone will fail them, and the next lesson but one is about the fix.\n\n**Negation nearly vanishes.** \"I want a refund\" and \"I do not want a refund\" sit very close together. A sentence's vector is dominated by its content words; \"not\" is one token among many, and the model was rarely trained on pairs whose only difference was a negation. This matters for intent classification built on embeddings.\n\n**Languages only cross if the model was trained to cross them.** An English-only model given Devanagari produces vectors, but they are close to noise. A Hindi query against English documents needs a multilingual model (`bge-m3`, `multilingual-e5`, or one of the hosted models that lists language coverage) and even then, test Hinglish in Latin script separately, because that is what people type and it appears in almost no training set.\n\n## What it costs\n\nEmbedding is the cheap part of retrieval. Hosted `text-embedding-3-small` costs about $0.02 per million tokens; a 10,000-page corpus is roughly five million tokens, so embedding all of it costs around ten cents. Query-time embedding is a rounding error.\n\nStorage is not free. 1,536 dimensions at four bytes each is about 6 KB per chunk, so a million chunks is 6 GB of vectors before any index. Reducing dimensions or quantising to 8-bit integers cuts that four-fold with a small loss in recall, and a later lesson covers when that trade is worth it.\n\nThe free path runs on a laptop with no GPU:\n\n```python\nfrom sentence_transformers import SentenceTransformer\nmodel = SentenceTransformer(\"all-MiniLM-L6-v2\")   # 22 MB download\nvecs = model.encode([\"how do I reset my password\",\n                     \"Password reset is under Settings > Security.\"],\n                    normalize_embeddings=True)\nprint(float(vecs[0] @ vecs[1]))\n```\n\nThat model embeds several hundred short sentences per second on an ordinary CPU, and runs on a phone through ONNX. `bge-small` and `nomic-embed-text` are a step up in quality at a similar size. You do not need a hosted API to learn this or to ship a first version of it.\n\n## The check you must run before trusting any of it\n\nTake twenty real questions your users ask. For each, find the passage in your corpus that actually answers it. Embed everything, and compute two sets of numbers: the similarity of each question to its answering passage, and the similarity of each question to twenty random passages. Sort both lists and look at them side by side.\n\nIf the lowest \"right answer\" score sits comfortably above the highest \"random\" score, the model separates your data, and the gap tells you where a threshold can go. If the two lists overlap heavily, no threshold exists that works, and no amount of tuning downstream will recover what this step lost. This takes an afternoon and it is the single most informative experiment in the whole pipeline."},"check":{"question":"A team using bge-base sets a similarity cutoff of 0.8, taken from a tutorial, and their search returns almost nothing. What is the most likely cause?","options":[{"text":"The corpus was chunked too coarsely, so each chunk mixes several topics and must be split into smaller pieces before the scores will rise.","correct":false,"tempting":"Confuses a retrieval-quality problem with a chunking problem; a chunking change moves scores a little, not from nothing to something."},{"text":"Score ranges are model-specific; for this model relevant pairs may sit near 0.65, so 0.8 excludes almost everything.","correct":true},{"text":"Cosine similarity is the wrong measure for text and Euclidean distance should be used instead.","correct":false,"tempting":"On normalised vectors, cosine and Euclidean rank identically; the measure is not the problem."},{"text":"768 dimensions is too few to separate relevant from irrelevant passages.","correct":false,"tempting":"Dimension count affects quality at the margin; it does not shift the whole score scale."}]},"prev":{"slug":"retrieval-that-helps","title":"Retrieval: what it fixes and what it does not","url":"https://addaly.com/learn/building-with-ai/retrieval-that-helps"},"next":{"slug":"chunking-and-embeddings","title":"Chunking and embeddings in practice","url":"https://addaly.com/learn/building-with-ai/chunking-and-embeddings"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'What an embedding is, and what \"close\" actually means', in Building with AI. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/building-with-ai/embeddings-what-the-numbers-mean (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"What an embedding is, and what “close” actually means","courseTitle":"Building with AI","module":{"index":4,"slug":"retrieval-and-the-knowledge-problem","title":"Retrieval and the knowledge problem"},"answer":null,"keyIdea":"An embedding’s closeness means only what the model’s training pairs meant by closeness, so similarity thresholds are model-specific, exact identifiers blur, negation nearly vanishes, and calibrating on your own data is the step that cannot be skipped.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"a-list-of-numbers-that-stands-for-meaning","runs":[{"text":"A list of numbers that stands for meaning"}]},{"t":"p","runs":[{"text":"An embedding model takes a piece of text and returns a fixed-length list of numbers. For "},{"text":"all-MiniLM-L6-v2","code":true},{"text":" that list is 384 numbers long. For "},{"text":"bge-base","code":true},{"text":" it is 768. For OpenAI’s "},{"text":"text-embedding-3-small","code":true},{"text":" it is 1,536, and for "},{"text":"text-embedding-3-large","code":true},{"text":" 3,072. No individual number means anything you can name. What matters is the whole list, treated as a point in a space with that many dimensions, and where that point sits relative to other points."}]},{"t":"p","runs":[{"text":"The model was trained by contrastive learning. It was shown pairs that belong together (a question and its answer, a title and its body, two phrasings of the same query) and pairs that do not, and it was adjusted until the belonging pairs landed near each other and the others landed far apart. That sentence carries the whole caveat of this lesson. “Close” means “the training data treated these as belonging together”. It does not mean synonymous, and it does not mean true."}]},{"t":"h2","id":"measuring-closeness","runs":[{"text":"Measuring closeness"}]},{"t":"p","runs":[{"text":"The standard measure is cosine similarity: the cosine of the angle between two vectors. Most embedding APIs return vectors already normalised to length one, and then cosine similarity is just a dot product."}]},{"t":"code","lang":"python","label":"Python","text":"import numpy as np\n\ndef cosine(a, b):\n    a, b = np.asarray(a), np.asarray(b)\n    return float(a @ b / (np.linalg.norm(a) * np.linalg.norm(b)))"},{"t":"p","runs":[{"text":"Here is the thing nobody tells you on a marketing page: the number you get back lives on a scale that belongs to the model, not to the idea of similarity. With some models, unrelated text scores around 0.2 to 0.4 and a relevant passage scores 0.6 to 0.8. With others, everything sits between 0.7 and 0.95, and the gap between relevant and irrelevant is a few hundredths. A threshold of 0.8 copied from a tutorial written against one model will, against another, either return everything or nothing. The mechanism is that the geometry of the space is an artefact of training; nothing forces two models to spread their points over the same range."}]},{"t":"p","runs":[{"text":"The consequence is a rule: never set a threshold from a blog post. Set it from your own data, using the check at the end of this lesson."}]},{"t":"h2","id":"a-question-is-not-a-passage","runs":[{"text":"A question is not a passage"}]},{"t":"p","runs":[{"text":"Queries are short and passages are long, and several of the best open models were trained to treat them differently. The "},{"text":"e5","code":true},{"text":" family expects the literal prefix "},{"text":"query: ","code":true},{"text":" on searches and "},{"text":"passage: ","code":true},{"text":" on documents. The "},{"text":"bge","code":true},{"text":" models take an instruction on the query side. Leave the prefix off and the model still returns vectors, but they land in a slightly different region of the space than the ones it was trained to match, and recall drops by a measurable margin: several points on standard benchmarks, more on short queries. This is the most common silent mistake in a first retrieval system, because nothing errors. Read the model card before you embed a single document."}]},{"t":"h2","id":"what-an-embedding-cannot-see","runs":[{"text":"What an embedding cannot see"}]},{"t":"p","runs":[{"text":"Three blind spots, each explained by how the vector is made."}]},{"t":"p","runs":[{"text":"Exact identifiers blur.","b":true},{"text":" "},{"text":"Order 48213","code":true},{"text":" and "},{"text":"Order 48219","code":true},{"text":" embed almost identically. The model breaks rare strings into subword pieces that carry no consistent meaning, so two order numbers that differ in one digit contribute nearly the same signal. If your users search for invoice numbers, part codes or people’s names, an embedding alone will fail them, and the next lesson but one is about the fix."}]},{"t":"p","runs":[{"text":"Negation nearly vanishes.","b":true},{"text":" “I want a refund” and “I do not want a refund” sit very close together. A sentence’s vector is dominated by its content words; “not” is one token among many, and the model was rarely trained on pairs whose only difference was a negation. This matters for intent classification built on embeddings."}]},{"t":"p","runs":[{"text":"Languages only cross if the model was trained to cross them.","b":true},{"text":" An English-only model given Devanagari produces vectors, but they are close to noise. A Hindi query against English documents needs a multilingual model ("},{"text":"bge-m3","code":true},{"text":", "},{"text":"multilingual-e5","code":true},{"text":", or one of the hosted models that lists language coverage) and even then, test Hinglish in Latin script separately, because that is what people type and it appears in almost no training set."}]},{"t":"h2","id":"what-it-costs","runs":[{"text":"What it costs"}]},{"t":"p","runs":[{"text":"Embedding is the cheap part of retrieval. Hosted "},{"text":"text-embedding-3-small","code":true},{"text":" costs about $0.02 per million tokens; a 10,000-page corpus is roughly five million tokens, so embedding all of it costs around ten cents. Query-time embedding is a rounding error."}]},{"t":"p","runs":[{"text":"Storage is not free. 1,536 dimensions at four bytes each is about 6 KB per chunk, so a million chunks is 6 GB of vectors before any index. Reducing dimensions or quantising to 8-bit integers cuts that four-fold with a small loss in recall, and a later lesson covers when that trade is worth it."}]},{"t":"p","runs":[{"text":"The free path runs on a laptop with no GPU:"}]},{"t":"code","lang":"python","label":"Python","text":"from sentence_transformers import SentenceTransformer\nmodel = SentenceTransformer(\"all-MiniLM-L6-v2\")   # 22 MB download\nvecs = model.encode([\"how do I reset my password\",\n                     \"Password reset is under Settings > Security.\"],\n                    normalize_embeddings=True)\nprint(float(vecs[0] @ vecs[1]))"},{"t":"p","runs":[{"text":"That model embeds several hundred short sentences per second on an ordinary CPU, and runs on a phone through ONNX. "},{"text":"bge-small","code":true},{"text":" and "},{"text":"nomic-embed-text","code":true},{"text":" are a step up in quality at a similar size. You do not need a hosted API to learn this or to ship a first version of it."}]},{"t":"h2","id":"the-check-you-must-run-before-trusting-any-of-it","runs":[{"text":"The check you must run before trusting any of it"}]},{"t":"p","runs":[{"text":"Take twenty real questions your users ask. For each, find the passage in your corpus that actually answers it. Embed everything, and compute two sets of numbers: the similarity of each question to its answering passage, and the similarity of each question to twenty random passages. Sort both lists and look at them side by side."}]},{"t":"p","runs":[{"text":"If the lowest “right answer” score sits comfortably above the highest “random” score, the model separates your data, and the gap tells you where a threshold can go. If the two lists overlap heavily, no threshold exists that works, and no amount of tuning downstream will recover what this step lost. This takes an afternoon and it is the single most informative experiment in the whole pipeline."}]}],"check":{"question":[{"text":"A team using bge-base sets a similarity cutoff of 0.8, taken from a tutorial, and their search returns almost nothing. What is the most likely cause?"}],"options":[{"text":[{"text":"The corpus was chunked too coarsely, so each chunk mixes several topics and must be split into smaller pieces before the scores will rise."}],"correct":false,"tempting":[{"text":"Confuses a retrieval-quality problem with a chunking problem; a chunking change moves scores a little, not from nothing to something."}]},{"text":[{"text":"Score ranges are model-specific; for this model relevant pairs may sit near 0.65, so 0.8 excludes almost everything."}],"correct":true,"tempting":null},{"text":[{"text":"Cosine similarity is the wrong measure for text and Euclidean distance should be used instead."}],"correct":false,"tempting":[{"text":"On normalised vectors, cosine and Euclidean rank identically; the measure is not the problem."}]},{"text":[{"text":"768 dimensions is too few to separate relevant from irrelevant passages."}],"correct":false,"tempting":[{"text":"Dimension count affects quality at the margin; it does not shift the whole score scale."}]}],"order":[3,0,1,2],"why":null},"prev":{"slug":"retrieval-that-helps","title":"Retrieval: what it fixes and what it does not","minutes":8},"next":{"slug":"chunking-and-embeddings","title":"Chunking and embeddings in practice","minutes":8}}}