{"course":{"slug":"context-engineering","title":"Context Engineering","url":"https://addaly.com/learn/context-engineering","level":"advanced","track":"build"},"lesson":{"slug":"relevance-is-not-similarity","title":"Similar is not the same as relevant","url":"https://addaly.com/learn/context-engineering/relevance-is-not-similarity","position":12,"of":87,"minutes":9,"answer":"Vector similarity measures whether two texts keep similar company, not whether one answers the other, so negations, on-topic non-answers and rare identifiers all rank high.","keyIdea":"Vector similarity measures whether two texts keep similar company, not whether one answers the other, so negations, on-topic non-answers and rare identifiers all rank high, and a raw score means nothing until you have calibrated it on your own judged pairs."},"body":{"format":"markdown","text":"## What a cosine score is measuring\n\nA vector search returns the passages whose embeddings point in a similar direction to the question's embedding. That is a statement about distributional similarity (these texts appear in similar company) and it is a useful proxy for relevance most of the time. It is not relevance, and the gap has a specific shape worth knowing, because it produces confident wrong answers rather than obvious failures.\n\nFour ways similarity and relevance come apart:\n\n**Negation is nearly invisible.** \"Refunds are available within 30 days\" and \"Refunds are not available within 30 days\" embed very close together in most models. They share almost every token and every topic. A retriever that ranks by similarity cannot prefer one, and whichever it returns will be believed.\n\n**The topic matches, the question does not.** A page about your refund policy is similar to a question about a refund deadline whether or not it states a deadline. You get an on-topic passage that does not contain the answer, and the model, reading an authoritative-looking passage on exactly the right subject, fills the gap.\n\n**The answer is phrased differently from the question.** Users write \"my money hasn't come back\"; your documentation says \"reimbursement processing timelines\". Lexical overlap is nil and topical overlap is weak. This is where embeddings usually beat keywords, and also where they most often miss entirely.\n\n**Rare identifiers do not embed well.** An order number, an error code, a part number, a legal citation. These are exactly the queries where an exact match is the only correct answer, and they are exactly where dense retrieval is worst. Keyword search handles them; the next module covers running both.\n\n## Scores are not probabilities and not comparable\n\nA cosine of 0.83 means nothing on its own. Embedding models differ enormously in how they spread scores: some put almost all unrelated pairs between 0.7 and 0.85, others spread them from 0.05 to 0.6. Copying a threshold of 0.75 from a blog post written about a different model is a way to accept everything or reject everything, and both look plausible in a demo.\n\nTwo habits fix this.\n\n**Rank, do not threshold, unless you calibrated the threshold yourself.** Take 200 query–passage pairs you have judged by hand, plot the score distributions of the relevant and irrelevant ones, and pick the point that gives the precision you need. Repeat whenever you change the embedding model: the number does not transfer.\n\n**Look at the gap, not the level.** The distance between the top score and the fifth is often more informative than the top score itself. A tight cluster of similar scores usually means nothing in the corpus is a strong match and the retriever is returning its best guesses. A large gap means one passage stands out. This is a cheap, model-independent abstention signal.\n\n```python\nscores = [s for _, s in results]\nmargin = scores[0] - scores[4]      # top vs fifth\nif margin < CALIBRATED_MARGIN:\n    return abstain()                 # nothing clearly matched\n```\n\n## The asymmetry of the failure\n\nMissing a relevant passage and including an irrelevant one are not equally bad, and which is worse depends on the task.\n\nFor a factual question with a single answer, an irrelevant but on-topic passage is worse than nothing, because it gives the model something to be confident about. For an open-ended summary, a missing passage is worse, because the output silently omits a whole theme.\n\nDecide which of those two your product mostly does, and tune the retriever's aggressiveness in that direction. Systems that treat both failures as one number end up tuned for neither.\n\n## Ask what a good passage would look like\n\nThe most useful diagnostic in retrieval is not a metric. It is this question, asked about a failing case: *what would a perfect passage for this query look like, and does it exist in the corpus at all?*\n\nThree different answers, three different fixes:\n\n- **It exists and was not retrieved.** A retrieval problem. Try hybrid search, query rewriting, a reranker, better chunking.\n- **It exists but is split across two chunks.** A chunking problem, and no amount of retriever tuning fixes it.\n- **It does not exist.** A content problem. No pipeline can retrieve a policy nobody wrote down, and the correct product behaviour is to say so rather than to synthesise one.\n\nTeams routinely spend weeks on the first when the truth is the third. Half an hour of reading the corpus by hand for ten failing queries will tell you which world you are in, and it is the single highest-return half hour in retrieval work.\n\n## The free path\n\nEverything here runs without a hosted service. `sentence-transformers` gives you strong open embedding models on CPU; `scikit-learn` or plain NumPy computes cosine similarity over a few hundred thousand vectors faster than you would expect; a Jupyter notebook and a scatter plot of relevant against irrelevant scores is the whole calibration exercise. The managed vector databases are worth money at scale and buy you nothing you need to learn this."},"check":{"question":"A retriever consistently returns the passage \"Refunds are not available on sale items\" for the question \"Can I get a refund on a sale item?\" and the assistant answers yes. Which explanation identifies the mechanism?","options":[{"text":"The embedding is stale and predates the policy change, so the wrong version was indexed.","correct":false,"tempting":"Blames freshness for a case where the retrieved text is current and correct; the failure is downstream of retrieval."},{"text":"The similarity score was below the abstention threshold, so a low-confidence passage was used as though it were high-confidence.","correct":false,"tempting":"Assumes a low score, when a negated sentence on exactly the right topic scores very high: that is precisely why it was retrieved."},{"text":"Retrieval worked; the model misread a negation it was given, which is a generation failure, not a selection one.","correct":true},{"text":"Cosine similarity cannot represent negation, so the retriever returned a passage that does not contain the answer.","correct":false,"tempting":"Correctly names the embedding limitation but applies it to the wrong stage: here the retrieved passage does contain the answer."}]},"prev":{"slug":"retrieval-versus-long-context","title":"Retrieval versus long context: the real trade","url":"https://addaly.com/learn/context-engineering/retrieval-versus-long-context"},"next":{"slug":"filter-before-you-rank","title":"Filter before you rank, not after","url":"https://addaly.com/learn/context-engineering/filter-before-you-rank"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Similar is not the same as relevant', in Context Engineering. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/context-engineering/relevance-is-not-similarity (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Similar is not the same as relevant","courseTitle":"Context Engineering","module":{"index":2,"slug":"what-earns-a-place","title":"Selection: what earns a place"},"answer":null,"keyIdea":"Vector similarity measures whether two texts keep similar company, not whether one answers the other, so negations, on-topic non-answers and rare identifiers all rank high, and a raw score means nothing until you have calibrated it on your own judged pairs.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"what-a-cosine-score-is-measuring","runs":[{"text":"What a cosine score is measuring"}]},{"t":"p","runs":[{"text":"A vector search returns the passages whose embeddings point in a similar direction to the question’s embedding. That is a statement about distributional similarity (these texts appear in similar company) and it is a useful proxy for relevance most of the time. It is not relevance, and the gap has a specific shape worth knowing, because it produces confident wrong answers rather than obvious failures."}]},{"t":"p","runs":[{"text":"Four ways similarity and relevance come apart:"}]},{"t":"p","runs":[{"text":"Negation is nearly invisible.","b":true},{"text":" “Refunds are available within 30 days” and “Refunds are not available within 30 days” embed very close together in most models. They share almost every token and every topic. A retriever that ranks by similarity cannot prefer one, and whichever it returns will be believed."}]},{"t":"p","runs":[{"text":"The topic matches, the question does not.","b":true},{"text":" A page about your refund policy is similar to a question about a refund deadline whether or not it states a deadline. You get an on-topic passage that does not contain the answer, and the model, reading an authoritative-looking passage on exactly the right subject, fills the gap."}]},{"t":"p","runs":[{"text":"The answer is phrased differently from the question.","b":true},{"text":" Users write “my money hasn’t come back”; your documentation says “reimbursement processing timelines”. Lexical overlap is nil and topical overlap is weak. This is where embeddings usually beat keywords, and also where they most often miss entirely."}]},{"t":"p","runs":[{"text":"Rare identifiers do not embed well.","b":true},{"text":" An order number, an error code, a part number, a legal citation. These are exactly the queries where an exact match is the only correct answer, and they are exactly where dense retrieval is worst. Keyword search handles them; the next module covers running both."}]},{"t":"h2","id":"scores-are-not-probabilities-and-not-comparable","runs":[{"text":"Scores are not probabilities and not comparable"}]},{"t":"p","runs":[{"text":"A cosine of 0.83 means nothing on its own. Embedding models differ enormously in how they spread scores: some put almost all unrelated pairs between 0.7 and 0.85, others spread them from 0.05 to 0.6. Copying a threshold of 0.75 from a blog post written about a different model is a way to accept everything or reject everything, and both look plausible in a demo."}]},{"t":"p","runs":[{"text":"Two habits fix this."}]},{"t":"p","runs":[{"text":"Rank, do not threshold, unless you calibrated the threshold yourself.","b":true},{"text":" Take 200 query–passage pairs you have judged by hand, plot the score distributions of the relevant and irrelevant ones, and pick the point that gives the precision you need. Repeat whenever you change the embedding model: the number does not transfer."}]},{"t":"p","runs":[{"text":"Look at the gap, not the level.","b":true},{"text":" The distance between the top score and the fifth is often more informative than the top score itself. A tight cluster of similar scores usually means nothing in the corpus is a strong match and the retriever is returning its best guesses. A large gap means one passage stands out. This is a cheap, model-independent abstention signal."}]},{"t":"code","lang":"python","label":"Python","text":"scores = [s for _, s in results]\nmargin = scores[0] - scores[4]      # top vs fifth\nif margin < CALIBRATED_MARGIN:\n    return abstain()                 # nothing clearly matched"},{"t":"h2","id":"the-asymmetry-of-the-failure","runs":[{"text":"The asymmetry of the failure"}]},{"t":"p","runs":[{"text":"Missing a relevant passage and including an irrelevant one are not equally bad, and which is worse depends on the task."}]},{"t":"p","runs":[{"text":"For a factual question with a single answer, an irrelevant but on-topic passage is worse than nothing, because it gives the model something to be confident about. For an open-ended summary, a missing passage is worse, because the output silently omits a whole theme."}]},{"t":"p","runs":[{"text":"Decide which of those two your product mostly does, and tune the retriever’s aggressiveness in that direction. Systems that treat both failures as one number end up tuned for neither."}]},{"t":"h2","id":"ask-what-a-good-passage-would-look-like","runs":[{"text":"Ask what a good passage would look like"}]},{"t":"p","runs":[{"text":"The most useful diagnostic in retrieval is not a metric. It is this question, asked about a failing case: "},{"text":"what would a perfect passage for this query look like, and does it exist in the corpus at all?","i":true}]},{"t":"p","runs":[{"text":"Three different answers, three different fixes:"}]},{"t":"ul","items":[[{"text":"It exists and was not retrieved.","b":true},{"text":" A retrieval problem. Try hybrid search, query rewriting, a reranker, better chunking."}],[{"text":"It exists but is split across two chunks.","b":true},{"text":" A chunking problem, and no amount of retriever tuning fixes it."}],[{"text":"It does not exist.","b":true},{"text":" A content problem. No pipeline can retrieve a policy nobody wrote down, and the correct product behaviour is to say so rather than to synthesise one."}]]},{"t":"p","runs":[{"text":"Teams routinely spend weeks on the first when the truth is the third. Half an hour of reading the corpus by hand for ten failing queries will tell you which world you are in, and it is the single highest-return half hour in retrieval work."}]},{"t":"h2","id":"the-free-path","runs":[{"text":"The free path"}],"role":"free"},{"t":"p","runs":[{"text":"Everything here runs without a hosted service. "},{"text":"sentence-transformers","code":true},{"text":" gives you strong open embedding models on CPU; "},{"text":"scikit-learn","code":true},{"text":" or plain NumPy computes cosine similarity over a few hundred thousand vectors faster than you would expect; a Jupyter notebook and a scatter plot of relevant against irrelevant scores is the whole calibration exercise. The managed vector databases are worth money at scale and buy you nothing you need to learn this."}]}],"check":{"question":[{"text":"A retriever consistently returns the passage “Refunds are not available on sale items” for the question “Can I get a refund on a sale item?” and the assistant answers yes. Which explanation identifies the mechanism?"}],"options":[{"text":[{"text":"The embedding is stale and predates the policy change, so the wrong version was indexed."}],"correct":false,"tempting":[{"text":"Blames freshness for a case where the retrieved text is current and correct; the failure is downstream of retrieval."}]},{"text":[{"text":"The similarity score was below the abstention threshold, so a low-confidence passage was used as though it were high-confidence."}],"correct":false,"tempting":[{"text":"Assumes a low score, when a negated sentence on exactly the right topic scores very high: that is precisely why it was retrieved."}]},{"text":[{"text":"Retrieval worked; the model misread a negation it was given, which is a generation failure, not a selection one."}],"correct":true,"tempting":null},{"text":[{"text":"Cosine similarity cannot represent negation, so the retriever returned a passage that does not contain the answer."}],"correct":false,"tempting":[{"text":"Correctly names the embedding limitation but applies it to the wrong stage: here the retrieved passage does contain the answer."}]}],"order":[0,3,2,1],"why":null},"prev":{"slug":"retrieval-versus-long-context","title":"Retrieval versus long context: the real trade","minutes":9},"next":{"slug":"filter-before-you-rank","title":"Filter before you rank, not after","minutes":8}}}