Similar is not the same as relevant

What a cosine score is measuring

A vector search returns the passages whose embeddings point in a similar direction to the question’s embedding. That is a statement about distributional similarity (these texts appear in similar company) and it is a useful proxy for relevance most of the time. It is not relevance, and the gap has a specific shape worth knowing, because it produces confident wrong answers rather than obvious failures.

Four ways similarity and relevance come apart:

Negation is nearly invisible. “Refunds are available within 30 days” and “Refunds are not available within 30 days” embed very close together in most models. They share almost every token and every topic. A retriever that ranks by similarity cannot prefer one, and whichever it returns will be believed.

The topic matches, the question does not. A page about your refund policy is similar to a question about a refund deadline whether or not it states a deadline. You get an on-topic passage that does not contain the answer, and the model, reading an authoritative-looking passage on exactly the right subject, fills the gap.

The answer is phrased differently from the question. Users write “my money hasn’t come back”; your documentation says “reimbursement processing timelines”. Lexical overlap is nil and topical overlap is weak. This is where embeddings usually beat keywords, and also where they most often miss entirely.

Rare identifiers do not embed well. An order number, an error code, a part number, a legal citation. These are exactly the queries where an exact match is the only correct answer, and they are exactly where dense retrieval is worst. Keyword search handles them; the next module covers running both.

Scores are not probabilities and not comparable

A cosine of 0.83 means nothing on its own. Embedding models differ enormously in how they spread scores: some put almost all unrelated pairs between 0.7 and 0.85, others spread them from 0.05 to 0.6. Copying a threshold of 0.75 from a blog post written about a different model is a way to accept everything or reject everything, and both look plausible in a demo.

Two habits fix this.

Rank, do not threshold, unless you calibrated the threshold yourself. Take 200 query–passage pairs you have judged by hand, plot the score distributions of the relevant and irrelevant ones, and pick the point that gives the precision you need. Repeat whenever you change the embedding model: the number does not transfer.

Look at the gap, not the level. The distance between the top score and the fifth is often more informative than the top score itself. A tight cluster of similar scores usually means nothing in the corpus is a strong match and the retriever is returning its best guesses. A large gap means one passage stands out. This is a cheap, model-independent abstention signal.

scores = [s for _, s in results]
margin = scores[0] - scores[4]      # top vs fifth
if margin < CALIBRATED_MARGIN:
    return abstain()                 # nothing clearly matched

The asymmetry of the failure

Missing a relevant passage and including an irrelevant one are not equally bad, and which is worse depends on the task.

For a factual question with a single answer, an irrelevant but on-topic passage is worse than nothing, because it gives the model something to be confident about. For an open-ended summary, a missing passage is worse, because the output silently omits a whole theme.

Decide which of those two your product mostly does, and tune the retriever’s aggressiveness in that direction. Systems that treat both failures as one number end up tuned for neither.

Ask what a good passage would look like

The most useful diagnostic in retrieval is not a metric. It is this question, asked about a failing case: what would a perfect passage for this query look like, and does it exist in the corpus at all?

Three different answers, three different fixes:

• It exists and was not retrieved. A retrieval problem. Try hybrid search, query rewriting, a reranker, better chunking.
• It exists but is split across two chunks. A chunking problem, and no amount of retriever tuning fixes it.
• It does not exist. A content problem. No pipeline can retrieve a policy nobody wrote down, and the correct product behaviour is to say so rather than to synthesise one.

Teams routinely spend weeks on the first when the truth is the third. Half an hour of reading the corpus by hand for ten failing queries will tell you which world you are in, and it is the single highest-return half hour in retrieval work.

The free path

Everything here runs without a hosted service. sentence-transformers gives you strong open embedding models on CPU; scikit-learn or plain NumPy computes cosine similarity over a few hundred thousand vectors faster than you would expect; a Jupyter notebook and a scatter plot of relevant against irrelevant scores is the whole calibration exercise. The managed vector databases are worth money at scale and buy you nothing you need to learn this.