Cosine, dot product, and the normalisation that connects them

Three measures, and when they differ

Cosine similarity is the cosine of the angle between two vectors. It ignores length entirely and ranges from −1 to 1, though in practice text embeddings rarely go below about 0 because the vectors occupy a narrow cone of the space.

Dot product multiplies element-wise and sums. It is cosine multiplied by both vectors’ lengths, so it rewards longer vectors.

Euclidean distance is straight-line distance. Smaller is better, which inverts every comparison and is a routine source of reversed results.

The connection that makes this simple: on unit-length vectors, ranking by cosine, dot product and Euclidean distance gives the identical order. Normalise once at encode time and the choice stops mattering:

v = model.encode(texts, normalize_embeddings=True)
scores = v @ q          # dot product == cosine, and it is the fastest

If you do not normalise, dot product favours documents whose vectors happen to be long, which correlates weakly with length and topic frequency, so unnormalised dot-product search has a subtle bias toward a particular kind of document and nobody can see it in the results.

Normalise everything at encode time, then use dot product. That one rule removes the entire category.

What the numbers mean, which is less than you think

A cosine of 0.82 between two texts means nothing on its own. Embedding models differ in how they spread their space: one model’s “clearly related” is 0.85, another’s is 0.55. Absolute thresholds do not transfer between models, and a threshold copied from a tutorial written for a different model is a wrong threshold.

Two consequences:

1. Rank, do not threshold, wherever you can. “Top five” is portable; “above 0.8” is not.
2. When you must threshold (deduplication, a cutoff for “no good answer”) derive it from your own data. Encode a few hundred known-related and known-unrelated pairs, plot the two distributions, and put the threshold where they separate. That takes fifteen minutes and produces a number that is true for your model and your text.

The high floor

Encode two unrelated sentences with a typical model and you may see 0.3 or 0.4 rather than 0. This is normal. Text embeddings occupy a narrow region of the space, so nothing is ever truly orthogonal to anything else.

It means a similarity of 0.5 is not “half related”. Calibrate on your own pairs before you interpret any score, and be suspicious of any product that shows users a raw similarity as a percentage: it reads as a confidence and is not one.

When Euclidean is the right choice

For clustering algorithms that assume Euclidean geometry (k-means in particular) use Euclidean distance on normalised vectors, which is equivalent to cosine for ranking and is what the algorithm’s mathematics expects. For pure retrieval, dot product on normalised vectors is faster and equivalent.

Vector databases ask you to choose a metric when you create an index. Choose the one matching how you normalised, and write it down, because changing it later means rebuilding the index.

A practical check

import numpy as np
v = model.encode(sample_texts)
print("norms:", np.linalg.norm(v, axis=1)[:5])

If those print as 1.0, the vectors are normalised. If they print as 8.3, 11.2, 6.7, they are not, and any dot-product search over them is length-biased. This one line catches a bug that otherwise shows up as “search is a bit worse than I expected” and is never diagnosed.

Maximal marginal relevance, briefly

Top-k by similarity often returns five near-identical chunks, because if one paragraph matches, its neighbours do too. Maximal marginal relevance picks the next result by balancing similarity to the query against dissimilarity to what is already selected:

# pick k items trading off relevance (to q) against redundancy (to chosen)
score = lam * sim_to_query[i] - (1 - lam) * max(sim(i, j) for j in chosen)

With lam around 0.7 you get results that cover the question rather than repeating one part of it. For retrieval feeding a language model this matters a great deal: five copies of one paragraph waste the context window that five different paragraphs would have filled.

Do this now

Encode your corpus twice, once normalised and once not, and compare the top five for three queries under dot product. Where the results differ, look at the lengths of the documents that moved. That is the bias, made visible.