Working in embedding space, with free tools

Build the thing, then argue about it

You have the idea from the previous lesson: meaning is a direction. This lesson is the practical half, because the limits of embeddings only become obvious when you have watched them fail on your own data.

Everything here runs offline on a laptop with no GPU. sentence-transformers is free and open source, the small models are a few hundred megabytes, and a corpus of a thousand documents embeds in under a minute on a CPU.

pip install sentence-transformers numpy

from sentence_transformers import SentenceTransformer
import numpy as np

model = SentenceTransformer("all-MiniLM-L6-v2")   # 384 dimensions, ~80 MB

docs = [
    "Refunds are issued within 7 working days of approval.",
    "Our delivery partner covers Pune, Nashik and Aurangabad.",
    "To reset your password, use the link on the sign-in page.",
    "Orders above 2,000 rupees ship free.",
]
D = model.encode(docs, normalize_embeddings=True)  # shape (4, 384)

def search(q, k=2):
    v = model.encode([q], normalize_embeddings=True)[0]
    scores = D @ v                      # cosine similarity, because rows are unit length
    order = np.argsort(-scores)[:k]
    return [(round(float(scores[i]), 3), docs[i]) for i in order]

print(search("my money hasn't come back yet"))

The query shares no words with the refund line and still retrieves it. That is the entire trick behind semantic search, and it is four lines once the vectors exist.

Normalise, then dot product

Note normalize_embeddings=True. Once every vector has length 1, cosine similarity is the dot product, so a search over a million documents becomes one matrix multiplication. This is why vector databases are fast: they are not doing anything clever with meaning, they are doing linear algebra with good indexing.

For up to a few hundred thousand documents you do not need a vector database at all. A NumPy array in memory and one @ is faster than most people expect and has no operational cost.

Four failure modes to see for yourself

Negation is nearly invisible. Embed “the medicine is safe for children” and “the medicine is not safe for children”. They will typically score above 0.85. Cosine similarity measures what the sentence is about, not what it asserts. Retrieval systems built on embeddings alone will happily hand a model the opposite of what it needed, and the model, having no other source, will use it.

Length distorts. Very short queries and very long documents compare badly. Chunking documents into passages of a few hundred tokens is not a stylistic choice; it is how you keep the comparison meaningful.

Domain shift is brutal. A general model has never seen your internal product codes. “ERR_4412 on the DX2 unit” is nearly meaningless to it, and its neighbours will be arbitrary. This is where keyword search still wins, and why serious systems combine both.

Spaces are not comparable. Vectors from all-MiniLM-L6-v2 mean nothing to another provider’s model. Changing embedding model means re-embedding every document you have stored. Budget for it.

The fix that actually works: hybrid and rerank

The standard production shape, and it is not complicated:

1. Retrieve 50 candidates by embedding similarity.
2. Retrieve 50 more by keyword search: BM25, available free in rank_bm25 or any search engine.
3. Merge the two lists.
4. Score the merged set with a cross-encoder, a small model that reads the query and the passage together rather than comparing two independent vectors.
5. Keep the top 5.

The cross-encoder is the step that recovers negation and fine distinctions, because it does not compress each side into one vector before comparing. It is slower (it must run once per candidate), which is exactly why you use it on 100 candidates rather than a million documents. cross-encoder/ms-marco-MiniLM-L-6-v2 is free and runs on CPU.

What the vector is not

An embedding is not a summary, not a compression you can decode, and not evidence. You cannot read a vector to recover the text. You cannot compare a similarity score across two different corpora and conclude anything. And a score of 0.83 has no absolute meaning at all: only the ranking matters, which is why thresholds tuned on one dataset break on the next.

Build the small version first, on fifty of your own documents, and look at the ten worst results by hand. That habit is worth more than any published benchmark, and it is the same discipline the evaluation course teaches for everything else.