Building a working semantic search with no GPU

Getting embeddings for free

You do not train these yourself. Pretrained models are free, small enough to run on a CPU, and good enough for most applications.

from sentence_transformers import SentenceTransformer
m = SentenceTransformer("all-MiniLM-L6-v2")
vecs = m.encode(list_of_texts, normalize_embeddings=True)
vecs.shape        # (n, 384)

all-MiniLM-L6-v2 is about 80 MB, produces 384-dimensional vectors, and encodes a few hundred short texts per second on an ordinary laptop CPU. It is the sensible default.

When you need better quality, all-mpnet-base-v2 is larger and slower. For non-English or mixed-language text, the multilingual models (paraphrase-multilingual-MiniLM-L12-v2 and its relatives) cover 50 or more languages in one shared space, which means a Hindi query can retrieve an English document. For Indian languages specifically, MuRIL from Google Research is free on Hugging Face and trained on 17 Indian languages including transliterated text.

All of these run locally. No API key, no per-call cost, no data leaving the machine, which matters when the documents are confidential.

Searching a million vectors

Comparing a query against a million vectors by brute force is a million dot products of 384 numbers each. In NumPy that is one matrix multiplication and takes a few hundred milliseconds, fine for many applications, and you should measure before assuming otherwise.

sims = corpus_vecs @ query_vec        # both normalised
top = np.argpartition(-sims, 10)[:10]

Above a few million, use an approximate index. These trade a small amount of recall for very large speedups.

• FAISS: free, from Meta, the standard. Runs on CPU. IndexFlatIP is exact; IndexHNSWFlat is the usual approximate choice.
• hnswlib: a small, fast standalone HNSW implementation.
• Annoy: from Spotify, memory-mapped so an index larger than RAM can be searched.

HNSW builds a layered graph of neighbours and walks it from a coarse layer down to a fine one. It typically achieves 95 to 99% recall of the true nearest neighbours at a hundred times the speed of brute force. That recall figure is a parameter you set, and it is worth measuring on your own data rather than accepting the default, because “the tenth result is sometimes wrong” matters more for some products than others.

Chunking, which decides more than the model

For documents longer than the model’s input limit (often 256 or 512 tokens) you must split before embedding. This choice affects retrieval quality more than the choice of model, and it is usually made carelessly.

• Too large, and one vector averages several topics, so it matches everything weakly and nothing strongly.
• Too small, and the chunk lacks the context needed to be interpretable. A sentence beginning “This is not recommended” is useless without its antecedent.
• Overlap of 10 to 20% keeps sentences that straddle a boundary retrievable from both sides.
• Split on structure where you can (paragraphs, sections, headings) rather than on a fixed character count. A chunk that respects the document’s own boundaries is more coherent than one that ends mid-sentence.

A reasonable starting point for prose is 300 to 500 tokens with 50 tokens of overlap, split at paragraph boundaries, and then measured.

Measuring retrieval

Do not deploy on the impression that results look reasonable. Build a small evaluation set: 50 real queries with the document that should be returned for each, written by hand. Then compute recall at 5 and mean reciprocal rank.

Fifty queries takes an afternoon and turns every subsequent decision (model, chunk size, index type) into a measurement rather than an argument. This is the same discipline as the rest of this course applied to a system that superficially seems not to need it.

Where pure embedding search fails

Three failure modes worth knowing before you commit.

Exact identifiers. Searching for an order number, a part code or an error string. Embeddings represent meaning, and a part number has no meaning to represent. Keyword search wins outright.

Negation and specificity. “Restaurants without parking” embeds close to “restaurants with parking”. The vectors capture topic much better than logical structure.

Rare exact terms. A drug name appearing three times in the corpus may be poorly represented, where an inverted index finds it perfectly.

The standard answer is hybrid search: run keyword search (BM25) and embedding search in parallel and combine the rankings, usually with reciprocal rank fusion, which needs no score calibration between the two systems. Hybrid reliably beats either alone, it is not much more code, and it is what production systems do.