Vector search: when NumPy is enough, and what an index trades away

The baseline nobody sells

A hundred thousand chunks at 768 dimensions in 32-bit floats is 300 MB. A single matrix-vector product over that on an ordinary CPU takes tens of milliseconds. The result is exact: every chunk is compared, nothing is missed.

import numpy as np
M = np.load("vectors.npy")            # shape (100_000, 768), rows normalised
def search(q, k=10):
    scores = M @ q                     # one dot product per chunk
    top = np.argpartition(-scores, k)[:k]
    return top[np.argsort(-scores[top])]

That is a complete vector search engine. No server, no index, no recall loss, no operations. A large share of internal tools (a company handbook, a product’s documentation, a support archive) never grow past this, and a team that starts by deploying a vector database for 20,000 chunks has bought a year of maintenance for a problem that fits in a NumPy array.

Where it stops

Scale the corpus to a million chunks and the array is 3 GB, and a query takes a few hundred milliseconds of pure arithmetic. Add concurrent users and the CPU saturates. That is the point where you need an index, and the honest way to say what an index does is: it answers faster by not looking at everything.

What HNSW actually does

The most common index, Hierarchical Navigable Small World, builds a graph where every vector is linked to a handful of near neighbours, with sparser layers on top for long jumps. A query enters at the top, walks greedily towards its target, drops a layer, and repeats. It touches perhaps a few thousand vectors out of a million and returns in one to five milliseconds.

The cost is that a greedy walk can be steered into a region of the graph near the target and stop there, missing a closer vector reachable only by a path it did not take. Two parameters govern the trade: M, the number of links per node, and ef_search, the width of the search beam. Raise them and recall goes up and speed goes down. Typical settings land at 95 to 99 per cent of the exact top ten.

The other common family, IVF, clusters the vectors first and searches only the nprobe nearest clusters. It is simpler, uses less memory, and loses recall for vectors that sit near a cluster boundary.

Recall loss is real, and you must measure it

Ninety-five per cent recall sounds fine until you write out what it means: one in twenty of the relevant chunks is missing before the language model sees anything, and no prompt downstream can recover it.

Measure it. Take two hundred queries, compute the exact top ten with the brute-force search above, then ask the index for its top ten, and count the overlap. That number is your index’s recall, and it goes in the same report as every other retrieval metric. If it is 0.90, you have a decision to make about ef_search before you tune anything else.

The options

• pgvector in the Postgres you already run. HNSW and IVF indexes, filtering in SQL, transactions, backups you already have. Comfortable to tens of millions of vectors. Free.
• SQLite with sqlite-vec for something that runs on a laptop or a phone. Free.
• FAISS, an in-process library from Meta, for batch and offline work where you want the fastest possible search and no server. Free.
• Qdrant, Weaviate, Milvus, Chroma, self-hosted. Free to run, with the usual cost of running one more service.
• Managed services such as Pinecone. What you pay for is operations, not better neighbours.

The ranking of results is the same everywhere; the maths is the maths. Choose on what you already operate.

Filtering: the trap in every first system

Real queries carry conditions: chunks the user is allowed to see, documents from this year, this product’s manuals. There are two ways to combine a filter with a vector search and one of them is wrong.

Post-filtering retrieves the top ten by similarity and then discards those that fail the condition. If the user can see two per cent of the corpus, the top ten will usually contain none of them, and the search returns nothing. Pre-filtering restricts the candidates first and then searches among them, which is correct but expensive for an approximate index, because the graph was built over everything and a walk that keeps hitting excluded nodes degrades. Modern pgvector handles this with iterative scanning, which keeps walking until it has enough results that pass; other engines integrate the filter into the traversal. Whichever you use, test it with a selective filter, because this is the case that returns an empty list in production.

Quantisation

Storing each dimension as an 8-bit integer instead of a 32-bit float cuts memory four-fold; binary quantisation cuts it thirty-two-fold. Recall drops a little, and the standard remedy is to retrieve a wider set with the small vectors and re-score the candidates with the full ones. For a corpus that no longer fits in RAM this is the difference between a search that works and one that thrashes the disk.

A sizing example

Two million chunks at 1,024 dimensions: 8 GB of raw vectors, roughly 10 to 12 GB with an HNSW index, which needs to sit in memory. Quantised to 8-bit: about 3 GB. Query time: a few milliseconds indexed, half a second brute-force. Index build: tens of minutes to hours depending on M. Those are the numbers to have before choosing a machine.