{"course":{"slug":"building-with-ai","title":"Building with AI","url":"https://addaly.com/learn/building-with-ai","level":"intermediate","track":"build"},"lesson":{"slug":"vector-search-without-a-vector-database","title":"Vector search: when NumPy is enough, and what an index trades away","url":"https://addaly.com/learn/building-with-ai/vector-search-without-a-vector-database","position":36,"of":80,"minutes":9,"answer":"Below about a million vectors a brute-force dot product on a CPU is exact and fast enough, and above it an approximate index buys speed by giving up a measurable fraction of recall.","keyIdea":"Below about a million vectors a brute-force dot product on a CPU is exact and fast enough, and above it an approximate index buys speed by giving up a measurable fraction of recall, which you must measure against exact results rather than assume away."},"body":{"format":"markdown","text":"## The baseline nobody sells\n\nA hundred thousand chunks at 768 dimensions in 32-bit floats is 300 MB. A single matrix-vector product over that on an ordinary CPU takes tens of milliseconds. The result is exact: every chunk is compared, nothing is missed.\n\n```python\nimport numpy as np\nM = np.load(\"vectors.npy\")            # shape (100_000, 768), rows normalised\ndef search(q, k=10):\n    scores = M @ q                     # one dot product per chunk\n    top = np.argpartition(-scores, k)[:k]\n    return top[np.argsort(-scores[top])]\n```\n\nThat is a complete vector search engine. No server, no index, no recall loss, no operations. A large share of internal tools (a company handbook, a product's documentation, a support archive) never grow past this, and a team that starts by deploying a vector database for 20,000 chunks has bought a year of maintenance for a problem that fits in a NumPy array.\n\n## Where it stops\n\nScale the corpus to a million chunks and the array is 3 GB, and a query takes a few hundred milliseconds of pure arithmetic. Add concurrent users and the CPU saturates. That is the point where you need an index, and the honest way to say what an index does is: it answers faster by not looking at everything.\n\n## What HNSW actually does\n\nThe most common index, Hierarchical Navigable Small World, builds a graph where every vector is linked to a handful of near neighbours, with sparser layers on top for long jumps. A query enters at the top, walks greedily towards its target, drops a layer, and repeats. It touches perhaps a few thousand vectors out of a million and returns in one to five milliseconds.\n\nThe cost is that a greedy walk can be steered into a region of the graph near the target and stop there, missing a closer vector reachable only by a path it did not take. Two parameters govern the trade: `M`, the number of links per node, and `ef_search`, the width of the search beam. Raise them and recall goes up and speed goes down. Typical settings land at 95 to 99 per cent of the exact top ten.\n\nThe other common family, IVF, clusters the vectors first and searches only the `nprobe` nearest clusters. It is simpler, uses less memory, and loses recall for vectors that sit near a cluster boundary.\n\n## Recall loss is real, and you must measure it\n\nNinety-five per cent recall sounds fine until you write out what it means: one in twenty of the relevant chunks is missing before the language model sees anything, and no prompt downstream can recover it.\n\nMeasure it. Take two hundred queries, compute the exact top ten with the brute-force search above, then ask the index for its top ten, and count the overlap. That number is your index's recall, and it goes in the same report as every other retrieval metric. If it is 0.90, you have a decision to make about `ef_search` before you tune anything else.\n\n## The options\n\n- **pgvector** in the Postgres you already run. HNSW and IVF indexes, filtering in SQL, transactions, backups you already have. Comfortable to tens of millions of vectors. Free.\n- **SQLite with `sqlite-vec`** for something that runs on a laptop or a phone. Free.\n- **FAISS**, an in-process library from Meta, for batch and offline work where you want the fastest possible search and no server. Free.\n- **Qdrant, Weaviate, Milvus, Chroma**, self-hosted. Free to run, with the usual cost of running one more service.\n- **Managed services** such as Pinecone. What you pay for is operations, not better neighbours.\n\nThe ranking of results is the same everywhere; the maths is the maths. Choose on what you already operate.\n\n## Filtering: the trap in every first system\n\nReal queries carry conditions: chunks the user is allowed to see, documents from this year, this product's manuals. There are two ways to combine a filter with a vector search and one of them is wrong.\n\nPost-filtering retrieves the top ten by similarity and then discards those that fail the condition. If the user can see two per cent of the corpus, the top ten will usually contain none of them, and the search returns nothing. Pre-filtering restricts the candidates first and then searches among them, which is correct but expensive for an approximate index, because the graph was built over everything and a walk that keeps hitting excluded nodes degrades. Modern pgvector handles this with iterative scanning, which keeps walking until it has enough results that pass; other engines integrate the filter into the traversal. Whichever you use, test it with a selective filter, because this is the case that returns an empty list in production.\n\n## Quantisation\n\nStoring each dimension as an 8-bit integer instead of a 32-bit float cuts memory four-fold; binary quantisation cuts it thirty-two-fold. Recall drops a little, and the standard remedy is to retrieve a wider set with the small vectors and re-score the candidates with the full ones. For a corpus that no longer fits in RAM this is the difference between a search that works and one that thrashes the disk.\n\n## A sizing example\n\nTwo million chunks at 1,024 dimensions: 8 GB of raw vectors, roughly 10 to 12 GB with an HNSW index, which needs to sit in memory. Quantised to 8-bit: about 3 GB. Query time: a few milliseconds indexed, half a second brute-force. Index build: tens of minutes to hours depending on `M`. Those are the numbers to have before choosing a machine."},"check":{"question":"A document search restricts results to files the current user may read, which is typically two per cent of the corpus. It retrieves the top ten by similarity and then removes files the user cannot see. Searches often return nothing. Why?","options":[{"text":"The HNSW index has too low a recall setting and keeps missing the small number of documents that happen to belong to this particular user's team.","correct":false,"tempting":"Index recall is a different, smaller loss; even a perfect index would fill the top ten with documents from the 98 per cent the user cannot see."},{"text":"Two per cent of the corpus is too little data for embeddings to work well.","correct":false,"tempting":"Mistakes an access restriction for a data-volume problem; the vectors exist and are fine."},{"text":"The filter runs after retrieval, so the top ten is drawn almost entirely from documents the user cannot see, and the filter empties it.","correct":true},{"text":"The similarity threshold is set too high for this user's documents.","correct":false,"tempting":"No threshold is involved in a top-ten search; the emptiness comes from the order of operations, not a cutoff."}]},"prev":{"slug":"choosing-an-embedding-model","title":"Choosing an embedding model: dimensions, languages and the leaderboard trap","url":"https://addaly.com/learn/building-with-ai/choosing-an-embedding-model"},"next":{"slug":"hybrid-search-keywords-still-win","title":"Hybrid search: why keyword search is still half the answer","url":"https://addaly.com/learn/building-with-ai/hybrid-search-keywords-still-win"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Vector search: when NumPy is enough, and what an index trades away', in Building with AI. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/building-with-ai/vector-search-without-a-vector-database (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Vector search: when NumPy is enough, and what an index trades away","courseTitle":"Building with AI","module":{"index":4,"slug":"retrieval-and-the-knowledge-problem","title":"Retrieval and the knowledge problem"},"answer":null,"keyIdea":"Below about a million vectors a brute-force dot product on a CPU is exact and fast enough, and above it an approximate index buys speed by giving up a measurable fraction of recall, which you must measure against exact results rather than assume away.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"the-baseline-nobody-sells","runs":[{"text":"The baseline nobody sells"}]},{"t":"p","runs":[{"text":"A hundred thousand chunks at 768 dimensions in 32-bit floats is 300 MB. A single matrix-vector product over that on an ordinary CPU takes tens of milliseconds. The result is exact: every chunk is compared, nothing is missed."}]},{"t":"code","lang":"python","label":"Python","text":"import numpy as np\nM = np.load(\"vectors.npy\")            # shape (100_000, 768), rows normalised\ndef search(q, k=10):\n    scores = M @ q                     # one dot product per chunk\n    top = np.argpartition(-scores, k)[:k]\n    return top[np.argsort(-scores[top])]"},{"t":"p","runs":[{"text":"That is a complete vector search engine. No server, no index, no recall loss, no operations. A large share of internal tools (a company handbook, a product’s documentation, a support archive) never grow past this, and a team that starts by deploying a vector database for 20,000 chunks has bought a year of maintenance for a problem that fits in a NumPy array."}]},{"t":"h2","id":"where-it-stops","runs":[{"text":"Where it stops"}]},{"t":"p","runs":[{"text":"Scale the corpus to a million chunks and the array is 3 GB, and a query takes a few hundred milliseconds of pure arithmetic. Add concurrent users and the CPU saturates. That is the point where you need an index, and the honest way to say what an index does is: it answers faster by not looking at everything."}]},{"t":"h2","id":"what-hnsw-actually-does","runs":[{"text":"What HNSW actually does"}]},{"t":"p","runs":[{"text":"The most common index, Hierarchical Navigable Small World, builds a graph where every vector is linked to a handful of near neighbours, with sparser layers on top for long jumps. A query enters at the top, walks greedily towards its target, drops a layer, and repeats. It touches perhaps a few thousand vectors out of a million and returns in one to five milliseconds."}]},{"t":"p","runs":[{"text":"The cost is that a greedy walk can be steered into a region of the graph near the target and stop there, missing a closer vector reachable only by a path it did not take. Two parameters govern the trade: "},{"text":"M","code":true},{"text":", the number of links per node, and "},{"text":"ef_search","code":true},{"text":", the width of the search beam. Raise them and recall goes up and speed goes down. Typical settings land at 95 to 99 per cent of the exact top ten."}]},{"t":"p","runs":[{"text":"The other common family, IVF, clusters the vectors first and searches only the "},{"text":"nprobe","code":true},{"text":" nearest clusters. It is simpler, uses less memory, and loses recall for vectors that sit near a cluster boundary."}]},{"t":"h2","id":"recall-loss-is-real-and-you-must-measure-it","runs":[{"text":"Recall loss is real, and you must measure it"}]},{"t":"p","runs":[{"text":"Ninety-five per cent recall sounds fine until you write out what it means: one in twenty of the relevant chunks is missing before the language model sees anything, and no prompt downstream can recover it."}]},{"t":"p","runs":[{"text":"Measure it. Take two hundred queries, compute the exact top ten with the brute-force search above, then ask the index for its top ten, and count the overlap. That number is your index’s recall, and it goes in the same report as every other retrieval metric. If it is 0.90, you have a decision to make about "},{"text":"ef_search","code":true},{"text":" before you tune anything else."}]},{"t":"h2","id":"the-options","runs":[{"text":"The options"}]},{"t":"ul","items":[[{"text":"pgvector","b":true},{"text":" in the Postgres you already run. HNSW and IVF indexes, filtering in SQL, transactions, backups you already have. Comfortable to tens of millions of vectors. Free."}],[{"text":"SQLite with ","b":true},{"text":"sqlite-vec","b":true,"code":true},{"text":" for something that runs on a laptop or a phone. Free."}],[{"text":"FAISS","b":true},{"text":", an in-process library from Meta, for batch and offline work where you want the fastest possible search and no server. Free."}],[{"text":"Qdrant, Weaviate, Milvus, Chroma","b":true},{"text":", self-hosted. Free to run, with the usual cost of running one more service."}],[{"text":"Managed services","b":true},{"text":" such as Pinecone. What you pay for is operations, not better neighbours."}]]},{"t":"p","runs":[{"text":"The ranking of results is the same everywhere; the maths is the maths. Choose on what you already operate."}]},{"t":"h2","id":"filtering-the-trap-in-every-first-system","runs":[{"text":"Filtering: the trap in every first system"}]},{"t":"p","runs":[{"text":"Real queries carry conditions: chunks the user is allowed to see, documents from this year, this product’s manuals. There are two ways to combine a filter with a vector search and one of them is wrong."}]},{"t":"p","runs":[{"text":"Post-filtering retrieves the top ten by similarity and then discards those that fail the condition. If the user can see two per cent of the corpus, the top ten will usually contain none of them, and the search returns nothing. Pre-filtering restricts the candidates first and then searches among them, which is correct but expensive for an approximate index, because the graph was built over everything and a walk that keeps hitting excluded nodes degrades. Modern pgvector handles this with iterative scanning, which keeps walking until it has enough results that pass; other engines integrate the filter into the traversal. Whichever you use, test it with a selective filter, because this is the case that returns an empty list in production."}]},{"t":"h2","id":"quantisation","runs":[{"text":"Quantisation"}]},{"t":"p","runs":[{"text":"Storing each dimension as an 8-bit integer instead of a 32-bit float cuts memory four-fold; binary quantisation cuts it thirty-two-fold. Recall drops a little, and the standard remedy is to retrieve a wider set with the small vectors and re-score the candidates with the full ones. For a corpus that no longer fits in RAM this is the difference between a search that works and one that thrashes the disk."}]},{"t":"h2","id":"a-sizing-example","runs":[{"text":"A sizing example"}]},{"t":"p","runs":[{"text":"Two million chunks at 1,024 dimensions: 8 GB of raw vectors, roughly 10 to 12 GB with an HNSW index, which needs to sit in memory. Quantised to 8-bit: about 3 GB. Query time: a few milliseconds indexed, half a second brute-force. Index build: tens of minutes to hours depending on "},{"text":"M","code":true},{"text":". Those are the numbers to have before choosing a machine."}]}],"check":{"question":[{"text":"A document search restricts results to files the current user may read, which is typically two per cent of the corpus. It retrieves the top ten by similarity and then removes files the user cannot see. Searches often return nothing. Why?"}],"options":[{"text":[{"text":"The HNSW index has too low a recall setting and keeps missing the small number of documents that happen to belong to this particular user’s team."}],"correct":false,"tempting":[{"text":"Index recall is a different, smaller loss; even a perfect index would fill the top ten with documents from the 98 per cent the user cannot see."}]},{"text":[{"text":"Two per cent of the corpus is too little data for embeddings to work well."}],"correct":false,"tempting":[{"text":"Mistakes an access restriction for a data-volume problem; the vectors exist and are fine."}]},{"text":[{"text":"The filter runs after retrieval, so the top ten is drawn almost entirely from documents the user cannot see, and the filter empties it."}],"correct":true,"tempting":null},{"text":[{"text":"The similarity threshold is set too high for this user’s documents."}],"correct":false,"tempting":[{"text":"No threshold is involved in a top-ten search; the emptiness comes from the order of operations, not a cutoff."}]}],"order":[1,0,2,3],"why":null},"prev":{"slug":"choosing-an-embedding-model","title":"Choosing an embedding model: dimensions, languages and the leaderboard trap","minutes":9},"next":{"slug":"hybrid-search-keywords-still-win","title":"Hybrid search: why keyword search is still half the answer","minutes":9}}}