NumPy first, FAISS second, a database rarely

The version that is enough for most projects

import numpy as np, json
from sentence_transformers import SentenceTransformer

model = SentenceTransformer("BAAI/bge-small-en-v1.5")
V = model.encode(chunks, normalize_embeddings=True, batch_size=64).astype("float32")
np.save("vectors.npy", V)
json.dump(meta, open("meta.json", "w"))          # same order as V

def search(q, k=5):
    qv = model.encode(q, normalize_embeddings=True, prompt_name="query")
    scores = V @ qv
    idx = np.argpartition(-scores, k)[:k]
    idx = idx[np.argsort(-scores[idx])]
    return [(float(scores[i]), meta[i]) for i in idx]

That is a complete semantic search engine in twelve lines, with no server and no dependency beyond NumPy. On 100,000 vectors of 384 dimensions it searches in a handful of milliseconds. On a million it is tens of milliseconds, which is still fine behind a web request.

argpartition rather than a full sort matters at scale: it finds the top k without ordering everything, and on a million items that is several times faster.

When NumPy stops being enough

Two thresholds, and they are further away than people assume:

• Memory. A million 384-dimensional float32 vectors is 1.5 GB. Ten million is 15 GB, and now you have a problem.
• Latency. Exact search is linear in corpus size. Beyond a few million, milliseconds become hundreds of milliseconds.

Below those, exact search is not only sufficient: it is better, because it is exact. Every approximate index trades recall for speed, and paying that price before you need it is a common and avoidable mistake.

FAISS, when you do cross the line

pip install faiss-cpu

import faiss

index = faiss.IndexFlatIP(V.shape[1])     # exact inner product
index.add(V)
D, I = index.search(qv.reshape(1, -1), 5)

IndexFlatIP is still exact: FAISS with a flat index is a faster matrix multiply, not an approximation, and it is a sensible first step. Approximation comes when you ask for it:

index = faiss.IndexHNSWFlat(V.shape[1], 32)   # graph index, approximate
index.hnsw.efConstruction = 200
index.add(V)
index.hnsw.efSearch = 64                      # higher = better recall, slower

HNSW builds a navigable graph and walks it. It is fast, it uses more memory than the raw vectors, and efSearch is the dial between recall and latency. Measure the recall you are losing by comparing against exact search on a few hundred queries: approximate indexes are routinely deployed with nobody knowing whether they return 99% or 80% of the right answers.

IndexIVFPQ compresses vectors as well, trading more accuracy for much less memory. It is what you reach for at tens of millions of vectors on one machine.

Vector databases, and when they are actually warranted

Qdrant, Weaviate, Milvus, Chroma, pgvector and the rest give you persistence, metadata filtering, incremental updates, concurrency and replication. Those are real features and none of them is about search quality.

Reach for one when you need:

• Filtered search: “nearest neighbours where language = 'hi' and date > 2024”. Doing this properly is harder than it looks, because filtering after retrieval can leave you with two results out of five.
• Frequent updates: documents added and deleted continuously, without rebuilding.
• More vectors than fit on one machine.
• Several services sharing one index.

If you already run PostgreSQL, pgvector is usually the right answer: your metadata and your vectors live in one place, filtering is ordinary SQL, and you add no new operational surface. That is a free option and it is under-recommended because it is not exciting.

Keyword search is not obsolete

Embeddings are bad at exact identifiers: a part number, an error code, a surname. BM25 (classic keyword ranking) handles those perfectly and runs in-process:

pip install rank_bm25

Combining both and merging the ranked lists (reciprocal rank fusion is a four-line function) reliably beats either alone, typically by a wide margin on real corpora with codes and names in them. Hybrid retrieval is not an advanced technique; it is the default that most systems should start with.

Do this now

Build the NumPy index over your own documents and search it. Then add BM25 over the same chunks and compare the top five for three queries containing a specific code or name. The gap you see is the argument for hybrid search.