What an embedding actually is

The object itself

An embedding model takes a piece of text and returns a fixed-length list of floating-point numbers: 384 of them for a small model, 1,536 or 3,072 for a large one. Nothing about any individual number means anything. The only thing that carries meaning is distance: text about similar things lands close together, text about different things lands apart.

You can see this on a laptop in about five minutes, with no GPU and no account:

from sentence_transformers import SentenceTransformer
m = SentenceTransformer("all-MiniLM-L6-v2")     # 384 dims, ~80 MB
v = m.encode(["cancel my order", "I want a refund", "the roof is leaking"])
print(v.shape)                                   # (3, 384)

Take cosine similarity of the pairs and the first two will score high and the third low. That is the entire idea, and everything built on embeddings (semantic search, clustering, recommendation, deduplication candidates, classification by nearest neighbour) is that idea with plumbing around it.

Cosine similarity, and the number that means nothing

Cosine similarity is the dot product of the two vectors divided by their lengths: 1.0 for identical direction, 0 for unrelated, negative for opposed. In practice, with most modern models, everything lands in a narrow band: a great many unrelated pairs score 0.3 to 0.5, and 0.8 can be either a strong match or a coincidence.

The important consequence: a threshold you found on one model does not transfer to another. The vectors live in a different space with a different scale, so 0.82 is not a level of similarity, it is a reading on an instrument you have not calibrated. Calibrate by sampling: take two hundred pairs from your own data, label them, and look at where the distributions of matching and non-matching pairs overlap. That overlap, not a number from a blog post, is your threshold.

The same fact explains a bigger operational point. If you change embedding model, every stored vector is worthless: you cannot compare old vectors with new ones, so you must re-encode the entire corpus. Store the model name and version alongside every vector, and treat a model change as a migration with a backfill, not a config edit.

Chunking is the decision that matters most

The unit you embed is the unit you retrieve. Embed a whole 40-page manual and you get one vector averaging everything it says, which is close to nothing. Embed single sentences and you retrieve fragments with no context: “This is not permitted under clause 4” without clause 4.

A reasonable starting point is 200 to 500 words with a sentence or two of overlap, split on real boundaries (headings, paragraphs, list items) rather than at a fixed character count that cuts a table in half. Keep a pointer from every chunk to its parent document, and give the agent a tool to fetch the surrounding region. Retrieve small, read large.

Queries and passages are not the same kind of text

A question (“how do I cancel a subscription”) and the passage that answers it (“Subscriptions may be terminated by written notice…”) do not look alike, so a model trained only to match similar sentences handles the pairing badly. Models trained for retrieval (the E5 and BGE families, both free on Hugging Face) expect a prefix, typically query: on the question and passage: on the document.

Omitting those prefixes does not raise an error. It quietly costs you accuracy, and it is one of the most common silent misconfigurations in retrieval systems.

You probably do not need a vector database

Fifty thousand chunks at 384 dimensions in 32-bit floats is 50,000 × 384 × 4 bytes, about 77 MB. That fits in memory on any machine, and searching it is one matrix multiplication:

import numpy as np
scores = matrix @ query_vec        # (50000,) — a few milliseconds
top = np.argsort(-scores)[:10]

That is the whole search engine. sqlite-vec adds persistence in a single file, FAISS adds approximate search when you reach millions, and pgvector puts it beside the rest of your data in PostgreSQL, which is usually the right answer for a business system because the filters and the permissions are already there. Reach for a dedicated vector service when you have measured that you need one.

What embeddings cannot do

Four hard limits, each with a mechanism worth knowing.

• Negation. “Invoices that are not paid” sits near “invoices that are paid”, because a short negation word barely moves a vector summarising the topic. Use a filter on a field, not a sentence.
• Numbers and quantities. “Over £10,000” is not distinguished from “over £100”. Filter in SQL.
• Exact identity. As in the deduplication lesson: near-identical strings are what embeddings are built to collapse.
• Recency and authority. A vector has no idea which of two documents supersedes the other. That has to come from metadata you attach and rank on yourself.

Each of these is fixed the same way: keep structured fields beside the vector, and let the vector do the one job it is good at.