Choosing an embedding model: dimensions, languages and the leaderboard trap

The menu, as it stands

Hosted: OpenAI text-embedding-3-small and -large, Cohere embed (strong on many languages), Voyage, Google’s Gemini embedding. Open weights: bge-m3 (multilingual, 8,192-token input, 1,024 dimensions), multilingual-e5-large, nomic-embed-text, the gte family, and Qwen’s embedding models. The open ones run through sentence-transformers on a CPU, through Ollama with ollama pull nomic-embed-text, or through Hugging Face’s hosted inference. The names will have moved by the time you read this; the way of choosing will not.

The leaderboard trap

MTEB, the Massive Text Embedding Benchmark, ranks models by an average over dozens of tasks (retrieval, classification, clustering, similarity) across many datasets. It is the right place to start and the wrong place to stop.

The mechanism is simple. An average over fifty tasks hides the variance on any one of them. Two models one point apart on the average can be ten points apart, in either direction, on a corpus like yours. And the datasets are public, so models get trained on data that resembles the test, which inflates scores without improving anything you will experience. The top of the table changes monthly, mostly by fractions.

Use it to shortlist three models that fit your constraints on language, size and licence. Then run the bake-off at the end of this lesson. Never choose on the average.

The limit that silently cuts your data

Every embedding model has a maximum input length, and it is usually shorter than people assume. all-MiniLM-L6-v2 truncates at 256 word pieces. Most of the bge and e5 base models take 512. bge-m3 and the hosted models take 8,192.

Truncation is silent. Send a 900-token chunk to a 256-token model and you get a perfectly good vector for the first quarter of the chunk. The remaining three-quarters is not in the index at all; a question answered in the last paragraph of that chunk will never retrieve it. Teams discover this weeks later as “retrieval is bad for some documents” when it is in fact “retrieval is bad for the ends of all documents”. Check the limit, and make your chunker respect it.

Dimensions, and the Matryoshka trick

More dimensions cost storage, memory and index-build time, not API money. They buy a little quality, with diminishing returns past a thousand or so.

Several recent models are trained with Matryoshka representation learning, which arranges the vector so that the first N dimensions are usable on their own. OpenAI reports that text-embedding-3-large cut to 256 dimensions still beats its earlier 1,536-dimension ada-002 on MTEB. If your model supports it, storing 256 or 512 dimensions and keeping the full vector only for a final re-scoring pass is a real saving on a large corpus.

Languages, and the test that matters

If your users write in Hindi, Tamil, Bengali or a mix of English and any of them, an English-only model is not a weaker choice, it is a broken one: Devanagari through MiniLM produces near-random vectors. bge-m3 and multilingual-e5 are trained on those scripts. But run one more test that no leaderboard covers: transliterated Hinglish in Latin script (“password reset kaise karu”) against English documentation. That is what a large share of Indian users actually type, and it is almost absent from training data. Some multilingual models handle it acceptably; some do not. Only your test will tell you.

Speed, and where the embedding runs

At query time a hosted call costs a network round trip, typically 50 to 150 milliseconds. A small local model on CPU answers in about five. That difference matters when retrieval is one stage in a pipeline with a latency budget.

At ingestion time the shape reverses. MiniLM on a laptop CPU embeds a few hundred chunks per second; a million-chunk corpus is an hour or two. bge-large on the same CPU manages perhaps ten to twenty a second, which is a day. That is where a GPU, or a hosted API at a fraction of a cent per thousand chunks, earns its place.

Lock-in, and the field you must store

Vectors from two different models are not comparable, even at the same dimension count. Switching models means re-embedding the entire corpus. Hosted models are retired (ada-002 was superseded) and open models get new versions.

So store the model name and version alongside every vector, keep the raw chunk text so you can always re-embed, and know the re-embed cost before you need it: five million tokens is ten cents hosted or a few hours on a CPU. That is cheap. Discovering that you deleted the source text is not.

The bake-off

Fifty real queries, each paired with the chunk that answers it. Three shortlisted models. For each, embed the corpus, run the fifty queries, and record recall at five: the fraction of queries whose answering chunk appears in the top five. One afternoon, one number per model, and a decision you can defend to anyone who asks why you did not pick the one at the top of the table.