Picking the model, and reading MTEB without being fooled

Four specifications, in order of how often they decide it

1. Maximum sequence length. Most embedding models truncate at 512 tokens, and many at 256. Text beyond that is silently discarded: the same silent truncation as module two, with the same consequence. If your chunks are longer than the limit, you are embedding their beginnings. Check model.max_seq_length and chunk to fit it.

2. Language. An English-trained model on Hindi, Bengali or Tamil produces vectors that are not random but are close to useless: everything is moderately similar to everything. The multilingual families (paraphrase-multilingual-MiniLM, multilingual-e5, BGE-M3, LaBSE, jina-embeddings-v3) are trained across many languages and, crucially, place translations of the same sentence close together. That cross-lingual property lets somebody search Hindi documents with an English query, which for a lot of this readership is the whole feature.

3. Dimensions. 384 is small and fast; 768 is the common middle; 1,024 and above are slower and heavier to store. More dimensions is not reliably better: a well-trained 384 model regularly beats a poorly trained 1,024 one. Dimensions decide cost with certainty and quality only loosely.

4. Size on disk. 90 MB to 2 GB. This decides whether it runs on a phone, in a browser or on a free Space.

MTEB, and its three traps

The Massive Text Embedding Benchmark scores models across dozens of tasks in several categories. It is the best available survey and it is a leaderboard, with everything the leaderboard lesson says about leaderboards.

Trap one: the average. The headline column averages retrieval, classification, clustering, reranking and semantic similarity. If you are building search, only the retrieval column matters, and the ordering within it differs from the overall ordering. Sort by the task you actually have.

Trap two: overfitting to the benchmark. MTEB tasks are public. Training on data that resembles them raises the score without raising real quality, and several high-ranking models have been credibly argued to do this. A model that is two points better on MTEB and was released by a team optimising for MTEB is not reliably two points better on your tickets.

Trap three: the language you need is not there. MTEB’s multilingual coverage has grown and is still thin for many Indian languages. A model absent from a language’s table is not necessarily bad at it; nobody measured.

Use MTEB to build a shortlist of three. Decide between them yourself.

The decision in practice

• English, small, fast, free everywhere: all-MiniLM-L6-v2 (90 MB, 384 dims). Still the right default for a first version, and the most-exercised embedding model on the Hub.
• English, better quality: BAAI/bge-base-en-v1.5 or gte-base. 768 dims, around 400 MB.
• Multilingual, including Indic languages: intfloat/multilingual-e5-base or BAAI/bge-m3. The latter handles 8,192 tokens and does dense and sparse retrieval together.
• In a browser or on a phone: a quantized ONNX build of MiniLM, which transformers.js loads directly.
• Long documents: a model with a long window, or chunking, and chunking is usually better.

The test that actually decides it

Thirty questions, written by you, against your own documents, each with the document you know should come back.

for name in shortlist:
    m = SentenceTransformer(name)
    D = m.encode(docs, normalize_embeddings=True)
    Q = m.encode(queries, normalize_embeddings=True, prompt_name="query")
    ranks = [(-(Q[i] @ D.T)).argsort().tolist().index(gold[i]) for i in range(len(Q))]
    print(name,
          "recall@1", sum(r == 0 for r in ranks) / len(ranks),
          "recall@5", sum(r < 5 for r in ranks) / len(ranks))

Twenty minutes, and it answers the question MTEB cannot: which of these is best on your text, in your languages, at your chunk length. Differences of five or ten points show up clearly at thirty questions. Differences of one point do not, and you should not act on them.

Cost, since people ask

Hosted embedding APIs charge per token and are convenient. A local MiniLM costs nothing after the download and runs on the hardware you have. For under a few million documents, local is cheaper, private, and fast enough, and it does not change under you when a provider deprecates a model, which is a real failure mode that invalidates an entire stored index.

Do this now

Build the thirty-question set before you pick a model. It is the artefact that makes every later decision in this module checkable, and it is the one thing a leaderboard can never give you.