How many passages is the right number

The default nobody chose

Almost every retrieval-augmented system in the world uses k=5 or k=10, and almost nobody measured it. The number came from a tutorial. It is sometimes right and it is never right for a reason.

Three quantities move as k grows, and they do not move together.

Recall rises, with diminishing returns. The chance that the necessary passage is somewhere in the set goes up steeply from k=1 to k=5, then flattens. Going from 10 to 40 typically adds a couple of percentage points.

Tokens and cost rise linearly. Twenty passages of 400 tokens is 8,000 tokens, on every request, forever. At k=40 you are paying four times the retrieval budget of k=10 for those last few points of recall.

Answer accuracy rises, peaks, and falls. This is the part people do not expect. Past some point, adding more passages makes answers worse, not merely more expensive.

Why more evidence can hurt

Two mechanisms, both worth being able to name.

Distraction. Passages ranked 15 to 30 are, by construction, on-topic and not the answer. They are the most dangerous kind of wrong material, because they look exactly like what the model is looking for. Probability mass moves towards them. The model produces an answer that is a blend of the correct passage and three plausible neighbours.

Dilution and position. More material means each passage holds a smaller share of attention, and a longer context puts the true passage further from the question. The position lesson later in the course covers the shape; the effect here is that the same correct passage is less likely to be used at k=40 than at k=8.

The combined result is a curve with a peak. Where the peak sits depends on your chunk size, your reranker, your model and your task, which is why the number cannot be borrowed.

Measure it in an afternoon

for k in [1, 3, 5, 8, 12, 20, 40]:
    acc, toks = [], []
    for q, gold in eval_set:
        ctx = retrieve(q, k=k)
        toks.append(count_tokens(ctx))
        acc.append(judge(answer(q, ctx), gold))
    print(k, round(mean(acc), 3), int(mean(toks)))

A hundred questions with known answers is enough to see the shape. Plot accuracy against k and tokens against k on the same axis, and choose the point where accuracy has flattened rather than the point where it peaks: the peak is usually inside the noise, and the flat region’s left edge is cheaper and just as good.

Teams running this for the first time commonly find the answer is smaller than they were using. Going from k=10 to k=5 with a reranker in front is one of the more reliable ways to cut a bill without losing quality.

Two failures the sweep separates

While you have the harness up, record a second number: whether the gold passage was in the retrieved set at all.

That splits every failure into two kinds.

• Not retrieved. Fix retrieval: hybrid search, query rewriting, chunking, a better embedding model.
• Retrieved and not used. Fix assembly: position, ordering, delimiters, instructions, or the fact that the passage is buried in 20 others.

The ratio between them tells you which module of this course to work in next. A system with 90% retrieval recall and 60% answer accuracy has an assembly problem, and no amount of retriever tuning will move it.

k is not one number

A fixed k treats every question the same. Two refinements, both cheap:

Cut on the score gap. Retrieve 30, then keep only those within some distance of the top score. A question with one obvious answer takes three passages; a broad question takes fifteen. This adapts the budget to the query rather than to the average query.

Cut on token budget rather than count. Chunks vary in size. Ten chunks might be 2,000 tokens or 9,000. Fill to a token budget, which is what you actually care about, and let the count fall where it falls. The packing lesson later in this module has the code.

The limitation

All of this assumes your evaluation set is representative. A k sweep on 100 head queries will pick a k that is right for head queries and possibly wrong for the tail, where recall matters more because the relevant material is rarer. If your traffic has a long tail, and most does, sweep separately for the tail, or accept that you have optimised for the common case and say so.