Packing the budget

Fill by tokens, not by count

The retriever hands you 40 ranked candidates. The budget says 12,000 tokens. Chunks are between 180 and 900 tokens because real documents are uneven. Taking “the top ten” gives you anywhere from 1,800 to 9,000 tokens, which means your budget is being set by document formatting rather than by you.

The fix is a few lines and it is the correct default:

def pack(candidates, budget, count_fn):
    out, used = [], 0
    for c in candidates:                # already ranked
        n = count_fn(c.header + c.text)
        if used + n > budget:
            continue                    # skip, keep trying smaller ones
        out.append(c); used += n
    return out, used

Note the continue rather than break. Skipping an oversized chunk and continuing down the list uses the budget better than stopping at the first thing that does not fit. It also introduces a subtle bias towards short chunks, which is usually acceptable and occasionally not: if your most valuable material is systematically long, cap chunk size at ingestion instead of relying on the packer to be fair.

Never include a partial chunk. Half a passage is a passage that ends mid-sentence, and a model reading a truncated clause will complete it from imagination. If it does not fit, it does not go in.

Diversity: stop packing eight versions of one answer

Ranking by relevance alone packs the window with whatever the retriever liked most, which is often eight passages saying nearly the same thing. Maximal marginal relevance is the standard fix: at each step, choose the candidate that is most relevant to the query and least similar to what you have already chosen.

def mmr(cands, qvec, k, lam=0.7):
    chosen = []
    while cands and len(chosen) < k:
        best = max(cands, key=lambda c:
            lam * cos(c.vec, qvec)
            - (1 - lam) * max([cos(c.vec, s.vec) for s in chosen], default=0))
        chosen.append(best); cands.remove(best)
    return chosen

lam at 1.0 is pure relevance; at 0.5 diversity dominates. Start at 0.7 and tune it on your evaluation set: this is a parameter that differs by corpus, and it matters most where your corpus has heavy duplication.

The honest limitation: MMR optimises for coverage of embedding space, which is a proxy for coverage of topics, which is a proxy for coverage of the facts the answer needs. Two of those steps can fail. It is a better default than pure relevance and it is not a guarantee.

Pin what must be there

Some material is not competing for a slot. The document the user just uploaded and asked about. The row for the order they named. The specific clause the previous turn referred to.

Reserve for it first, subtract from the budget, then pack the rest:

budget = RETRIEVED_BUDGET - count(pinned)
chosen, used = pack(candidates, budget, count)
context = pinned + chosen

The failure this prevents is one users notice immediately and describe as the assistant being stupid: they attach a file, ask about it, and the retriever ranks three help-centre pages above their own document because those pages match the question wording better. Pinning is the difference between a system that feels attentive and one that feels deaf.

The knapsack refinement, and when to skip it

Greedy fill by rank is not optimal. The optimal packing maximises total value for a token cost, which is a knapsack problem, and if you have a per-candidate value estimate (a reranker score is the usual one) you can do better by ordering on value per token rather than value alone.

candidates.sort(key=lambda c: c.score / max(count(c), 1), reverse=True)

This is worth trying and worth measuring, because it has a known failure: it favours short chunks, and a short chunk with a high score is often a heading or a fragment rather than a substantive passage. Measure it against plain ranked fill before adopting it. On most corpora the difference is small; on corpora with very uneven chunk sizes it is not.

Order after packing, not during

One discipline that saves confusion later: packing decides what, ordering decides where. Keep them as separate functions.

The packer works down a relevance-ranked list because that is the right way to choose. The final order in the window is a different question with a different answer (sometimes best-first, sometimes best-last, sometimes chronological) and it is the subject of a whole module later. If your packer also writes the final sequence, you cannot change the ordering without touching selection, and every ordering experiment becomes a selection experiment too.

Log the packing decision

Emit, per request: how many candidates were considered, how many were packed, how many tokens used against the budget, and which were skipped for size. Two failure signatures show up immediately in that data: a system where the packer almost always exhausts its budget is one where a bigger budget would probably help, and a system that regularly skips its top-ranked candidate for being too long has a chunking problem upstream, not a packing one.