{"course":{"slug":"context-engineering","title":"Context Engineering","url":"https://addaly.com/learn/context-engineering","level":"advanced","track":"build"},"lesson":{"slug":"packing-to-a-budget","title":"Packing the budget","url":"https://addaly.com/learn/context-engineering/packing-to-a-budget","position":19,"of":87,"minutes":9,"answer":"Pack the retrieved budget in tokens rather than in chunk counts, never include a partial chunk, pin the material the user explicitly referred to before anything competes for space.","keyIdea":"Pack the retrieved budget in tokens rather than in chunk counts, never include a partial chunk, pin the material the user explicitly referred to before anything competes for space, and keep packing separate from ordering so each can be changed alone."},"body":{"format":"markdown","text":"## Fill by tokens, not by count\n\nThe retriever hands you 40 ranked candidates. The budget says 12,000 tokens. Chunks are between 180 and 900 tokens because real documents are uneven. Taking \"the top ten\" gives you anywhere from 1,800 to 9,000 tokens, which means your budget is being set by document formatting rather than by you.\n\nThe fix is a few lines and it is the correct default:\n\n```python\ndef pack(candidates, budget, count_fn):\n    out, used = [], 0\n    for c in candidates:                # already ranked\n        n = count_fn(c.header + c.text)\n        if used + n > budget:\n            continue                    # skip, keep trying smaller ones\n        out.append(c); used += n\n    return out, used\n```\n\nNote the `continue` rather than `break`. Skipping an oversized chunk and continuing down the list uses the budget better than stopping at the first thing that does not fit. It also introduces a subtle bias towards short chunks, which is usually acceptable and occasionally not: if your most valuable material is systematically long, cap chunk size at ingestion instead of relying on the packer to be fair.\n\n**Never include a partial chunk.** Half a passage is a passage that ends mid-sentence, and a model reading a truncated clause will complete it from imagination. If it does not fit, it does not go in.\n\n## Diversity: stop packing eight versions of one answer\n\nRanking by relevance alone packs the window with whatever the retriever liked most, which is often eight passages saying nearly the same thing. Maximal marginal relevance is the standard fix: at each step, choose the candidate that is most relevant to the query *and* least similar to what you have already chosen.\n\n```python\ndef mmr(cands, qvec, k, lam=0.7):\n    chosen = []\n    while cands and len(chosen) < k:\n        best = max(cands, key=lambda c:\n            lam * cos(c.vec, qvec)\n            - (1 - lam) * max([cos(c.vec, s.vec) for s in chosen], default=0))\n        chosen.append(best); cands.remove(best)\n    return chosen\n```\n\n`lam` at 1.0 is pure relevance; at 0.5 diversity dominates. Start at 0.7 and tune it on your evaluation set: this is a parameter that differs by corpus, and it matters most where your corpus has heavy duplication.\n\nThe honest limitation: MMR optimises for coverage of embedding space, which is a proxy for coverage of topics, which is a proxy for coverage of the facts the answer needs. Two of those steps can fail. It is a better default than pure relevance and it is not a guarantee.\n\n## Pin what must be there\n\nSome material is not competing for a slot. The document the user just uploaded and asked about. The row for the order they named. The specific clause the previous turn referred to.\n\nReserve for it first, subtract from the budget, then pack the rest:\n\n> Figure: Filling a retrieved budget, in order. Steps: 1. Reserve the pinned material (The file just attached, the order named, the clause the last turn referred to); 2. Subtract it from the budget (What is left is what the ranked candidates may compete for); 3. Walk the ranked list (Skip a chunk that does not fit, then carry on down rather than stopping); 4. Never cut a chunk in half (A passage ending mid-clause is one the model finishes from imagination); 5. Stop on tokens, not on count (Ten real chunks are anywhere between 1,800 and 9,000 tokens); 6. Log what was skipped (Candidates seen, tokens used against budget, and every chunk dropped for size). Packing decides what goes in; ordering decides where it sits. Keep them as two functions so either can be changed without touching the other.\n\n```python\nbudget = RETRIEVED_BUDGET - count(pinned)\nchosen, used = pack(candidates, budget, count)\ncontext = pinned + chosen\n```\n\nThe failure this prevents is one users notice immediately and describe as the assistant being stupid: they attach a file, ask about it, and the retriever ranks three help-centre pages above their own document because those pages match the question wording better. Pinning is the difference between a system that feels attentive and one that feels deaf.\n\n## The knapsack refinement, and when to skip it\n\nGreedy fill by rank is not optimal. The optimal packing maximises total value for a token cost, which is a knapsack problem, and if you have a per-candidate value estimate (a reranker score is the usual one) you can do better by ordering on value per token rather than value alone.\n\n```python\ncandidates.sort(key=lambda c: c.score / max(count(c), 1), reverse=True)\n```\n\nThis is worth trying and worth measuring, because it has a known failure: it favours short chunks, and a short chunk with a high score is often a heading or a fragment rather than a substantive passage. Measure it against plain ranked fill before adopting it. On most corpora the difference is small; on corpora with very uneven chunk sizes it is not.\n\n## Order after packing, not during\n\nOne discipline that saves confusion later: packing decides *what*, ordering decides *where*. Keep them as separate functions.\n\nThe packer works down a relevance-ranked list because that is the right way to choose. The final order in the window is a different question with a different answer (sometimes best-first, sometimes best-last, sometimes chronological) and it is the subject of a whole module later. If your packer also writes the final sequence, you cannot change the ordering without touching selection, and every ordering experiment becomes a selection experiment too.\n\n## Log the packing decision\n\nEmit, per request: how many candidates were considered, how many were packed, how many tokens used against the budget, and which were skipped for size. Two failure signatures show up immediately in that data: a system where the packer almost always exhausts its budget is one where a bigger budget would probably help, and a system that regularly skips its top-ranked candidate for being too long has a chunking problem upstream, not a packing one."},"check":{"question":"A user uploads a document and asks a question about it. The assistant answers from generic help-centre pages instead. Retrieval is working correctly by its own metrics. What is missing?","options":[{"text":"The uploaded document was never pinned, so it competed on relevance score with pages whose wording matches the question better.","correct":true},{"text":"The upload was not embedded with the same model as the corpus, making its scores incomparable.","correct":false,"tempting":"A real hazard when mixing embedding models, but it would produce erratic scores rather than a consistent loss to on-topic help pages."},{"text":"The packer stopped at the first oversized chunk, so the document never got a chance to be considered.","correct":false,"tempting":"Describes a `break`-instead-of-`continue` bug that would drop material unpredictably, not specifically the user's own file."},{"text":"Help-centre pages are shorter, so value-per-token ordering favoured them.","correct":false,"tempting":"Assumes a knapsack refinement that most pipelines do not use, and would not by itself exclude the document entirely."}]},"prev":{"slug":"summarise-or-include","title":"The source or a summary of it","url":"https://addaly.com/learn/context-engineering/summarise-or-include"},"next":{"slug":"saying-what-was-left-out","title":"Admitting the set is partial","url":"https://addaly.com/learn/context-engineering/saying-what-was-left-out"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Packing the budget', in Context Engineering. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/context-engineering/packing-to-a-budget (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Packing the budget","courseTitle":"Context Engineering","module":{"index":2,"slug":"what-earns-a-place","title":"Selection: what earns a place"},"answer":null,"keyIdea":"Pack the retrieved budget in tokens rather than in chunk counts, never include a partial chunk, pin the material the user explicitly referred to before anything competes for space, and keep packing separate from ordering so each can be changed alone.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"type":"flow","steps":[{"label":"Reserve the pinned material","detail":"The file just attached, the order named, the clause the last turn referred to"},{"label":"Subtract it from the budget","detail":"What is left is what the ranked candidates may compete for"},{"label":"Walk the ranked list","detail":"Skip a chunk that does not fit, then carry on down rather than stopping"},{"label":"Never cut a chunk in half","detail":"A passage ending mid-clause is one the model finishes from imagination"},{"label":"Stop on tokens, not on count","detail":"Ten real chunks are anywhere between 1,800 and 9,000 tokens"},{"label":"Log what was skipped","detail":"Candidates seen, tokens used against budget, and every chunk dropped for size"}],"title":"Filling a retrieved budget, in order","caption":"Packing decides what goes in; ordering decides where it sits. Keep them as two functions so either can be changed without touching the other."},"alt":"Figure: Filling a retrieved budget, in order. Steps: 1. Reserve the pinned material (The file just attached, the order named, the clause the last turn referred to); 2. Subtract it from the budget (What is left is what the ranked candidates may compete for); 3. Walk the ranked list (Skip a chunk that does not fit, then carry on down rather than stopping); 4. Never cut a chunk in half (A passage ending mid-clause is one the model finishes from imagination); 5. Stop on tokens, not on count (Ten real chunks are anywhere between 1,800 and 9,000 tokens); 6. Log what was skipped (Candidates seen, tokens used against budget, and every chunk dropped for size). Packing decides what goes in; ordering decides where it sits. Keep them as two functions so either can be changed without touching the other."}},"blocks":[{"t":"h2","id":"fill-by-tokens-not-by-count","runs":[{"text":"Fill by tokens, not by count"}]},{"t":"p","runs":[{"text":"The retriever hands you 40 ranked candidates. The budget says 12,000 tokens. Chunks are between 180 and 900 tokens because real documents are uneven. Taking “the top ten” gives you anywhere from 1,800 to 9,000 tokens, which means your budget is being set by document formatting rather than by you."}]},{"t":"p","runs":[{"text":"The fix is a few lines and it is the correct default:"}]},{"t":"code","lang":"python","label":"Python","text":"def pack(candidates, budget, count_fn):\n    out, used = [], 0\n    for c in candidates:                # already ranked\n        n = count_fn(c.header + c.text)\n        if used + n > budget:\n            continue                    # skip, keep trying smaller ones\n        out.append(c); used += n\n    return out, used"},{"t":"p","runs":[{"text":"Note the "},{"text":"continue","code":true},{"text":" rather than "},{"text":"break","code":true},{"text":". Skipping an oversized chunk and continuing down the list uses the budget better than stopping at the first thing that does not fit. It also introduces a subtle bias towards short chunks, which is usually acceptable and occasionally not: if your most valuable material is systematically long, cap chunk size at ingestion instead of relying on the packer to be fair."}]},{"t":"p","runs":[{"text":"Never include a partial chunk.","b":true},{"text":" Half a passage is a passage that ends mid-sentence, and a model reading a truncated clause will complete it from imagination. If it does not fit, it does not go in."}]},{"t":"h2","id":"diversity-stop-packing-eight-versions-of-one-answer","runs":[{"text":"Diversity: stop packing eight versions of one answer"}]},{"t":"p","runs":[{"text":"Ranking by relevance alone packs the window with whatever the retriever liked most, which is often eight passages saying nearly the same thing. Maximal marginal relevance is the standard fix: at each step, choose the candidate that is most relevant to the query "},{"text":"and","i":true},{"text":" least similar to what you have already chosen."}]},{"t":"code","lang":"python","label":"Python","text":"def mmr(cands, qvec, k, lam=0.7):\n    chosen = []\n    while cands and len(chosen) < k:\n        best = max(cands, key=lambda c:\n            lam * cos(c.vec, qvec)\n            - (1 - lam) * max([cos(c.vec, s.vec) for s in chosen], default=0))\n        chosen.append(best); cands.remove(best)\n    return chosen"},{"t":"p","runs":[{"text":"lam","code":true},{"text":" at 1.0 is pure relevance; at 0.5 diversity dominates. Start at 0.7 and tune it on your evaluation set: this is a parameter that differs by corpus, and it matters most where your corpus has heavy duplication."}]},{"t":"p","runs":[{"text":"The honest limitation: MMR optimises for coverage of embedding space, which is a proxy for coverage of topics, which is a proxy for coverage of the facts the answer needs. Two of those steps can fail. It is a better default than pure relevance and it is not a guarantee."}]},{"t":"h2","id":"pin-what-must-be-there","runs":[{"text":"Pin what must be there"}]},{"t":"p","runs":[{"text":"Some material is not competing for a slot. The document the user just uploaded and asked about. The row for the order they named. The specific clause the previous turn referred to."}]},{"t":"p","runs":[{"text":"Reserve for it first, subtract from the budget, then pack the rest:"}]},{"t":"code","lang":"python","label":"Python","text":"budget = RETRIEVED_BUDGET - count(pinned)\nchosen, used = pack(candidates, budget, count)\ncontext = pinned + chosen"},{"t":"p","runs":[{"text":"The failure this prevents is one users notice immediately and describe as the assistant being stupid: they attach a file, ask about it, and the retriever ranks three help-centre pages above their own document because those pages match the question wording better. Pinning is the difference between a system that feels attentive and one that feels deaf."}]},{"t":"h2","id":"the-knapsack-refinement-and-when-to-skip-it","runs":[{"text":"The knapsack refinement, and when to skip it"}]},{"t":"p","runs":[{"text":"Greedy fill by rank is not optimal. The optimal packing maximises total value for a token cost, which is a knapsack problem, and if you have a per-candidate value estimate (a reranker score is the usual one) you can do better by ordering on value per token rather than value alone."}]},{"t":"code","lang":"python","label":"Python","text":"candidates.sort(key=lambda c: c.score / max(count(c), 1), reverse=True)"},{"t":"p","runs":[{"text":"This is worth trying and worth measuring, because it has a known failure: it favours short chunks, and a short chunk with a high score is often a heading or a fragment rather than a substantive passage. Measure it against plain ranked fill before adopting it. On most corpora the difference is small; on corpora with very uneven chunk sizes it is not."}]},{"t":"h2","id":"order-after-packing-not-during","runs":[{"text":"Order after packing, not during"}]},{"t":"p","runs":[{"text":"One discipline that saves confusion later: packing decides "},{"text":"what","i":true},{"text":", ordering decides "},{"text":"where","i":true},{"text":". Keep them as separate functions."}]},{"t":"p","runs":[{"text":"The packer works down a relevance-ranked list because that is the right way to choose. The final order in the window is a different question with a different answer (sometimes best-first, sometimes best-last, sometimes chronological) and it is the subject of a whole module later. If your packer also writes the final sequence, you cannot change the ordering without touching selection, and every ordering experiment becomes a selection experiment too."}]},{"t":"h2","id":"log-the-packing-decision","runs":[{"text":"Log the packing decision"}]},{"t":"p","runs":[{"text":"Emit, per request: how many candidates were considered, how many were packed, how many tokens used against the budget, and which were skipped for size. Two failure signatures show up immediately in that data: a system where the packer almost always exhausts its budget is one where a bigger budget would probably help, and a system that regularly skips its top-ranked candidate for being too long has a chunking problem upstream, not a packing one."}]}],"check":{"question":[{"text":"A user uploads a document and asks a question about it. The assistant answers from generic help-centre pages instead. Retrieval is working correctly by its own metrics. What is missing?"}],"options":[{"text":[{"text":"The uploaded document was never pinned, so it competed on relevance score with pages whose wording matches the question better."}],"correct":true,"tempting":null},{"text":[{"text":"The upload was not embedded with the same model as the corpus, making its scores incomparable."}],"correct":false,"tempting":[{"text":"A real hazard when mixing embedding models, but it would produce erratic scores rather than a consistent loss to on-topic help pages."}]},{"text":[{"text":"The packer stopped at the first oversized chunk, so the document never got a chance to be considered."}],"correct":false,"tempting":[{"text":"Describes a "},{"text":"break","code":true},{"text":"-instead-of-"},{"text":"continue","code":true},{"text":" bug that would drop material unpredictably, not specifically the user’s own file."}]},{"text":[{"text":"Help-centre pages are shorter, so value-per-token ordering favoured them."}],"correct":false,"tempting":[{"text":"Assumes a knapsack refinement that most pipelines do not use, and would not by itself exclude the document entirely."}]}],"order":[1,0,2,3],"why":null},"prev":{"slug":"summarise-or-include","title":"The source or a summary of it","minutes":9},"next":{"slug":"saying-what-was-left-out","title":"Admitting the set is partial","minutes":8}}}