{"course":{"slug":"context-engineering","title":"Context Engineering","url":"https://addaly.com/learn/context-engineering","level":"advanced","track":"build"},"lesson":{"slug":"batching-and-the-slow-lane","title":"Work that can wait","url":"https://addaly.com/learn/context-engineering/batching-and-the-slow-lane","position":46,"of":87,"minutes":8,"answer":"Batch endpoints trade latency for roughly half price, so route everything with no human waiting (ingestion, evaluation, backfills, scheduled work) through the same assembler into a batch file with a joinable id.","keyIdea":"Batch endpoints trade latency for roughly half price, so route everything with no human waiting (ingestion, evaluation, backfills, scheduled work) through the same assembler into a batch file with a joinable id, and size fast-lane concurrency against the tokens-per-minute limit rather than the request count."},"body":{"format":"markdown","text":"## Two lanes, one codebase\n\nMost providers run a batch endpoint: submit a file of requests, get results within a stated window, usually up to 24 hours, at roughly half the interactive price. Open-source serving stacks get a similar effect from larger batch sizes and better hardware utilisation when nobody is waiting.\n\nThe discount is real and the constraint is simple. **Anything with a human waiting goes in the fast lane. Everything else should not.**\n\nWhat belongs in the slow lane, in most systems:\n\n- **Ingestion.** Contextual chunk augmentation, document summaries, extracting metadata. This is the largest batch workload in a typical retrieval system and it is entirely asynchronous by nature.\n- **Evaluation runs.** A thousand test cases at half price, overnight, is the same result as a thousand at full price during the day.\n- **Backfills.** Re-processing a corpus after a prompt change or a model upgrade.\n- **Scheduled reports and digests.** Nobody is watching at 04:00.\n- **Offline classification** of yesterday's tickets, messages or logs.\n\nFor a system with a large corpus, moving ingestion to batch can halve the total bill on its own, because ingestion is often a bigger token consumer than serving.\n\n## Design so one function serves both\n\nThe mistake is writing the batch path as a separate program. It drifts: the prompt is updated in one place and not the other, and six weeks later your index was built with a different instruction than your evaluation assumes.\n\nThe pure assembler from module one is what makes this easy. `build_context()` produces a request object; a thin adapter either sends it now or serialises it into a batch file.\n\n```python\nreq = build_context(inputs, cfg)          # identical either way\nif mode == \"interactive\":\n    return client.send(req)\nbatch_file.write(json.dumps({\"custom_id\": inputs.id, **req.as_dict()}) + \"\\n\")\n```\n\nThe `custom_id` is the part people forget. Batch results come back unordered and sometimes partial, so every request needs an id you can join on, and the joining code needs to handle rows that are missing entirely.\n\n## What batch does not give you\n\nThree honest limitations.\n\n**No latency guarantee inside the window.** \"Within 24 hours\" means it may take 24 hours. Anything on a deadline (a nightly job that must finish before the business day) needs either a large margin or the interactive lane.\n\n**Partial failure is normal.** Some rows fail for reasons unrelated to your input: capacity, transient errors, a single malformed request. Your consumer must be idempotent and able to re-submit the missing ids without redoing the rest.\n\n**Debugging is slower.** A mistake in a batch of 200,000 requests is discovered hours later, after you have paid for all of them. Always run a hundred rows interactively first, read the outputs with your own eyes, and only then submit the batch. This one habit saves more money than the discount does.\n\n## Concurrency in the fast lane\n\nSeparately from batching: when you have many independent requests and someone is waiting, the lever is concurrency, and the limit is the rate limits in both directions: requests per minute and tokens per minute.\n\nThe token limit is the one that surprises people. Ten concurrent requests of 50,000 tokens each is 500,000 tokens in flight, which exceeds many accounts' per-minute token allowance even though ten requests is a trivial request count. You get throttled by a limit you were not watching.\n\nHandle it with a bounded worker pool sized against the token limit rather than the request limit, and with retry that respects the `retry-after` header rather than hammering. Exponential backoff with jitter, and a cap on total attempts, because retrying a request that failed for being too large will fail identically forever.\n\n## A worked example of the saving\n\nA corpus of 40,000 documents, averaging 30 chunks each, with a contextual sentence generated per chunk. That is 1.2 million small-model calls. At interactive rates with the document cached, call it a few hundred pounds; through the batch endpoint, roughly half that, and the difference is larger again on the first full build before caching is warm.\n\nThe re-run is where it compounds. Every change to the augmentation prompt, every new extraction field, every embedding-model upgrade means reprocessing the corpus. A team that does this quarterly and runs it interactively is paying the premium four times a year for work nobody is waiting on.\n\n## The decision, in one question\n\nFor any model call in your system, ask: *is a person waiting for this specific result?*\n\nIf no, it belongs in the slow lane and you are currently paying twice what you need to. If yes, then latency work applies and batching does not. Most teams have more work in the first category than they realise, and the reason it is in the fast lane is usually that it was written there first and nobody revisited it."},"check":{"question":"A team runs ten concurrent requests of 50,000 tokens each and starts receiving throttling errors, despite being far below their requests-per-minute limit. Why?","options":[{"text":"Concurrent requests share a prefix cache entry, and contention on it causes rejections.","correct":false,"tempting":"Invents a contention mechanism; concurrent reads of a cache entry are fine and would not produce throttling."},{"text":"Long inputs are queued behind batch traffic, which has priority during busy periods.","correct":false,"tempting":"Reverses the priority: batch work is deprioritised, which is why it is cheaper."},{"text":"The tokens-per-minute limit is being exceeded, which a low request count does not protect against.","correct":true},{"text":"Requests above a size threshold are automatically routed to the batch endpoint.","correct":false,"tempting":"No such automatic routing exists; batch submission is an explicit, separate call."}]},"prev":{"slug":"two-more-caches","title":"The other two caches, and the dangerous one","url":"https://addaly.com/learn/context-engineering/two-more-caches"},"next":{"slug":"routing-by-context-size","title":"Not every request needs the big model","url":"https://addaly.com/learn/context-engineering/routing-by-context-size"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Work that can wait', in Context Engineering. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/context-engineering/batching-and-the-slow-lane (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Work that can wait","courseTitle":"Context Engineering","module":{"index":5,"slug":"cost-cache-and-latency","title":"Cost, caching and latency"},"answer":null,"keyIdea":"Batch endpoints trade latency for roughly half price, so route everything with no human waiting (ingestion, evaluation, backfills, scheduled work) through the same assembler into a batch file with a joinable id, and size fast-lane concurrency against the tokens-per-minute limit rather than the request count.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"two-lanes-one-codebase","runs":[{"text":"Two lanes, one codebase"}]},{"t":"p","runs":[{"text":"Most providers run a batch endpoint: submit a file of requests, get results within a stated window, usually up to 24 hours, at roughly half the interactive price. Open-source serving stacks get a similar effect from larger batch sizes and better hardware utilisation when nobody is waiting."}]},{"t":"p","runs":[{"text":"The discount is real and the constraint is simple. "},{"text":"Anything with a human waiting goes in the fast lane. Everything else should not.","b":true}]},{"t":"p","runs":[{"text":"What belongs in the slow lane, in most systems:"}]},{"t":"ul","items":[[{"text":"Ingestion.","b":true},{"text":" Contextual chunk augmentation, document summaries, extracting metadata. This is the largest batch workload in a typical retrieval system and it is entirely asynchronous by nature."}],[{"text":"Evaluation runs.","b":true},{"text":" A thousand test cases at half price, overnight, is the same result as a thousand at full price during the day."}],[{"text":"Backfills.","b":true},{"text":" Re-processing a corpus after a prompt change or a model upgrade."}],[{"text":"Scheduled reports and digests.","b":true},{"text":" Nobody is watching at 04:00."}],[{"text":"Offline classification","b":true},{"text":" of yesterday’s tickets, messages or logs."}]]},{"t":"p","runs":[{"text":"For a system with a large corpus, moving ingestion to batch can halve the total bill on its own, because ingestion is often a bigger token consumer than serving."}]},{"t":"h2","id":"design-so-one-function-serves-both","runs":[{"text":"Design so one function serves both"}]},{"t":"p","runs":[{"text":"The mistake is writing the batch path as a separate program. It drifts: the prompt is updated in one place and not the other, and six weeks later your index was built with a different instruction than your evaluation assumes."}]},{"t":"p","runs":[{"text":"The pure assembler from module one is what makes this easy. "},{"text":"build_context()","code":true},{"text":" produces a request object; a thin adapter either sends it now or serialises it into a batch file."}]},{"t":"code","lang":"python","label":"Python","text":"req = build_context(inputs, cfg)          # identical either way\nif mode == \"interactive\":\n    return client.send(req)\nbatch_file.write(json.dumps({\"custom_id\": inputs.id, **req.as_dict()}) + \"\\n\")"},{"t":"p","runs":[{"text":"The "},{"text":"custom_id","code":true},{"text":" is the part people forget. Batch results come back unordered and sometimes partial, so every request needs an id you can join on, and the joining code needs to handle rows that are missing entirely."}]},{"t":"h2","id":"what-batch-does-not-give-you","runs":[{"text":"What batch does not give you"}]},{"t":"p","runs":[{"text":"Three honest limitations."}]},{"t":"p","runs":[{"text":"No latency guarantee inside the window.","b":true},{"text":" “Within 24 hours” means it may take 24 hours. Anything on a deadline (a nightly job that must finish before the business day) needs either a large margin or the interactive lane."}]},{"t":"p","runs":[{"text":"Partial failure is normal.","b":true},{"text":" Some rows fail for reasons unrelated to your input: capacity, transient errors, a single malformed request. Your consumer must be idempotent and able to re-submit the missing ids without redoing the rest."}]},{"t":"p","runs":[{"text":"Debugging is slower.","b":true},{"text":" A mistake in a batch of 200,000 requests is discovered hours later, after you have paid for all of them. Always run a hundred rows interactively first, read the outputs with your own eyes, and only then submit the batch. This one habit saves more money than the discount does."}]},{"t":"h2","id":"concurrency-in-the-fast-lane","runs":[{"text":"Concurrency in the fast lane"}]},{"t":"p","runs":[{"text":"Separately from batching: when you have many independent requests and someone is waiting, the lever is concurrency, and the limit is the rate limits in both directions: requests per minute and tokens per minute."}]},{"t":"p","runs":[{"text":"The token limit is the one that surprises people. Ten concurrent requests of 50,000 tokens each is 500,000 tokens in flight, which exceeds many accounts’ per-minute token allowance even though ten requests is a trivial request count. You get throttled by a limit you were not watching."}]},{"t":"p","runs":[{"text":"Handle it with a bounded worker pool sized against the token limit rather than the request limit, and with retry that respects the "},{"text":"retry-after","code":true},{"text":" header rather than hammering. Exponential backoff with jitter, and a cap on total attempts, because retrying a request that failed for being too large will fail identically forever."}]},{"t":"h2","id":"a-worked-example-of-the-saving","runs":[{"text":"A worked example of the saving"}]},{"t":"p","runs":[{"text":"A corpus of 40,000 documents, averaging 30 chunks each, with a contextual sentence generated per chunk. That is 1.2 million small-model calls. At interactive rates with the document cached, call it a few hundred pounds; through the batch endpoint, roughly half that, and the difference is larger again on the first full build before caching is warm."}]},{"t":"p","runs":[{"text":"The re-run is where it compounds. Every change to the augmentation prompt, every new extraction field, every embedding-model upgrade means reprocessing the corpus. A team that does this quarterly and runs it interactively is paying the premium four times a year for work nobody is waiting on."}]},{"t":"h2","id":"the-decision-in-one-question","runs":[{"text":"The decision, in one question"}]},{"t":"p","runs":[{"text":"For any model call in your system, ask: "},{"text":"is a person waiting for this specific result?","i":true}]},{"t":"p","runs":[{"text":"If no, it belongs in the slow lane and you are currently paying twice what you need to. If yes, then latency work applies and batching does not. Most teams have more work in the first category than they realise, and the reason it is in the fast lane is usually that it was written there first and nobody revisited it."}]}],"check":{"question":[{"text":"A team runs ten concurrent requests of 50,000 tokens each and starts receiving throttling errors, despite being far below their requests-per-minute limit. Why?"}],"options":[{"text":[{"text":"Concurrent requests share a prefix cache entry, and contention on it causes rejections."}],"correct":false,"tempting":[{"text":"Invents a contention mechanism; concurrent reads of a cache entry are fine and would not produce throttling."}]},{"text":[{"text":"Long inputs are queued behind batch traffic, which has priority during busy periods."}],"correct":false,"tempting":[{"text":"Reverses the priority: batch work is deprioritised, which is why it is cheaper."}]},{"text":[{"text":"The tokens-per-minute limit is being exceeded, which a low request count does not protect against."}],"correct":true,"tempting":null},{"text":[{"text":"Requests above a size threshold are automatically routed to the batch endpoint."}],"correct":false,"tempting":[{"text":"No such automatic routing exists; batch submission is an explicit, separate call."}]}],"order":[1,0,2,3],"why":null},"prev":{"slug":"two-more-caches","title":"The other two caches, and the dangerous one","minutes":9},"next":{"slug":"routing-by-context-size","title":"Not every request needs the big model","minutes":8}}}