{"course":{"slug":"building-with-ai","title":"Building with AI","url":"https://addaly.com/learn/building-with-ai","level":"intermediate","track":"build"},"lesson":{"slug":"the-batch-window","title":"The batch window: half price for anything that can wait","url":"https://addaly.com/learn/building-with-ai/the-batch-window","position":74,"of":80,"minutes":8,"answer":"Providers sell a batch endpoint at around half the interactive price in exchange for results within a day rather than seconds, so every model call that is not a person waiting (classification, backfills, evaluation runs.","keyIdea":"Providers sell a batch endpoint at around half the interactive price in exchange for results within a day rather than seconds, so every model call that is not a person waiting (classification, backfills, evaluation runs, embedding, summaries) belongs in a queue that drains through it, with idempotent items and per-item failure handling."},"body":{"format":"markdown","text":"## The offer\n\nThe major providers offer a batch mode: submit a file of requests, receive the results within a window, usually twenty-four hours and often much sooner, at a discount of about half against the interactive price. The provider runs your work when it has spare capacity, and the discount is what they pay you for the flexibility.\n\nHalf is a large number. A feature spending ten million tokens a day interactively at $3 per million pays $30; the same work through the batch endpoint pays $15. Over a year that is the difference between five and a half thousand dollars and eleven.\n\n## What qualifies\n\nThe question is only whether a person is waiting. If nobody is, the work is batch work. In most products that is more than people think:\n\n- **Classification and tagging** of records as they arrive (support tickets, reviews, documents) where a label within an hour is as good as one within a second.\n- **Backfills**: applying a new prompt or a new model to everything that already exists, which the earlier module on extraction at scale set up.\n- **Evaluation runs**: the harness's fifty cases times three, the judge on shadow-mode pairs. Nobody is waiting on CI for a day, but a nightly evaluation certainly can run through batch.\n- **Embedding** at ingestion; some providers batch this too.\n- **Summaries and digests**: the thread summary from the interface module, a nightly report, memory extraction after a conversation ends.\n- **Synthetic data** for the evaluation set.\n\nThe interactive endpoint is for the chat reply, the inline completion, the tool call in a live agent loop. Everything else is a candidate.\n\n## The design\n\nA queue table, in the database you already have:\n\n```sql\ncreate table model_jobs (\n  id          bigserial primary key,\n  kind        text not null,           -- classify, summarise, extract, ...\n  input       jsonb not null,\n  idem_key    text unique not null,    -- so a retry never duplicates\n  status      text default 'pending',  -- pending, submitted, done, failed\n  batch_id    text,\n  output      jsonb,\n  error       text,\n  created_at  timestamptz default now()\n);\n```\n\nA worker collects pending jobs every few minutes, writes them into the provider's batch format with the job id as the custom identifier, submits, and records the batch id against each. Another worker polls for completed batches, reads the results file, and writes each output back against its job by that identifier. Postgres is enough for this; a dedicated queue is not needed until the volume is large.\n\n## Failures are per item\n\nA batch does not fail as a whole. Individual requests fail (a malformed input, a content filter, a length limit) and the results file marks them. Handle each: retry the transient ones in the next batch, mark the permanent ones failed with the error text, and never let one bad row block the other ten thousand. The idempotency key is what makes the retry safe: a job resubmitted after a crash in the worker produces one result, not two.\n\n## The window is a promise, not a schedule\n\nResults within twenty-four hours means results within twenty-four hours. Most batches complete far sooner, and some, when the provider is busy, take the full window. Design the consumer for that: the classification that usually lands in an hour must be fine landing tomorrow, and the interface must not show a spinner waiting for it. If any part of the product cannot tolerate the full window, that part is not batch work.\n\n## Combining with the other discounts\n\nBatch requests can often use prompt caching too, when many items share a prefix (the same system prompt and instructions across ten thousand classifications) and the two discounts stack. A batch of cached-prefix requests can land at a small fraction of the interactive uncached price. Check the provider's page for whether they combine; it varies.\n\n## The arithmetic to run\n\nList every model call in the product. Mark each as \"a person is waiting\" or not. Sum the tokens of the second group. Halve their cost. That number, for many products, is larger than any saving available from prompt engineering, and it requires no change to a single prompt: only a queue, two workers and the patience to let the work take a day."},"check":{"question":"A team moves nightly ticket classification to the batch endpoint and reports that results sometimes arrive after the morning dashboard is built, leaving tickets unclassified. What was misjudged?","options":[{"text":"The batch endpoint is unsuitable for classification, which needs the interactive endpoint's consistency.","correct":false,"tempting":"Classification is the archetypal batch workload; the endpoint's outputs are the same, only the timing differs."},{"text":"The window is a promise of completion within a day, not a schedule; a consumer that cannot wait the full window is not batch work.","correct":true},{"text":"The batch was submitted too late in the evening; submitting it earlier in the day guarantees completion well before the morning dashboard runs.","correct":false,"tempting":"Earlier submission raises the odds but the provider still promises only the window, so the guarantee is illusory."},{"text":"Per-item failures were blocking the whole batch, delaying every result.","correct":false,"tempting":"Batches do not fail as a whole on item errors; the delay is the provider's scheduling, not a blocking failure."}]},"prev":{"slug":"caching-at-three-levels","title":"Caching at three levels: exact, prefix and semantic","url":"https://addaly.com/learn/building-with-ai/caching-at-three-levels"},"next":{"slug":"routing-and-cascades","title":"Routing and cascades: paying the high price only where it is needed","url":"https://addaly.com/learn/building-with-ai/routing-and-cascades"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'The batch window: half price for anything that can wait', in Building with AI. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/building-with-ai/the-batch-window (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"The batch window: half price for anything that can wait","courseTitle":"Building with AI","module":{"index":8,"slug":"cost-latency-and-running-it","title":"Cost, latency and running it in production"},"answer":null,"keyIdea":"Providers sell a batch endpoint at around half the interactive price in exchange for results within a day rather than seconds, so every model call that is not a person waiting (classification, backfills, evaluation runs, embedding, summaries) belongs in a queue that drains through it, with idempotent items and per-item failure handling.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"the-offer","runs":[{"text":"The offer"}]},{"t":"p","runs":[{"text":"The major providers offer a batch mode: submit a file of requests, receive the results within a window, usually twenty-four hours and often much sooner, at a discount of about half against the interactive price. The provider runs your work when it has spare capacity, and the discount is what they pay you for the flexibility."}]},{"t":"p","runs":[{"text":"Half is a large number. A feature spending ten million tokens a day interactively at $3 per million pays $30; the same work through the batch endpoint pays $15. Over a year that is the difference between five and a half thousand dollars and eleven."}]},{"t":"h2","id":"what-qualifies","runs":[{"text":"What qualifies"}]},{"t":"p","runs":[{"text":"The question is only whether a person is waiting. If nobody is, the work is batch work. In most products that is more than people think:"}]},{"t":"ul","items":[[{"text":"Classification and tagging","b":true},{"text":" of records as they arrive (support tickets, reviews, documents) where a label within an hour is as good as one within a second."}],[{"text":"Backfills","b":true},{"text":": applying a new prompt or a new model to everything that already exists, which the earlier module on extraction at scale set up."}],[{"text":"Evaluation runs","b":true},{"text":": the harness’s fifty cases times three, the judge on shadow-mode pairs. Nobody is waiting on CI for a day, but a nightly evaluation certainly can run through batch."}],[{"text":"Embedding","b":true},{"text":" at ingestion; some providers batch this too."}],[{"text":"Summaries and digests","b":true},{"text":": the thread summary from the interface module, a nightly report, memory extraction after a conversation ends."}],[{"text":"Synthetic data","b":true},{"text":" for the evaluation set."}]]},{"t":"p","runs":[{"text":"The interactive endpoint is for the chat reply, the inline completion, the tool call in a live agent loop. Everything else is a candidate."}]},{"t":"h2","id":"the-design","runs":[{"text":"The design"}]},{"t":"p","runs":[{"text":"A queue table, in the database you already have:"}]},{"t":"code","lang":"sql","label":"SQL","text":"create table model_jobs (\n  id          bigserial primary key,\n  kind        text not null,           -- classify, summarise, extract, ...\n  input       jsonb not null,\n  idem_key    text unique not null,    -- so a retry never duplicates\n  status      text default 'pending',  -- pending, submitted, done, failed\n  batch_id    text,\n  output      jsonb,\n  error       text,\n  created_at  timestamptz default now()\n);"},{"t":"p","runs":[{"text":"A worker collects pending jobs every few minutes, writes them into the provider’s batch format with the job id as the custom identifier, submits, and records the batch id against each. Another worker polls for completed batches, reads the results file, and writes each output back against its job by that identifier. Postgres is enough for this; a dedicated queue is not needed until the volume is large."}]},{"t":"h2","id":"failures-are-per-item","runs":[{"text":"Failures are per item"}]},{"t":"p","runs":[{"text":"A batch does not fail as a whole. Individual requests fail (a malformed input, a content filter, a length limit) and the results file marks them. Handle each: retry the transient ones in the next batch, mark the permanent ones failed with the error text, and never let one bad row block the other ten thousand. The idempotency key is what makes the retry safe: a job resubmitted after a crash in the worker produces one result, not two."}]},{"t":"h2","id":"the-window-is-a-promise-not-a-schedule","runs":[{"text":"The window is a promise, not a schedule"}]},{"t":"p","runs":[{"text":"Results within twenty-four hours means results within twenty-four hours. Most batches complete far sooner, and some, when the provider is busy, take the full window. Design the consumer for that: the classification that usually lands in an hour must be fine landing tomorrow, and the interface must not show a spinner waiting for it. If any part of the product cannot tolerate the full window, that part is not batch work."}]},{"t":"h2","id":"combining-with-the-other-discounts","runs":[{"text":"Combining with the other discounts"}]},{"t":"p","runs":[{"text":"Batch requests can often use prompt caching too, when many items share a prefix (the same system prompt and instructions across ten thousand classifications) and the two discounts stack. A batch of cached-prefix requests can land at a small fraction of the interactive uncached price. Check the provider’s page for whether they combine; it varies."}]},{"t":"h2","id":"the-arithmetic-to-run","runs":[{"text":"The arithmetic to run"}]},{"t":"p","runs":[{"text":"List every model call in the product. Mark each as “a person is waiting” or not. Sum the tokens of the second group. Halve their cost. That number, for many products, is larger than any saving available from prompt engineering, and it requires no change to a single prompt: only a queue, two workers and the patience to let the work take a day."}]}],"check":{"question":[{"text":"A team moves nightly ticket classification to the batch endpoint and reports that results sometimes arrive after the morning dashboard is built, leaving tickets unclassified. What was misjudged?"}],"options":[{"text":[{"text":"The batch endpoint is unsuitable for classification, which needs the interactive endpoint’s consistency."}],"correct":false,"tempting":[{"text":"Classification is the archetypal batch workload; the endpoint’s outputs are the same, only the timing differs."}]},{"text":[{"text":"The window is a promise of completion within a day, not a schedule; a consumer that cannot wait the full window is not batch work."}],"correct":true,"tempting":null},{"text":[{"text":"The batch was submitted too late in the evening; submitting it earlier in the day guarantees completion well before the morning dashboard runs."}],"correct":false,"tempting":[{"text":"Earlier submission raises the odds but the provider still promises only the window, so the guarantee is illusory."}]},{"text":[{"text":"Per-item failures were blocking the whole batch, delaying every result."}],"correct":false,"tempting":[{"text":"Batches do not fail as a whole on item errors; the delay is the provider’s scheduling, not a blocking failure."}]}],"order":[2,1,3,0],"why":null},"prev":{"slug":"caching-at-three-levels","title":"Caching at three levels: exact, prefix and semantic","minutes":9},"next":{"slug":"routing-and-cascades","title":"Routing and cascades: paying the high price only where it is needed","minutes":9}}}