The batch window: half price for anything that can wait

The offer

The major providers offer a batch mode: submit a file of requests, receive the results within a window, usually twenty-four hours and often much sooner, at a discount of about half against the interactive price. The provider runs your work when it has spare capacity, and the discount is what they pay you for the flexibility.

Half is a large number. A feature spending ten million tokens a day interactively at $3 per million pays $30; the same work through the batch endpoint pays $15. Over a year that is the difference between five and a half thousand dollars and eleven.

What qualifies

The question is only whether a person is waiting. If nobody is, the work is batch work. In most products that is more than people think:

• Classification and tagging of records as they arrive (support tickets, reviews, documents) where a label within an hour is as good as one within a second.
• Backfills: applying a new prompt or a new model to everything that already exists, which the earlier module on extraction at scale set up.
• Evaluation runs: the harness’s fifty cases times three, the judge on shadow-mode pairs. Nobody is waiting on CI for a day, but a nightly evaluation certainly can run through batch.
• Embedding at ingestion; some providers batch this too.
• Summaries and digests: the thread summary from the interface module, a nightly report, memory extraction after a conversation ends.
• Synthetic data for the evaluation set.

The interactive endpoint is for the chat reply, the inline completion, the tool call in a live agent loop. Everything else is a candidate.

The design

A queue table, in the database you already have:

create table model_jobs (
  id          bigserial primary key,
  kind        text not null,           -- classify, summarise, extract, ...
  input       jsonb not null,
  idem_key    text unique not null,    -- so a retry never duplicates
  status      text default 'pending',  -- pending, submitted, done, failed
  batch_id    text,
  output      jsonb,
  error       text,
  created_at  timestamptz default now()
);

A worker collects pending jobs every few minutes, writes them into the provider’s batch format with the job id as the custom identifier, submits, and records the batch id against each. Another worker polls for completed batches, reads the results file, and writes each output back against its job by that identifier. Postgres is enough for this; a dedicated queue is not needed until the volume is large.

Failures are per item

A batch does not fail as a whole. Individual requests fail (a malformed input, a content filter, a length limit) and the results file marks them. Handle each: retry the transient ones in the next batch, mark the permanent ones failed with the error text, and never let one bad row block the other ten thousand. The idempotency key is what makes the retry safe: a job resubmitted after a crash in the worker produces one result, not two.

The window is a promise, not a schedule

Results within twenty-four hours means results within twenty-four hours. Most batches complete far sooner, and some, when the provider is busy, take the full window. Design the consumer for that: the classification that usually lands in an hour must be fine landing tomorrow, and the interface must not show a spinner waiting for it. If any part of the product cannot tolerate the full window, that part is not batch work.

Combining with the other discounts

Batch requests can often use prompt caching too, when many items share a prefix (the same system prompt and instructions across ten thousand classifications) and the two discounts stack. A batch of cached-prefix requests can land at a small fraction of the interactive uncached price. Check the provider’s page for whether they combine; it varies.

The arithmetic to run

List every model call in the product. Mark each as “a person is waiting” or not. Sum the tokens of the second group. Halve their cost. That number, for many products, is larger than any saving available from prompt engineering, and it requires no change to a single prompt: only a queue, two workers and the patience to let the work take a day.