Work that can wait

Two lanes, one codebase

Most providers run a batch endpoint: submit a file of requests, get results within a stated window, usually up to 24 hours, at roughly half the interactive price. Open-source serving stacks get a similar effect from larger batch sizes and better hardware utilisation when nobody is waiting.

The discount is real and the constraint is simple. Anything with a human waiting goes in the fast lane. Everything else should not.

What belongs in the slow lane, in most systems:

• Ingestion. Contextual chunk augmentation, document summaries, extracting metadata. This is the largest batch workload in a typical retrieval system and it is entirely asynchronous by nature.
• Evaluation runs. A thousand test cases at half price, overnight, is the same result as a thousand at full price during the day.
• Backfills. Re-processing a corpus after a prompt change or a model upgrade.
• Scheduled reports and digests. Nobody is watching at 04:00.
• Offline classification of yesterday’s tickets, messages or logs.

For a system with a large corpus, moving ingestion to batch can halve the total bill on its own, because ingestion is often a bigger token consumer than serving.

Design so one function serves both

The mistake is writing the batch path as a separate program. It drifts: the prompt is updated in one place and not the other, and six weeks later your index was built with a different instruction than your evaluation assumes.

The pure assembler from module one is what makes this easy. build_context() produces a request object; a thin adapter either sends it now or serialises it into a batch file.

req = build_context(inputs, cfg)          # identical either way
if mode == "interactive":
    return client.send(req)
batch_file.write(json.dumps({"custom_id": inputs.id, **req.as_dict()}) + "\n")

The custom_id is the part people forget. Batch results come back unordered and sometimes partial, so every request needs an id you can join on, and the joining code needs to handle rows that are missing entirely.

What batch does not give you

Three honest limitations.

No latency guarantee inside the window. “Within 24 hours” means it may take 24 hours. Anything on a deadline (a nightly job that must finish before the business day) needs either a large margin or the interactive lane.

Partial failure is normal. Some rows fail for reasons unrelated to your input: capacity, transient errors, a single malformed request. Your consumer must be idempotent and able to re-submit the missing ids without redoing the rest.

Debugging is slower. A mistake in a batch of 200,000 requests is discovered hours later, after you have paid for all of them. Always run a hundred rows interactively first, read the outputs with your own eyes, and only then submit the batch. This one habit saves more money than the discount does.

Concurrency in the fast lane

Separately from batching: when you have many independent requests and someone is waiting, the lever is concurrency, and the limit is the rate limits in both directions: requests per minute and tokens per minute.

The token limit is the one that surprises people. Ten concurrent requests of 50,000 tokens each is 500,000 tokens in flight, which exceeds many accounts’ per-minute token allowance even though ten requests is a trivial request count. You get throttled by a limit you were not watching.

Handle it with a bounded worker pool sized against the token limit rather than the request limit, and with retry that respects the retry-after header rather than hammering. Exponential backoff with jitter, and a cap on total attempts, because retrying a request that failed for being too large will fail identically forever.

A worked example of the saving

A corpus of 40,000 documents, averaging 30 chunks each, with a contextual sentence generated per chunk. That is 1.2 million small-model calls. At interactive rates with the document cached, call it a few hundred pounds; through the batch endpoint, roughly half that, and the difference is larger again on the first full build before caching is warm.

The re-run is where it compounds. Every change to the augmentation prompt, every new extraction field, every embedding-model upgrade means reprocessing the corpus. A team that does this quarterly and runs it interactively is paying the premium four times a year for work nobody is waiting on.

The decision, in one question

For any model call in your system, ask: is a person waiting for this specific result?

If no, it belongs in the slow lane and you are currently paying twice what you need to. If yes, then latency work applies and batching does not. Most teams have more work in the first category than they realise, and the reason it is in the fast lane is usually that it was written there first and nobody revisited it.