Rate limits, workers and the thundering herd

The limit that actually binds

Providers publish two limits: requests per minute and tokens per minute. Chat applications hit the request limit. Agents almost never do, and the reason is the arithmetic from the context-window lesson.

Suppose your allowance is 200,000 tokens per minute and 500 requests per minute. An agent on turn fifteen is sending a 40,000-token transcript. Divide: five such requests exhaust the token allowance in a minute, while you are using one per cent of the request allowance. Your effective concurrency is five, and no amount of raising the request limit changes it.

This surprises teams whose load test used short prompts and reported plenty of headroom. Model your capacity in tokens per minute, using the transcript size at the end of a run rather than the beginning, and the number will be far smaller and correct.

Backoff needs jitter

When you get a 429, read the Retry-After header and obey it. When there is no header, back off exponentially, and add randomness.

Without jitter, every worker that failed at the same moment retries at the same moment, collides again, and repeats. This is the thundering herd, and it turns a brief limit into a sustained outage of your own making. Full jitter (sleep a random duration between zero and the current backoff ceiling) spreads them out:

delay = random.uniform(0, min(cap, base * 2 ** attempt))

Cap the attempts, and distinguish what is worth retrying: 429 and 5xx yes, 400 and 401 no. Retrying a malformed request forever is a way to spend an afternoon’s quota on the same rejection.

How many workers

Little’s Law gives the answer without guessing. Workers needed equals arrival rate multiplied by service time. Six hundred items an hour is 0.167 per second; if each takes thirty seconds, you need 0.167 × 30 = 5 workers to keep up, with a little headroom for variance.

Fix that number with a semaphore or a worker pool. Unbounded concurrency (spawning a task per item because the language makes it easy) produces a system that works in testing and, on the morning a backlog arrives, opens four thousand simultaneous requests, hits every limit at once and takes longer than doing it serially.

A fixed pool also makes cost predictable, which matters more than it sounds: parallel agents multiply your spend per minute, and a runaway parallel job can produce a month’s budget in an hour. Put a spend-per-minute ceiling next to the worker count, not only a per-run cap.

Fairness

One customer uploading nine thousand documents should not stop everybody else’s work for the afternoon. If a single queue is processed in order, that is exactly what happens.

Either keep a queue per tenant and round-robin between them, or cap how many items from one source may be in flight at once. Both are a few lines and both prevent the support conversation where a small customer’s urgent item sat behind a large customer’s bulk import.

Timeouts, from the inside out

Every network call gets a timeout. Every tool gets a timeout longer than its slowest call. Every run gets a timeout longer than its steps allow for.

The rule that catches people: an outer timeout shorter than the sum of what it contains will kill work that is still running and cannot be cancelled. The HTTP request is abandoned, the provider still bills for it, and the write it was about to perform may still land, which produces the duplicate that idempotency keys exist to absorb.

Poison items

Some item will fail forever: a corrupt PDF, a record with a null where the schema promised a value. Without a limit, it is retried indefinitely and consumes a worker permanently.

Count attempts on the item itself. After three, move it to a failed state with the error, and carry on. Then alert on the failure rate, because one poison item is normal and forty in an hour means something upstream changed.

The free path

You almost certainly do not need a message broker. PostgreSQL with FOR UPDATE SKIP LOCKED gives you a durable, observable queue with transactions, and it is already running. Redis with RQ or Celery is the next step up. Managed options exist on every platform. Whatever you use, the properties to insist on are the same: at-least-once delivery, a visible backlog, per-item attempt counts, and somewhere for failures to land where a person will see them.