Errors, retries, and how a retry loop takes down your own service

Not all failures are the same failure

Every model API returns a small set of status codes, and the only useful question about any of them is: will doing this again help?

• 400 Bad Request: your JSON is malformed, or the prompt exceeds the context window. Retrying sends the same broken request again. Never retry.
• 401 / 403: the key is wrong, revoked, or lacks permission. Never retry. Page someone.
• 404: usually a model name that no longer exists, which happens when a provider retires a version. Never retry; fall back to another model if you have one.
• 413 / context length exceeded: too much input. Retrying is pointless. Trim and resend, once.
• 429 Too Many Requests: you hit a rate limit. Retry, but only after waiting, and only as many times as the deadline allows.
• 500 / 502 / 503: something broke on their side. Retry.
• 529 / “overloaded”: the provider is at capacity. Retry, with a longer wait than a 500.
• Timeouts and dropped sockets: retry, with the caveat below.

The distinction that matters is between retryable and not retryable, and about half of production incidents involving model APIs come from code that retried something in the second group.

Backoff, with jitter, and why jitter is not optional

import random, time

RETRYABLE = {429, 500, 502, 503, 529}

def call_with_retries(fn, attempts=4, deadline_s=25):
    started = time.monotonic()
    for i in range(attempts):
        try:
            return fn()
        except APIError as e:
            if e.status not in RETRYABLE or i == attempts - 1:
                raise
            wait = e.retry_after or min(2 ** i, 8) * (0.5 + random.random())
            if time.monotonic() - started + wait > deadline_s:
                raise DeadlineExceeded() from e
            time.sleep(wait)

Three things in that snippet are load-bearing.

retry_after first. Rate-limit responses usually carry a retry-after header telling you exactly how long the window is. Obeying it beats guessing.

Random jitter. Without it, every client that failed at the same instant retries at the same instant. A provider recovering from a blip gets hit by a perfectly synchronised wall of traffic and falls over again. This is the thundering herd, and multiplying the wait by a random factor between 0.5 and 1.5 is the entire fix.

A deadline, not just an attempt count. Four attempts with exponential backoff can consume 15 seconds. If the user’s request already had a 10-second budget, three of those attempts were spent answering a browser tab that closed. Retries must respect the wall clock of the request that spawned them, not their own counter.

The retry storm that is your fault

Here is the failure mode worth internalising, because it is self-inflicted.

The provider slows down. Your requests start taking 20 seconds instead of 2. Your timeout fires at 10 and you retry. Now there are two requests in flight for one user. The first one is still being generated, and still being billed. Every user in the queue does the same. Your concurrent connections triple, you hit your own rate limit, so now you are generating 429s yourself, and your retry logic responds to those by retrying. Your dashboard shows an outage caused by the provider. In truth the provider was merely slow, and you converted slow into down.

The defences are all boring:

• A concurrency cap: a semaphore around outbound calls. Above it, queue or shed load. A number you chose beats a number your connection pool chose for you.
• A circuit breaker. After N consecutive failures, stop calling for 30 seconds and fail fast. This is counter-intuitive and correct: when a dependency is unhealthy, the kindest thing you can do is stop sending it traffic.
• A total budget per user request: attempts and seconds, both capped.
• Shed the retry on non-interactive paths. A background job can wait a minute. A user cannot.

Timeouts on a streaming call

One subtlety. A total timeout is wrong for streaming, because a long answer legitimately keeps the connection open for a minute. What you want is an inactivity timeout: abort if no chunk has arrived in 15 seconds. A stalled stream is a failure; a slow one is not.

Retries and duplicates

A retried call is a second generation, billed separately, producing different text. That is fine when the call is a pure question. It is not fine when the call has already caused a side effect: a tool that sent an email, wrote a row, issued a refund.

So: keep side effects out of the retried region, or make them idempotent. If your loop retries an entire agent turn, and step three of that turn charged a card, a network blip has now charged it twice. Give every externally visible action a key derived from the request, store the key, and make the second attempt a no-op. This is ordinary distributed-systems hygiene, and putting a model in the loop does not exempt you from it: it makes it more likely, because agent loops retry more.