Timeouts and retries: the code that decides whether you pay twice

Two timeouts, not one

timeout=10 is shorthand for a pair:

r = session.post(url, json=payload, timeout=(5, 120))

The first number is the connect timeout: how long to wait for the server to accept the connection. If this expires, nothing was sent, and you know it. The second is the read timeout: how long to wait, between bytes, for the server to send something. If this expires, the request was sent and the server may be working on it, or may have finished.

For a model API the two need different values. Connecting should take under a second; five is generous. Generating 1,000 tokens can legitimately take a minute, so a read timeout of 10 seconds guarantees failures on the longest, most expensive requests — precisely the ones you least want to abandon. Set it from the longest response you expect, then add half again.

Without a timeout at all, a call can hang forever. A hung process looks like a slow one until you notice it has been three hours.

The retry question is about state, not errors

A retry is safe when the failed attempt provably did nothing on the server. Sort failures by that test.

Safe to retry, because nothing happened:

• A connect timeout or a DNS failure. The request never left.
• A ConnectionError raised while sending. Almost always nothing was processed.
• A 429: the server refused the request because you are over your rate limit. It did no work.
• A 503: the server said it was unavailable and did no work.

Not safe to retry blindly:

• A read timeout on a POST. The server received the request. A model API will generate the full response, count the tokens, bill your account, and send a reply that nobody is listening for. Retrying makes a second charged generation.
• A 500. Something broke; whether it broke before or after the work was done is unknown.
• A 400, 401, 403, 404. Retrying the same wrong request produces the same answer.

GET requests are different: they are defined to have no side effects, so any of them can be retried. This is what the word idempotent means in the documentation — repeating it changes nothing further. A GET is idempotent by definition. A POST that creates a chat completion is not.

Some providers accept an Idempotency-Key header: you generate a random ID per logical request, send it on every attempt, and the server returns the first result rather than redoing the work. Where it exists, use it; then a read-timeout retry is safe.

Backoff, and why it is exponential

When a 429 arrives, retrying immediately is the worst response. The limit is per second or per minute; hammering it extends the ban. Wait, then wait longer:

import random, time

def wait_time(attempt, base=1.0, cap=60.0):
    return min(cap, base * 2 ** attempt) + random.uniform(0, 1)

Attempt 0 waits about a second, then two, four, eight, capped at sixty. The random fraction is jitter. Without it, a hundred clients rejected at the same moment all retry at the same moment and are all rejected again; a little randomness spreads them out.

If the response carries a Retry-After header, the server has told you exactly how long to wait, and you should believe it over your own formula:

delay = float(r.headers.get("Retry-After", wait_time(attempt)))

A retry that respects the rules

import requests

RETRY_STATUSES = {429, 502, 503, 504}

def post_with_retry(session, url, payload, attempts=5, timeout=(5, 120)):
    for attempt in range(attempts):
        try:
            r = session.post(url, json=payload, timeout=timeout)
        except requests.ConnectTimeout:
            time.sleep(wait_time(attempt)); continue          # nothing was sent
        except requests.ReadTimeout:
            raise                                             # may have been billed; let the caller decide
        if r.status_code in RETRY_STATUSES:
            delay = float(r.headers.get("Retry-After", wait_time(attempt)))
            time.sleep(delay); continue
        r.raise_for_status()
        return r
    raise RuntimeError(f"gave up after {attempts} attempts")

The ReadTimeout branch re-raises on purpose. The right response to "the server may have done the work" depends on what the work was, and that decision belongs one level up.

Letting urllib3 do it

requests sits on urllib3, which has a Retry object that handles most of this:

from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry

retry = Retry(
    total=5,
    backoff_factor=1,                       # 1, 2, 4, 8, 16 seconds
    status_forcelist=[429, 502, 503, 504],
    allowed_methods=["GET", "POST"],        # POST is not retried unless you say so
    respect_retry_after_header=True,
)
session.mount("https://", HTTPAdapter(max_retries=retry))

Notice allowed_methods. By default Retry does not retry POST, for exactly the reason above. Adding it is a decision you should make knowing that a read timeout will still not be retried by this path — Retry handles status codes and connection errors, not a body that stopped arriving.

What the SDKs do

The openai and anthropic clients retry twice by default with backoff on 408, 409, 429 and 5xx, and honour Retry-After. They do not retry a read timeout on a completed send, for the same reason. max_retries= on the client changes the count. Knowing this stops you wrapping their calls in a second retry loop and multiplying the attempts.

Try this now

Point post_with_retry at https://httpbin.org/status/503 and watch it back off and give up. Then at https://httpbin.org/delay/5 with timeout=(5, 2) and confirm you get a ReadTimeout, not a retry. Print the attempt number and the delay each time round; the shape of the waits is the lesson.