Retries, timeouts and doing it twice

Two kinds of failure, two policies

Sort every failure into transient or deterministic, because they need opposite treatment.

Transient: the same request would probably succeed in three seconds. Timeouts, connection resets, HTTP 429, 500, 502, 503, 504. The harness retries these itself. The model never needs to know.

Deterministic: the same request will fail identically forever. HTTP 400, 401, 403, 404, 422, validation errors, “no such order”. Retrying is pure waste. These go back to the model as an informative observation, because that is the retry: a different request, chosen by something that can reason about why.

Getting this backwards produces the two classic bugs. Retrying deterministic failures gives you the retry loop from the failure catalogue. Not retrying transient ones gives you an agent that gives up whenever a network hiccups.

Backoff, and specifically jitter

Fixed backoff is worse than it looks. Twenty clients all fail at once, all wait exactly two seconds, and all retry at exactly the same instant: you have built a metronome for hammering a service that is already struggling.

Full jitter fixes it:

delay = random.uniform(0, min(cap, base * 2 ** attempt))
# base = 1s, cap = 30s, max 5 attempts

Randomising the whole interval, not just adding a little noise, is what spreads the load. This is the “full jitter” strategy from AWS’s well-known write-up on backoff, and it beats the alternatives in their simulations.

And when the server tells you the number, use it. A 429 usually carries a Retry-After header. Sleeping your own guess when the answer is in the response is how you get rate-limited harder.

Idempotency, at the harness level

You met the idea in tool design. Here is the operational rule, which is short and gets broken constantly:

Generate the idempotency key once, before the first attempt, and reuse it for every retry of that logical operation.

key = f"{run_id}:{step}:{sha256(canonical_json(args))[:16]}"   # once
for attempt in range(5):
    resp = call(url, headers={"Idempotency-Key": key}, json=args)

If the key is generated inside the loop, every retry is a brand-new request as far as the other side is concerned, and you have implemented duplicate creation with extra steps.

At-least-once is what the world gives you

Exactly-once delivery is not available over a network you do not control. The write can succeed and the acknowledgement can be lost; nothing you do at your end distinguishes that from the write failing.

So design side effects so that repetition is either harmless (setting a field to a value, which is naturally idempotent) or detectable, by checking whether the thing already exists before creating it. INSERT ... ON CONFLICT DO NOTHING is a load-bearing piece of automation engineering.

Retry budgets

Retries multiply load precisely when a dependency is least able to take it. Cap them twice:

• Per operation: five attempts, then fail with a clear message.
• Per run and per minute across runs: if more than some share of calls are retries, stop retrying and start failing fast. This is a circuit breaker, and a crude one made of two counters is enough.

An agent with unlimited retries against a struggling API is a load generator with a personality.