{"course":{"slug":"agents-and-automation","title":"Agents and Automation","url":"https://addaly.com/learn/agents-and-automation","level":"intermediate","track":"build"},"lesson":{"slug":"retries-and-idempotency","title":"Retries, timeouts and doing it twice","url":"https://addaly.com/learn/agents-and-automation/retries-and-idempotency","position":34,"of":72,"minutes":8,"answer":"Retry only what is transient, back off with jitter, and generate the idempotency key once before the first attempt: a new key per attempt is simply a new request.","keyIdea":"Retry only what is transient, back off with jitter, and generate the idempotency key once before the first attempt: a new key per attempt is simply a new request."},"body":{"format":"markdown","text":"## Two kinds of failure, two policies\n\nSort every failure into transient or deterministic, because they need opposite treatment.\n\n**Transient**: the same request would probably succeed in three seconds. Timeouts, connection resets, HTTP 429, 500, 502, 503, 504. The harness retries these itself. The model never needs to know.\n\n**Deterministic**: the same request will fail identically forever. HTTP 400, 401, 403, 404, 422, validation errors, \"no such order\". Retrying is pure waste. These go back to the model as an informative observation, because *that* is the retry: a different request, chosen by something that can reason about why.\n\nGetting this backwards produces the two classic bugs. Retrying deterministic failures gives you the retry loop from the failure catalogue. Not retrying transient ones gives you an agent that gives up whenever a network hiccups.\n\n## Backoff, and specifically jitter\n\nFixed backoff is worse than it looks. Twenty clients all fail at once, all wait exactly two seconds, and all retry at exactly the same instant: you have built a metronome for hammering a service that is already struggling.\n\nFull jitter fixes it:\n\n```python\ndelay = random.uniform(0, min(cap, base * 2 ** attempt))\n# base = 1s, cap = 30s, max 5 attempts\n```\n\nRandomising the whole interval, not just adding a little noise, is what spreads the load. This is the \"full jitter\" strategy from AWS's well-known write-up on backoff, and it beats the alternatives in their simulations.\n\n**And when the server tells you the number, use it.** A 429 usually carries a `Retry-After` header. Sleeping your own guess when the answer is in the response is how you get rate-limited harder.\n\n## Idempotency, at the harness level\n\nYou met the idea in tool design. Here is the operational rule, which is short and gets broken constantly:\n\n**Generate the idempotency key once, before the first attempt, and reuse it for every retry of that logical operation.**\n\n```python\nkey = f\"{run_id}:{step}:{sha256(canonical_json(args))[:16]}\"   # once\nfor attempt in range(5):\n    resp = call(url, headers={\"Idempotency-Key\": key}, json=args)\n```\n\nIf the key is generated inside the loop, every retry is a brand-new request as far as the other side is concerned, and you have implemented duplicate creation with extra steps.\n\n## At-least-once is what the world gives you\n\nExactly-once delivery is not available over a network you do not control. The write can succeed and the acknowledgement can be lost; nothing you do at your end distinguishes that from the write failing.\n\nSo design side effects so that repetition is either **harmless** (setting a field to a value, which is naturally idempotent) or **detectable**, by checking whether the thing already exists before creating it. `INSERT ... ON CONFLICT DO NOTHING` is a load-bearing piece of automation engineering.\n\n## Retry budgets\n\nRetries multiply load precisely when a dependency is least able to take it. Cap them twice:\n\n- **Per operation**: five attempts, then fail with a clear message.\n- **Per run and per minute across runs**: if more than some share of calls are retries, stop retrying and start failing fast. This is a circuit breaker, and a crude one made of two counters is enough.\n\nAn agent with unlimited retries against a struggling API is a load generator with a personality."},"check":{"question":"A tool call to a partner API returns HTTP 429 with `Retry-After: 30`. The harness retries after 1s, 2s, 4s and 8s, all failing, then gives up and reports an error to the model. What is the most important thing wrong here?","options":[{"text":"429 is a deterministic error and should not have been retried at all.","correct":false,"tempting":"Miscategorises the archetypal transient failure; a rate limit is precisely the case where waiting and retrying is correct."},{"text":"The backoff lacked jitter, so the four attempts synchronised with other clients.","correct":false,"tempting":"A real weakness in general, but with a server-supplied wait time already present, jitter is not the main failure here."},{"text":"The server said how long to wait and the harness ignored it, so all four attempts landed inside the window and the call failed for a reason that was never a failure.","correct":true},{"text":"The error should have been returned to the model immediately so it could choose a different tool.","correct":false,"tempting":"Sends a transient condition to the reasoning layer, where the model can only guess; the harness had the exact answer in a header."}]},"prev":{"slug":"state-machines-over-loops","title":"When the loop should be a graph","url":"https://addaly.com/learn/agents-and-automation/state-machines-over-loops"},"next":{"slug":"validators-not-vibes","title":"The validator is the product","url":"https://addaly.com/learn/agents-and-automation/validators-not-vibes"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Retries, timeouts and doing it twice', in Agents and Automation. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/agents-and-automation/retries-and-idempotency (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Retries, timeouts and doing it twice","courseTitle":"Agents and Automation","module":{"index":4,"slug":"making-it-reliable","title":"Surviving real data"},"answer":null,"keyIdea":"Retry only what is transient, back off with jitter, and generate the idempotency key once before the first attempt: a new key per attempt is simply a new request.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"two-kinds-of-failure-two-policies","runs":[{"text":"Two kinds of failure, two policies"}]},{"t":"p","runs":[{"text":"Sort every failure into transient or deterministic, because they need opposite treatment."}]},{"t":"p","runs":[{"text":"Transient","b":true},{"text":": the same request would probably succeed in three seconds. Timeouts, connection resets, HTTP 429, 500, 502, 503, 504. The harness retries these itself. The model never needs to know."}]},{"t":"p","runs":[{"text":"Deterministic","b":true},{"text":": the same request will fail identically forever. HTTP 400, 401, 403, 404, 422, validation errors, “no such order”. Retrying is pure waste. These go back to the model as an informative observation, because "},{"text":"that","i":true},{"text":" is the retry: a different request, chosen by something that can reason about why."}]},{"t":"p","runs":[{"text":"Getting this backwards produces the two classic bugs. Retrying deterministic failures gives you the retry loop from the failure catalogue. Not retrying transient ones gives you an agent that gives up whenever a network hiccups."}]},{"t":"h2","id":"backoff-and-specifically-jitter","runs":[{"text":"Backoff, and specifically jitter"}]},{"t":"p","runs":[{"text":"Fixed backoff is worse than it looks. Twenty clients all fail at once, all wait exactly two seconds, and all retry at exactly the same instant: you have built a metronome for hammering a service that is already struggling."}]},{"t":"p","runs":[{"text":"Full jitter fixes it:"}]},{"t":"code","lang":"python","label":"Python","text":"delay = random.uniform(0, min(cap, base * 2 ** attempt))\n# base = 1s, cap = 30s, max 5 attempts"},{"t":"p","runs":[{"text":"Randomising the whole interval, not just adding a little noise, is what spreads the load. This is the “full jitter” strategy from AWS’s well-known write-up on backoff, and it beats the alternatives in their simulations."}]},{"t":"p","runs":[{"text":"And when the server tells you the number, use it.","b":true},{"text":" A 429 usually carries a "},{"text":"Retry-After","code":true},{"text":" header. Sleeping your own guess when the answer is in the response is how you get rate-limited harder."}]},{"t":"h2","id":"idempotency-at-the-harness-level","runs":[{"text":"Idempotency, at the harness level"}]},{"t":"p","runs":[{"text":"You met the idea in tool design. Here is the operational rule, which is short and gets broken constantly:"}]},{"t":"p","runs":[{"text":"Generate the idempotency key once, before the first attempt, and reuse it for every retry of that logical operation.","b":true}]},{"t":"code","lang":"python","label":"Python","text":"key = f\"{run_id}:{step}:{sha256(canonical_json(args))[:16]}\"   # once\nfor attempt in range(5):\n    resp = call(url, headers={\"Idempotency-Key\": key}, json=args)"},{"t":"p","runs":[{"text":"If the key is generated inside the loop, every retry is a brand-new request as far as the other side is concerned, and you have implemented duplicate creation with extra steps."}]},{"t":"h2","id":"at-least-once-is-what-the-world-gives-you","runs":[{"text":"At-least-once is what the world gives you"}]},{"t":"p","runs":[{"text":"Exactly-once delivery is not available over a network you do not control. The write can succeed and the acknowledgement can be lost; nothing you do at your end distinguishes that from the write failing."}]},{"t":"p","runs":[{"text":"So design side effects so that repetition is either "},{"text":"harmless","b":true},{"text":" (setting a field to a value, which is naturally idempotent) or "},{"text":"detectable","b":true},{"text":", by checking whether the thing already exists before creating it. "},{"text":"INSERT ... ON CONFLICT DO NOTHING","code":true},{"text":" is a load-bearing piece of automation engineering."}]},{"t":"h2","id":"retry-budgets","runs":[{"text":"Retry budgets"}]},{"t":"p","runs":[{"text":"Retries multiply load precisely when a dependency is least able to take it. Cap them twice:"}]},{"t":"ul","items":[[{"text":"Per operation","b":true},{"text":": five attempts, then fail with a clear message."}],[{"text":"Per run and per minute across runs","b":true},{"text":": if more than some share of calls are retries, stop retrying and start failing fast. This is a circuit breaker, and a crude one made of two counters is enough."}]]},{"t":"p","runs":[{"text":"An agent with unlimited retries against a struggling API is a load generator with a personality."}]}],"check":{"question":[{"text":"A tool call to a partner API returns HTTP 429 with "},{"text":"Retry-After: 30","code":true},{"text":". The harness retries after 1s, 2s, 4s and 8s, all failing, then gives up and reports an error to the model. What is the most important thing wrong here?"}],"options":[{"text":[{"text":"429 is a deterministic error and should not have been retried at all."}],"correct":false,"tempting":[{"text":"Miscategorises the archetypal transient failure; a rate limit is precisely the case where waiting and retrying is correct."}]},{"text":[{"text":"The backoff lacked jitter, so the four attempts synchronised with other clients."}],"correct":false,"tempting":[{"text":"A real weakness in general, but with a server-supplied wait time already present, jitter is not the main failure here."}]},{"text":[{"text":"The server said how long to wait and the harness ignored it, so all four attempts landed inside the window and the call failed for a reason that was never a failure."}],"correct":true,"tempting":null},{"text":[{"text":"The error should have been returned to the model immediately so it could choose a different tool."}],"correct":false,"tempting":[{"text":"Sends a transient condition to the reasoning layer, where the model can only guess; the harness had the exact answer in a header."}]}],"order":[0,3,1,2],"why":null},"prev":{"slug":"state-machines-over-loops","title":"When the loop should be a graph","minutes":9},"next":{"slug":"validators-not-vibes","title":"The validator is the product","minutes":9}}}