{"course":{"slug":"building-with-ai","title":"Building with AI","url":"https://addaly.com/learn/building-with-ai","level":"intermediate","track":"build"},"lesson":{"slug":"retries-that-do-not-make-it-worse","title":"Errors, retries, and how a retry loop takes down your own service","url":"https://addaly.com/learn/building-with-ai/retries-that-do-not-make-it-worse","position":7,"of":80,"minutes":9,"answer":"Retry only what a second attempt could fix, with jittered backoff under a wall-clock deadline and a concurrency cap, or your retry logic will convert a slow provider into your own outage.","keyIdea":"Retry only what a second attempt could fix, with jittered backoff under a wall-clock deadline and a concurrency cap, or your retry logic will convert a slow provider into your own outage."},"body":{"format":"markdown","text":"## Not all failures are the same failure\n\nEvery model API returns a small set of status codes, and the only useful question about any of them is: *will doing this again help?*\n\n- **400 Bad Request**: your JSON is malformed, or the prompt exceeds the context window. Retrying sends the same broken request again. Never retry.\n- **401 / 403**: the key is wrong, revoked, or lacks permission. Never retry. Page someone.\n- **404**: usually a model name that no longer exists, which happens when a provider retires a version. Never retry; fall back to another model if you have one.\n- **413 / context length exceeded**: too much input. Retrying is pointless. Trim and resend, once.\n- **429 Too Many Requests**: you hit a rate limit. Retry, but only after waiting, and only as many times as the deadline allows.\n- **500 / 502 / 503**: something broke on their side. Retry.\n- **529 / \"overloaded\"**: the provider is at capacity. Retry, with a longer wait than a 500.\n- **Timeouts and dropped sockets**: retry, with the caveat below.\n\nThe distinction that matters is between **retryable** and **not retryable**, and about half of production incidents involving model APIs come from code that retried something in the second group.\n\n## Backoff, with jitter, and why jitter is not optional\n\n```python\nimport random, time\n\nRETRYABLE = {429, 500, 502, 503, 529}\n\ndef call_with_retries(fn, attempts=4, deadline_s=25):\n    started = time.monotonic()\n    for i in range(attempts):\n        try:\n            return fn()\n        except APIError as e:\n            if e.status not in RETRYABLE or i == attempts - 1:\n                raise\n            wait = e.retry_after or min(2 ** i, 8) * (0.5 + random.random())\n            if time.monotonic() - started + wait > deadline_s:\n                raise DeadlineExceeded() from e\n            time.sleep(wait)\n```\n\nThree things in that snippet are load-bearing.\n\n**`retry_after` first.** Rate-limit responses usually carry a `retry-after` header telling you exactly how long the window is. Obeying it beats guessing.\n\n**Random jitter.** Without it, every client that failed at the same instant retries at the same instant. A provider recovering from a blip gets hit by a perfectly synchronised wall of traffic and falls over again. This is the thundering herd, and multiplying the wait by a random factor between 0.5 and 1.5 is the entire fix.\n\n**A deadline, not just an attempt count.** Four attempts with exponential backoff can consume 15 seconds. If the user's request already had a 10-second budget, three of those attempts were spent answering a browser tab that closed. Retries must respect the wall clock of the request that spawned them, not their own counter.\n\n## The retry storm that is your fault\n\nHere is the failure mode worth internalising, because it is self-inflicted.\n\nThe provider slows down. Your requests start taking 20 seconds instead of 2. Your timeout fires at 10 and you retry. Now there are two requests in flight for one user. The first one is still being generated, and still being billed. Every user in the queue does the same. Your concurrent connections triple, you hit your own rate limit, so now you are generating 429s yourself, and your retry logic responds to those by retrying. Your dashboard shows an outage caused by the provider. In truth the provider was merely slow, and you converted slow into down.\n\nThe defences are all boring:\n\n- **A concurrency cap**: a semaphore around outbound calls. Above it, queue or shed load. A number you chose beats a number your connection pool chose for you.\n- **A circuit breaker.** After N consecutive failures, stop calling for 30 seconds and fail fast. This is counter-intuitive and correct: when a dependency is unhealthy, the kindest thing you can do is stop sending it traffic.\n- **A total budget per user request**: attempts and seconds, both capped.\n- **Shed the retry on non-interactive paths.** A background job can wait a minute. A user cannot.\n\n## Timeouts on a streaming call\n\nOne subtlety. A total timeout is wrong for streaming, because a long answer legitimately keeps the connection open for a minute. What you want is an **inactivity timeout**: abort if no chunk has arrived in 15 seconds. A stalled stream is a failure; a slow one is not.\n\n## Retries and duplicates\n\nA retried call is a second generation, billed separately, producing different text. That is fine when the call is a pure question. It is not fine when the call has already caused a side effect: a tool that sent an email, wrote a row, issued a refund.\n\nSo: **keep side effects out of the retried region**, or make them idempotent. If your loop retries an entire agent turn, and step three of that turn charged a card, a network blip has now charged it twice. Give every externally visible action a key derived from the request, store the key, and make the second attempt a no-op. This is ordinary distributed-systems hygiene, and putting a model in the loop does not exempt you from it: it makes it more likely, because agent loops retry more.\n\n> Figure: How a retry loop turns slow into down. Steps: 1. The provider slows down (Requests that took two seconds take twenty. Nothing has failed.); 2. Your ten-second timeout fires (The first generation continues on their side, and is still billed.); 3. You retry, so two calls are in flight per user (Every user in the queue does the same. Concurrency triples.); 4. You cross your own rate limit (Now you are generating 429s yourself, without any help from them.); 5. The loop retries the 429s (An attempt counter with no deadline keeps spending the request's budget.); 6. Your service is down (The dashboard blames the provider. The provider was only slow.). Nothing in this chain is a provider outage. Every step is your own code responding reasonably to the step before it. A concurrency cap breaks it at step three, a circuit breaker at step five, and a wall-clock deadline at step six."},"check":{"question":"During a provider slowdown, a service's own error rate goes from 0% to 60% even though the provider never returns errors, only slow responses. What is the most likely mechanism?","options":[{"text":"Slow responses are truncated by the provider, so the results fail validation.","correct":false,"tempting":"Confuses latency with truncation; a slow response still completes normally and carries a normal stop reason."},{"text":"The model degrades under load and produces malformed output.","correct":false,"tempting":"Attributes a queueing problem to generation quality, when the same weights produce the same distribution regardless of how busy the servers are."},{"text":"Timeouts fire and trigger retries while the original requests are still generating, multiplying in-flight calls until the client's own rate limit and connection pool fail.","correct":true},{"text":"The retries are being served by a different, weaker model as a fallback.","correct":false,"tempting":"Assumes a fallback that only exists if you wrote one, and would not by itself produce a sixfold error rate."}]},"prev":{"slug":"reading-the-response","title":"Every field in the response, and the two that catch bugs","url":"https://addaly.com/learn/building-with-ai/reading-the-response"},"next":{"slug":"choosing-a-model","title":"Choosing a model, and designing so you can change your mind","url":"https://addaly.com/learn/building-with-ai/choosing-a-model"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Errors, retries, and how a retry loop takes down your own service', in Building with AI. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/building-with-ai/retries-that-do-not-make-it-worse (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Errors, retries, and how a retry loop takes down your own service","courseTitle":"Building with AI","module":{"index":1,"slug":"the-call-underneath-everything","title":"The call underneath everything"},"answer":null,"keyIdea":"Retry only what a second attempt could fix, with jittered backoff under a wall-clock deadline and a concurrency cap, or your retry logic will convert a slow provider into your own outage.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"type":"flow","steps":[{"label":"The provider slows down","detail":"Requests that took two seconds take twenty. Nothing has failed."},{"label":"Your ten-second timeout fires","detail":"The first generation continues on their side, and is still billed."},{"label":"You retry, so two calls are in flight per user","detail":"Every user in the queue does the same. Concurrency triples."},{"label":"You cross your own rate limit","detail":"Now you are generating 429s yourself, without any help from them."},{"label":"The loop retries the 429s","detail":"An attempt counter with no deadline keeps spending the request’s budget."},{"label":"Your service is down","detail":"The dashboard blames the provider. The provider was only slow."}],"title":"How a retry loop turns slow into down","caption":"Nothing in this chain is a provider outage. Every step is your own code responding reasonably to the step before it. A concurrency cap breaks it at step three, a circuit breaker at step five, and a wall-clock deadline at step six."},"alt":"Figure: How a retry loop turns slow into down. Steps: 1. The provider slows down (Requests that took two seconds take twenty. Nothing has failed.); 2. Your ten-second timeout fires (The first generation continues on their side, and is still billed.); 3. You retry, so two calls are in flight per user (Every user in the queue does the same. Concurrency triples.); 4. You cross your own rate limit (Now you are generating 429s yourself, without any help from them.); 5. The loop retries the 429s (An attempt counter with no deadline keeps spending the request’s budget.); 6. Your service is down (The dashboard blames the provider. The provider was only slow.). Nothing in this chain is a provider outage. Every step is your own code responding reasonably to the step before it. A concurrency cap breaks it at step three, a circuit breaker at step five, and a wall-clock deadline at step six."}},"blocks":[{"t":"h2","id":"not-all-failures-are-the-same-failure","runs":[{"text":"Not all failures are the same failure"}]},{"t":"p","runs":[{"text":"Every model API returns a small set of status codes, and the only useful question about any of them is: "},{"text":"will doing this again help?","i":true}]},{"t":"ul","items":[[{"text":"400 Bad Request","b":true},{"text":": your JSON is malformed, or the prompt exceeds the context window. Retrying sends the same broken request again. Never retry."}],[{"text":"401 / 403","b":true},{"text":": the key is wrong, revoked, or lacks permission. Never retry. Page someone."}],[{"text":"404","b":true},{"text":": usually a model name that no longer exists, which happens when a provider retires a version. Never retry; fall back to another model if you have one."}],[{"text":"413 / context length exceeded","b":true},{"text":": too much input. Retrying is pointless. Trim and resend, once."}],[{"text":"429 Too Many Requests","b":true},{"text":": you hit a rate limit. Retry, but only after waiting, and only as many times as the deadline allows."}],[{"text":"500 / 502 / 503","b":true},{"text":": something broke on their side. Retry."}],[{"text":"529 / “overloaded”","b":true},{"text":": the provider is at capacity. Retry, with a longer wait than a 500."}],[{"text":"Timeouts and dropped sockets","b":true},{"text":": retry, with the caveat below."}]]},{"t":"p","runs":[{"text":"The distinction that matters is between "},{"text":"retryable","b":true},{"text":" and "},{"text":"not retryable","b":true},{"text":", and about half of production incidents involving model APIs come from code that retried something in the second group."}]},{"t":"h2","id":"backoff-with-jitter-and-why-jitter-is-not-optional","runs":[{"text":"Backoff, with jitter, and why jitter is not optional"}]},{"t":"code","lang":"python","label":"Python","text":"import random, time\n\nRETRYABLE = {429, 500, 502, 503, 529}\n\ndef call_with_retries(fn, attempts=4, deadline_s=25):\n    started = time.monotonic()\n    for i in range(attempts):\n        try:\n            return fn()\n        except APIError as e:\n            if e.status not in RETRYABLE or i == attempts - 1:\n                raise\n            wait = e.retry_after or min(2 ** i, 8) * (0.5 + random.random())\n            if time.monotonic() - started + wait > deadline_s:\n                raise DeadlineExceeded() from e\n            time.sleep(wait)"},{"t":"p","runs":[{"text":"Three things in that snippet are load-bearing."}]},{"t":"p","runs":[{"text":"retry_after","b":true,"code":true},{"text":" first.","b":true},{"text":" Rate-limit responses usually carry a "},{"text":"retry-after","code":true},{"text":" header telling you exactly how long the window is. Obeying it beats guessing."}]},{"t":"p","runs":[{"text":"Random jitter.","b":true},{"text":" Without it, every client that failed at the same instant retries at the same instant. A provider recovering from a blip gets hit by a perfectly synchronised wall of traffic and falls over again. This is the thundering herd, and multiplying the wait by a random factor between 0.5 and 1.5 is the entire fix."}]},{"t":"p","runs":[{"text":"A deadline, not just an attempt count.","b":true},{"text":" Four attempts with exponential backoff can consume 15 seconds. If the user’s request already had a 10-second budget, three of those attempts were spent answering a browser tab that closed. Retries must respect the wall clock of the request that spawned them, not their own counter."}]},{"t":"h2","id":"the-retry-storm-that-is-your-fault","runs":[{"text":"The retry storm that is your fault"}]},{"t":"p","runs":[{"text":"Here is the failure mode worth internalising, because it is self-inflicted."}]},{"t":"p","runs":[{"text":"The provider slows down. Your requests start taking 20 seconds instead of 2. Your timeout fires at 10 and you retry. Now there are two requests in flight for one user. The first one is still being generated, and still being billed. Every user in the queue does the same. Your concurrent connections triple, you hit your own rate limit, so now you are generating 429s yourself, and your retry logic responds to those by retrying. Your dashboard shows an outage caused by the provider. In truth the provider was merely slow, and you converted slow into down."}]},{"t":"p","runs":[{"text":"The defences are all boring:"}]},{"t":"ul","items":[[{"text":"A concurrency cap","b":true},{"text":": a semaphore around outbound calls. Above it, queue or shed load. A number you chose beats a number your connection pool chose for you."}],[{"text":"A circuit breaker.","b":true},{"text":" After N consecutive failures, stop calling for 30 seconds and fail fast. This is counter-intuitive and correct: when a dependency is unhealthy, the kindest thing you can do is stop sending it traffic."}],[{"text":"A total budget per user request","b":true},{"text":": attempts and seconds, both capped."}],[{"text":"Shed the retry on non-interactive paths.","b":true},{"text":" A background job can wait a minute. A user cannot."}]]},{"t":"h2","id":"timeouts-on-a-streaming-call","runs":[{"text":"Timeouts on a streaming call"}]},{"t":"p","runs":[{"text":"One subtlety. A total timeout is wrong for streaming, because a long answer legitimately keeps the connection open for a minute. What you want is an "},{"text":"inactivity timeout","b":true},{"text":": abort if no chunk has arrived in 15 seconds. A stalled stream is a failure; a slow one is not."}]},{"t":"h2","id":"retries-and-duplicates","runs":[{"text":"Retries and duplicates"}]},{"t":"p","runs":[{"text":"A retried call is a second generation, billed separately, producing different text. That is fine when the call is a pure question. It is not fine when the call has already caused a side effect: a tool that sent an email, wrote a row, issued a refund."}]},{"t":"p","runs":[{"text":"So: "},{"text":"keep side effects out of the retried region","b":true},{"text":", or make them idempotent. If your loop retries an entire agent turn, and step three of that turn charged a card, a network blip has now charged it twice. Give every externally visible action a key derived from the request, store the key, and make the second attempt a no-op. This is ordinary distributed-systems hygiene, and putting a model in the loop does not exempt you from it: it makes it more likely, because agent loops retry more."}]}],"check":{"question":[{"text":"During a provider slowdown, a service’s own error rate goes from 0% to 60% even though the provider never returns errors, only slow responses. What is the most likely mechanism?"}],"options":[{"text":[{"text":"Slow responses are truncated by the provider, so the results fail validation."}],"correct":false,"tempting":[{"text":"Confuses latency with truncation; a slow response still completes normally and carries a normal stop reason."}]},{"text":[{"text":"The model degrades under load and produces malformed output."}],"correct":false,"tempting":[{"text":"Attributes a queueing problem to generation quality, when the same weights produce the same distribution regardless of how busy the servers are."}]},{"text":[{"text":"Timeouts fire and trigger retries while the original requests are still generating, multiplying in-flight calls until the client’s own rate limit and connection pool fail."}],"correct":true,"tempting":null},{"text":[{"text":"The retries are being served by a different, weaker model as a fallback."}],"correct":false,"tempting":[{"text":"Assumes a fallback that only exists if you wrote one, and would not by itself produce a sixfold error rate."}]}],"order":[3,2,0,1],"why":null},"prev":{"slug":"reading-the-response","title":"Every field in the response, and the two that catch bugs","minutes":8},"next":{"slug":"choosing-a-model","title":"Choosing a model, and designing so you can change your mind","minutes":9}}}