{"course":{"slug":"python-for-ai","title":"Python, from Zero, for AI","url":"https://addaly.com/learn/python-for-ai","level":"beginner","track":"build"},"lesson":{"slug":"timeouts-retries-and-backoff","title":"Timeouts and retries: the code that decides whether you pay twice","url":"https://addaly.com/learn/python-for-ai/timeouts-retries-and-backoff","position":52,"of":89,"minutes":9,"answer":"Retry a request only when you can prove it did not take effect (a connect failure, a 429, a 503) and never blindly retry a POST that timed out while reading.","keyIdea":"Retry a request only when you can prove it did not take effect (a connect failure, a 429, a 503) and never blindly retry a POST that timed out while reading, because the server may have finished the work and billed you before the reply was lost."},"body":{"format":"markdown","text":"## Two timeouts, not one\n\n`timeout=10` is shorthand for a pair:\n\n```python\nr = session.post(url, json=payload, timeout=(5, 120))\n```\n\nThe first number is the **connect** timeout: how long to wait for the server to accept the connection. If this expires, nothing was sent, and you know it. The second is the **read** timeout: how long to wait, between bytes, for the server to send something. If this expires, the request was sent and the server may be working on it, or may have finished.\n\nFor a model API the two need different values. Connecting should take under a second; five is generous. Generating 1,000 tokens can legitimately take a minute, so a read timeout of 10 seconds guarantees failures on the longest, most expensive requests: precisely the ones you least want to abandon. Set it from the longest response you expect, then add half again.\n\nWithout a timeout at all, a call can hang forever. A hung process looks like a slow one until you notice it has been three hours.\n\n## The retry question is about state, not errors\n\nA retry is safe when the failed attempt provably did nothing on the server. Sort failures by that test.\n\n**Safe to retry, because nothing happened:**\n\n- A connect timeout or a DNS failure. The request never left.\n- A `ConnectionError` raised while sending. Almost always nothing was processed.\n- A **429**: the server refused the request because you are over your rate limit. It did no work.\n- A **503**: the server said it was unavailable and did no work.\n\n**Not safe to retry blindly:**\n\n- A **read timeout** on a `POST`. The server received the request. A model API will generate the full response, count the tokens, bill your account, and send a reply that nobody is listening for. Retrying makes a second charged generation.\n- A **500**. Something broke; whether it broke before or after the work was done is unknown.\n- A **400**, **401**, **403**, **404**. Retrying the same wrong request produces the same answer.\n\n`GET` requests are different: they are defined to have no side effects, so any of them can be retried. This is what the word **idempotent** means in the documentation: repeating it changes nothing further. A `GET` is idempotent by definition. A `POST` that creates a chat completion is not.\n\n> Figure: Which failures are safe to retry. A two by two grid, rows Failed to connect and Read timed out, columns GET and POST that bills: Failed to connect and GET: Retry; Failed to connect and POST that bills: Retry; Read timed out and GET: Retry; Read timed out and POST that bills: Do not retry. The question is never how bad the error looks; it is whether the attempt provably did nothing. 429 and 503 are safe for the same reason as a connect failure: both mean not now. A 500 is not, because you cannot tell whether it broke before or after the work. An Idempotency-Key, where the provider offers one, makes the bottom right cell safe.\n\nSome providers accept an `Idempotency-Key` header: you generate a random ID per logical request, send it on every attempt, and the server returns the first result rather than redoing the work. Where it exists, use it; then a read-timeout retry is safe.\n\n## Backoff, and why it is exponential\n\nWhen a 429 arrives, retrying immediately is the worst response. The limit is per second or per minute; hammering it extends the ban. Wait, then wait longer:\n\n```python\nimport random, time\n\ndef wait_time(attempt, base=1.0, cap=60.0):\n    return min(cap, base * 2 ** attempt) + random.uniform(0, 1)\n```\n\nAttempt 0 waits about a second, then two, four, eight, capped at sixty. The random fraction is **jitter**. Without it, a hundred clients rejected at the same moment all retry at the same moment and are all rejected again; a little randomness spreads them out.\n\nIf the response carries a `Retry-After` header, the server has told you exactly how long to wait, and you should believe it over your own formula:\n\n```python\ndelay = float(r.headers.get(\"Retry-After\", wait_time(attempt)))\n```\n\n## A retry that respects the rules\n\n```python\nimport requests\n\nRETRY_STATUSES = {429, 502, 503, 504}\n\ndef post_with_retry(\n    session, url, payload, attempts=5, timeout=(5, 120)\n):\n    for attempt in range(attempts):\n        try:\n            r = session.post(\n                url, json=payload, timeout=timeout\n            )\n        except requests.ConnectTimeout:\n            # nothing was sent\n            time.sleep(wait_time(attempt))\n            continue\n        except requests.ReadTimeout:\n            # may have been billed; let the caller decide\n            raise\n        if r.status_code in RETRY_STATUSES:\n            delay = float(\n                r.headers.get(\n                    \"Retry-After\", wait_time(attempt)\n                )\n            )\n            time.sleep(delay)\n            continue\n        r.raise_for_status()\n        return r\n    raise RuntimeError(\n        f\"gave up after {attempts} attempts\"\n    )\n```\n\nThe `ReadTimeout` branch re-raises on purpose. The right response to \"the server may have done the work\" depends on what the work was, and that decision belongs one level up.\n\n## Letting urllib3 do it\n\n`requests` sits on `urllib3`, which has a `Retry` object that handles most of this:\n\n```python\nfrom requests.adapters import HTTPAdapter\nfrom urllib3.util.retry import Retry\n\nretry = Retry(\n    total=5,\n    backoff_factor=1,                       # 1, 2, 4, 8, 16 seconds\n    status_forcelist=[429, 502, 503, 504],\n    allowed_methods=[\"GET\", \"POST\"],        # POST is not retried unless you say so\n    respect_retry_after_header=True,\n)\nsession.mount(\"https://\", HTTPAdapter(max_retries=retry))\n```\n\nNotice `allowed_methods`. By default `Retry` does not retry `POST`, for exactly the reason above. Adding it is a decision you should make knowing what it switches on: with `POST` allowed, `Retry` also resends a `POST` whose reply never started arriving (a read timeout) and one that came back 502 or 504, which is exactly the double charge above. If you allow `POST`, also pass `read=False`, so a read timeout reaches you as `ReadTimeout` instead of being resent.\n\n## What the SDKs do\n\nThe `openai` and `anthropic` clients retry twice by default with backoff on 408, 409, 429 and 5xx, and honour `Retry-After`. They also retry connection errors and timeouts, a read timeout included, so with the defaults a slow generation can be sent, and billed, up to three times; set a longer `timeout=` or `max_retries=0` when that matters. `max_retries=` on the client changes the count. Knowing this stops you wrapping their calls in a second retry loop and multiplying the attempts.\n\n## Try this now\n\nPoint `post_with_retry` at `https://httpbin.org/status/503` and watch it back off and give up. Then at `https://httpbin.org/delay/5` with `timeout=(5, 2)` and confirm you get a `ReadTimeout`, not a retry. Print the attempt number and the delay each time round; the shape of the waits is the lesson."},"check":{"question":"A script sends a 2,000-token prompt with `timeout=10`. The connection opens fine, but after ten seconds of waiting for the reply `requests` raises `ReadTimeout`. The developer wraps the call in a loop that retries up to five times on any exception. What is the likely consequence?","options":[{"text":"The retries succeed on the second attempt because the server has cached the answer to the identical prompt.","correct":false,"tempting":"Model APIs do not cache full answers by prompt; each attempt is a fresh generation."},{"text":"Nothing changes, because a timed-out request is discarded by the server the moment the client stops listening, so no work is done and nothing is billed for the abandoned attempts.","correct":false,"tempting":"HTTP has no such mechanism; a server that has started generating keeps going and completes the request whether or not anyone is waiting for the reply."},{"text":"The server may finish and bill every attempt, so one slow reply becomes up to five charged generations, of which the script keeps one.","correct":true},{"text":"The loop raises `RecursionError` because retrying inside an exception handler re-enters the same frame.","correct":false,"tempting":"A loop does not recurse; it repeats in the same frame, and the exception handler exits normally each pass."}]},"prev":{"slug":"sessions-and-connection-reuse","title":"Sessions: paying for the handshake once instead of every call","url":"https://addaly.com/learn/python-for-ai/sessions-and-connection-reuse"},"next":{"slug":"pagination-and-cursors","title":"Pagination: walking a result that arrives in pages","url":"https://addaly.com/learn/python-for-ai/pagination-and-cursors"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Source and assessment review continues. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Timeouts and retries: the code that decides whether you pay twice', in Python, from Zero, for AI. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/python-for-ai/timeouts-retries-and-backoff (Accessed: 21 November 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Timeouts and retries: the code that decides whether you pay twice","courseTitle":"Python, from Zero, for AI","module":{"index":6,"slug":"talking-to-a-service","title":"Talking to a service: HTTP from Python, done properly"},"answer":null,"keyIdea":"Retry a request only when you can prove it did not take effect (a connect failure, a 429, a 503) and never blindly retry a POST that timed out while reading, because the server may have finished the work and billed you before the reply was lost.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"caption":"The question is never how bad the error looks; it is whether the attempt provably did nothing. 429 and 503 are safe for the same reason as a connect failure: both mean not now. A 500 is not, because you cannot tell whether it broke before or after the work. An Idempotency-Key, where the provider offers one, makes the bottom right cell safe.","cellLabels":[["nothing was sent","nothing was sent"],["no side effects","may already be done"]],"cells":[["Retry","Retry"],["Retry","Do not retry"]],"cols":["GET","POST that bills"],"rows":["Failed to connect","Read timed out"],"title":"Which failures are safe to retry","type":"matrix"},"alt":"Figure: Which failures are safe to retry. A two by two grid, rows Failed to connect and Read timed out, columns GET and POST that bills: Failed to connect and GET: Retry; Failed to connect and POST that bills: Retry; Read timed out and GET: Retry; Read timed out and POST that bills: Do not retry. The question is never how bad the error looks; it is whether the attempt provably did nothing. 429 and 503 are safe for the same reason as a connect failure: both mean not now. A 500 is not, because you cannot tell whether it broke before or after the work. An Idempotency-Key, where the provider offers one, makes the bottom right cell safe."}},"blocks":[{"t":"h2","id":"two-timeouts-not-one","runs":[{"text":"Two timeouts, not one"}]},{"t":"p","runs":[{"text":"timeout=10","code":true},{"text":" is shorthand for a pair:"}]},{"t":"code","lang":"python","label":"Python","text":"r = session.post(url, json=payload, timeout=(5, 120))"},{"t":"p","runs":[{"text":"The first number is the "},{"text":"connect","b":true},{"text":" timeout: how long to wait for the server to accept the connection. If this expires, nothing was sent, and you know it. The second is the "},{"text":"read","b":true},{"text":" timeout: how long to wait, between bytes, for the server to send something. If this expires, the request was sent and the server may be working on it, or may have finished."}]},{"t":"p","runs":[{"text":"For a model API the two need different values. Connecting should take under a second; five is generous. Generating 1,000 tokens can legitimately take a minute, so a read timeout of 10 seconds guarantees failures on the longest, most expensive requests: precisely the ones you least want to abandon. Set it from the longest response you expect, then add half again."}]},{"t":"p","runs":[{"text":"Without a timeout at all, a call can hang forever. A hung process looks like a slow one until you notice it has been three hours."}]},{"t":"h2","id":"the-retry-question-is-about-state-not-errors","runs":[{"text":"The retry question is about state, not errors"}]},{"t":"p","runs":[{"text":"A retry is safe when the failed attempt provably did nothing on the server. Sort failures by that test."}]},{"t":"p","runs":[{"text":"Safe to retry, because nothing happened:","b":true}]},{"t":"ul","items":[[{"text":"A connect timeout or a DNS failure. The request never left."}],[{"text":"A "},{"text":"ConnectionError","code":true},{"text":" raised while sending. Almost always nothing was processed."}],[{"text":"A "},{"text":"429","b":true},{"text":": the server refused the request because you are over your rate limit. It did no work."}],[{"text":"A "},{"text":"503","b":true},{"text":": the server said it was unavailable and did no work."}]]},{"t":"p","runs":[{"text":"Not safe to retry blindly:","b":true}]},{"t":"ul","items":[[{"text":"A "},{"text":"read timeout","b":true},{"text":" on a "},{"text":"POST","code":true},{"text":". The server received the request. A model API will generate the full response, count the tokens, bill your account, and send a reply that nobody is listening for. Retrying makes a second charged generation."}],[{"text":"A "},{"text":"500","b":true},{"text":". Something broke; whether it broke before or after the work was done is unknown."}],[{"text":"A "},{"text":"400","b":true},{"text":", "},{"text":"401","b":true},{"text":", "},{"text":"403","b":true},{"text":", "},{"text":"404","b":true},{"text":". Retrying the same wrong request produces the same answer."}]]},{"t":"p","runs":[{"text":"GET","code":true},{"text":" requests are different: they are defined to have no side effects, so any of them can be retried. This is what the word "},{"text":"idempotent","b":true},{"text":" means in the documentation: repeating it changes nothing further. A "},{"text":"GET","code":true},{"text":" is idempotent by definition. A "},{"text":"POST","code":true},{"text":" that creates a chat completion is not."}]},{"t":"p","runs":[{"text":"Some providers accept an "},{"text":"Idempotency-Key","code":true},{"text":" header: you generate a random ID per logical request, send it on every attempt, and the server returns the first result rather than redoing the work. Where it exists, use it; then a read-timeout retry is safe."}]},{"t":"h2","id":"backoff-and-why-it-is-exponential","runs":[{"text":"Backoff, and why it is exponential"}]},{"t":"p","runs":[{"text":"When a 429 arrives, retrying immediately is the worst response. The limit is per second or per minute; hammering it extends the ban. Wait, then wait longer:"}]},{"t":"code","lang":"python","label":"Python","text":"import random, time\n\ndef wait_time(attempt, base=1.0, cap=60.0):\n    return min(cap, base * 2 ** attempt) + random.uniform(0, 1)"},{"t":"p","runs":[{"text":"Attempt 0 waits about a second, then two, four, eight, capped at sixty. The random fraction is "},{"text":"jitter","b":true},{"text":". Without it, a hundred clients rejected at the same moment all retry at the same moment and are all rejected again; a little randomness spreads them out."}]},{"t":"p","runs":[{"text":"If the response carries a "},{"text":"Retry-After","code":true},{"text":" header, the server has told you exactly how long to wait, and you should believe it over your own formula:"}]},{"t":"code","lang":"python","label":"Python","text":"delay = float(r.headers.get(\"Retry-After\", wait_time(attempt)))"},{"t":"h2","id":"a-retry-that-respects-the-rules","runs":[{"text":"A retry that respects the rules"}]},{"t":"code","lang":"python","label":"Python","text":"import requests\n\nRETRY_STATUSES = {429, 502, 503, 504}\n\n\ndef post_with_retry(\n    session, url, payload, attempts=5, timeout=(5, 120)\n):\n    for attempt in range(attempts):\n        try:\n            r = session.post(\n                url, json=payload, timeout=timeout\n            )\n        except requests.ConnectTimeout:\n            # nothing was sent\n            time.sleep(wait_time(attempt))\n            continue\n        except requests.ReadTimeout:\n            # may have been billed; let the caller decide\n            raise\n        if r.status_code in RETRY_STATUSES:\n            delay = float(\n                r.headers.get(\n                    \"Retry-After\", wait_time(attempt)\n                )\n            )\n            time.sleep(delay)\n            continue\n        r.raise_for_status()\n        return r\n    raise RuntimeError(\n        f\"gave up after {attempts} attempts\"\n    )"},{"t":"p","runs":[{"text":"The "},{"text":"ReadTimeout","code":true},{"text":" branch re-raises on purpose. The right response to “the server may have done the work” depends on what the work was, and that decision belongs one level up."}]},{"t":"h2","id":"letting-urllib3-do-it","runs":[{"text":"Letting urllib3 do it"}]},{"t":"p","runs":[{"text":"requests","code":true},{"text":" sits on "},{"text":"urllib3","code":true},{"text":", which has a "},{"text":"Retry","code":true},{"text":" object that handles most of this:"}]},{"t":"code","lang":"python","label":"Python","text":"from requests.adapters import HTTPAdapter\nfrom urllib3.util.retry import Retry\n\nretry = Retry(\n    total=5,\n    backoff_factor=1,                       # 1, 2, 4, 8, 16 seconds\n    status_forcelist=[429, 502, 503, 504],\n    allowed_methods=[\"GET\", \"POST\"],        # POST is not retried unless you say so\n    respect_retry_after_header=True,\n)\nsession.mount(\"https://\", HTTPAdapter(max_retries=retry))"},{"t":"p","runs":[{"text":"Notice "},{"text":"allowed_methods","code":true},{"text":". By default "},{"text":"Retry","code":true},{"text":" does not retry "},{"text":"POST","code":true},{"text":", for exactly the reason above. Adding it is a decision you should make knowing what it switches on: with "},{"text":"POST","code":true},{"text":" allowed, "},{"text":"Retry","code":true},{"text":" also resends a "},{"text":"POST","code":true},{"text":" whose reply never started arriving (a read timeout) and one that came back 502 or 504, which is exactly the double charge above. If you allow "},{"text":"POST","code":true},{"text":", also pass "},{"text":"read=False","code":true},{"text":", so a read timeout reaches you as "},{"text":"ReadTimeout","code":true},{"text":" instead of being resent."}]},{"t":"h2","id":"what-the-sdks-do","runs":[{"text":"What the SDKs do"}]},{"t":"p","runs":[{"text":"The "},{"text":"openai","code":true},{"text":" and "},{"text":"anthropic","code":true},{"text":" clients retry twice by default with backoff on 408, 409, 429 and 5xx, and honour "},{"text":"Retry-After","code":true},{"text":". They also retry connection errors and timeouts, a read timeout included, so with the defaults a slow generation can be sent, and billed, up to three times; set a longer "},{"text":"timeout=","code":true},{"text":" or "},{"text":"max_retries=0","code":true},{"text":" when that matters. "},{"text":"max_retries=","code":true},{"text":" on the client changes the count. Knowing this stops you wrapping their calls in a second retry loop and multiplying the attempts."}]},{"t":"h2","id":"try-this-now","runs":[{"text":"Try this now"}],"role":"try"},{"t":"p","runs":[{"text":"Point "},{"text":"post_with_retry","code":true},{"text":" at "},{"text":"https://httpbin.org/status/503","code":true},{"text":" and watch it back off and give up. Then at "},{"text":"https://httpbin.org/delay/5","code":true},{"text":" with "},{"text":"timeout=(5, 2)","code":true},{"text":" and confirm you get a "},{"text":"ReadTimeout","code":true},{"text":", not a retry. Print the attempt number and the delay each time round; the shape of the waits is the lesson."}]}],"check":{"question":[{"text":"A script sends a 2,000-token prompt with "},{"text":"timeout=10","code":true},{"text":". The connection opens fine, but after ten seconds of waiting for the reply "},{"text":"requests","code":true},{"text":" raises "},{"text":"ReadTimeout","code":true},{"text":". The developer wraps the call in a loop that retries up to five times on any exception. What is the likely consequence?"}],"options":[{"text":[{"text":"The retries succeed on the second attempt because the server has cached the answer to the identical prompt."}],"correct":false,"tempting":[{"text":"Model APIs do not cache full answers by prompt; each attempt is a fresh generation."}]},{"text":[{"text":"Nothing changes, because a timed-out request is discarded by the server the moment the client stops listening, so no work is done and nothing is billed for the abandoned attempts."}],"correct":false,"tempting":[{"text":"HTTP has no such mechanism; a server that has started generating keeps going and completes the request whether or not anyone is waiting for the reply."}]},{"text":[{"text":"The server may finish and bill every attempt, so one slow reply becomes up to five charged generations, of which the script keeps one."}],"correct":true,"tempting":null},{"text":[{"text":"The loop raises "},{"text":"RecursionError","code":true},{"text":" because retrying inside an exception handler re-enters the same frame."}],"correct":false,"tempting":[{"text":"A loop does not recurse; it repeats in the same frame, and the exception handler exits normally each pass."}]}],"order":[3,1,0,2],"why":null},"prev":{"slug":"sessions-and-connection-reuse","title":"Sessions: paying for the handshake once instead of every call","minutes":7},"next":{"slug":"pagination-and-cursors","title":"Pagination: walking a result that arrives in pages","minutes":8}}}