{"course":{"slug":"python-for-ai","title":"Python, from Zero, for AI","url":"https://addaly.com/learn/python-for-ai","level":"beginner","track":"build"},"lesson":{"slug":"asyncio-and-concurrent-calls","title":"asyncio: twenty calls in the time of one","url":"https://addaly.com/learn/python-for-ai/asyncio-and-concurrent-calls","position":56,"of":89,"minutes":10,"answer":"async lets one thread overlap the waiting of many network calls, so twenty one-second requests finish in about a second; gather them, cap them with a Semaphore so the rate limit holds.","keyIdea":"async lets one thread overlap the waiting of many network calls, so twenty one-second requests finish in about a second; gather them, cap them with a Semaphore so the rate limit holds, and remember that any blocking call inside a coroutine stalls all of them."},"body":{"format":"markdown","text":"## Where the time goes\n\nCall a model API in a loop and time it:\n\n```python\nfor prompt in prompts:          # 20 prompts, about 1 second each\n    results.append(ask(prompt)) # ≈ 20 seconds\n```\n\nYour program does almost nothing during those twenty seconds. It sends a request, then waits, then sends the next. The CPU is idle; the network is idle most of the time too. The waiting is the cost, and the waits could overlap: if all twenty requests were in flight at once, the whole batch would take about as long as the slowest one.\n\n`asyncio` is Python's way of overlapping waits on a single thread.\n\n> Figure: Wall-clock time for one-second API calls. Line chart of Seconds until all are done against Number of one-second requests: Plain loop, one at a time (5, 5) (10, 10) (20, 20) (50, 50) (100, 100); asyncio.gather with Semaphore(5) (5, 1.1) (10, 2.1) (20, 4.1) (50, 10.2) (100, 20.2). Sequential time is the sum of the waits. Concurrent time is the longest wait, plus a little. The Semaphore line is what respecting a rate limit costs (five at a time rather than all at once) and it is still an order of magnitude faster than the loop. None of this makes one request quicker.\n\n## The shape\n\n```python\nimport asyncio, httpx\n\nasync def ask(client, prompt):\n    r = await client.post(url, headers=headers, json=payload_for(prompt), timeout=60)\n    r.raise_for_status()\n    return r.json()\n\nasync def main(prompts):\n    async with httpx.AsyncClient() as client:\n        tasks = [ask(client, p) for p in prompts]\n        return await asyncio.gather(*tasks)\n\nresults = asyncio.run(main(prompts))    # ≈ 1–2 seconds for 20\n```\n\nRead it as three new words.\n\n`async def` declares a **coroutine**: a function that can pause. Calling it does not run it; it returns a coroutine object that the event loop will run.\n\n`await` marks a pause point: \"this will take a while: run something else and come back when it is done\". Only things designed for asyncio can be awaited: `httpx` calls, `asyncio.sleep`, other coroutines.\n\n`asyncio.gather` starts all the coroutines and waits for all of them, returning their results in the same order as the inputs.\n\n`asyncio.run` creates the event loop, runs `main` to completion, and shuts the loop down. It is the one place your synchronous program hands over to the asynchronous one.\n\n## Why requests cannot do this\n\n`requests` is synchronous. Its `get` blocks the thread until the reply arrives, and asyncio has no way to pause it. That is why the example uses `httpx`, whose `AsyncClient` mirrors the `requests` API but with `await`. The provider SDKs offer the same split: `AsyncOpenAI` and `AsyncAnthropic` have identical methods to their synchronous twins, each awaited.\n\n## The one rule\n\nInside a coroutine, never call anything that blocks. `time.sleep(1)` blocks. So does `requests.get`, reading a large file with plain `open`, and a heavy loop of arithmetic. Each of them freezes the single thread the event loop lives on, and every other coroutine (all twenty in-flight requests) stops advancing until it returns. The symptom is that your beautifully concurrent code runs exactly as slowly as the loop it replaced, and nothing tells you why.\n\nThe replacements: `await asyncio.sleep(1)`, `await client.get(...)`, and for CPU work or a stubborn synchronous library, `await asyncio.to_thread(func, args)`, which runs the blocking call on a worker thread and awaits the result.\n\n## Respecting the rate limit\n\nTwenty at once is fine. Two thousand at once is a 429 storm and possibly a suspended key. Cap the concurrency:\n\n```python\nsem = asyncio.Semaphore(5)\n\nasync def ask(client, prompt):\n    async with sem:\n        r = await client.post(...)\n        ...\n```\n\nA `Semaphore(5)` lets five coroutines through the `async with` at a time; the sixth waits until one finishes. All two thousand tasks still exist and gather still returns them in order; only the in-flight count is limited. Set it from the provider's published limit and the size of your requests. Combine it with the backoff from earlier in this module, because five concurrent requests can still hit a per-minute token limit.\n\n## Errors in a batch\n\nBy default, `gather` raises the first exception and the others keep running in the background with their results lost. For a batch where one failure should not discard the rest:\n\n```python\nresults = await asyncio.gather(*tasks, return_exceptions=True)\nfor prompt, res in zip(prompts, results):\n    if isinstance(res, Exception):\n        log.warning(\"failed %s: %s\", prompt[:30], res)\n    else:\n        save(prompt, res)\n```\n\nEach slot is either a result or the exception object. Check with `isinstance` before using it.\n\n## The warning you will see\n\n```\nRuntimeWarning: coroutine 'ask' was never awaited\n```\n\nThis means you called `ask(prompt)` and did nothing with the returned coroutine: no `await`, no `gather`, no `create_task`. Nothing ran. It is the async version of forgetting to call a function, and it is the first thing to check when an async program finishes suspiciously fast with no results.\n\n## Where async does not help\n\nAsync overlaps *waiting*. It does not make a single request faster, and it does nothing for CPU-bound work such as parsing a large file or computing similarities over a million vectors, because there is still one thread. For that, the next lesson covers threads and processes and the reason Python needs both.\n\nAsync also spreads. A function that awaits must be `async`, so its caller must be `async`, and so on up to `asyncio.run`. That is fine for a program built around network calls; it is awkward to bolt onto a synchronous one. When only one corner needs concurrency, a thread pool is often the smaller change.\n\n## Try this now\n\nPoint the example at Ollama's local endpoint with ten short prompts and time it against the plain loop. Then insert `time.sleep(0.5)` inside `ask`, watch the total climb, and replace it with `await asyncio.sleep(0.5)` to watch it fall again."},"check":{"question":"A developer converts a loop of 50 API calls to `asyncio.gather` with `httpx.AsyncClient`. Total time drops from 50 seconds to 2. They then add a line inside the coroutine that saves each result with `time.sleep(1)` to \"go easy on the disk\", and total time goes back to about 52 seconds. Why?","options":[{"text":"`asyncio.gather` runs coroutines in creation order, so sleeps cannot overlap regardless of what function is used.","correct":false,"tempting":"gather interleaves coroutines whenever they await; the ordering claim is false and the earlier 2-second result proves it."},{"text":"The event loop enforces a one-second minimum between task switches to prevent starvation.","correct":false,"tempting":"There is no such rule; the loop switches whenever a coroutine awaits, with no minimum interval."},{"text":"`time.sleep` blocks the one thread the event loop runs on, so every other coroutine stalls until it returns; `await asyncio.sleep(1)` would yield.","correct":true},{"text":"Disk writes are synchronous in every language, so the saves serialise the coroutines whatever sleep is used, and the only fix is to move saving into a separate process.","correct":false,"tempting":"The saves are not the slow part; a one-second blocking sleep per call is, and removing the blocking sleep restores the overlap."}]},"prev":{"slug":"consuming-a-stream","title":"Consuming a streamed reply: server-sent events, line by line","url":"https://addaly.com/learn/python-for-ai/consuming-a-stream"},"next":{"slug":"threads-processes-and-the-gil","title":"Threads, processes, and the lock that decides which one you need","url":"https://addaly.com/learn/python-for-ai/threads-processes-and-the-gil"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Source and assessment review continues. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'asyncio: twenty calls in the time of one', in Python, from Zero, for AI. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/python-for-ai/asyncio-and-concurrent-calls (Accessed: 21 November 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"asyncio: twenty calls in the time of one","courseTitle":"Python, from Zero, for AI","module":{"index":6,"slug":"talking-to-a-service","title":"Talking to a service: HTTP from Python, done properly"},"answer":null,"keyIdea":"async lets one thread overlap the waiting of many network calls, so twenty one-second requests finish in about a second; gather them, cap them with a Semaphore so the rate limit holds, and remember that any blocking call inside a coroutine stalls all of them.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"caption":"Sequential time is the sum of the waits. Concurrent time is the longest wait, plus a little. The Semaphore line is what respecting a rate limit costs (five at a time rather than all at once) and it is still an order of magnitude faster than the loop. None of this makes one request quicker.","series":[{"label":"Plain loop, one at a time","points":[[5,5],[10,10],[20,20],[50,50],[100,100]]},{"label":"asyncio.gather with Semaphore(5)","points":[[5,1.1],[10,2.1],[20,4.1],[50,10.2],[100,20.2]]}],"title":"Wall-clock time for one-second API calls","type":"line","xLabel":"Number of one-second requests","yLabel":"Seconds until all are done"},"alt":"Figure: Wall-clock time for one-second API calls. Line chart of Seconds until all are done against Number of one-second requests: Plain loop, one at a time (5, 5) (10, 10) (20, 20) (50, 50) (100, 100); asyncio.gather with Semaphore(5) (5, 1.1) (10, 2.1) (20, 4.1) (50, 10.2) (100, 20.2). Sequential time is the sum of the waits. Concurrent time is the longest wait, plus a little. The Semaphore line is what respecting a rate limit costs (five at a time rather than all at once) and it is still an order of magnitude faster than the loop. None of this makes one request quicker."}},"blocks":[{"t":"h2","id":"where-the-time-goes","runs":[{"text":"Where the time goes"}]},{"t":"p","runs":[{"text":"Call a model API in a loop and time it:"}]},{"t":"code","lang":"python","label":"Python","text":"for prompt in prompts:          # 20 prompts, about 1 second each\n    results.append(ask(prompt)) # ≈ 20 seconds"},{"t":"p","runs":[{"text":"Your program does almost nothing during those twenty seconds. It sends a request, then waits, then sends the next. The CPU is idle; the network is idle most of the time too. The waiting is the cost, and the waits could overlap: if all twenty requests were in flight at once, the whole batch would take about as long as the slowest one."}]},{"t":"p","runs":[{"text":"asyncio","code":true},{"text":" is Python’s way of overlapping waits on a single thread."}]},{"t":"h2","id":"the-shape","runs":[{"text":"The shape"}]},{"t":"code","lang":"python","label":"Python","text":"import asyncio, httpx\n\nasync def ask(client, prompt):\n    r = await client.post(url, headers=headers, json=payload_for(prompt), timeout=60)\n    r.raise_for_status()\n    return r.json()\n\nasync def main(prompts):\n    async with httpx.AsyncClient() as client:\n        tasks = [ask(client, p) for p in prompts]\n        return await asyncio.gather(*tasks)\n\nresults = asyncio.run(main(prompts))    # ≈ 1–2 seconds for 20"},{"t":"p","runs":[{"text":"Read it as three new words."}]},{"t":"p","runs":[{"text":"async def","code":true},{"text":" declares a "},{"text":"coroutine","b":true},{"text":": a function that can pause. Calling it does not run it; it returns a coroutine object that the event loop will run."}]},{"t":"p","runs":[{"text":"await","code":true},{"text":" marks a pause point: “this will take a while: run something else and come back when it is done”. Only things designed for asyncio can be awaited: "},{"text":"httpx","code":true},{"text":" calls, "},{"text":"asyncio.sleep","code":true},{"text":", other coroutines."}]},{"t":"p","runs":[{"text":"asyncio.gather","code":true},{"text":" starts all the coroutines and waits for all of them, returning their results in the same order as the inputs."}]},{"t":"p","runs":[{"text":"asyncio.run","code":true},{"text":" creates the event loop, runs "},{"text":"main","code":true},{"text":" to completion, and shuts the loop down. It is the one place your synchronous program hands over to the asynchronous one."}]},{"t":"h2","id":"why-requests-cannot-do-this","runs":[{"text":"Why requests cannot do this"}]},{"t":"p","runs":[{"text":"requests","code":true},{"text":" is synchronous. Its "},{"text":"get","code":true},{"text":" blocks the thread until the reply arrives, and asyncio has no way to pause it. That is why the example uses "},{"text":"httpx","code":true},{"text":", whose "},{"text":"AsyncClient","code":true},{"text":" mirrors the "},{"text":"requests","code":true},{"text":" API but with "},{"text":"await","code":true},{"text":". The provider SDKs offer the same split: "},{"text":"AsyncOpenAI","code":true},{"text":" and "},{"text":"AsyncAnthropic","code":true},{"text":" have identical methods to their synchronous twins, each awaited."}]},{"t":"h2","id":"the-one-rule","runs":[{"text":"The one rule"}]},{"t":"p","runs":[{"text":"Inside a coroutine, never call anything that blocks. "},{"text":"time.sleep(1)","code":true},{"text":" blocks. So does "},{"text":"requests.get","code":true},{"text":", reading a large file with plain "},{"text":"open","code":true},{"text":", and a heavy loop of arithmetic. Each of them freezes the single thread the event loop lives on, and every other coroutine (all twenty in-flight requests) stops advancing until it returns. The symptom is that your beautifully concurrent code runs exactly as slowly as the loop it replaced, and nothing tells you why."}]},{"t":"p","runs":[{"text":"The replacements: "},{"text":"await asyncio.sleep(1)","code":true},{"text":", "},{"text":"await client.get(...)","code":true},{"text":", and for CPU work or a stubborn synchronous library, "},{"text":"await asyncio.to_thread(func, args)","code":true},{"text":", which runs the blocking call on a worker thread and awaits the result."}]},{"t":"h2","id":"respecting-the-rate-limit","runs":[{"text":"Respecting the rate limit"}]},{"t":"p","runs":[{"text":"Twenty at once is fine. Two thousand at once is a 429 storm and possibly a suspended key. Cap the concurrency:"}]},{"t":"code","lang":"python","label":"Python","text":"sem = asyncio.Semaphore(5)\n\nasync def ask(client, prompt):\n    async with sem:\n        r = await client.post(...)\n        ..."},{"t":"p","runs":[{"text":"A "},{"text":"Semaphore(5)","code":true},{"text":" lets five coroutines through the "},{"text":"async with","code":true},{"text":" at a time; the sixth waits until one finishes. All two thousand tasks still exist and gather still returns them in order; only the in-flight count is limited. Set it from the provider’s published limit and the size of your requests. Combine it with the backoff from earlier in this module, because five concurrent requests can still hit a per-minute token limit."}]},{"t":"h2","id":"errors-in-a-batch","runs":[{"text":"Errors in a batch"}]},{"t":"p","runs":[{"text":"By default, "},{"text":"gather","code":true},{"text":" raises the first exception and the others keep running in the background with their results lost. For a batch where one failure should not discard the rest:"}]},{"t":"code","lang":"python","label":"Python","text":"results = await asyncio.gather(*tasks, return_exceptions=True)\nfor prompt, res in zip(prompts, results):\n    if isinstance(res, Exception):\n        log.warning(\"failed %s: %s\", prompt[:30], res)\n    else:\n        save(prompt, res)"},{"t":"p","runs":[{"text":"Each slot is either a result or the exception object. Check with "},{"text":"isinstance","code":true},{"text":" before using it."}]},{"t":"h2","id":"the-warning-you-will-see","runs":[{"text":"The warning you will see"}]},{"t":"code","lang":"","label":"Text","text":"RuntimeWarning: coroutine 'ask' was never awaited"},{"t":"p","runs":[{"text":"This means you called "},{"text":"ask(prompt)","code":true},{"text":" and did nothing with the returned coroutine: no "},{"text":"await","code":true},{"text":", no "},{"text":"gather","code":true},{"text":", no "},{"text":"create_task","code":true},{"text":". Nothing ran. It is the async version of forgetting to call a function, and it is the first thing to check when an async program finishes suspiciously fast with no results."}]},{"t":"h2","id":"where-async-does-not-help","runs":[{"text":"Where async does not help"}]},{"t":"p","runs":[{"text":"Async overlaps "},{"text":"waiting","i":true},{"text":". It does not make a single request faster, and it does nothing for CPU-bound work such as parsing a large file or computing similarities over a million vectors, because there is still one thread. For that, the next lesson covers threads and processes and the reason Python needs both."}]},{"t":"p","runs":[{"text":"Async also spreads. A function that awaits must be "},{"text":"async","code":true},{"text":", so its caller must be "},{"text":"async","code":true},{"text":", and so on up to "},{"text":"asyncio.run","code":true},{"text":". That is fine for a program built around network calls; it is awkward to bolt onto a synchronous one. When only one corner needs concurrency, a thread pool is often the smaller change."}]},{"t":"h2","id":"try-this-now","runs":[{"text":"Try this now"}],"role":"try"},{"t":"p","runs":[{"text":"Point the example at Ollama’s local endpoint with ten short prompts and time it against the plain loop. Then insert "},{"text":"time.sleep(0.5)","code":true},{"text":" inside "},{"text":"ask","code":true},{"text":", watch the total climb, and replace it with "},{"text":"await asyncio.sleep(0.5)","code":true},{"text":" to watch it fall again."}]}],"check":{"question":[{"text":"A developer converts a loop of 50 API calls to "},{"text":"asyncio.gather","code":true},{"text":" with "},{"text":"httpx.AsyncClient","code":true},{"text":". Total time drops from 50 seconds to 2. They then add a line inside the coroutine that saves each result with "},{"text":"time.sleep(1)","code":true},{"text":" to “go easy on the disk”, and total time goes back to about 52 seconds. Why?"}],"options":[{"text":[{"text":"asyncio.gather","code":true},{"text":" runs coroutines in creation order, so sleeps cannot overlap regardless of what function is used."}],"correct":false,"tempting":[{"text":"gather interleaves coroutines whenever they await; the ordering claim is false and the earlier 2-second result proves it."}]},{"text":[{"text":"The event loop enforces a one-second minimum between task switches to prevent starvation."}],"correct":false,"tempting":[{"text":"There is no such rule; the loop switches whenever a coroutine awaits, with no minimum interval."}]},{"text":[{"text":"time.sleep","code":true},{"text":" blocks the one thread the event loop runs on, so every other coroutine stalls until it returns; "},{"text":"await asyncio.sleep(1)","code":true},{"text":" would yield."}],"correct":true,"tempting":null},{"text":[{"text":"Disk writes are synchronous in every language, so the saves serialise the coroutines whatever sleep is used, and the only fix is to move saving into a separate process."}],"correct":false,"tempting":[{"text":"The saves are not the slow part; a one-second blocking sleep per call is, and removing the blocking sleep restores the overlap."}]}],"order":[1,3,2,0],"why":null},"prev":{"slug":"consuming-a-stream","title":"Consuming a streamed reply: server-sent events, line by line","minutes":9},"next":{"slug":"threads-processes-and-the-gil","title":"Threads, processes, and the lock that decides which one you need","minutes":9}}}