asyncio: twenty calls in the time of one

Where the time goes

Call a model API in a loop and time it:

for prompt in prompts:          # 20 prompts, about 1 second each
    results.append(ask(prompt)) # ≈ 20 seconds

Your program does almost nothing during those twenty seconds. It sends a request, then waits, then sends the next. The CPU is idle; the network is idle most of the time too. The waiting is the cost, and the waits could overlap: if all twenty requests were in flight at once, the whole batch would take about as long as the slowest one.

asyncio is Python's way of overlapping waits on a single thread.

The shape

import asyncio, httpx

async def ask(client, prompt):
    r = await client.post(url, headers=headers, json=payload_for(prompt), timeout=60)
    r.raise_for_status()
    return r.json()

async def main(prompts):
    async with httpx.AsyncClient() as client:
        tasks = [ask(client, p) for p in prompts]
        return await asyncio.gather(*tasks)

results = asyncio.run(main(prompts))    # ≈ 1–2 seconds for 20

Read it as three new words.

async def declares a coroutine: a function that can pause. Calling it does not run it; it returns a coroutine object that the event loop will run.

await marks a pause point: "this will take a while — run something else and come back when it is done". Only things designed for asyncio can be awaited: httpx calls, asyncio.sleep, other coroutines.

asyncio.gather starts all the coroutines and waits for all of them, returning their results in the same order as the inputs.

asyncio.run creates the event loop, runs main to completion, and shuts the loop down. It is the one place your synchronous program hands over to the asynchronous one.

Why requests cannot do this

requests is synchronous. Its get blocks the thread until the reply arrives, and asyncio has no way to pause it. That is why the example uses httpx, whose AsyncClient mirrors the requests API but with await. The provider SDKs offer the same split: AsyncOpenAI and AsyncAnthropic have identical methods to their synchronous twins, each awaited.

The one rule

Inside a coroutine, never call anything that blocks. time.sleep(1) blocks. So does requests.get, reading a large file with plain open, and a heavy loop of arithmetic. Each of them freezes the single thread the event loop lives on, and every other coroutine — all twenty in-flight requests — stops advancing until it returns. The symptom is that your beautifully concurrent code runs exactly as slowly as the loop it replaced, and nothing tells you why.

The replacements: await asyncio.sleep(1), await client.get(...), and for CPU work or a stubborn synchronous library, await asyncio.to_thread(func, args), which runs the blocking call on a worker thread and awaits the result.

Respecting the rate limit

Twenty at once is fine. Two thousand at once is a 429 storm and possibly a suspended key. Cap the concurrency:

sem = asyncio.Semaphore(5)

async def ask(client, prompt):
    async with sem:
        r = await client.post(...)
        ...

A Semaphore(5) lets five coroutines through the async with at a time; the sixth waits until one finishes. All two thousand tasks still exist and gather still returns them in order; only the in-flight count is limited. Set it from the provider's published limit and the size of your requests. Combine it with the backoff from earlier in this module, because five concurrent requests can still hit a per-minute token limit.

Errors in a batch

By default, gather raises the first exception and the others keep running in the background with their results lost. For a batch where one failure should not discard the rest:

results = await asyncio.gather(*tasks, return_exceptions=True)
for prompt, res in zip(prompts, results):
    if isinstance(res, Exception):
        log.warning("failed %s: %s", prompt[:30], res)
    else:
        save(prompt, res)

Each slot is either a result or the exception object. Check with isinstance before using it.

The warning you will see

RuntimeWarning: coroutine 'ask' was never awaited

This means you called ask(prompt) and did nothing with the returned coroutine — no await, no gather, no create_task. Nothing ran. It is the async version of forgetting to call a function, and it is the first thing to check when an async program finishes suspiciously fast with no results.

Where async does not help

Async overlaps waiting. It does not make a single request faster, and it does nothing for CPU-bound work such as parsing a large file or computing similarities over a million vectors, because there is still one thread. For that, the next lesson covers threads and processes and the reason Python needs both.

Async also spreads. A function that awaits must be async, so its caller must be async, and so on up to asyncio.run. That is fine for a program built around network calls; it is awkward to bolt onto a synchronous one. When only one corner needs concurrency, a thread pool is often the smaller change.

Try this now

Point the example at Ollama's local endpoint with ten short prompts and time it against the plain loop. Then insert time.sleep(0.5) inside ask, watch the total climb, and replace it with await asyncio.sleep(0.5) to watch it fall again.