{"course":{"slug":"agents-and-automation","title":"Agents and Automation","url":"https://addaly.com/learn/agents-and-automation","level":"intermediate","track":"build"},"lesson":{"slug":"queues-and-concurrency","title":"Rate limits, workers and the thundering herd","url":"https://addaly.com/learn/agents-and-automation/queues-and-concurrency","position":59,"of":72,"minutes":9,"answer":"For agents the tokens-per-minute limit usually binds long before the requests-per-minute one, because a twenty-turn transcript is enormous compared with a single chat message.","keyIdea":"For agents the tokens-per-minute limit usually binds long before the requests-per-minute one, because a twenty-turn transcript is enormous compared with a single chat message."},"body":{"format":"markdown","text":"## The limit that actually binds\n\nProviders publish two limits: requests per minute and tokens per minute. Chat applications hit the request limit. Agents almost never do, and the reason is the arithmetic from the context-window lesson.\n\nSuppose your allowance is 200,000 tokens per minute and 500 requests per minute. An agent on turn fifteen is sending a 40,000-token transcript. Divide: five such requests exhaust the token allowance in a minute, while you are using one per cent of the request allowance. Your effective concurrency is five, and no amount of raising the request limit changes it.\n\n> Figure: Which limit actually binds, turn by turn. Line chart of Requests per minute the allowance permits against Turn in the run: Permitted by 200,000 tokens per minute (1, 74) (5, 15) (10, 7) (15, 5) (20, 4); Permitted by 500 requests per minute (1, 500) (5, 500) (10, 500) (15, 500) (20, 500). The request allowance is never touched. At turn fifteen a forty-thousand-token transcript means five requests exhaust the token allowance for that minute, so your effective concurrency is five and raising the request limit changes nothing.\n\nThis surprises teams whose load test used short prompts and reported plenty of headroom. Model your capacity in tokens per minute, using the transcript size at the *end* of a run rather than the beginning, and the number will be far smaller and correct.\n\n## Backoff needs jitter\n\nWhen you get a 429, read the `Retry-After` header and obey it. When there is no header, back off exponentially, and add randomness.\n\nWithout jitter, every worker that failed at the same moment retries at the same moment, collides again, and repeats. This is the thundering herd, and it turns a brief limit into a sustained outage of your own making. Full jitter (sleep a random duration between zero and the current backoff ceiling) spreads them out:\n\n```python\ndelay = random.uniform(0, min(cap, base * 2 ** attempt))\n```\n\nCap the attempts, and distinguish what is worth retrying: 429 and 5xx yes, 400 and 401 no. Retrying a malformed request forever is a way to spend an afternoon's quota on the same rejection.\n\n## How many workers\n\nLittle's Law gives the answer without guessing. Workers needed equals arrival rate multiplied by service time. Six hundred items an hour is 0.167 per second; if each takes thirty seconds, you need 0.167 × 30 = 5 workers to keep up, with a little headroom for variance.\n\nFix that number with a semaphore or a worker pool. Unbounded concurrency (spawning a task per item because the language makes it easy) produces a system that works in testing and, on the morning a backlog arrives, opens four thousand simultaneous requests, hits every limit at once and takes longer than doing it serially.\n\nA fixed pool also makes cost predictable, which matters more than it sounds: parallel agents multiply your **spend per minute**, and a runaway parallel job can produce a month's budget in an hour. Put a spend-per-minute ceiling next to the worker count, not only a per-run cap.\n\n## Fairness\n\nOne customer uploading nine thousand documents should not stop everybody else's work for the afternoon. If a single queue is processed in order, that is exactly what happens.\n\nEither keep a queue per tenant and round-robin between them, or cap how many items from one source may be in flight at once. Both are a few lines and both prevent the support conversation where a small customer's urgent item sat behind a large customer's bulk import.\n\n## Timeouts, from the inside out\n\nEvery network call gets a timeout. Every tool gets a timeout longer than its slowest call. Every run gets a timeout longer than its steps allow for.\n\nThe rule that catches people: an outer timeout shorter than the sum of what it contains will kill work that is still running and cannot be cancelled. The HTTP request is abandoned, the provider still bills for it, and the write it was about to perform may still land, which produces the duplicate that idempotency keys exist to absorb.\n\n## Poison items\n\nSome item will fail forever: a corrupt PDF, a record with a null where the schema promised a value. Without a limit, it is retried indefinitely and consumes a worker permanently.\n\nCount attempts on the item itself. After three, move it to a failed state with the error, and carry on. Then alert on the failure *rate*, because one poison item is normal and forty in an hour means something upstream changed.\n\n## The free path\n\nYou almost certainly do not need a message broker. PostgreSQL with `FOR UPDATE SKIP LOCKED` gives you a durable, observable queue with transactions, and it is already running. **Redis** with **RQ** or **Celery** is the next step up. Managed options exist on every platform. Whatever you use, the properties to insist on are the same: at-least-once delivery, a visible backlog, per-item attempt counts, and somewhere for failures to land where a person will see them."},"check":{"question":"An agent service has limits of 500 requests per minute and 200,000 tokens per minute. Late in a run each request carries a 40,000-token transcript. What is the practical concurrency, and why do load tests miss it?","options":[{"text":"About five concurrent requests, because the token limit binds first, and load tests using short prompts never approach it.","correct":true},{"text":"About 500, since the request limit is the published ceiling and token limits apply to billing rather than throttling.","correct":false,"tempting":"Treats the token limit as an accounting figure; providers enforce it as a rate limit and it is usually the tighter of the two."},{"text":"It depends on latency, since concurrency is determined by how long each request takes rather than by the limits.","correct":false,"tempting":"Latency sets how many workers you need, but it cannot lift you past a quota that is consumed by transcript size."},{"text":"About 250, because both limits apply and the effective ceiling is the average of the two.","correct":false,"tempting":"Invents an averaging rule; concurrent limits bind independently and the tighter one governs."}]},"prev":{"slug":"where-automation-runs","title":"Where it runs when your laptop is shut","url":"https://addaly.com/learn/agents-and-automation/where-automation-runs"},"next":{"slug":"pinning-and-migrating-models","title":"The model you tested is not the model you deployed","url":"https://addaly.com/learn/agents-and-automation/pinning-and-migrating-models"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Rate limits, workers and the thundering herd', in Agents and Automation. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/agents-and-automation/queues-and-concurrency (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Rate limits, workers and the thundering herd","courseTitle":"Agents and Automation","module":{"index":7,"slug":"shipping-and-operating","title":"Shipping it, and keeping it alive"},"answer":null,"keyIdea":"For agents the tokens-per-minute limit usually binds long before the requests-per-minute one, because a twenty-turn transcript is enormous compared with a single chat message.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"type":"line","title":"Which limit actually binds, turn by turn","series":[{"label":"Permitted by 200,000 tokens per minute","points":[[1,74],[5,15],[10,7],[15,5],[20,4]]},{"label":"Permitted by 500 requests per minute","points":[[1,500],[5,500],[10,500],[15,500],[20,500]]}],"xLabel":"Turn in the run","yLabel":"Requests per minute the allowance permits","caption":"The request allowance is never touched. At turn fifteen a forty-thousand-token transcript means five requests exhaust the token allowance for that minute, so your effective concurrency is five and raising the request limit changes nothing."},"alt":"Figure: Which limit actually binds, turn by turn. Line chart of Requests per minute the allowance permits against Turn in the run: Permitted by 200,000 tokens per minute (1, 74) (5, 15) (10, 7) (15, 5) (20, 4); Permitted by 500 requests per minute (1, 500) (5, 500) (10, 500) (15, 500) (20, 500). The request allowance is never touched. At turn fifteen a forty-thousand-token transcript means five requests exhaust the token allowance for that minute, so your effective concurrency is five and raising the request limit changes nothing."}},"blocks":[{"t":"h2","id":"the-limit-that-actually-binds","runs":[{"text":"The limit that actually binds"}],"role":"limit"},{"t":"p","runs":[{"text":"Providers publish two limits: requests per minute and tokens per minute. Chat applications hit the request limit. Agents almost never do, and the reason is the arithmetic from the context-window lesson."}]},{"t":"p","runs":[{"text":"Suppose your allowance is 200,000 tokens per minute and 500 requests per minute. An agent on turn fifteen is sending a 40,000-token transcript. Divide: five such requests exhaust the token allowance in a minute, while you are using one per cent of the request allowance. Your effective concurrency is five, and no amount of raising the request limit changes it."}]},{"t":"p","runs":[{"text":"This surprises teams whose load test used short prompts and reported plenty of headroom. Model your capacity in tokens per minute, using the transcript size at the "},{"text":"end","i":true},{"text":" of a run rather than the beginning, and the number will be far smaller and correct."}]},{"t":"h2","id":"backoff-needs-jitter","runs":[{"text":"Backoff needs jitter"}]},{"t":"p","runs":[{"text":"When you get a 429, read the "},{"text":"Retry-After","code":true},{"text":" header and obey it. When there is no header, back off exponentially, and add randomness."}]},{"t":"p","runs":[{"text":"Without jitter, every worker that failed at the same moment retries at the same moment, collides again, and repeats. This is the thundering herd, and it turns a brief limit into a sustained outage of your own making. Full jitter (sleep a random duration between zero and the current backoff ceiling) spreads them out:"}]},{"t":"code","lang":"python","label":"Python","text":"delay = random.uniform(0, min(cap, base * 2 ** attempt))"},{"t":"p","runs":[{"text":"Cap the attempts, and distinguish what is worth retrying: 429 and 5xx yes, 400 and 401 no. Retrying a malformed request forever is a way to spend an afternoon’s quota on the same rejection."}]},{"t":"h2","id":"how-many-workers","runs":[{"text":"How many workers"}]},{"t":"p","runs":[{"text":"Little’s Law gives the answer without guessing. Workers needed equals arrival rate multiplied by service time. Six hundred items an hour is 0.167 per second; if each takes thirty seconds, you need 0.167 × 30 = 5 workers to keep up, with a little headroom for variance."}]},{"t":"p","runs":[{"text":"Fix that number with a semaphore or a worker pool. Unbounded concurrency (spawning a task per item because the language makes it easy) produces a system that works in testing and, on the morning a backlog arrives, opens four thousand simultaneous requests, hits every limit at once and takes longer than doing it serially."}]},{"t":"p","runs":[{"text":"A fixed pool also makes cost predictable, which matters more than it sounds: parallel agents multiply your "},{"text":"spend per minute","b":true},{"text":", and a runaway parallel job can produce a month’s budget in an hour. Put a spend-per-minute ceiling next to the worker count, not only a per-run cap."}]},{"t":"h2","id":"fairness","runs":[{"text":"Fairness"}]},{"t":"p","runs":[{"text":"One customer uploading nine thousand documents should not stop everybody else’s work for the afternoon. If a single queue is processed in order, that is exactly what happens."}]},{"t":"p","runs":[{"text":"Either keep a queue per tenant and round-robin between them, or cap how many items from one source may be in flight at once. Both are a few lines and both prevent the support conversation where a small customer’s urgent item sat behind a large customer’s bulk import."}]},{"t":"h2","id":"timeouts-from-the-inside-out","runs":[{"text":"Timeouts, from the inside out"}]},{"t":"p","runs":[{"text":"Every network call gets a timeout. Every tool gets a timeout longer than its slowest call. Every run gets a timeout longer than its steps allow for."}]},{"t":"p","runs":[{"text":"The rule that catches people: an outer timeout shorter than the sum of what it contains will kill work that is still running and cannot be cancelled. The HTTP request is abandoned, the provider still bills for it, and the write it was about to perform may still land, which produces the duplicate that idempotency keys exist to absorb."}]},{"t":"h2","id":"poison-items","runs":[{"text":"Poison items"}]},{"t":"p","runs":[{"text":"Some item will fail forever: a corrupt PDF, a record with a null where the schema promised a value. Without a limit, it is retried indefinitely and consumes a worker permanently."}]},{"t":"p","runs":[{"text":"Count attempts on the item itself. After three, move it to a failed state with the error, and carry on. Then alert on the failure "},{"text":"rate","i":true},{"text":", because one poison item is normal and forty in an hour means something upstream changed."}]},{"t":"h2","id":"the-free-path","runs":[{"text":"The free path"}],"role":"free"},{"t":"p","runs":[{"text":"You almost certainly do not need a message broker. PostgreSQL with "},{"text":"FOR UPDATE SKIP LOCKED","code":true},{"text":" gives you a durable, observable queue with transactions, and it is already running. "},{"text":"Redis","b":true},{"text":" with "},{"text":"RQ","b":true},{"text":" or "},{"text":"Celery","b":true},{"text":" is the next step up. Managed options exist on every platform. Whatever you use, the properties to insist on are the same: at-least-once delivery, a visible backlog, per-item attempt counts, and somewhere for failures to land where a person will see them."}]}],"check":{"question":[{"text":"An agent service has limits of 500 requests per minute and 200,000 tokens per minute. Late in a run each request carries a 40,000-token transcript. What is the practical concurrency, and why do load tests miss it?"}],"options":[{"text":[{"text":"About five concurrent requests, because the token limit binds first, and load tests using short prompts never approach it."}],"correct":true,"tempting":null},{"text":[{"text":"About 500, since the request limit is the published ceiling and token limits apply to billing rather than throttling."}],"correct":false,"tempting":[{"text":"Treats the token limit as an accounting figure; providers enforce it as a rate limit and it is usually the tighter of the two."}]},{"text":[{"text":"It depends on latency, since concurrency is determined by how long each request takes rather than by the limits."}],"correct":false,"tempting":[{"text":"Latency sets how many workers you need, but it cannot lift you past a quota that is consumed by transcript size."}]},{"text":[{"text":"About 250, because both limits apply and the effective ceiling is the average of the two."}],"correct":false,"tempting":[{"text":"Invents an averaging rule; concurrent limits bind independently and the tighter one governs."}]}],"order":[3,2,1,0],"why":null},"prev":{"slug":"where-automation-runs","title":"Where it runs when your laptop is shut","minutes":8},"next":{"slug":"pinning-and-migrating-models","title":"The model you tested is not the model you deployed","minutes":9}}}