{"course":{"slug":"building-with-ai","title":"Building with AI","url":"https://addaly.com/learn/building-with-ai","level":"intermediate","track":"build"},"lesson":{"slug":"rate-limits-in-both-directions","title":"Rate limits in both directions","url":"https://addaly.com/learn/building-with-ai/rate-limits-in-both-directions","position":77,"of":80,"minutes":8,"answer":"The provider limits your requests and tokens per minute by tier and raises limits only on request over days, so a feature must be planned at half its allowance to survive bursts.","keyIdea":"The provider limits your requests and tokens per minute by tier and raises limits only on request over days, so a feature must be planned at half its allowance to survive bursts, must throttle itself client-side rather than discover the limit through 429s, and must set its own ceiling for users below the provider's so that refusals are yours, clear and recoverable."},"body":{"format":"markdown","text":"## The provider's side\n\nEvery provider limits how much you can send: requests per minute, tokens per minute, sometimes tokens per day, with separate figures per model. The limits are tiered by your spending history, so a new account starts tight, and an increase is a request that takes days or weeks to be granted. Read the numbers for your tier before launch and write them into the cost model beside the prices. A feature designed for traffic the account is not allowed to send fails on launch day with a wall of 429s.\n\n## What a 429 means, and the deeper fix\n\nThe first module covered the retry: honour the `Retry-After` header, back off exponentially with jitter, and never retry in a tight loop that turns a throttle into an outage. That handles the occasional 429. The deeper fix is not to send the request that will be refused.\n\nThrottle on your side. A semaphore that allows at most N model calls in flight; a token bucket that mirrors the provider's per-minute allowance and makes a request wait for a token rather than fire and fail; a queue in front of both. The provider's limit then becomes a ceiling you approach smoothly rather than a wall you hit.\n\n## Priority lanes\n\nNot every request deserves the same place in the queue. A person waiting for a chat reply is first. A background classification, a memory extraction, a re-index is last, and better still goes through the batch endpoint from the earlier lesson, which has its own separate allowance and does not compete at all. A queue with two lanes (interactive and background) where background work is dispatched only when the interactive lane has headroom, stops a nightly job from starving the product at nine in the morning.\n\n## Headroom\n\nPlan at half the limit. Traffic is not flat: a marketing email lands and two thousand people open the product in the same minute; a news story mentions you; a customer runs a script. Peaks run at several times the average, and a feature planned at ninety per cent of its allowance on the average hits the wall on the first peak.\n\nThe arithmetic: users active in the peak hour, times turns per user, times tokens per turn, divided by sixty, against the tokens-per-minute limit. If the result is over half the limit, file the increase request now, because it will take longer than you have.\n\n## Keys per feature\n\nOne key shared by every feature means one feature's burst throttles all of them. Separate keys (or separate projects, where the provider supports them) per feature give each its own allowance, its own cost line in the bill, and its own kill switch. The inline completion feature from the interface module, with its thousands of small calls, should never share an allowance with the chat.\n\n## Your own limits\n\nThe denial-of-wallet lesson set them for cost; here they are set for capacity. Per-user requests per minute by tier; a global ceiling on requests per minute for the whole feature; both enforced before any call to the provider. Your global ceiling sits below the provider's limit, with the headroom above, so that when the feature is saturated the refusals come from you, with a `Retry-After` of your own, a countdown in the interface, and the user's typed text preserved, rather than from the provider, unpredictably, with an error the interface was not built to explain.\n\n## Long requests eat the minute\n\nToken limits count tokens. A single request with a hundred thousand tokens of context consumes a large share of a minute's allowance on its own, and ten of them in a minute exhaust it for everyone. A feature that sends long contexts needs its own key and its own limit, and the caching lesson's prefix discount does not reduce the tokens counted against the allowance on most providers. Check yours.\n\n## Monitor the approach, not the collision\n\nFrom the traces: the 429 rate from the provider, queue depth, time spent waiting in the queue, and tokens per minute as a share of the limit. Alert when the share passes seventy per cent for more than a few minutes, and on any sustained 429s. The request for a higher limit should be filed when the trend line says you will need it in a month, not when the wall is reached, because the provider's clock runs slower than yours."},"check":{"question":"A feature is planned to use 90 per cent of its tokens-per-minute allowance at average traffic. What happens on the first day a marketing email drives a burst of users?","options":[{"text":"Requests are refused with 429s because peaks run at several times the average and the allowance was already nearly consumed.","correct":true},{"text":"The provider automatically raises the limit in response to the burst, since usage justifies it.","correct":false,"tempting":"Increases are granted on request, over days; nothing raises a limit in real time."},{"text":"Nothing changes, because tokens per minute is averaged over the hour and a short burst is absorbed.","correct":false,"tempting":"The allowance is measured per minute, and a burst is exactly what a per-minute limit is designed to stop."},{"text":"Only background jobs are affected, because interactive requests are prioritised by the provider.","correct":false,"tempting":"The provider does not know which of your requests are interactive; priority is something you build on your side."}]},"prev":{"slug":"running-your-own-inference","title":"Running your own inference: the break-even arithmetic","url":"https://addaly.com/learn/building-with-ai/running-your-own-inference"},"next":{"slug":"outages-and-switching-providers","title":"Outages, fallbacks and what does not port","url":"https://addaly.com/learn/building-with-ai/outages-and-switching-providers"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Rate limits in both directions', in Building with AI. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/building-with-ai/rate-limits-in-both-directions (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Rate limits in both directions","courseTitle":"Building with AI","module":{"index":8,"slug":"cost-latency-and-running-it","title":"Cost, latency and running it in production"},"answer":null,"keyIdea":"The provider limits your requests and tokens per minute by tier and raises limits only on request over days, so a feature must be planned at half its allowance to survive bursts, must throttle itself client-side rather than discover the limit through 429s, and must set its own ceiling for users below the provider’s so that refusals are yours, clear and recoverable.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"the-provider-s-side","runs":[{"text":"The provider’s side"}]},{"t":"p","runs":[{"text":"Every provider limits how much you can send: requests per minute, tokens per minute, sometimes tokens per day, with separate figures per model. The limits are tiered by your spending history, so a new account starts tight, and an increase is a request that takes days or weeks to be granted. Read the numbers for your tier before launch and write them into the cost model beside the prices. A feature designed for traffic the account is not allowed to send fails on launch day with a wall of 429s."}]},{"t":"h2","id":"what-a-429-means-and-the-deeper-fix","runs":[{"text":"What a 429 means, and the deeper fix"}]},{"t":"p","runs":[{"text":"The first module covered the retry: honour the "},{"text":"Retry-After","code":true},{"text":" header, back off exponentially with jitter, and never retry in a tight loop that turns a throttle into an outage. That handles the occasional 429. The deeper fix is not to send the request that will be refused."}]},{"t":"p","runs":[{"text":"Throttle on your side. A semaphore that allows at most N model calls in flight; a token bucket that mirrors the provider’s per-minute allowance and makes a request wait for a token rather than fire and fail; a queue in front of both. The provider’s limit then becomes a ceiling you approach smoothly rather than a wall you hit."}]},{"t":"h2","id":"priority-lanes","runs":[{"text":"Priority lanes"}]},{"t":"p","runs":[{"text":"Not every request deserves the same place in the queue. A person waiting for a chat reply is first. A background classification, a memory extraction, a re-index is last, and better still goes through the batch endpoint from the earlier lesson, which has its own separate allowance and does not compete at all. A queue with two lanes (interactive and background) where background work is dispatched only when the interactive lane has headroom, stops a nightly job from starving the product at nine in the morning."}]},{"t":"h2","id":"headroom","runs":[{"text":"Headroom"}]},{"t":"p","runs":[{"text":"Plan at half the limit. Traffic is not flat: a marketing email lands and two thousand people open the product in the same minute; a news story mentions you; a customer runs a script. Peaks run at several times the average, and a feature planned at ninety per cent of its allowance on the average hits the wall on the first peak."}]},{"t":"p","runs":[{"text":"The arithmetic: users active in the peak hour, times turns per user, times tokens per turn, divided by sixty, against the tokens-per-minute limit. If the result is over half the limit, file the increase request now, because it will take longer than you have."}]},{"t":"h2","id":"keys-per-feature","runs":[{"text":"Keys per feature"}]},{"t":"p","runs":[{"text":"One key shared by every feature means one feature’s burst throttles all of them. Separate keys (or separate projects, where the provider supports them) per feature give each its own allowance, its own cost line in the bill, and its own kill switch. The inline completion feature from the interface module, with its thousands of small calls, should never share an allowance with the chat."}]},{"t":"h2","id":"your-own-limits","runs":[{"text":"Your own limits"}]},{"t":"p","runs":[{"text":"The denial-of-wallet lesson set them for cost; here they are set for capacity. Per-user requests per minute by tier; a global ceiling on requests per minute for the whole feature; both enforced before any call to the provider. Your global ceiling sits below the provider’s limit, with the headroom above, so that when the feature is saturated the refusals come from you, with a "},{"text":"Retry-After","code":true},{"text":" of your own, a countdown in the interface, and the user’s typed text preserved, rather than from the provider, unpredictably, with an error the interface was not built to explain."}]},{"t":"h2","id":"long-requests-eat-the-minute","runs":[{"text":"Long requests eat the minute"}]},{"t":"p","runs":[{"text":"Token limits count tokens. A single request with a hundred thousand tokens of context consumes a large share of a minute’s allowance on its own, and ten of them in a minute exhaust it for everyone. A feature that sends long contexts needs its own key and its own limit, and the caching lesson’s prefix discount does not reduce the tokens counted against the allowance on most providers. Check yours."}]},{"t":"h2","id":"monitor-the-approach-not-the-collision","runs":[{"text":"Monitor the approach, not the collision"}]},{"t":"p","runs":[{"text":"From the traces: the 429 rate from the provider, queue depth, time spent waiting in the queue, and tokens per minute as a share of the limit. Alert when the share passes seventy per cent for more than a few minutes, and on any sustained 429s. The request for a higher limit should be filed when the trend line says you will need it in a month, not when the wall is reached, because the provider’s clock runs slower than yours."}]}],"check":{"question":[{"text":"A feature is planned to use 90 per cent of its tokens-per-minute allowance at average traffic. What happens on the first day a marketing email drives a burst of users?"}],"options":[{"text":[{"text":"Requests are refused with 429s because peaks run at several times the average and the allowance was already nearly consumed."}],"correct":true,"tempting":null},{"text":[{"text":"The provider automatically raises the limit in response to the burst, since usage justifies it."}],"correct":false,"tempting":[{"text":"Increases are granted on request, over days; nothing raises a limit in real time."}]},{"text":[{"text":"Nothing changes, because tokens per minute is averaged over the hour and a short burst is absorbed."}],"correct":false,"tempting":[{"text":"The allowance is measured per minute, and a burst is exactly what a per-minute limit is designed to stop."}]},{"text":[{"text":"Only background jobs are affected, because interactive requests are prioritised by the provider."}],"correct":false,"tempting":[{"text":"The provider does not know which of your requests are interactive; priority is something you build on your side."}]}],"order":[2,0,3,1],"why":null},"prev":{"slug":"running-your-own-inference","title":"Running your own inference: the break-even arithmetic","minutes":9},"next":{"slug":"outages-and-switching-providers","title":"Outages, fallbacks and what does not port","minutes":9}}}