Rate limits in both directions

The provider’s side

Every provider limits how much you can send: requests per minute, tokens per minute, sometimes tokens per day, with separate figures per model. The limits are tiered by your spending history, so a new account starts tight, and an increase is a request that takes days or weeks to be granted. Read the numbers for your tier before launch and write them into the cost model beside the prices. A feature designed for traffic the account is not allowed to send fails on launch day with a wall of 429s.

What a 429 means, and the deeper fix

The first module covered the retry: honour the Retry-After header, back off exponentially with jitter, and never retry in a tight loop that turns a throttle into an outage. That handles the occasional 429. The deeper fix is not to send the request that will be refused.

Throttle on your side. A semaphore that allows at most N model calls in flight; a token bucket that mirrors the provider’s per-minute allowance and makes a request wait for a token rather than fire and fail; a queue in front of both. The provider’s limit then becomes a ceiling you approach smoothly rather than a wall you hit.

Priority lanes

Not every request deserves the same place in the queue. A person waiting for a chat reply is first. A background classification, a memory extraction, a re-index is last, and better still goes through the batch endpoint from the earlier lesson, which has its own separate allowance and does not compete at all. A queue with two lanes (interactive and background) where background work is dispatched only when the interactive lane has headroom, stops a nightly job from starving the product at nine in the morning.

Headroom

Plan at half the limit. Traffic is not flat: a marketing email lands and two thousand people open the product in the same minute; a news story mentions you; a customer runs a script. Peaks run at several times the average, and a feature planned at ninety per cent of its allowance on the average hits the wall on the first peak.

The arithmetic: users active in the peak hour, times turns per user, times tokens per turn, divided by sixty, against the tokens-per-minute limit. If the result is over half the limit, file the increase request now, because it will take longer than you have.

Keys per feature

One key shared by every feature means one feature’s burst throttles all of them. Separate keys (or separate projects, where the provider supports them) per feature give each its own allowance, its own cost line in the bill, and its own kill switch. The inline completion feature from the interface module, with its thousands of small calls, should never share an allowance with the chat.

Your own limits

The denial-of-wallet lesson set them for cost; here they are set for capacity. Per-user requests per minute by tier; a global ceiling on requests per minute for the whole feature; both enforced before any call to the provider. Your global ceiling sits below the provider’s limit, with the headroom above, so that when the feature is saturated the refusals come from you, with a Retry-After of your own, a countdown in the interface, and the user’s typed text preserved, rather than from the provider, unpredictably, with an error the interface was not built to explain.

Long requests eat the minute

Token limits count tokens. A single request with a hundred thousand tokens of context consumes a large share of a minute’s allowance on its own, and ten of them in a minute exhaust it for everyone. A feature that sends long contexts needs its own key and its own limit, and the caching lesson’s prefix discount does not reduce the tokens counted against the allowance on most providers. Check yours.

Monitor the approach, not the collision

From the traces: the 429 rate from the provider, queue depth, time spent waiting in the queue, and tokens per minute as a share of the limit. Alert when the share passes seventy per cent for more than a few minutes, and on any sustained 429s. The request for a higher limit should be filed when the trend line says you will need it in a month, not when the wall is reached, because the provider’s clock runs slower than yours.