Capping what one user can cost you

The arithmetic

Providers charge per token, separately for tokens in and tokens out, and output is several times the price of input. The exact figures are on each provider’s pricing page and change often enough that this lesson will not quote them; the shape is what to learn.

A summarise call sends a 2,000-token document plus a 200-token prompt and gets 300 tokens back. Look up your model’s two prices per million tokens, and the call costs 2,200 × input price + 300 × output price, divided by a million. For a small hosted model that is a fraction of a cent; for a frontier model it is a few cents. Neither matters at ten calls a day. At ten thousand, the difference between the two models is a phone bill and a rent payment.

data.usage in every response reports the actual counts. Log them, sum them per user per day, and the arithmetic becomes a report rather than a guess.

Three caps, in three places

The provider’s hard limit. Every dashboard has a monthly spending cap or a prepaid balance. Set it before the key exists. This is the one cap that holds when every line of your code is wrong, because it is enforced by the people sending the bill. Set it to a number you would be annoyed to lose and not devastated.

A per-user allowance in your own table. A row per user per day with a count, checked before the call:

const used = await claimUsage(session.userId, today);   // atomic increment, returns new count
if (used > DAILY_LIMIT) return new Response("Daily limit reached", { status: 429 });
try {
  return await callModel(...);
} catch (e) {
  await refundUsage(session.userId, today);              // the call never happened
  throw e;
}

The order matters and is the point of the question above. Increment first, atomically, in one statement: UPDATE ... SET count = count + 1 RETURNING count, and refund on failure. A counter incremented after the response is a counter that a burst of parallel requests passes before it has moved. Twenty checks read zero at the same instant, twenty calls go out, and the script that sent them sends nine hundred more.

A maximum on every request. max_tokens on the call caps the output. A cap on the input length (characters, before the call) caps the other half. A user pasting a 200-page document into a summariser is either a mistake or a test of your limits, and both should get a polite refusal rather than a $4 call.

Who is doing the calling

Your route is reachable by anything that has a session cookie, including a script written in five minutes. The per-user cap handles the signed-in abuser. Two more layers handle the rest: a per-IP rate limit on the route (a few requests a minute is generous for a person and hopeless for a script) and the sign-up flood defence from the security block, because a thousand accounts at twenty calls each is twenty thousand calls.

On the free tiers, the provider’s own per-minute limit acts as a crude cap: the route returns 429 when the tier is exhausted and the cost is zero. That is a feature of the free tier and it goes away the day you add a card.

The failure that costs the most

A retry loop with no upper bound. A route that, on a 429 or a timeout, tries again immediately, and again, and again, generates thousands of calls from one user’s single click and stops only when the function is killed. The resilience lesson next sets the shape: a small fixed number of retries with growing delays, and then an honest failure. Search the generated code for while near fetch and read what ends the loop.

Say the number

Put the limit in the interface. “You have used 14 of 20 today” is honest, sets expectations, and turns the 429 from a mystery into a feature. It also means the day you raise the limit, users notice. A cap nobody can see reads as a bug.

What this looks like at the end of the month

Total tokens by feature, by day. Calls per user, sorted. The top user should be a person, and you should be able to name them. If the top user is an account with no other activity that made three thousand calls at 4am, the caps above were not all in place, and the arithmetic at the top of this lesson tells you what it cost to learn that.