Reasoning models, thinking budgets, and when they are a waste of money

What “thinking” actually is

A reasoning model is not a different kind of machine. It is a model trained so that, before producing its answer, it generates a long stretch of intermediate text (working, dead ends, self-correction) and then answers. Some providers hide that text and bill you for it anyway; some show a summary; some let you set how many tokens it may spend.

The mechanism is worth stating plainly, because it explains everything about when it helps. A transformer does a fixed amount of computation per token. It cannot think harder about a single token. The only way to spend more computation on a problem is to emit more tokens. Intermediate reasoning is not a metaphor for deliberation; it is literally the model buying itself more forward passes, with each step conditioned on the last.

That is why chain-of-thought prompting worked in the first place, and why the trained-in version works better: the model has been rewarded for reasoning traces that lead to correct answers, rather than merely imitating the appearance of working.

What it costs

Thinking tokens are output tokens, and output tokens are the expensive ones: often three to five times the input rate. A model that spends 4,000 tokens thinking before writing a 200-token answer has produced 4,200 billable output tokens for a 200-token result.

And it is slow. Those 4,000 tokens are generated serially at 30 to 80 tokens per second, so the user waits 50 to 130 seconds before the first word of the answer. There is no streaming fix for this: there is nothing to show.

For a batch job at midnight, both costs are irrelevant. For a chat box, both are fatal.

Where it helps

The honest split, from a large amount of published and internal measurement:

Substantial gains on competition mathematics, multi-step logic puzzles, algorithm design and debugging, planning under constraints, tasks where an error at step two invalidates step seven, and anything where the model needs to notice its own mistake and back up.

Little or no gain on classification, extraction from a document, summarising, translation, rewriting, tone changes, straightforward retrieval answering, and most format-following. On these, thinking often makes things slightly worse: the model reasons its way past the obvious answer, or the extended trace drifts from the requested format.

The pattern is consistent. Thinking helps when the difficulty is in deciding what the answer is. It does not help when the difficulty is in knowing something or in producing a shape. Retrieval and schemas fix those, respectively, and both are far cheaper.

Setting a budget

Where the API exposes a token budget, treat it as a dial with a knee:

resp = client.messages.create(
    model=REASONING_MODEL,
    thinking={"type": "enabled", "budget_tokens": 4000},
    max_tokens=6000,   # must exceed the thinking budget
    messages=[...])

Run your evaluation set at 1,000, 4,000 and 16,000. Almost every task has a point past which more budget buys nothing measurable: commonly a few thousand tokens for ordinary product work. Paying for 32,000 when the curve flattened at 3,000 is the most common way to waste money on reasoning models.

Note the constraint in that snippet: max_tokens must leave room for both the thinking and the answer. Set them too close and the model spends its whole budget reasoning and gets truncated before it answers; you pay full price for nothing, and stop_reason will say max_tokens, which is another reason to check that field.

Do not treat the trace as an explanation

The most important limitation, and the one most often ignored in product design.

A reasoning trace is generated text. It is correlated with the computation that produced the answer, but it is not a log of it. Models have been shown to reach an answer and then produce a trace that rationalises it, to use a hint in the prompt while never mentioning the hint in their reasoning, and to write a plausible chain that does not match what actually determined the output.

So: do not show the trace to a user as a justification, do not audit decisions by reading it, and do not build a compliance story on it. If you need a defensible reason for a decision, require the model to cite a source you can verify independently. A verifiable quote is evidence. A reasoning trace is a narrative.

The cheap version still works

Before reaching for a reasoning model, try the free version on an ordinary one: ask it to work through the problem in a <scratchpad> block, then give the final answer in a <answer> block, and strip the scratchpad in your code. You get most of the benefit on moderately hard tasks, you can read the working, and you control the budget by asking for brevity.

And measure. On roughly half the tasks people reach for reasoning models to solve, a well-retrieved mid-tier model answers correctly in 700 milliseconds for a fortieth of the price.