{"course":{"slug":"building-with-ai","title":"Building with AI","url":"https://addaly.com/learn/building-with-ai","level":"intermediate","track":"build"},"lesson":{"slug":"thinking-budgets","title":"Reasoning models, thinking budgets, and when they are a waste of money","url":"https://addaly.com/learn/building-with-ai/thinking-budgets","position":14,"of":80,"minutes":9,"answer":"Extended thinking is the model buying itself more forward passes by emitting more output tokens.","keyIdea":"Extended thinking is the model buying itself more forward passes by emitting more output tokens, so it helps when the difficulty is deciding the answer and not when the difficulty is knowing a fact or producing a format."},"body":{"format":"markdown","text":"## What \"thinking\" actually is\n\nA reasoning model is not a different kind of machine. It is a model trained so that, before producing its answer, it generates a long stretch of intermediate text (working, dead ends, self-correction) and then answers. Some providers hide that text and bill you for it anyway; some show a summary; some let you set how many tokens it may spend.\n\nThe mechanism is worth stating plainly, because it explains everything about when it helps. A transformer does a fixed amount of computation per token. It cannot think harder about a single token. The only way to spend more computation on a problem is to **emit more tokens**. Intermediate reasoning is not a metaphor for deliberation; it is literally the model buying itself more forward passes, with each step conditioned on the last.\n\nThat is why chain-of-thought prompting worked in the first place, and why the trained-in version works better: the model has been rewarded for reasoning traces that lead to correct answers, rather than merely imitating the appearance of working.\n\n## What it costs\n\nThinking tokens are output tokens, and output tokens are the expensive ones: often three to five times the input rate. A model that spends 4,000 tokens thinking before writing a 200-token answer has produced 4,200 billable output tokens for a 200-token result.\n\nAnd it is slow. Those 4,000 tokens are generated serially at 30 to 80 tokens per second, so the user waits 50 to 130 seconds before the first word of the answer. There is no streaming fix for this: there is nothing to show.\n\nFor a batch job at midnight, both costs are irrelevant. For a chat box, both are fatal.\n\n## Where it helps\n\nThe honest split, from a large amount of published and internal measurement:\n\n> Figure: How long the screen stays empty while a model thinks. Bars (seconds before the first word of the answer): Mid-tier model, no thinking 0.7; Reasoning, 1,000-token budget 20; Reasoning, 4,000-token budget 80; Reasoning, 16,000-token budget 320. Thinking tokens are output tokens, generated serially at roughly 50 a second, and there is nothing to stream because the trace is not the answer. For a batch job at midnight this does not matter. For a chat box it is the whole product.\n\n**Substantial gains** on competition mathematics, multi-step logic puzzles, algorithm design and debugging, planning under constraints, tasks where an error at step two invalidates step seven, and anything where the model needs to notice its own mistake and back up.\n\n**Little or no gain** on classification, extraction from a document, summarising, translation, rewriting, tone changes, straightforward retrieval answering, and most format-following. On these, thinking often makes things slightly *worse*: the model reasons its way past the obvious answer, or the extended trace drifts from the requested format.\n\nThe pattern is consistent. Thinking helps when the difficulty is in **deciding what the answer is**. It does not help when the difficulty is in **knowing something** or in **producing a shape**. Retrieval and schemas fix those, respectively, and both are far cheaper.\n\n## Setting a budget\n\nWhere the API exposes a token budget, treat it as a dial with a knee:\n\n```python\nresp = client.messages.create(\n    model=REASONING_MODEL,\n    thinking={\"type\": \"enabled\", \"budget_tokens\": 4000},\n    max_tokens=6000,   # must exceed the thinking budget\n    messages=[...])\n```\n\nRun your evaluation set at 1,000, 4,000 and 16,000. Almost every task has a point past which more budget buys nothing measurable: commonly a few thousand tokens for ordinary product work. Paying for 32,000 when the curve flattened at 3,000 is the most common way to waste money on reasoning models.\n\nNote the constraint in that snippet: `max_tokens` must leave room for both the thinking and the answer. Set them too close and the model spends its whole budget reasoning and gets truncated before it answers; you pay full price for nothing, and `stop_reason` will say `max_tokens`, which is another reason to check that field.\n\n## Do not treat the trace as an explanation\n\nThe most important limitation, and the one most often ignored in product design.\n\nA reasoning trace is generated text. It is *correlated* with the computation that produced the answer, but it is not a log of it. Models have been shown to reach an answer and then produce a trace that rationalises it, to use a hint in the prompt while never mentioning the hint in their reasoning, and to write a plausible chain that does not match what actually determined the output.\n\nSo: do not show the trace to a user as a justification, do not audit decisions by reading it, and do not build a compliance story on it. If you need a defensible reason for a decision, require the model to cite a source you can verify independently. A verifiable quote is evidence. A reasoning trace is a narrative.\n\n## The cheap version still works\n\nBefore reaching for a reasoning model, try the free version on an ordinary one: ask it to work through the problem in a `<scratchpad>` block, then give the final answer in a `<answer>` block, and strip the scratchpad in your code. You get most of the benefit on moderately hard tasks, you can read the working, and you control the budget by asking for brevity.\n\nAnd measure. On roughly half the tasks people reach for reasoning models to solve, a well-retrieved mid-tier model answers correctly in 700 milliseconds for a fortieth of the price."},"check":{"question":"A team enables a large thinking budget on a document-summarisation feature and sees no quality improvement, much higher cost, and slower responses. Why is this the expected result?","options":[{"text":"The thinking budget was too small to make a difference on long documents.","correct":false,"tempting":"Assumes the curve keeps rising with budget, when the limiting factor here is the nature of the task rather than the amount of computation."},{"text":"Summarisation needs a longer context window rather than more reasoning, so the budget was spent on the wrong resource.","correct":false,"tempting":"Names a real but separate constraint; the documents fit, and window size is not what the thinking budget trades against."},{"text":"The difficulty in summarising is selecting and compressing material already present, not deciding an answer through multiple dependent steps, so extra forward passes have nothing to buy.","correct":true},{"text":"Reasoning models are trained on mathematics and code and are simply weaker at prose.","correct":false,"tempting":"Overstates specialisation; these models are strong at prose, and the absence of gain here is about the task's structure rather than a weakness."}]},"prev":{"slug":"decomposition-and-chaining","title":"One big prompt, or four small ones","url":"https://addaly.com/learn/building-with-ai/decomposition-and-chaining"},"next":{"slug":"prompt-caching","title":"Prompt caching, and the ordering rule that pays for itself","url":"https://addaly.com/learn/building-with-ai/prompt-caching"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Reasoning models, thinking budgets, and when they are a waste of money', in Building with AI. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/building-with-ai/thinking-budgets (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Reasoning models, thinking budgets, and when they are a waste of money","courseTitle":"Building with AI","module":{"index":2,"slug":"prompting-as-engineering","title":"Prompting as engineering"},"answer":null,"keyIdea":"Extended thinking is the model buying itself more forward passes by emitting more output tokens, so it helps when the difficulty is deciding the answer and not when the difficulty is knowing a fact or producing a format.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"type":"bars","unit":"seconds before the first word of the answer","title":"How long the screen stays empty while a model thinks","series":[{"label":"Mid-tier model, no thinking","value":0.7,"highlight":true},{"label":"Reasoning, 1,000-token budget","value":20},{"label":"Reasoning, 4,000-token budget","value":80},{"label":"Reasoning, 16,000-token budget","value":320}],"caption":"Thinking tokens are output tokens, generated serially at roughly 50 a second, and there is nothing to stream because the trace is not the answer. For a batch job at midnight this does not matter. For a chat box it is the whole product."},"alt":"Figure: How long the screen stays empty while a model thinks. Bars (seconds before the first word of the answer): Mid-tier model, no thinking 0.7; Reasoning, 1,000-token budget 20; Reasoning, 4,000-token budget 80; Reasoning, 16,000-token budget 320. Thinking tokens are output tokens, generated serially at roughly 50 a second, and there is nothing to stream because the trace is not the answer. For a batch job at midnight this does not matter. For a chat box it is the whole product."}},"blocks":[{"t":"h2","id":"what-thinking-actually-is","runs":[{"text":"What “thinking” actually is"}]},{"t":"p","runs":[{"text":"A reasoning model is not a different kind of machine. It is a model trained so that, before producing its answer, it generates a long stretch of intermediate text (working, dead ends, self-correction) and then answers. Some providers hide that text and bill you for it anyway; some show a summary; some let you set how many tokens it may spend."}]},{"t":"p","runs":[{"text":"The mechanism is worth stating plainly, because it explains everything about when it helps. A transformer does a fixed amount of computation per token. It cannot think harder about a single token. The only way to spend more computation on a problem is to "},{"text":"emit more tokens","b":true},{"text":". Intermediate reasoning is not a metaphor for deliberation; it is literally the model buying itself more forward passes, with each step conditioned on the last."}]},{"t":"p","runs":[{"text":"That is why chain-of-thought prompting worked in the first place, and why the trained-in version works better: the model has been rewarded for reasoning traces that lead to correct answers, rather than merely imitating the appearance of working."}]},{"t":"h2","id":"what-it-costs","runs":[{"text":"What it costs"}]},{"t":"p","runs":[{"text":"Thinking tokens are output tokens, and output tokens are the expensive ones: often three to five times the input rate. A model that spends 4,000 tokens thinking before writing a 200-token answer has produced 4,200 billable output tokens for a 200-token result."}]},{"t":"p","runs":[{"text":"And it is slow. Those 4,000 tokens are generated serially at 30 to 80 tokens per second, so the user waits 50 to 130 seconds before the first word of the answer. There is no streaming fix for this: there is nothing to show."}]},{"t":"p","runs":[{"text":"For a batch job at midnight, both costs are irrelevant. For a chat box, both are fatal."}]},{"t":"h2","id":"where-it-helps","runs":[{"text":"Where it helps"}]},{"t":"p","runs":[{"text":"The honest split, from a large amount of published and internal measurement:"}]},{"t":"p","runs":[{"text":"Substantial gains","b":true},{"text":" on competition mathematics, multi-step logic puzzles, algorithm design and debugging, planning under constraints, tasks where an error at step two invalidates step seven, and anything where the model needs to notice its own mistake and back up."}]},{"t":"p","runs":[{"text":"Little or no gain","b":true},{"text":" on classification, extraction from a document, summarising, translation, rewriting, tone changes, straightforward retrieval answering, and most format-following. On these, thinking often makes things slightly "},{"text":"worse","i":true},{"text":": the model reasons its way past the obvious answer, or the extended trace drifts from the requested format."}]},{"t":"p","runs":[{"text":"The pattern is consistent. Thinking helps when the difficulty is in "},{"text":"deciding what the answer is","b":true},{"text":". It does not help when the difficulty is in "},{"text":"knowing something","b":true},{"text":" or in "},{"text":"producing a shape","b":true},{"text":". Retrieval and schemas fix those, respectively, and both are far cheaper."}]},{"t":"h2","id":"setting-a-budget","runs":[{"text":"Setting a budget"}]},{"t":"p","runs":[{"text":"Where the API exposes a token budget, treat it as a dial with a knee:"}]},{"t":"code","lang":"python","label":"Python","text":"resp = client.messages.create(\n    model=REASONING_MODEL,\n    thinking={\"type\": \"enabled\", \"budget_tokens\": 4000},\n    max_tokens=6000,   # must exceed the thinking budget\n    messages=[...])"},{"t":"p","runs":[{"text":"Run your evaluation set at 1,000, 4,000 and 16,000. Almost every task has a point past which more budget buys nothing measurable: commonly a few thousand tokens for ordinary product work. Paying for 32,000 when the curve flattened at 3,000 is the most common way to waste money on reasoning models."}]},{"t":"p","runs":[{"text":"Note the constraint in that snippet: "},{"text":"max_tokens","code":true},{"text":" must leave room for both the thinking and the answer. Set them too close and the model spends its whole budget reasoning and gets truncated before it answers; you pay full price for nothing, and "},{"text":"stop_reason","code":true},{"text":" will say "},{"text":"max_tokens","code":true},{"text":", which is another reason to check that field."}]},{"t":"h2","id":"do-not-treat-the-trace-as-an-explanation","runs":[{"text":"Do not treat the trace as an explanation"}]},{"t":"p","runs":[{"text":"The most important limitation, and the one most often ignored in product design."}]},{"t":"p","runs":[{"text":"A reasoning trace is generated text. It is "},{"text":"correlated","i":true},{"text":" with the computation that produced the answer, but it is not a log of it. Models have been shown to reach an answer and then produce a trace that rationalises it, to use a hint in the prompt while never mentioning the hint in their reasoning, and to write a plausible chain that does not match what actually determined the output."}]},{"t":"p","runs":[{"text":"So: do not show the trace to a user as a justification, do not audit decisions by reading it, and do not build a compliance story on it. If you need a defensible reason for a decision, require the model to cite a source you can verify independently. A verifiable quote is evidence. A reasoning trace is a narrative."}]},{"t":"h2","id":"the-cheap-version-still-works","runs":[{"text":"The cheap version still works"}]},{"t":"p","runs":[{"text":"Before reaching for a reasoning model, try the free version on an ordinary one: ask it to work through the problem in a "},{"text":"<scratchpad>","code":true},{"text":" block, then give the final answer in a "},{"text":"<answer>","code":true},{"text":" block, and strip the scratchpad in your code. You get most of the benefit on moderately hard tasks, you can read the working, and you control the budget by asking for brevity."}]},{"t":"p","runs":[{"text":"And measure. On roughly half the tasks people reach for reasoning models to solve, a well-retrieved mid-tier model answers correctly in 700 milliseconds for a fortieth of the price."}]}],"check":{"question":[{"text":"A team enables a large thinking budget on a document-summarisation feature and sees no quality improvement, much higher cost, and slower responses. Why is this the expected result?"}],"options":[{"text":[{"text":"The thinking budget was too small to make a difference on long documents."}],"correct":false,"tempting":[{"text":"Assumes the curve keeps rising with budget, when the limiting factor here is the nature of the task rather than the amount of computation."}]},{"text":[{"text":"Summarisation needs a longer context window rather than more reasoning, so the budget was spent on the wrong resource."}],"correct":false,"tempting":[{"text":"Names a real but separate constraint; the documents fit, and window size is not what the thinking budget trades against."}]},{"text":[{"text":"The difficulty in summarising is selecting and compressing material already present, not deciding an answer through multiple dependent steps, so extra forward passes have nothing to buy."}],"correct":true,"tempting":null},{"text":[{"text":"Reasoning models are trained on mathematics and code and are simply weaker at prose."}],"correct":false,"tempting":[{"text":"Overstates specialisation; these models are strong at prose, and the absence of gain here is about the task’s structure rather than a weakness."}]}],"order":[1,3,2,0],"why":null},"prev":{"slug":"decomposition-and-chaining","title":"One big prompt, or four small ones","minutes":9},"next":{"slug":"prompt-caching","title":"Prompt caching, and the ordering rule that pays for itself","minutes":8}}}