{"course":{"slug":"context-engineering","title":"Context Engineering","url":"https://addaly.com/learn/context-engineering","level":"advanced","track":"build"},"lesson":{"slug":"too-many-tools","title":"When the toolset outgrows the window","url":"https://addaly.com/learn/context-engineering/too-many-tools","position":64,"of":87,"minutes":9,"answer":"Past a few dozen tools the answer is to retrieve a subset per request rather than to list them all.","keyIdea":"Past a few dozen tools the answer is to retrieve a subset per request rather than to list them all, which trades a fixed token tax and a routing problem for a recall problem whose failures are invisible."},"body":{"format":"markdown","text":"## How a toolset gets to two hundred\n\nNobody designs a two-hundred-tool system. It accumulates.\n\nThe usual route is servers. You connect one that exposes your issue tracker: 18 tools. One for the document store: 22. One for the calendar: 11. One for the cloud provider: 40. One for the database: 9. Five connections, exactly the number a working assistant needs, and you are now at a hundred tools and somewhere around **18,000 tokens of schema** on every request, before a single word of the conversation.\n\nAt that size two things are true at once. The token cost is the largest single block in your request. And the selection problem has become hard, because a hundred descriptions written by five different teams will overlap in ways none of those teams could have anticipated.\n\n## Retrieving tools instead of listing them\n\nThe move is to treat the toolset the way you treat a corpus: index it, and select per request.\n\nEmbed each tool's name and description once. At request time, embed the user's message, retrieve the top ten or fifteen tools, and pass only those. Your 18,000-token block becomes 2,500, and the model is choosing among fifteen options rather than a hundred.\n\nThis works, and reported results on it are good. It also introduces a new failure mode that you must understand before you deploy it, because it is silent.\n\n**If the retriever misses the right tool, the model does not know a tool is missing.** There is no error. There is no empty result. The model answers from its own knowledge, or says it cannot do the thing, and both of those look exactly like ordinary behaviour. Compare this with a bad retrieval of passages, where the model at least has the retrieved text and can say it does not answer the question. A missed tool leaves no trace at all.\n\nSo tool retrieval needs its own recall measurement, on the same routing set you built for descriptions: for each message, was the correct tool in the retrieved subset? That number caps everything. If recall@15 is 0.94, six percent of your traffic is silently degraded and no dashboard will show it.\n\n## The cache consequence, and the layout that fixes it\n\nA variable tool block is a variable prefix, and a variable prefix does not cache. If the tools change per request and they sit at the top of the request, you have just destroyed the prefix cache for the whole system prompt above them, which is usually a far bigger block.\n\nThe layout that keeps both:\n\n1. System prompt: stable.\n2. **Core tools**: the eight or ten called on most requests, in a fixed sorted order: stable.\n3. **Cache breakpoint here.**\n4. **Retrieved tools**: the request-specific extras: variable.\n5. Conversation, retrieved passages, user message.\n\nEverything above the breakpoint hits the cache every time. The variable tail is small. You keep most of the discount and most of the token saving.\n\nDeciding which tools are core is an empirical question with an easy answer: log tool calls for a week and take the ones above a usage threshold. Teams consistently find the distribution is extremely skewed: a handful of tools account for the large majority of calls, and dozens are called a few times a month.\n\n## The alternatives, and when each is better\n\n**Grouping behind a router.** Expose five coarse tools (`calendar`, `issues`, `documents`) each of which takes an action name and arguments, and dispatch inside. This collapses the schema dramatically. The cost is that you have moved the specificity out of the schema, so constrained decoding can no longer validate the arguments, and the model is now writing a free-form action name that your code has to handle when it is wrong. Reasonable when the sub-actions are numerous and similar; poor when their parameters differ wildly.\n\n**Progressive disclosure.** Expose a small set plus a `find_tools(query)` meta-tool that returns matching definitions, which the model then calls. Honest about the missing-tool problem: the model can go looking. Costs an extra round trip on every request that needs a rare tool, so it is a latency trade rather than a token trade.\n\n**Splitting into sub-agents.** Give each sub-agent its own small, coherent toolset and have the parent delegate. This is the cleanest by far when the work decomposes, and the next lesson but one is about what it costs.\n\n**Deleting tools.** Unfashionable and usually the correct first step. Before building retrieval over a hundred tools, find out how many of them were called at all last month. The answer is often twenty-something. Removing seventy unused tools is a smaller change than any architecture above, and it improves selection accuracy for the ones that remain by removing their competitors.\n\n## What is not settled\n\nThere is no agreed threshold at which listing stops working and retrieval starts being worth it. Claims circulate in both directions and most rest on one team's benchmark with one model. The reason there is no number is that the variable that matters is not the count but the distinctness: forty tools that do obviously different things route better than twelve that overlap.\n\nWhich means the threshold is yours to find. Run the routing test at your current size. Add the next server. Run it again. When accuracy falls below what your product can tolerate, you have your answer, measured on your tools rather than borrowed from someone else's."},"check":{"question":"A system retrieves the top 15 tools per request instead of listing all 120. What failure does this introduce that listing did not have?","options":[{"text":"Schema validation weakens, because retrieved definitions are passed as text rather than as structured tools.","correct":false,"tempting":"Retrieved tools are passed through the same tools parameter and validate identically; retrieval changes which are present, not their form."},{"text":"A missed tool raises no error: the model simply answers without it.","correct":true},{"text":"Latency rises on every request, because the model must make a second call to fetch the tool it wants.","correct":false,"tempting":"Describes progressive disclosure, a different design; per-request retrieval happens before the model call, adding only an embedding lookup."},{"text":"Token cost rises, because the retrieved subset is appended to the full list rather than replacing it.","correct":false,"tempting":"Retrieval replaces the list; the token cost is the main thing it reduces."}]},"prev":{"slug":"errors-that-teach","title":"An error message is context too","url":"https://addaly.com/learn/context-engineering/errors-that-teach"},"next":{"slug":"the-agent-loop-grows","title":"Why a twenty-step agent run costs what it does","url":"https://addaly.com/learn/context-engineering/the-agent-loop-grows"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'When the toolset outgrows the window', in Context Engineering. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/context-engineering/too-many-tools (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"When the toolset outgrows the window","courseTitle":"Context Engineering","module":{"index":7,"slug":"tools-schemas-and-the-agent-window","title":"Tools, schemas and the agent’s window"},"answer":null,"keyIdea":"Past a few dozen tools the answer is to retrieve a subset per request rather than to list them all, which trades a fixed token tax and a routing problem for a recall problem whose failures are invisible.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"how-a-toolset-gets-to-two-hundred","runs":[{"text":"How a toolset gets to two hundred"}]},{"t":"p","runs":[{"text":"Nobody designs a two-hundred-tool system. It accumulates."}]},{"t":"p","runs":[{"text":"The usual route is servers. You connect one that exposes your issue tracker: 18 tools. One for the document store: 22. One for the calendar: 11. One for the cloud provider: 40. One for the database: 9. Five connections, exactly the number a working assistant needs, and you are now at a hundred tools and somewhere around "},{"text":"18,000 tokens of schema","b":true},{"text":" on every request, before a single word of the conversation."}]},{"t":"p","runs":[{"text":"At that size two things are true at once. The token cost is the largest single block in your request. And the selection problem has become hard, because a hundred descriptions written by five different teams will overlap in ways none of those teams could have anticipated."}]},{"t":"h2","id":"retrieving-tools-instead-of-listing-them","runs":[{"text":"Retrieving tools instead of listing them"}]},{"t":"p","runs":[{"text":"The move is to treat the toolset the way you treat a corpus: index it, and select per request."}]},{"t":"p","runs":[{"text":"Embed each tool’s name and description once. At request time, embed the user’s message, retrieve the top ten or fifteen tools, and pass only those. Your 18,000-token block becomes 2,500, and the model is choosing among fifteen options rather than a hundred."}]},{"t":"p","runs":[{"text":"This works, and reported results on it are good. It also introduces a new failure mode that you must understand before you deploy it, because it is silent."}]},{"t":"p","runs":[{"text":"If the retriever misses the right tool, the model does not know a tool is missing.","b":true},{"text":" There is no error. There is no empty result. The model answers from its own knowledge, or says it cannot do the thing, and both of those look exactly like ordinary behaviour. Compare this with a bad retrieval of passages, where the model at least has the retrieved text and can say it does not answer the question. A missed tool leaves no trace at all."}]},{"t":"p","runs":[{"text":"So tool retrieval needs its own recall measurement, on the same routing set you built for descriptions: for each message, was the correct tool in the retrieved subset? That number caps everything. If recall@15 is 0.94, six percent of your traffic is silently degraded and no dashboard will show it."}]},{"t":"h2","id":"the-cache-consequence-and-the-layout-that-fixes-it","runs":[{"text":"The cache consequence, and the layout that fixes it"}]},{"t":"p","runs":[{"text":"A variable tool block is a variable prefix, and a variable prefix does not cache. If the tools change per request and they sit at the top of the request, you have just destroyed the prefix cache for the whole system prompt above them, which is usually a far bigger block."}]},{"t":"p","runs":[{"text":"The layout that keeps both:"}]},{"t":"ol","start":1,"items":[[{"text":"System prompt: stable."}],[{"text":"Core tools","b":true},{"text":": the eight or ten called on most requests, in a fixed sorted order: stable."}],[{"text":"Cache breakpoint here.","b":true}],[{"text":"Retrieved tools","b":true},{"text":": the request-specific extras: variable."}],[{"text":"Conversation, retrieved passages, user message."}]]},{"t":"p","runs":[{"text":"Everything above the breakpoint hits the cache every time. The variable tail is small. You keep most of the discount and most of the token saving."}]},{"t":"p","runs":[{"text":"Deciding which tools are core is an empirical question with an easy answer: log tool calls for a week and take the ones above a usage threshold. Teams consistently find the distribution is extremely skewed: a handful of tools account for the large majority of calls, and dozens are called a few times a month."}]},{"t":"h2","id":"the-alternatives-and-when-each-is-better","runs":[{"text":"The alternatives, and when each is better"}]},{"t":"p","runs":[{"text":"Grouping behind a router.","b":true},{"text":" Expose five coarse tools ("},{"text":"calendar","code":true},{"text":", "},{"text":"issues","code":true},{"text":", "},{"text":"documents","code":true},{"text":") each of which takes an action name and arguments, and dispatch inside. This collapses the schema dramatically. The cost is that you have moved the specificity out of the schema, so constrained decoding can no longer validate the arguments, and the model is now writing a free-form action name that your code has to handle when it is wrong. Reasonable when the sub-actions are numerous and similar; poor when their parameters differ wildly."}]},{"t":"p","runs":[{"text":"Progressive disclosure.","b":true},{"text":" Expose a small set plus a "},{"text":"find_tools(query)","code":true},{"text":" meta-tool that returns matching definitions, which the model then calls. Honest about the missing-tool problem: the model can go looking. Costs an extra round trip on every request that needs a rare tool, so it is a latency trade rather than a token trade."}]},{"t":"p","runs":[{"text":"Splitting into sub-agents.","b":true},{"text":" Give each sub-agent its own small, coherent toolset and have the parent delegate. This is the cleanest by far when the work decomposes, and the next lesson but one is about what it costs."}]},{"t":"p","runs":[{"text":"Deleting tools.","b":true},{"text":" Unfashionable and usually the correct first step. Before building retrieval over a hundred tools, find out how many of them were called at all last month. The answer is often twenty-something. Removing seventy unused tools is a smaller change than any architecture above, and it improves selection accuracy for the ones that remain by removing their competitors."}]},{"t":"h2","id":"what-is-not-settled","runs":[{"text":"What is not settled"}]},{"t":"p","runs":[{"text":"There is no agreed threshold at which listing stops working and retrieval starts being worth it. Claims circulate in both directions and most rest on one team’s benchmark with one model. The reason there is no number is that the variable that matters is not the count but the distinctness: forty tools that do obviously different things route better than twelve that overlap."}]},{"t":"p","runs":[{"text":"Which means the threshold is yours to find. Run the routing test at your current size. Add the next server. Run it again. When accuracy falls below what your product can tolerate, you have your answer, measured on your tools rather than borrowed from someone else’s."}]}],"check":{"question":[{"text":"A system retrieves the top 15 tools per request instead of listing all 120. What failure does this introduce that listing did not have?"}],"options":[{"text":[{"text":"Schema validation weakens, because retrieved definitions are passed as text rather than as structured tools."}],"correct":false,"tempting":[{"text":"Retrieved tools are passed through the same tools parameter and validate identically; retrieval changes which are present, not their form."}]},{"text":[{"text":"A missed tool raises no error: the model simply answers without it."}],"correct":true,"tempting":null},{"text":[{"text":"Latency rises on every request, because the model must make a second call to fetch the tool it wants."}],"correct":false,"tempting":[{"text":"Describes progressive disclosure, a different design; per-request retrieval happens before the model call, adding only an embedding lookup."}]},{"text":[{"text":"Token cost rises, because the retrieved subset is appended to the full list rather than replacing it."}],"correct":false,"tempting":[{"text":"Retrieval replaces the list; the token cost is the main thing it reduces."}]}],"order":[2,1,0,3],"why":null},"prev":{"slug":"errors-that-teach","title":"An error message is context too","minutes":8},"next":{"slug":"the-agent-loop-grows","title":"Why a twenty-step agent run costs what it does","minutes":10}}}