When the toolset outgrows the window

How a toolset gets to two hundred

Nobody designs a two-hundred-tool system. It accumulates.

The usual route is servers. You connect one that exposes your issue tracker: 18 tools. One for the document store: 22. One for the calendar: 11. One for the cloud provider: 40. One for the database: 9. Five connections, exactly the number a working assistant needs, and you are now at a hundred tools and somewhere around 18,000 tokens of schema on every request, before a single word of the conversation.

At that size two things are true at once. The token cost is the largest single block in your request. And the selection problem has become hard, because a hundred descriptions written by five different teams will overlap in ways none of those teams could have anticipated.

Retrieving tools instead of listing them

The move is to treat the toolset the way you treat a corpus: index it, and select per request.

Embed each tool’s name and description once. At request time, embed the user’s message, retrieve the top ten or fifteen tools, and pass only those. Your 18,000-token block becomes 2,500, and the model is choosing among fifteen options rather than a hundred.

This works, and reported results on it are good. It also introduces a new failure mode that you must understand before you deploy it, because it is silent.

If the retriever misses the right tool, the model does not know a tool is missing. There is no error. There is no empty result. The model answers from its own knowledge, or says it cannot do the thing, and both of those look exactly like ordinary behaviour. Compare this with a bad retrieval of passages, where the model at least has the retrieved text and can say it does not answer the question. A missed tool leaves no trace at all.

So tool retrieval needs its own recall measurement, on the same routing set you built for descriptions: for each message, was the correct tool in the retrieved subset? That number caps everything. If recall@15 is 0.94, six percent of your traffic is silently degraded and no dashboard will show it.

The cache consequence, and the layout that fixes it

A variable tool block is a variable prefix, and a variable prefix does not cache. If the tools change per request and they sit at the top of the request, you have just destroyed the prefix cache for the whole system prompt above them, which is usually a far bigger block.

The layout that keeps both:

1. System prompt: stable.
2. Core tools: the eight or ten called on most requests, in a fixed sorted order: stable.
3. Cache breakpoint here.
4. Retrieved tools: the request-specific extras: variable.
5. Conversation, retrieved passages, user message.

Everything above the breakpoint hits the cache every time. The variable tail is small. You keep most of the discount and most of the token saving.

Deciding which tools are core is an empirical question with an easy answer: log tool calls for a week and take the ones above a usage threshold. Teams consistently find the distribution is extremely skewed: a handful of tools account for the large majority of calls, and dozens are called a few times a month.

The alternatives, and when each is better

Grouping behind a router. Expose five coarse tools (calendar, issues, documents) each of which takes an action name and arguments, and dispatch inside. This collapses the schema dramatically. The cost is that you have moved the specificity out of the schema, so constrained decoding can no longer validate the arguments, and the model is now writing a free-form action name that your code has to handle when it is wrong. Reasonable when the sub-actions are numerous and similar; poor when their parameters differ wildly.

Progressive disclosure. Expose a small set plus a find_tools(query) meta-tool that returns matching definitions, which the model then calls. Honest about the missing-tool problem: the model can go looking. Costs an extra round trip on every request that needs a rare tool, so it is a latency trade rather than a token trade.

Splitting into sub-agents. Give each sub-agent its own small, coherent toolset and have the parent delegate. This is the cleanest by far when the work decomposes, and the next lesson but one is about what it costs.

Deleting tools. Unfashionable and usually the correct first step. Before building retrieval over a hundred tools, find out how many of them were called at all last month. The answer is often twenty-something. Removing seventy unused tools is a smaller change than any architecture above, and it improves selection accuracy for the ones that remain by removing their competitors.

What is not settled

There is no agreed threshold at which listing stops working and retrieval starts being worth it. Claims circulate in both directions and most rest on one team’s benchmark with one model. The reason there is no number is that the variable that matters is not the count but the distinctness: forty tools that do obviously different things route better than twelve that overlap.

Which means the threshold is yours to find. Run the routing test at your current size. Add the next server. Run it again. When accuracy falls below what your product can tolerate, you have your answer, measured on your tools rather than borrowed from someone else’s.