Retrieving tools the way you retrieve documents

Lesson 2 of 5 in Tool Selection at Scale: Forty Tools, One Context.

Hand-written loadouts stop scaling somewhere around the point where you cannot hold the catalog in your head — a few dozen tools, or a catalog that changes weekly because it is assembled from servers other teams own. The next rung looks exactly like RAG, with tools in place of documents: embed each tool’s name and description, index them, embed the incoming request, retrieve the top k, expose only those.

The arithmetic is genuinely good. Two hundred contracts at, say, 150 tokens each is 30,000 tokens riding along on every call whether used or not; the top eight is 1,200. That is the entire pitch, and it is a real one.

Now the part that is usually discovered in production. You have inserted a retriever into the control path of every request — a component with its own accuracy, its own latency, and its own failure mode. And that failure mode is the nastiest kind.

The silent failure: retrieval misses look exactly like model stupidity

When the right tool is not in the top k, nothing errors. There is no exception, no 4xx, no red span. The model receives a menu that does not contain the correct move and does the most sensible thing available: it uses a neighbouring tool, or it tells the user it cannot help. Read the trace afterwards and you will see a model reasoning competently over the wrong menu — which is indistinguishable, from the model’s output alone, from a model that had the right menu and chose badly.

Those two bugs have opposite fixes. One is rewrite the tool description so it wins the boundary contest; the other is fix the retriever, which never showed the model the tool at all. Guess wrong and you will spend a week polishing descriptions that were never read.

The one instrumentation rule that makes this tractable: log the exposed tool set — the actual list, or a stable hash plus the retrieval scores — on every model call. Without that line, every selection bug in your system is unfalsifiable. With it, the diagnosis takes seconds.

Retrieval-based selection, with the parts people skip

Request
Build the retrieval query
The naive version embeds the raw user message. Better: the current task plus recent turns, because “do that for the other region too” has none of the words that would find the right tool.
Retrieve candidates
Hybrid beats pure semantic here. Tool names and parameters are short, exact strings — keyword matching catches get_dns_record when the user says get_dns_record, which embeddings can blur.
Union with the floor set
Always-present tools: escalate/ask_human, and any discovery tool. A retriever that can hide the escape hatch will eventually hide it on the request that needed it most.
Model call with k + floor tools
Log the exposed set and the scores. This is the line the next lesson debugs from.
Task shifted? Re-retrieve
Retrieving once per conversation is the common shortcut and a common bug: a run that starts as a lookup and becomes a change request needs a different menu mid-flight.
Answer

Which metric governs the retriever — precision or recall?

Recall@k, decisively. A precise retriever that omits the needed tool produces a wrong answer with no error. An imprecise one that includes three irrelevant tools produces a slightly heavier prompt and a routing decision the model is usually fine at, because that is the same job it does with a hand-written loadout. The asymmetry says: be generous with k. Tune k down only when you have measured recall and can watch it hold.

What does the labeled dataset look like?

Rows of (request, run state, the tool that must be available) — and where several are acceptable, the acceptable set. A few hundred rows drawn from real traffic beats a synthetic thousand, because the requests you cannot imagine are exactly the ones that miss. Every production miss you diagnose becomes a new row; that is the regression suite for your retriever.

Why does pure semantic search underperform on tool catalogs?

A tool description is 30–200 tokens of terse, jargon-dense text, and its most discriminating token is often an exact identifier — a table name, a service name, an API verb. Embeddings are built to blur near-synonyms, which is the wrong instinct when get_dns_record and get_dns_zone are different tools. Hybrid retrieval — keyword plus vector — recovers the exact-match cases without giving up the paraphrase cases. If you are also reranking, rerank on the description, not the name.

What about tools that are never retrieved at all?

Audit for them deliberately: any tool with zero retrievals over a month is either dead or shadowed by a better-matching neighbour, and you cannot tell which from usage data alone. This is the retrieval-scale version of the shadowing problem from tool design — same cause, different layer. A monthly “never retrieved” report is cheap and finds real bugs.

When is retrieval not worth it?

Rule of thumb rather than a threshold from a paper: while you can still enumerate the catalog by hand and the loadouts are stable, code beats a model. A hand-written map from task to tool list is deterministic, reviewable in a pull request, free at runtime, and cannot suffer a recall miss. Retrieval earns its failure mode when the catalog is too large or too volatile to hand-maintain — most often because it is aggregated from servers you do not own. Adopting it earlier buys you an extra silent failure mode in exchange for nothing.

Key terms: tool retrieval, hybrid search, embedding, reranking, trace

Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.