Reading mis-selection in a trace

Right intent, wrong tool

Signature. The model states the goal correctly — “I’ll look up the customer’s recent orders” — and then calls something adjacent, like get_shipments.

Cause. Two contracts both plausibly answer the request; no description says when this one and not that one. Classic overlap, now at scale.

Fix. Boundary text in both descriptions, each naming the other. Merge them if you cannot state the boundary in one sentence.

Not the fix. A bigger model. If the boundary is genuinely ambiguous, a stronger model picks more confidently, not more correctly.

Argument cross-pollination

Signature. Correct tool, but arguments borrowed from a sibling: customer_id populated with an order number, or a parameter name that belongs to the neighbouring schema.

Cause. Sibling tools with near-identical parameter schemas blur together in context. The model has read both and interpolates.

Fix. Make the shapes visibly different — distinct required-parameter names, distinct formats, an enum where the sibling takes free text. Then let schema validation reject the blend loudly instead of coercing it.

Watch for. This one often passes validation when both schemas take a string. Add format constraints so the wrong value cannot type-check.

The tool was never there

Signature. The model apologises for a missing capability, or invents a tool name that does not exist.

Cause. A loadout filter or a retriever excluded it. The model behaved correctly given its menu.

Fix. Nothing in the model layer. Fix the filter or the retriever — and add the request to the retriever’s labeled set.

How you know. Only from the logged exposed set. This is why that log line exists; without it this case is systematically misdiagnosed as one of the two above.

Ping-pong

Signature. Across turns of one run, the model alternates between two tools for the same job, sometimes calling both.

Cause. A near-tie in the descriptions, resolved differently by sampling each turn. At scale, also caused by re-retrieval returning a different ordering per turn.

Fix. Break the tie explicitly in text. If re-retrieval is the source, stabilise it — cache the loadout for the duration of a task rather than recomputing per turn, which also keeps your cache warm.

Never called at all

Signature. A tool with weeks of zero invocations that the team is sure is needed.

Cause. Shadowed by a broader, better-named neighbour that absorbs every request it was built for — or never retrieved, which is the same shadowing one layer down.

Fix. Carve out its territory in the description explicitly. And check the retrieval logs first: “never selected” and “never offered” are different bugs with the same appearance.