Routing: classify, then dispatch

Lesson 2 of 6 in Workflow Patterns: Five Ways to Compose Model Calls.

A chain handles one kind of input. Real traffic is a mixture — refund requests next to password resets next to “your app deleted my thesis”. Forcing one prompt to handle all of them means every instruction competes for the model’s attention, and optimizing for one category quietly degrades the others.

Routing separates the concerns: a first call classifies the input, then code dispatches it to a specialist path — its own prompt, its own tools, often its own model. The router can be a small, cheap, fast model (classification is easy compared to generation), while each specialist is tuned for exactly one job. Specialization is the point: the refund path can be a strict chain with an approval gate, while the general-question path is a single cheap call.

Routing: classify, then dispatch

Ticket arrives
Router: classify intent
A small model with structured output: one label from a fixed set, plus a confidence signal. Cheap enough to run on every request.
Refund specialist
A chain with tools: look up the order, check the refund policy, draft the response, gate on refund amount.
Technical specialist
A different prompt, different tools (docs search, known-issues database), maybe a stronger model.
Fallback path
The escape hatch for low-confidence or unclassifiable inputs — a general prompt or a human queue. Every router needs one.
Respond

Failure mode: the misclassification cascade

A router error is invisible downstream. The refund specialist never asks “is this really a refund?” — it was built on the assumption that the router already decided. So a misrouted security incident gets a cheerful refund reply, executed with full specialist confidence. This is the cascade: one wrong label at the top, flawless-looking wrong work below it.

Three consequences. First, the router needs its own eval with its own labeled dataset — measured classification accuracy per category, not vibes. Second, every router needs a fallback route for low-confidence and out-of-distribution inputs; a router forced to pick from only happy-path categories will confidently mislabel everything unusual. Third, categories drift — new product, new failure modes, new abuse patterns — so router accuracy belongs in your regression suite, re-checked every release, not validated once at launch.

One design question decides most routing architectures: what does the router see, and what does it emit? Emit a bare label via structured outputs and dispatch in code — that is routing as taught here, and the control flow stays yours. The alternative — letting the model pick a “handoff” target mid-conversation — is drifting toward agent territory, where the model owns control flow. Both are legitimate; know which one you are building, because they fail differently.

Key terms: routing, structured outputs, eval, context window

Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.