The Two-Model Router

Route easy traffic to a small model and hard traffic to a flagship: when it actually pays, how to build the router in three rungs, and how quality erodes when you stop watching the boundary.

Plot the difficulty of a high-volume LLM workload and it is almost never flat. A large share of requests are formulaic — rephrase this, classify that, extract fields into a schema the model has seen ten thousand times — and a small share genuinely need a flagship's reasoning. Paying flagship rates for the formulaic share is, on the bills we get asked to look at, the single largest recoverable line item: nothing about the product has to change to claim it.

The two-model router claims it. A cheap decision sits in front of two models: requests the router is confident are easy go to a Small language model (SLM) (or a small tier of the same family); everything else goes to the flagship. The pattern is portable across providers because every major model catalog now carries small and flagship tiers with per-token rates far enough apart to matter — put live numbers in the Token Cost Estimator rather than trusting anyone's prose, ours included.

One disambiguation before the plumbing: this is application-level routing, a component you build and operate. It shares a name with the Router inside a Mixture of experts (MoE) layer — a learned gate dispatching tokens to experts inside one model — but nothing else. That router is the model vendor's problem. This one is yours: you choose its inputs, you set its threshold, and you own every request it sends to the wrong place.

When the pattern pays — all four, or skip it

Volume. The router is an engineering project with an on-call rotation. At low volume, the saved tokens never repay it; the honest alternative is just using the flagship everywhere.
Skewed difficulty, measured. Sample real traffic, run it through both models, grade the pairs (an LLM-as-judge with spot-check human passes is the usual harness). If the small model does not win or tie on a large fraction, there is no easy share to route — stop here.
Difficulty is detectable before generation. The router sees the request, not the answer. If easy and hard requests look identical at the prompt surface, no rung of the ladder below will separate them.
The product tolerates tiered quality. If you have promised every user the best model on every request, routing down is a contract violation, not an optimization.

The router itself comes in three rungs, and the discipline is to start at the bottom and climb only when the current rung's mistakes are measured and expensive.

Rung 1 — rules. Route on what the request already tells you: which endpoint or product surface it came from, input length, language, customer tier, whether documents are attached. Rules are free at request time, fully debuggable ("why did this go to the small model?" has a one-line answer), and deployable in an afternoon. A surprising number of deployments never need more, because product surface is a strong difficulty proxy — the autocomplete endpoint is not where the hard questions live.

Rung 2 — a trained classifier. When wording, not metadata, predicts difficulty, train a small classifier — a fine-tuned encoder or a head over an Embedding — to predict "the small model will handle this" from the prompt itself. The labels come from the same both-models replay you ran to validate the skew, so the grading harness you built for the preconditions becomes training data. You get a score you can threshold and tune, for single-digit-to-tens of milliseconds of added latency on modest hardware.

Rung 3 — a small-LLM judge. When the boundary is too semantic even for a classifier — difficulty hinges on what is being asked, not how — prompt a small LLM to assess the request and emit a routing verdict. This is the most flexible rung and the most expensive: it puts a real generation in front of every request, its judgments shift when its prompt or its underlying model changes, and it inherits the known biases of LLM-as-judge setups. It is a last rung, not a first one — and if rung 3 is not separating traffic either, the conclusion is not a bigger router. It is that your traffic does not have a routable boundary.

The router ladder — each rung trades operational simplicity for a finer decision boundary.
Rung	Decides from	Added latency	Upkeep	Climb when
Rules	Request metadata: endpoint, length, language, customer tier, attachments	Effectively zero	A rule list that needs a named owner, or it fossilizes as the product grows	Graded outcomes disagree with the rules on a growing slice of traffic
Trained classifier	The prompt text itself, embedded or encoded	Single-digit to tens of milliseconds	Labeled routing data plus periodic retraining as traffic shifts	The boundary is semantic — what is asked, not any surface feature, predicts difficulty
Small-LLM judge	The full request, judged in natural language against a routing rubric	A full model call ahead of every request — can rival the small model’s own generation time	A judge prompt to version and evaluate, plus per-request inference cost	There is no next rung. If this fails, reconsider routing itself

One request through the router — doubt always flows toward the flagship

Request arrives
Router scores the request
Rules, classifier, or judge — whichever rung you run, it must emit a score and a confidence, and both get logged with the request id.
Confidently easy?
The threshold is a product decision dressed as a config value. Everything that is not confidently easy — including "the router timed out" — goes up.
Small model generates
The routed-down path. This is where the savings are earned — and where quality erosion hides if nobody is grading it.
Output passes checks?
Cheap post-generation checks: schema validity, refusal detection, self-reported uncertainty, groundedness against provided context. Cheap — or the check eats the margin.
Flagship generates
Hard traffic lands here directly; escalations land here after paying the small model first. Keep this path a first-class code path — it carries your hardest requests.
Response returned

The diagram encodes the design rule that keeps this pattern safe: doubt flows up. The router does not decide between two models; it decides whether a request has earned the cheap path. Uncertainty, a malformed score, a router timeout — all of it defaults to the flagship, so the failure mode of the router failing is a bigger bill, not a worse answer. You grow the savings by widening the confidently-easy region with evidence: better features, more labels, a retrained threshold. You never grow it by lowering the bar under cost pressure — that is how routers rot.

There are two distinct fallbacks in the diagram, and they have different jobs. Pre-generation fallback (the "no / unsure" edge) is the normal case and costs nothing extra — the request simply goes where it would have gone without a router. Post-generation escalation (the "fail" edge) is the expensive one: the user waits through the small model's attempt and the flagship's, and you pay both meters. It exists as a safety net, not a routing strategy — its rate is one of the most useful health metrics the system emits. A rising escalation rate means the router is sending traffic down that should have gone up; treat it as an alarm, not as the mechanism working.

Two operational notes that save later grief. Log the routing decision — rung, score, confidence, threshold version — on every request; a routing boundary you cannot reconstruct is a routing boundary you cannot debug. And keep conversations sticky where the product allows it: a session that alternates between models mid-thread exposes the seam (Temperature and phrasing quirks differ), and re-answering with a different model after an escalation already exposes it once.

Whether the whole apparatus pays is one inequality, in the same named-variable style as break-even thinking. Define, per month:

V — requests
p_small, p_big — blended cost per request on each path: tokenize real traffic and price it with live rates in the Token Cost Estimator; per request, because that is the unit the router decides in
s — the share of traffic routed down
e — the escalation rate: the fraction of routed-down requests that bounce up after failing checks
c_r — the router's own cost per request: ~zero for rules, inference for a classifier, a model call for a judge

Without the router every request costs p_big. With it, a routed-down request costs p_small, an escalated one costs p_small + p_big, and everything pays c_r:

C_routed = c_r + (1 − s) × p_big + s × p_small + s × e × p_big

so the monthly saving is V × [s × (p_big − p_small − e × p_big) − c_r], and the router pays while s × (p_big − p_small − e × p_big) > c_r.

Read the inequality the way you will feel it. The prize rides on s × (p_big − p_small) — a big routable share times a big price gap; shrink either and the pattern quietly stops paying. Escalations are the double-billing term: each one pays p_small for nothing and p_big anyway, which is why e must stay small for the arithmetic to survive. c_r is multiplied by all of V — a judge-rung router with real per-call cost taxes every request, including the hard ones it merely waves through. And p_big − p_small is set by someone else's pricing page: a flagship price cut shrinks the prize overnight while your engineering cost stays sunk. Re-run the inequality whenever prices move; never hardcode the rates.

Notice what the formula does not contain: a term for quality. Tokens are what get metered; the cost of a routed-down request getting a slightly worse answer appears on no invoice. That asymmetry is exactly why the next section exists.

The eval discipline for a routed system has one organizing principle: the aggregate metric will lie to you. Flagship traffic props up the mean, so a fleet-wide quality score can hold steady while the routed-down slice decays. Everything below is a way of refusing the aggregate.

Per-route golden sets. Keep one Golden set per route — easy-set cases graded against the small model, hard-set cases against the flagship — and report them separately, always. The harness mechanics are the standard ones (building an eval harness); the routing-specific move is simply never letting the two routes share a scoreboard.

Watch the boundary, not the middle. The interesting failures live near the threshold, where the router was almost unsure. Sample requests from a band around the cutoff score, run both models on them, and grade the pairs. This is also where new labels come from: boundary cases the flagship wins decisively are your classifier's next training data.

Shadow-route a slice. Continuously send a small random sample of routed-down traffic to the flagship as well, compare the answers with an LLM-as-judge plus periodic human passes, and chart the small model's win-or-tie rate over time. This chart is the erosion detector — the direct measurement of the thing the cost formula cannot see. When it dips, you found Drift before your users did.

Re-baseline on every change at the boundary. A model swap on either path, a prompt change, a threshold moved to chase savings — each is a deploy, and each gets Regression testing against the per-route sets before it ships. The threshold especially: it looks like a config value, and it is actually the product's quality contract (production evals is the longer version of this argument).

The honest failure modes

Boundary drift. The router was fit to last quarter's traffic. A new user segment, a new product surface, a marketing push that changes who shows up — and yesterday's "easy" starts mislabeling today's requests. Rules fossilize the day their owner changes teams; classifiers decay politely and silently. The boundary is a model of your traffic, and traffic does not sign a contract to stay put.
Silent quality erosion on routed-down traffic. Nobody files a ticket titled "you gave me the small model." Routed-down users just get slightly worse answers, the aggregate dashboard stays green, and the failure is invisible by construction unless the per-route sets and the shadow slice exist. This is the pattern's signature failure: it degrades exactly the traffic it was built to save money on, at exactly the rate nobody is measuring.
The router's own latency and availability. The router sits in front of 100% of traffic — including the hard requests that gain nothing from it. Its latency adds to TTFT on every path, a judge rung can spend longer deciding than the small model spends answering, and the router is a new single point of failure that needs the same timeout-and-default story as any other hot-path service. The default, as always: doubt flows up.

What to steal

Measure the skew before building anything. Replay a traffic sample through both models and grade it. No large win-or-tie share for the small model, no project.
Climb the ladder on evidence. Rules first; a classifier when wording beats metadata; a judge only when the boundary is irreducibly semantic. Each rung's mistakes must be measured before the next rung is justified.
Doubt flows up. Uncertainty, timeouts, and failed checks all default to the flagship. Grow savings by widening the confidently-easy set with evidence — never by lowering the bar.
Keep the two fallbacks distinct. Pre-generation fallback is free and normal; post-generation escalation pays both bills and is an alarm signal, not a strategy.
Do the cost math in named variables — the router pays while s × (p_big − p_small − e × p_big) > c_r — and re-run it whenever a pricing page moves. The formula has no quality term; that absence is a warning label, not an oversight.
Refuse the aggregate. Per-route golden sets, a graded shadow slice of routed-down traffic, and boundary-band sampling. The one chart that must exist is the small model's win-or-tie rate over time on traffic the router sent down.
Log every routing decision with score, confidence, and threshold version. A boundary you cannot reconstruct is a boundary you cannot debug — or defend.