Cascading and task-specific small models
Lesson 3 of 3 in Model Deployment and Throughput Strategies.
There is a budget discipline hiding inside this architecture: right-size the model per request, not per application. Most production traffic is routine — classify this ticket, extract these fields, answer this FAQ — and a task-specific small model (a small pre-trained FM, or a distilled student of your big one) handles it at a fraction of the latency and token cost. The frontier model should be reserved for the requests that actually need it.
API-based model cascading is how that policy becomes a request path: every query tries the cheap model first; a gate — a confidence score, a validation check, a judge — decides whether the draft is good enough; only failures escalate. The economics follow the escalation rate: if 80% of traffic ends at the small model, the big model's cost applies to one request in five.
| Pattern | Who decides | Shape | Reach for it when |
|---|---|---|---|
Your gate, after the cheap attempt | Try small → check → escalate failures | Routine traffic dominates and a reliable quality check exists | |
Bedrock, before any model runs — it predicts per-prompt response quality | One router endpoint → best-fit model in a family | You want the decision managed: no gate to build, response reports which model served | |
Distillation to a task-specific small model | You, at training time | Teacher model trains a small student for ONE task | A narrow task runs at volume — the student replaces the big model there outright |
Key terms: Model cascading, Intelligent prompt routing (Bedrock), Model distillation
Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.