Making Small Good

Lesson 3 of 3 in Small Language Models.

A small model that wins in production is rarely small-and-generic. It is small-and-made-good — assembled from three levers this site has already taught, aimed at one workload.

Distillation imports quality. A strong teacher model generates outputs on your task distribution; the small student trains to match them, inheriting behavior it could not have reached from raw data alone. Distillation covers the mechanics; the landscape observation is that the small end lives downstream of the big end — the strength of available teachers keeps raising the floor for students.

Fine-tuning narrows the model to the job. A small model cannot know everything, but your endpoint does not need everything. Task-specific SFT — often as a LoRA — buys disproportionate quality at the small end, because you are spending the model’s limited capacity exactly where the workload lives. Fine-Tuning is the full treatment.

Quantization fits the target. For edge and single-accelerator serving, weights get compressed to lower precision to shrink memory and speed up decode. Small-model cards often ship the artifacts directly — SmolLM3’s card links quantized versions, and the gpt-oss cards note MoE weights shipped in MXFP4 quantization. One rule is absolute: quantization changes the model, so the quantized artifact re-enters evaluation as a new candidate. Quantization explains what you traded.

From big-model quality to small-model serving

  1. Flagship meets the quality bar

    First prove the task is solvable at all: run it on a big model and grade the results. This run also produces your teacher outputs and your baseline.

  2. Build the golden set

    Real traffic in, graded outputs out — the same set will train (via distillation data), evaluate, and regression-test every candidate that follows.

  3. Distill and/or fine-tune a small candidate

    Teacher outputs supervise the student; task-specific SFT or LoRA narrows it to the workload.

  4. Quantize for the target hardware

    Edge or single-accelerator targets usually need lower precision. The quantized artifact is a new model — it does not inherit the un-quantized eval.

  5. Evals match the big-model baseline on your tasks?

    Same harness, same golden set, side by side with the flagship baseline. “Close enough” is a product decision made on numbers, not vibes.

  6. Route: easy traffic small, hard traffic big

    A router — rules, a classifier, or confidence signals — sends the routine majority to the small model and escalates the hard tail.

  7. Small in production; flagship as ceiling and baseline

    The big model remains the escalation path and the regression baseline every future candidate is measured against.

Now the honest part. Two weaknesses are commonly observed at the small end — treat them as default hypotheses to test, not laws. Breadth: a small model holds less of the world, so it degrades faster the further a request strays from its training and tuning distribution. Long-tail robustness: the rare, weird, adversarial, or compound requests that a flagship shrugs off are where small candidates most often crack. Neither weakness announces itself in a demo — both surface exactly where your golden set is thinnest, which is an argument for building the tail into the Golden set deliberately.

That is why the last lever is non-negotiable: evaluate against the big-model baseline, always. Not against a public leaderboard — against the flagship running your task, on your data, in your Eval harness (Building an Eval Harness is the recipe). The baseline turns “is the small model good?” — unanswerable — into “what, exactly, do we lose by going small, and on which slices?” — a decision. And the harness stays on: every fine-tune, every quantization, every candidate swap is a model change, and model changes go through Regression testing.

Why capability per parameter keeps improving

The small end has a remarkable property: it keeps getting better without getting bigger. The precise causes are partly proprietary, but three forces are visible in what builders document, and they compound.

Data got better faster than models got bigger. Heavier curation and filtering concentrate more learnable signal into each training token. The verified Phi card is explicit about the philosophy — supervised fine-tuning on carefully curated data, with a stated training run of 240 B200 GPUs over 4 days, positioned against “models that rely on substantially more training data and compute.”

Training runs longer relative to size. The after-Chinchilla overtraining logic from lesson one — the Gemma 3 card’s 4 trillion tokens into a 4B model is a documented instance — means each new small generation absorbs far more data than compute-optimal prescriptions once suggested.

Teachers keep improving. Distillation ties the small end to the frontier: today’s students learn from teachers that did not exist when yesterday’s students trained, so frontier gains propagate downward with a lag. Qualitatively, small models inherit techniques and training signal from above.

The navigation consequence matters more than the mechanism: never carry a capability judgment about a size class across model generations. “A 3B model can’t do X” is a dated observation, not a durable fact — it expires when the next 3B generation trains on better data, more tokens, and stronger teachers. Re-run the class through your harness whenever the families refresh; the Model Release Log is the tripwire.

In production

Small language models are the standing answer to a sentence every platform team eventually says: “this workload cannot afford the flagship.” The evergreen pattern is routing, not replacement — the easy majority of traffic goes small, the hard tail escalates big, and one shared eval harness referees the split.

AWS

On AWS the split typically runs through Amazon Bedrock’s multi-model surface: one API in front of models at very different capability points, so a router — rules, a lightweight classifier, or confidence signals — can send each request to the cheapest adequate model with the flagship as the escalation path. Where data must stay inside your VPC, self-hosted small models on GPU instances or SageMaker endpoints join the same architecture, held to the same golden set.

Azure

Azure AI Foundry’s model catalog puts flagship APIs and small open-weights families side by side, which is what a routing architecture wants: a small-model deployment next to the flagship deployment, a router in front, and both endpoints graded by the same harness. Small models are also what make edge and on-premises stories real — where data cannot leave, the model that fits on hardware you control is the model you can use.

Google Cloud

On Google Cloud the same pattern runs through Vertex AI: Model Garden carries small open-weights models beside the flagship APIs, so routine traffic can run on a small dedicated endpoint while the hard tail escalates. For on-device products, small models turn “call our API” into “ship the capability with the app,” with quantized variants doing the fitting — and the shared eval harness deciding, per release, whether small is still holding its slice.

Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.