What Survives the Transfer
Lesson 3 of 3 in Distillation.
A distilled student is not a small teacher. It is a specialist trained on a snapshot of the teacher’s behavior in one region of input space — and that framing predicts what survives the transfer.
What transfers well: the behavior you actually sampled. Output format and schema discipline. Style and tone. The decision policy on in-distribution inputs — the mapping from “tickets like these” to “labels like those”. Narrow skills the corpus exercises densely. These are exactly the things SFT is good at instilling, and the synthetic corpus demonstrates them thousands of times.
What transfers poorly — commonly observed, and worth hedging because results vary by task and scale: the teacher’s broad world knowledge, which a model a fraction of the size simply lacks the capacity to hold; robustness to inputs unlike the corpus, where the teacher’s composure comes from breadth the student never saw; and graceful degradation at the edges — students tend to fail harder and more confidently than teachers when pushed off-distribution. If the capability you need is knowledge breadth, distillation is the wrong lever; that is retrieval’s job or a bigger model’s.
Bar chart comparing a teacher and a distilled student on three illustrative axes. On the target task the bars are close: teacher 90 percent, student 86 percent. On out-of-distribution inputs they diverge: teacher 78 percent, student 55 percent. On broad general knowledge they diverge further: teacher 84 percent, student 52 percent.
This is why the eval discipline around distillation is non-negotiable, and why it centers on one artifact: a fixed eval set drawn from your real task distribution, frozen before training starts.
Run the teacher on it first. That score is two things at once: your ceiling (the student will not beat what it imitates) and your benchmark (every future comparison is against this number, not against vibes). Then score the student — overall and per slice: per category, per locale, per input length, per format. Aggregate parity can hide a slice that collapsed. The teacher–student gap, slice by slice, is the true price tag of the smaller serving bill; decide deliberately whether each gap is worth it.
And keep the teacher. Not in the serving path — offline, as the standing benchmark: re-run it on fresh traffic samples so you notice when the world drifts away from the corpus, and re-compare on every student retrain. Some teams also keep the teacher reachable as an escalation target for inputs the student flags as hard; routing between models at serve time is orchestration — agent territory, one line and moving on.
Sequence distillation is SFT wearing a costume
Strip the vocabulary away and the LLM-era recipe contains no new training mathematics at all. The student is trained with the standard Cross-entropy Loss on the teacher’s generated tokens as hard targets — at each position, one “correct” next Token, probability one, everything else zero. That is exactly supervised fine-tuning; the only novelty is who authored the corpus. Every property of SFT follows: imitation of surface behavior, sensitivity to data quality, no notion of “this example might be wrong”.
Classic soft-label distillation differs at precisely this point. Instead of one target token per position, the student matches the teacher’s entire next-token distribution — typically by minimizing the divergence between the two distributions, with the teacher’s softmax softened by a temperature above 1 so that near-miss alternatives carry visible probability. Every position then teaches a full ranking over the vocabulary rather than a single answer: a strictly richer signal from the same text.
The catch is access. Soft labels require the teacher’s per-token probabilities and a student that shares (or is aligned to) the same tokenizer vocabulary — otherwise the two distributions are not even over the same set. Teams that own both models (an open-weights teacher distilled into a sibling architecture) can and do use it. Against an API-only teacher that returns text and at most a few top log-probabilities, you cannot — which is how “distillation” in industry practice came to mean the sequence-level, synthetic-corpus form by default.
In production
Distillation is the classic cost-optimization endgame of a served LLM feature: pay once to move a proven behavior into a smaller model, then collect the savings on every request for the lifetime of the feature. All three hyperscalers package the same mechanism — teacher generates, service fine-tunes a smaller student you deploy like any other custom model.
AWS
Amazon Bedrock offers a managed distillation-style flow at exactly the mechanism this module describes: you select a teacher and a smaller student model, supply prompts (or point at logged production invocations), and the service generates the teacher responses and fine-tunes the student on them, yielding a custom model you deploy and meter like other customized models. The pipeline stages — coverage, generation, training — are the same; what is managed is the plumbing between them.
Azure
Azure AI Foundry composes the same pipeline from its primitives: stored completions capture a deployed teacher’s real traffic as prompt–response pairs, those records become the training file for fine-tuning a smaller model, and the platform’s evaluation tooling compares student against teacher before you swap deployments. The stored-completions step is the “real logged prompts” corpus source from lesson two, productized.
Google Cloud
On Vertex AI the same shape is built from batch prediction plus tuning: run the teacher over your prompt set as an offline batch job (latency-free, exactly how teacher generation wants to run), curate the outputs into a supervised tuning dataset, and tune a smaller model that deploys to the same endpoints as any tuned model. The division of labor mirrors the module: batch generation for coverage, your filters for quality, managed tuning for the mechanical step.
Key terms: Distillation, Synthetic data, Supervised fine-tuning (SFT), Fine-tuning, Benchmark
Interactive checkpoint quiz (1 questions) — open this page in a browser to take it.