Teacher to Student

Lesson 1 of 3 in Distillation.

Distillation starts with an observation from Hinton, Vinyals and Dean (2015): a trained model knows more than its top answer. When an image classifier says “dog”, the relative probabilities it assigns to the wrong answers — cat far more likely than car, car far more likely than carrot — encode how the teacher carves up the world. Their recipe trained a small student to match the big teacher’s full output distribution (softened with a Temperature-style scaling so the small probabilities become visible), instead of training on hard labels alone. The soft targets carry more signal per example, and the small model learns faster and better from them than from the raw data.

The LLM era kept the name and changed the mechanics. A hosted frontier model hands you text, not probability distributions — so the field’s working definition shifted to sequence-level distillation: the teacher generates the training data. You prompt the strongest model you can access across your task, keep its answers, filter out the junk, and fine-tune a small model on what survives. The knowledge moves as a corpus of synthetic data, and the student never sees the teacher — only its homework.

That reframing matters because it makes distillation composable with everything you already know: the training step is plain Supervised fine-tuning (SFT), the corpus is just data you happen not to have written by hand, and every habit from the fine-tuning module — data formats, LoRA, eval sets — applies unchanged.

The sequence-distillation pipeline

  1. Prompts spanning your task

    Real logged traffic where you have it; teacher-expanded seed prompts where you do not. Coverage here bounds everything the student can learn.

  2. Teacher generates answers

    The biggest, best model you can access — run offline in batch, where latency is irrelevant and only quality counts.

  3. Filter for quality

    Heuristics, verifiers, judge models. Whatever survives becomes ground truth to the student — including any surviving mistakes.

  4. Synthetic training corpus

    Prompt–response pairs in ordinary SFT format. From here on, this is just fine-tuning data.

  5. Fine-tune the student (SFT)

    A much smaller model — full fine-tune or LoRA/QLoRA. Training the student is usually the cheapest stage of the pipeline.

  6. Student close enough to teacher on your eval set?

    Fix the eval set before training. The teacher’s score on it is the ceiling; the gap to the student is the price of the smaller bill.

  7. Deploy the student; keep the teacher as benchmark

    The teacher stays in the loop offline — as the regression benchmark for every retrain and the generator for every corpus extension.

When is this the right lever? Follow the money. Prompting a frontier teacher gives you quality immediately, but you pay frontier prices on every request, forever. Distillation inverts the shape: generation and training are a one-time cost, and the smaller student’s cheaper, faster serving is a recurring saving on every request for the lifetime of the feature. The higher your volume and the tighter your latency budget, the faster the payback.

If that shape sounds familiar, it should: pretraining teams routinely overtrain small models far past the compute-optimal point, because extra training cost is paid once while inference cost is paid on every call (the scaling-laws module works through this). Distillation is the same bet made at adaptation time — spend more up front so each serve costs less — except the “extra training data” is manufactured by a bigger model instead of scraped from the web.

The bet has preconditions. The task should be stable and well-defined — a moving target invalidates the corpus. Volume should be high enough that per-request savings repay the up-front spend. And the teacher must demonstrably solve the task already, because the student will not exceed what the teacher shows it.

Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.