How It’s Done Today

Lesson 2 of 3 in Distillation.

The pipeline has three moving parts — prompt coverage, filtering, training — and the first two do most of the work. Training is a solved, mechanical step; coverage and quality control are where distillations succeed or quietly fail.

Coverage first. The student learns only what the corpus shows it, so the prompt set must span the task the way production traffic does: routine cases, rare cases, hostile phrasing, every locale and format you serve. Real logged prompts are the gold standard because they are the distribution. When you have too few — a new product, a new slice — the standard move is teacher-powered expansion: hand the teacher a handful of seed examples and have it write variations, the pattern popularized by Self-Instruct (Wang et al., 2022), where a model bootstraps an instruction-tuning corpus from a small seed set.

Prompt-set expansion (Self-Instruct pattern)

You write inputs for the following task: {{TASK_DESCRIPTION}}.

Here are {{N}} real examples of inputs users send:

{{SEED_EXAMPLES}}

Write {{K}} new inputs for the same task. Vary:
- length: one-liners through multi-paragraph
- tone: terse, frustrated, formal, rambling
- difficulty: routine cases and rare edge cases
- surface details: names, products, quantities, dates

Do not copy or lightly rephrase any seed example. Output one input per line, with no numbering and no commentary.

This expands the prompt side only. The teacher then answers each generated input with your real production prompt, and those pairs — after filtering — become the student’s training corpus. Deduplicate aggressively against the seeds and against earlier batches: expansion runs drift toward each other.

Filtering is the quality gate. Teacher outputs contain mistakes, hallucinations, and format violations — and the student cannot tell an error from an insight. Whatever survives filtering becomes ground truth. Three filter families stack, cheapest first:

Heuristics — deduplication, length bounds, format and schema validity, language checks, banned-content patterns. Cheap, brutal, and they catch a surprising share of junk.
Verifiers — anything that can check an answer mechanically: run generated code against tests, compare extracted fields to source documents, validate JSON against a schema, re-compute arithmetic. Where a verifier exists, use it; verified data is the best data.
Judge models — a second model grades each pair against a rubric, catching what rules cannot: subtle wrongness, unhelpful tone, instructions ignored. How to build judges you can trust — rubrics, calibration, agreement with humans — is the Evaluation domain’s subject; here, treat the judge as one more filter with its own error rate.

Where the corpus comes from determines what quality control fits and which failure you are signing up for.
Data source	Quality control that fits	Main risk
Real logged prompts + teacher answers	Deduplication, PII scrubbing, judge scoring of answers	Privacy and consent obligations on user data; traffic skews toward common cases, so rare-but-important slices are underrepresented
Seed prompts expanded by the teacher (Self-Instruct style)	Diversity checks, dedup against seeds and prior batches, judge scoring	Mode collapse — variations cluster around the teacher’s habits, and the corpus drifts away from real traffic
Existing dataset relabeled by the teacher	Agreement checks against existing labels, human spot audits of disagreements	Systematic teacher labeling errors propagate silently at corpus scale
Teacher rationales / step-by-step traces	Verify the final answer; keep only traces that end in a verified answer	Fluent-but-wrong reasoning reads as high quality and teaches the student confident nonsense

Training the student is the easy part — by design. The filtered corpus is prompt–response pairs in ordinary SFT format, so everything from the fine-tuning module applies verbatim: full fine-tune or LoRA/QLoRA, the same data hygiene, the same held-out split. Pick the student empirically: start from the smallest Instruction-tuned model plausible for the task, train one size up as a comparison, and let your eval set arbitrate. If even the larger student misses the bar, the fix is almost never “train longer” — it is back at the top of the pipeline: widen coverage, filter harder, regenerate.

That loop is the real shape of the work. Teams iterate on generation and filtering many times for every time they touch training hyperparameters — the corpus is the product, and the student is just its compression.

Interactive checkpoint quiz (1 questions) — open this page in a browser to take it.