Instruction Data: Teaching by Demonstration
Lesson 1 of 4 in Supervised Fine-Tuning.
A Base model completes text. To turn it into an assistant, supervised fine-tuning shows it the behavior you want, directly: a demonstration pair — a prompt, and an ideal response written the way the finished assistant should answer. Then it trains the model, with the same Next-token prediction objective it already knows, to reproduce those responses token by token. No new mechanism, no reward signal, no judge. Just: here is the question, here is what a good answer looks like — imitate it.
That simplicity is why SFT is the first stage of essentially every post-training pipeline, and why this module matters even if you never train anything: every Instruction-tuned model you call inherited its answer shape, its tone, its formatting habits, and much of its Refusal style from somebody’s demonstration set. Instruction tuning is the broad name for this move — training on instruction-response pairs so the model treats instructions as things to follow rather than text to continue.
Where do tens of thousands of these pairs come from? Three lineages dominate, and every production dataset you will encounter is some blend of them.
Hired annotators. InstructGPT (Ouyang et al. 2022) is the canonical example: contracted labelers wrote demonstrations against a style guide, on prompts drawn largely from real API traffic. You get high quality and real-distribution coverage — and you pay for it, per example. Less obviously, the style guide itself becomes the product: whatever the guideline authors decided about hedging, refusals, and format is what the model learns as “good”.
Academic collections. FLAN (Wei et al. 2021) showed you can convert existing NLP datasets — translation, summarization, question answering — into instruction form with templates, cheaply producing enormous task variety. The risk is stylistic: academic datasets favor short, benchmark-shaped answers, and a model tuned only on them sounds like a benchmark, not an assistant.
Model-generated data. Self-Instruct (Wang et al. 2022) bootstraps from a small seed set: a model generates new instructions and responses, filters the weak ones, and the result trains the next round. It scales almost for free — and inherits every bias, blind spot, and error pattern of the generating model, at dataset scale.
| Source | How it works | Strengths | Characteristic risks |
|---|---|---|---|
Hired annotators (InstructGPT) | Contracted writers produce responses to a style guide, often on prompts sampled from real traffic | High quality; matches real user distribution; style is deliberate | Expensive per example; the style guide’s judgment calls — hedging, refusal tone — are baked in silently; annotator inconsistency becomes label noise |
Academic collections (FLAN) | Templates convert existing NLP datasets into instruction-response form | Huge scale at low cost; broad task coverage; well-documented provenance | Benchmark-shaped, terse answers; templated phrasing the model over-learns; coverage reflects what academia measured, not what users ask |
Model-generated (Self-Instruct) | A model generates instructions and responses from seed examples, then filters | Cheapest to scale; easy to steer toward a domain; fills coverage gaps on demand | Inherits the generator’s errors and biases at scale; quality filtering is imperfect; can drift toward what the generator finds easy to write |
Key terms: Supervised fine-tuning (SFT), Instruction tuning, Chat template, Preference data, Hallucination
Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.