Continued pre-training and distillation
Lesson 3 of 4 in Customizing Foundation Models.
Sometimes the base model's problem isn't behavior — it genuinely doesn't speak your domain. Clinical shorthand, actuarial tables, proprietary telemetry formats: if the pre-training corpus never contained your world, no amount of pair-tuning teaches it. Continued pre-training resumes the self-supervised objective on a large unlabeled domain corpus — raw documents, no prompt-response pairs — reshaping what the model knows rather than how it responds. Bedrock offers it for select models; like fine-tuning, the output is a private custom model.
| Fine-tuning | Continued pre-training | |
|---|---|---|
Training data | Labeled prompt-response pairs | Raw unlabeled domain text |
Changes | Behavior: format, tone, task skill | Knowledge: domain vocabulary and concepts |
Scale | Hundreds to thousands of pairs | Much larger corpora — and compute to match |
Reach for it when | The model understands the domain but responds wrongly | The model fundamentally misreads the domain's language |
The ladder also runs downward: distillation transfers a big teacher model's skill on a specific task into a smaller, cheaper, faster student. Amazon Bedrock Model Distillation automates the loop — it uses your prompts to generate teacher responses and trains the student on them. The deal: near-teacher quality within the distilled scope, lower latency and cost per request, and reduced general capability outside that scope. Exam cue: "match the large model's quality on this one high-volume task while cutting inference cost and latency."
Key terms: Continued pre-training, Model distillation, Fine-tuning, Catastrophic forgetting
Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.