Distillation

Teaching a small model with a big one: synthetic data, filtering, and what actually transfers.

Content last verified 2026-09.

Lessons

1. Teacher to Student
2. How It’s Done Today
3. What Survives the Transfer

Sources

• Hinton, Vinyals & Dean (2015) — Distilling the Knowledge in a Neural Network
• Wang et al. (2022) — Self-Instruct: Aligning Language Models with Self-Generated Instructions
• Hu et al. (2021) — LoRA: Low-Rank Adaptation of Large Language Models