What is knowledge distillation in AI?
Also called model distillation, distillation, teacher-student training
Definition
Knowledge distillation is a model compression technique in which a smaller student model is trained to reproduce the outputs or output probabilities of a larger, more capable teacher model.
Knowledge distillation is a training technique that transfers what a large model has learned into a smaller model that is cheaper and faster to run. The student model learns from the teacher's outputs instead of, or in addition to, the original labeled data. Geoffrey Hinton, Oriol Vinyals, and Jeff Dean formalized the modern approach in Distilling the Knowledge in a Neural Network (2015), building on earlier model compression work by Caruana and colleagues.
How knowledge distillation works#
In the Hinton et al. formulation, the student trains on the teacher's soft targets: the full probability distribution over possible outputs, not only the top answer. A plausible wrong answer that receives some probability tells the student something a one-hot label cannot, namely which mistakes are reasonable. Both models apply a raised temperature to the softmax during training, which spreads probability across more classes, and the student returns to temperature 1 at deployment. The training loss combines a term for matching the soft targets with a lower-weighted term for the true labels.
LLM distillation takes two main forms:
- Logit distillation: the student matches the teacher's next-token probabilities. Meta built the Llama 3.2 1B and 3B models by pruning Llama 3.1 8B and then using output probabilities from Llama 3.1 8B and 70B as token-level targets during pretraining (Meta, 2024).
- Sequence-level distillation: the teacher generates responses, and the student is fine-tuned on them with ordinary supervised training. DeepSeek released six DeepSeek-R1-Distill models, from 1.5B to 70B parameters, by fine-tuning Qwen2.5 and Llama models on samples generated by DeepSeek-R1, a reasoning model.
An earlier landmark is DistilBERT (Sanh et al., 2019), which is 40% smaller and 60% faster than BERT while retaining 97% of its language understanding capabilities.
Knowledge distillation vs quantization#
Both techniques cut inference cost, in different ways. Distillation trains a new, smaller model; quantization keeps the same model and stores its weights at lower precision. Distillation requires a training run and teacher outputs, while post-training quantization needs at most a small calibration dataset. The two combine well: a distilled small language model is often quantized again for local deployment. For other ways to cut serving cost, see the LLM inference optimization guide.
Related terms
Guides that cover knowledge distillation
Cite this definition
The AI Engineering Editors. "What is knowledge distillation in AI?" The AI Engineering, updated Nov 06, 2026. https://theaiengineering.com/glossary/knowledge-distillation
- Title
- What is knowledge distillation in AI?
- Author
- The AI Engineering Editors
- Publisher
- The AI Engineering
- Updated