---
title: "What is knowledge distillation in AI?"
description: "Knowledge distillation trains a small student model to imitate a larger teacher model's outputs. How it works, LLM examples, and distillation vs quantization."
type: "glossary-term"
canonical_url: "https://theaiengineering.com/glossary/knowledge-distillation"
markdown_url: "https://theaiengineering.com/glossary/knowledge-distillation.md"
topic: "LLM models and benchmarks"
author: "The AI Engineering Editors"
publisher: "The AI Engineering"
updated: "2026-11-06"
keywords: ["Knowledge distillation","model distillation","distillation","teacher-student training"]
citation: "The AI Engineering Editors. \"What is knowledge distillation in AI?\" The AI Engineering, updated Nov 06, 2026. https://theaiengineering.com/glossary/knowledge-distillation"
---

# What is knowledge distillation in AI?

Also called model distillation, distillation, teacher-student training.

> **Definition of Knowledge distillation:** Knowledge distillation is a model compression technique in which a smaller student model is trained to reproduce the outputs or output probabilities of a larger, more capable teacher model.

Knowledge distillation is a training technique that transfers what a large model has learned into a smaller model that is cheaper and faster to run. The student model learns from the teacher's outputs instead of, or in addition to, the original labeled data. Geoffrey Hinton, Oriol Vinyals, and Jeff Dean formalized the modern approach in [Distilling the Knowledge in a Neural Network](https://arxiv.org/abs/1503.02531) (2015), building on earlier model compression work by Caruana and colleagues.

## How knowledge distillation works

In the Hinton et al. formulation, the student trains on the teacher's soft targets: the full probability distribution over possible outputs, not only the top answer. A plausible wrong answer that receives some probability tells the student something a one-hot label cannot, namely which mistakes are reasonable. Both models apply a raised [temperature](https://theaiengineering.com/glossary/temperature) to the softmax during training, which spreads probability across more classes, and the student returns to temperature 1 at deployment. The training loss combines a term for matching the soft targets with a lower-weighted term for the true labels.

LLM distillation takes two main forms:

- **Logit distillation:** the student matches the teacher's next-token probabilities. Meta built the Llama 3.2 1B and 3B models by pruning Llama 3.1 8B and then using output probabilities from Llama 3.1 8B and 70B as token-level targets during [pretraining](https://theaiengineering.com/glossary/pretraining) ([Meta, 2024](https://ai.meta.com/blog/llama-3-2-connect-2024-vision-edge-mobile-devices/)).
- **Sequence-level distillation:** the teacher generates responses, and the student is [fine-tuned](https://theaiengineering.com/glossary/fine-tuning) on them with ordinary supervised training. DeepSeek released six DeepSeek-R1-Distill models, from 1.5B to 70B parameters, by fine-tuning Qwen2.5 and Llama models on samples generated by DeepSeek-R1, a [reasoning model](https://theaiengineering.com/glossary/reasoning-model).

An earlier landmark is DistilBERT ([Sanh et al., 2019](https://arxiv.org/abs/1910.01108)), which is 40% smaller and 60% faster than BERT while retaining 97% of its language understanding capabilities.

## Knowledge distillation vs quantization

Both techniques cut inference cost, in different ways. Distillation trains a new, smaller model; [quantization](https://theaiengineering.com/glossary/quantization) keeps the same model and stores its weights at lower precision. Distillation requires a training run and teacher outputs, while post-training quantization needs at most a small calibration dataset. The two combine well: a distilled [small language model](https://theaiengineering.com/glossary/small-language-model) is often quantized again for local deployment. For other ways to cut serving cost, see the [LLM inference optimization guide](https://theaiengineering.com/articles/llm-inference-optimization).

## Related terms

- [Small language model](https://theaiengineering.com/glossary/small-language-model): A small language model (SLM) is a language model compact enough to run on a consumer device such as a laptop or phone, which in 2025 generally meant fewer than about 10 billion parameters.
- [Quantization](https://theaiengineering.com/glossary/quantization): Quantization is a model compression technique that represents a neural network's weights, and sometimes its activations or KV cache, with fewer bits, such as 8-bit or 4-bit integers instead of 16-bit floats.
- [Fine-tuning](https://theaiengineering.com/glossary/fine-tuning): Fine-tuning is the process of further training a pretrained model on a smaller, task-specific dataset so its weights change to improve performance on a particular task, domain, style, or output format.
- [Reasoning model](https://theaiengineering.com/glossary/reasoning-model): A reasoning model is a large language model trained, usually with reinforcement learning, to generate intermediate reasoning tokens before its final answer, spending more inference compute to solve multi-step problems.
- [Temperature](https://theaiengineering.com/glossary/temperature): Temperature is a sampling parameter that rescales a language model's next-token probabilities: low values concentrate probability on the likeliest tokens, and high values flatten the distribution for more varied output.

## Guides that cover knowledge distillation

- [Reasoning models: how they work and when to use them](https://theaiengineering.com/articles/reasoning-models): Reasoning models explained: how test-time compute and reinforcement learning produce chain-of-thought reasoning, when to use them, and their cost and latency.
- [LLM inference optimization: a practical guide for engineers](https://theaiengineering.com/articles/llm-inference-optimization): LLM inference optimization guide: prefill vs decode, TTFT, KV cache and PagedAttention, continuous batching, quantization, speculative decoding, and GPUs.

Part of the [LLM models and benchmarks](https://theaiengineering.com/topics/models) topic hub.

## Cite this definition

The AI Engineering Editors. "What is knowledge distillation in AI?" The AI Engineering, updated Nov 06, 2026. https://theaiengineering.com/glossary/knowledge-distillation

- Title: What is knowledge distillation in AI?
- Author: The AI Engineering Editors
- Publisher: The AI Engineering
- Updated: 2026-11-06
- URL: https://theaiengineering.com/glossary/knowledge-distillation
- Markdown: https://theaiengineering.com/glossary/knowledge-distillation.md
