What is a small language model (SLM)? Examples and uses
Also called SLM, small LLM, compact language model
Definition
A small language model (SLM) is a language model compact enough to run on a consumer device such as a laptop or phone, which in 2025 generally meant fewer than about 10 billion parameters.
A small language model is built on the same architecture as a large language model, at a size that fits modest hardware. There is no official cutoff. NVIDIA researchers, in Small Language Models are the Future of Agentic AI (2025), define an SLM as a model that fits on a common consumer electronic device and runs fast enough to serve one user's agentic requests. The authors add that, as of 2025, most models under 10 billion parameters qualify, and they avoid a fixed parameter count because hardware keeps moving the line.
Microsoft's Phi-3-mini is a typical example: its technical report describes a 3.8-billion-parameter model trained on 3.3 trillion tokens that is small enough to deploy on a phone. Meta's Llama 3.2 family includes 1B and 3B text models aimed at on-device use.
Small language model vs LLM#
| Small language model | Large language model | |
|---|---|---|
| Typical size | Under about 10B parameters | Tens to hundreds of billions of parameters |
| Where it runs | Laptop, phone, or single GPU | GPU clusters or provider APIs |
| Latency and cost | Low | Higher |
| Breadth | Narrower knowledge and weaker multi-step reasoning | Broad knowledge and stronger reasoning |
| Best fit | Narrow, repetitive, well-defined tasks | Open-ended and complex tasks |
Small models get much of their capability from training technique, not size. Knowledge distillation transfers behavior from a larger teacher: Meta built Llama 3.2 1B and 3B with pruning and distillation from Llama 3.1 models. The Phi-3 report credits heavily filtered web data and synthetic data for Phi-3-mini's quality. Quantization then shrinks a small model further for local runtimes such as those compared in vLLM vs Ollama.
Why small language models matter#
SLMs make on-device and private inference practical: data can stay on the user's machine, and responses do not depend on a network round trip. SLMs are also cheaper to fine-tune for a single narrow task. The NVIDIA paper argues that agentic systems mostly run a small number of specialized, repetitive tasks that SLMs can already handle, and it recommends heterogeneous systems that call larger models only where general conversation or broad reasoning is needed. LLM routing is the mechanism that sends each request to the right model in that kind of setup.
Related terms
Guides that cover small language model
Cite this definition
The AI Engineering Editors. "What is a small language model (SLM)? Examples and uses." The AI Engineering, updated Nov 06, 2026. https://theaiengineering.com/glossary/small-language-model
- Title
- What is a small language model (SLM)? Examples and uses
- Author
- The AI Engineering Editors
- Publisher
- The AI Engineering
- Updated