What is a small language model (SLM)? Examples and uses

Also called SLM, small LLM, compact language model

Definition

A small language model (SLM) is a language model compact enough to run on a consumer device such as a laptop or phone, which in 2025 generally meant fewer than about 10 billion parameters.

A small language model is built on the same architecture as a large language model, at a size that fits modest hardware. There is no official cutoff. NVIDIA researchers, in Small Language Models are the Future of Agentic AI (2025), define an SLM as a model that fits on a common consumer electronic device and runs fast enough to serve one user's agentic requests. The authors add that, as of 2025, most models under 10 billion parameters qualify, and they avoid a fixed parameter count because hardware keeps moving the line.

Microsoft's Phi-3-mini is a typical example: its technical report describes a 3.8-billion-parameter model trained on 3.3 trillion tokens that is small enough to deploy on a phone. Meta's Llama 3.2 family includes 1B and 3B text models aimed at on-device use.

Small language model vs LLM#

Small language model Large language model
Typical size Under about 10B parameters Tens to hundreds of billions of parameters
Where it runs Laptop, phone, or single GPU GPU clusters or provider APIs
Latency and cost Low Higher
Breadth Narrower knowledge and weaker multi-step reasoning Broad knowledge and stronger reasoning
Best fit Narrow, repetitive, well-defined tasks Open-ended and complex tasks

Small models get much of their capability from training technique, not size. Knowledge distillation transfers behavior from a larger teacher: Meta built Llama 3.2 1B and 3B with pruning and distillation from Llama 3.1 models. The Phi-3 report credits heavily filtered web data and synthetic data for Phi-3-mini's quality. Quantization then shrinks a small model further for local runtimes such as those compared in vLLM vs Ollama.

Why small language models matter#

SLMs make on-device and private inference practical: data can stay on the user's machine, and responses do not depend on a network round trip. SLMs are also cheaper to fine-tune for a single narrow task. The NVIDIA paper argues that agentic systems mostly run a small number of specialized, repetitive tasks that SLMs can already handle, and it recommends heterogeneous systems that call larger models only where general conversation or broad reasoning is needed. LLM routing is the mechanism that sends each request to the right model in that kind of setup.

Cite this definition

The AI Engineering Editors. "What is a small language model (SLM)? Examples and uses." The AI Engineering, updated Nov 06, 2026. https://theaiengineering.com/glossary/small-language-model

Title
What is a small language model (SLM)? Examples and uses
Author
The AI Engineering Editors
Publisher
The AI Engineering
Updated
View as Markdown

Part of the LLM models and benchmarks hub · Updated