---
title: "What is a small language model (SLM)? Examples and uses"
description: "A small language model (SLM) is a compact language model, generally under about 10B parameters, that runs on a consumer device. Examples, uses, and SLM vs LLM."
type: "glossary-term"
canonical_url: "https://theaiengineering.com/glossary/small-language-model"
markdown_url: "https://theaiengineering.com/glossary/small-language-model.md"
topic: "LLM models and benchmarks"
author: "The AI Engineering Editors"
publisher: "The AI Engineering"
updated: "2026-11-06"
keywords: ["Small language model","SLM","small LLM","compact language model"]
citation: "The AI Engineering Editors. \"What is a small language model (SLM)? Examples and uses.\" The AI Engineering, updated Nov 06, 2026. https://theaiengineering.com/glossary/small-language-model"
---

# What is a small language model (SLM)? Examples and uses

Also called SLM, small LLM, compact language model.

> **Definition of Small language model:** A small language model (SLM) is a language model compact enough to run on a consumer device such as a laptop or phone, which in 2025 generally meant fewer than about 10 billion parameters.

A small language model is built on the same architecture as a [large language model](https://theaiengineering.com/glossary/large-language-model), at a size that fits modest hardware. There is no official cutoff. NVIDIA researchers, in [Small Language Models are the Future of Agentic AI](https://arxiv.org/abs/2506.02153) (2025), define an SLM as a model that fits on a common consumer electronic device and runs fast enough to serve one user's agentic requests. The authors add that, as of 2025, most models under 10 billion parameters qualify, and they avoid a fixed parameter count because hardware keeps moving the line.

Microsoft's Phi-3-mini is a typical example: its [technical report](https://arxiv.org/abs/2404.14219) describes a 3.8-billion-parameter model trained on 3.3 trillion tokens that is small enough to deploy on a phone. Meta's Llama 3.2 family includes 1B and 3B text models aimed at on-device use.

## Small language model vs LLM

| | Small language model | Large language model |
|---|---|---|
| Typical size | Under about 10B parameters | Tens to hundreds of billions of parameters |
| Where it runs | Laptop, phone, or single GPU | GPU clusters or provider APIs |
| Latency and cost | Low | Higher |
| Breadth | Narrower knowledge and weaker multi-step reasoning | Broad knowledge and stronger reasoning |
| Best fit | Narrow, repetitive, well-defined tasks | Open-ended and complex tasks |

Small models get much of their capability from training technique, not size. [Knowledge distillation](https://theaiengineering.com/glossary/knowledge-distillation) transfers behavior from a larger teacher: Meta built Llama 3.2 1B and 3B with pruning and distillation from Llama 3.1 models. The Phi-3 report credits heavily filtered web data and synthetic data for Phi-3-mini's quality. [Quantization](https://theaiengineering.com/glossary/quantization) then shrinks a small model further for local runtimes such as those compared in [vLLM vs Ollama](https://theaiengineering.com/articles/vllm-vs-ollama).

## Why small language models matter

SLMs make on-device and private inference practical: data can stay on the user's machine, and responses do not depend on a network round trip. SLMs are also cheaper to [fine-tune](https://theaiengineering.com/glossary/fine-tuning) for a single narrow task. The NVIDIA paper argues that agentic systems mostly run a small number of specialized, repetitive tasks that SLMs can already handle, and it recommends heterogeneous systems that call larger models only where general conversation or broad reasoning is needed. [LLM routing](https://theaiengineering.com/glossary/llm-routing) is the mechanism that sends each request to the right model in that kind of setup.

## Related terms

- [Large language model (LLM)](https://theaiengineering.com/glossary/large-language-model): A large language model (LLM) is a neural network, usually a transformer with billions of parameters, trained on web-scale text to predict the next token and then adapted to follow instructions.
- [Knowledge distillation](https://theaiengineering.com/glossary/knowledge-distillation): Knowledge distillation is a model compression technique in which a smaller student model is trained to reproduce the outputs or output probabilities of a larger, more capable teacher model.
- [Quantization](https://theaiengineering.com/glossary/quantization): Quantization is a model compression technique that represents a neural network's weights, and sometimes its activations or KV cache, with fewer bits, such as 8-bit or 4-bit integers instead of 16-bit floats.
- [Fine-tuning](https://theaiengineering.com/glossary/fine-tuning): Fine-tuning is the process of further training a pretrained model on a smaller, task-specific dataset so its weights change to improve performance on a particular task, domain, style, or output format.
- [LLM routing](https://theaiengineering.com/glossary/llm-routing): LLM routing is the practice of directing each request to one of several language models or providers, based on rules or a learned classifier, to balance cost, quality, latency, and availability.
- [Open-weights model](https://theaiengineering.com/glossary/open-weights-model): An open-weights model is an AI model whose trained parameters are published for download, so anyone can run, fine-tune, or quantize it on their own hardware under the terms of its license.
- [Model cascade](https://theaiengineering.com/glossary/model-cascade): A model cascade is an LLM serving pattern that calls models in order from cheapest to most capable and returns the first answer that passes a quality check, escalating only when it fails.

## Guides that cover small language model

- [vLLM vs Ollama vs llama.cpp vs SGLang: which engine to use](https://theaiengineering.com/articles/vllm-vs-ollama): vLLM vs Ollama compared with llama.cpp, SGLang, LM Studio, and TensorRT-LLM: scope, hardware, APIs, concurrency, and which engine fits local or production use.
- [LLM inference optimization: a practical guide for engineers](https://theaiengineering.com/articles/llm-inference-optimization): LLM inference optimization guide: prefill vs decode, TTFT, KV cache and PagedAttention, continuous batching, quantization, speculative decoding, and GPUs.
- [LLM routing: strategies, fallbacks, and learned routers](https://theaiengineering.com/articles/llm-routing): LLM routing guide: rule-based and cost-aware strategies, fallbacks, load balancing, semantic and learned routers like RouteLLM, router evals, and pitfalls.

Part of the [LLM models and benchmarks](https://theaiengineering.com/topics/models) topic hub.

## Cite this definition

The AI Engineering Editors. "What is a small language model (SLM)? Examples and uses." The AI Engineering, updated Nov 06, 2026. https://theaiengineering.com/glossary/small-language-model

- Title: What is a small language model (SLM)? Examples and uses
- Author: The AI Engineering Editors
- Publisher: The AI Engineering
- Updated: 2026-11-06
- URL: https://theaiengineering.com/glossary/small-language-model
- Markdown: https://theaiengineering.com/glossary/small-language-model.md
