What are tokens in AI and LLMs?
Also called token, tokenization, LLM tokens
Definition
Tokens are the units of text a language model processes, typically whole words, word fragments, punctuation, or bytes, each mapped to an integer ID from the model's fixed vocabulary.
Tokens are the units of text that a large language model reads and generates. A tokenizer splits input text into tokens, such as whole words, word fragments, punctuation, or raw bytes, and maps each one to an integer ID from a fixed vocabulary. The model never sees characters directly; it reads and predicts sequences of token IDs.
How tokenization works#
Tokenization works by applying a learned vocabulary of subword units to text. Most LLM tokenizers use byte pair encoding (BPE) or a close variant: training starts from single characters or bytes and repeatedly merges the most frequent adjacent pair into a new token until the vocabulary reaches a target size. Sennrich et al. (2015) applied BPE to neural translation so models could handle rare words by splitting them into known pieces.
The result is that common words become a single token, while rare words, names, and identifiers split into several. Inside the model, each token ID is looked up in a table to get an embedding vector. Tokenizers differ between model families, so the same text produces different token counts on different models.
Tokens vs words#
Tokens and words are not one-to-one. OpenAI's guidance for English text is roughly 4 characters per token, or about 75 words per 100 tokens. The ratio shifts with language and content. Petrov et al. (2023) found that the same text translated into different languages can tokenize to lengths that differ by up to 15 times, which means users of some languages pay more and fit less into the same budget.
Why tokens matter#
Tokens matter because they are the unit behind most limits and costs in an LLM system:
- Capacity: the context window is measured in tokens, and it includes the generated output.
- Billing: providers price API usage per input token and per output token.
- Latency: output is generated one token at a time, so long responses take longer, and time to first token measures the wait before the first one arrives.
Counting tokens with the target model's own tokenizer, rather than estimating from word counts, prevents truncated prompts and unexpected costs. Context engineering covers how to budget tokens across instructions, retrieved data, and conversation history.
Related terms
Guides that cover tokens
- What is context engineering? A practical guide for AI agents
- LLM inference optimization: a practical guide for engineers
- How to build an AI agent: a step-by-step guide in Python
- LLM observability: what to capture, which metrics, and tools
- What is an AI gateway? How it works and when you need one
- Reasoning models: how they work and when to use them
- Chunking strategies for RAG: how to choose, tune, and evaluate them
- AI engineer roadmap: skills, projects, and resources by stage
Cite this definition
The AI Engineering Editors. "What are tokens in AI and LLMs?" The AI Engineering, updated Nov 06, 2026. https://theaiengineering.com/glossary/tokens
- Title
- What are tokens in AI and LLMs?
- Author
- The AI Engineering Editors
- Publisher
- The AI Engineering
- Updated