---
title: "What are tokens in AI and LLMs?"
description: "Tokens are the units of text an LLM reads and writes, usually word pieces. Learn how tokenization works, how tokens compare to words, and why tokens matter."
type: "glossary-term"
canonical_url: "https://theaiengineering.com/glossary/tokens"
markdown_url: "https://theaiengineering.com/glossary/tokens.md"
topic: "AI engineering"
author: "The AI Engineering Editors"
publisher: "The AI Engineering"
updated: "2026-11-06"
keywords: ["Tokens","token","tokenization","LLM tokens"]
citation: "The AI Engineering Editors. \"What are tokens in AI and LLMs?\" The AI Engineering, updated Nov 06, 2026. https://theaiengineering.com/glossary/tokens"
---

# What are tokens in AI and LLMs?

Also called token, tokenization, LLM tokens.

> **Definition of Tokens:** Tokens are the units of text a language model processes, typically whole words, word fragments, punctuation, or bytes, each mapped to an integer ID from the model's fixed vocabulary.

Tokens are the units of text that a [large language model](https://theaiengineering.com/glossary/large-language-model) reads and generates. A tokenizer splits input text into tokens, such as whole words, word fragments, punctuation, or raw bytes, and maps each one to an integer ID from a fixed vocabulary. The model never sees characters directly; it reads and predicts sequences of token IDs.

## How tokenization works

Tokenization works by applying a learned vocabulary of subword units to text. Most LLM tokenizers use byte pair encoding (BPE) or a close variant: training starts from single characters or bytes and repeatedly merges the most frequent adjacent pair into a new token until the vocabulary reaches a target size. [Sennrich et al. (2015)](https://arxiv.org/abs/1508.07909) applied BPE to neural translation so models could handle rare words by splitting them into known pieces.

The result is that common words become a single token, while rare words, names, and identifiers split into several. Inside the model, each token ID is looked up in a table to get an [embedding](https://theaiengineering.com/glossary/embeddings) vector. Tokenizers differ between model families, so the same text produces different token counts on different models.

## Tokens vs words

Tokens and words are not one-to-one. [OpenAI's guidance](https://help.openai.com/en/articles/4936856-what-are-tokens-and-how-to-count-them) for English text is roughly 4 characters per token, or about 75 words per 100 tokens. The ratio shifts with language and content. [Petrov et al. (2023)](https://arxiv.org/abs/2305.15425) found that the same text translated into different languages can tokenize to lengths that differ by up to 15 times, which means users of some languages pay more and fit less into the same budget.

## Why tokens matter

Tokens matter because they are the unit behind most limits and costs in an LLM system:

- **Capacity**: the [context window](https://theaiengineering.com/glossary/context-window) is measured in tokens, and it includes the generated output.
- **Billing**: providers price API usage per input token and per output token.
- **Latency**: output is generated one token at a time, so long responses take longer, and [time to first token](https://theaiengineering.com/glossary/time-to-first-token) measures the wait before the first one arrives.

Counting tokens with the target model's own tokenizer, rather than estimating from word counts, prevents truncated prompts and unexpected costs. [Context engineering](https://theaiengineering.com/articles/context-engineering) covers how to budget tokens across instructions, retrieved data, and conversation history.

## Related terms

- [Context window](https://theaiengineering.com/glossary/context-window): A context window is the maximum number of tokens a language model can take into account in a single request, including the system prompt, conversation history, tool results, and the generated output.
- [Embeddings](https://theaiengineering.com/glossary/embeddings): Embeddings are fixed-length vectors of numbers that represent data such as text or images so that items with similar meaning sit close together, which lets software compare meaning with arithmetic.
- [Large language model (LLM)](https://theaiengineering.com/glossary/large-language-model): A large language model (LLM) is a neural network, usually a transformer with billions of parameters, trained on web-scale text to predict the next token and then adapted to follow instructions.
- [Time to first token](https://theaiengineering.com/glossary/time-to-first-token): Time to first token (TTFT) is the elapsed time from submitting a request to an LLM until the first token of the response arrives, covering request queuing, network transit, and prompt prefill.
- [Prompt caching](https://theaiengineering.com/glossary/prompt-caching): Prompt caching is an LLM API and inference-server feature that stores the computed attention state for a repeated prompt prefix, so later requests sharing that prefix skip reprocessing it.

## Guides that cover tokens

- [What is context engineering? A practical guide for AI agents](https://theaiengineering.com/articles/context-engineering): Context engineering is the practice of choosing what fills an LLM context window. Learn its components, why context rot happens, and how to measure results.
- [LLM inference optimization: a practical guide for engineers](https://theaiengineering.com/articles/llm-inference-optimization): LLM inference optimization guide: prefill vs decode, TTFT, KV cache and PagedAttention, continuous batching, quantization, speculative decoding, and GPUs.
- [How to build an AI agent: a step-by-step guide in Python](https://theaiengineering.com/articles/how-to-build-an-ai-agent): Learn how to build an AI agent from scratch in Python: when you need one, tool schemas, the agent loop, stop conditions, guardrails, evals, and tracing.
- [LLM observability: what to capture, which metrics, and tools](https://theaiengineering.com/articles/llm-observability): LLM observability guide: what to capture in traces and spans, metrics to track, OpenTelemetry GenAI conventions, tool categories, and keeping prompts private.
- [What is an AI gateway? How it works and when you need one](https://theaiengineering.com/articles/what-is-an-ai-gateway): An AI gateway is a proxy between your apps and model providers. Learn how it works, its core features, AI gateway vs API gateway, and open source options.
- [Reasoning models: how they work and when to use them](https://theaiengineering.com/articles/reasoning-models): Reasoning models explained: how test-time compute and reinforcement learning produce chain-of-thought reasoning, when to use them, and their cost and latency.
- [Chunking strategies for RAG: how to choose, tune, and evaluate them](https://theaiengineering.com/articles/chunking-strategies-for-rag): Chunking strategies for RAG compared: fixed-size, recursive, document-aware, semantic, parent-child, and late chunking, with tested Python and an eval method.
- [AI engineer roadmap: skills, projects, and resources by stage](https://theaiengineering.com/articles/ai-engineer-roadmap): A practical AI engineer roadmap in seven stages, each with the skills to learn, a project to ship, and resources worth the time, from Python to production.

Part of the [AI engineering](https://theaiengineering.com/topics/ai-engineering) topic hub.

## Cite this definition

The AI Engineering Editors. "What are tokens in AI and LLMs?" The AI Engineering, updated Nov 06, 2026. https://theaiengineering.com/glossary/tokens

- Title: What are tokens in AI and LLMs?
- Author: The AI Engineering Editors
- Publisher: The AI Engineering
- Updated: 2026-11-06
- URL: https://theaiengineering.com/glossary/tokens
- Markdown: https://theaiengineering.com/glossary/tokens.md
