What are tokens in AI and LLMs?

Also called token, tokenization, LLM tokens

Definition

Tokens are the units of text a language model processes, typically whole words, word fragments, punctuation, or bytes, each mapped to an integer ID from the model's fixed vocabulary.

Tokens are the units of text that a large language model reads and generates. A tokenizer splits input text into tokens, such as whole words, word fragments, punctuation, or raw bytes, and maps each one to an integer ID from a fixed vocabulary. The model never sees characters directly; it reads and predicts sequences of token IDs.

How tokenization works#

Tokenization works by applying a learned vocabulary of subword units to text. Most LLM tokenizers use byte pair encoding (BPE) or a close variant: training starts from single characters or bytes and repeatedly merges the most frequent adjacent pair into a new token until the vocabulary reaches a target size. Sennrich et al. (2015) applied BPE to neural translation so models could handle rare words by splitting them into known pieces.

The result is that common words become a single token, while rare words, names, and identifiers split into several. Inside the model, each token ID is looked up in a table to get an embedding vector. Tokenizers differ between model families, so the same text produces different token counts on different models.

Tokens vs words#

Tokens and words are not one-to-one. OpenAI's guidance for English text is roughly 4 characters per token, or about 75 words per 100 tokens. The ratio shifts with language and content. Petrov et al. (2023) found that the same text translated into different languages can tokenize to lengths that differ by up to 15 times, which means users of some languages pay more and fit less into the same budget.

Why tokens matter#

Tokens matter because they are the unit behind most limits and costs in an LLM system:

  • Capacity: the context window is measured in tokens, and it includes the generated output.
  • Billing: providers price API usage per input token and per output token.
  • Latency: output is generated one token at a time, so long responses take longer, and time to first token measures the wait before the first one arrives.

Counting tokens with the target model's own tokenizer, rather than estimating from word counts, prevents truncated prompts and unexpected costs. Context engineering covers how to budget tokens across instructions, retrieved data, and conversation history.

Cite this definition

The AI Engineering Editors. "What are tokens in AI and LLMs?" The AI Engineering, updated Nov 06, 2026. https://theaiengineering.com/glossary/tokens

Title
What are tokens in AI and LLMs?
Author
The AI Engineering Editors
Publisher
The AI Engineering
Updated
View as Markdown

Part of the AI engineering hub · Updated