What is chunking in RAG? Strategies explained

Also called document chunking, text chunking, chunking strategies

Definition

Chunking is the step in a retrieval pipeline that splits documents into smaller passages, called chunks, so each one can be embedded, indexed, and retrieved on its own.

Chunking is the step in a retrieval-augmented generation pipeline that splits source documents into smaller passages before indexing. Each chunk becomes one unit of retrieval: it gets its own embedding, its own entry in the index, and competes on its own for a place in the prompt. Chunk boundaries therefore determine which pieces of information can be retrieved together.

How chunking works#

Chunking works by applying a splitting rule to each document and, optionally, attaching metadata such as the document title, section heading, or page number to every chunk. The main strategies:

Strategy How it splits Trade-off
Fixed-size Every N tokens, usually with overlap Simple, but can cut sentences and ideas in half
Recursive On paragraphs, then sentences, then words, until under a size limit Keeps natural boundaries; a common default
Structure-aware On headings, sections, code blocks, or table rows Follows the document's layout; needs a parser per format
Semantic Where embedding similarity between adjacent sentences drops Groups related sentences; costs an extra embedding pass

Overlap, which repeats a few sentences between neighboring chunks, lowers the chance that an answer spans a boundary, at the cost of a larger index.

Why chunking matters#

Chunking matters because a retriever can only return what a chunk contains. Chunks that are too small lose context: a sentence such as "coverage begins after 90 days" is useless without the name of the policy it belongs to. Chunks that are too large dilute the embedding with unrelated text and consume more of the context window per retrieved result.

Anthropic's contextual retrieval method targets the lost-context problem. A model writes a short, chunk-specific note (typically 50 to 100 tokens) that situates the chunk within the whole document, and the note is prepended before embedding. In Anthropic's tests, this cut top-20 retrieval failures by 35% with contextual embeddings alone, and by 49% when combined with contextual BM25.

A practical approach is to start with a recursive splitter, build a small set of test questions with known answer passages, and measure retrieval recall while changing chunk size and overlap. Reranking can recover precision when chunks are small, and What is RAG walks through the full pipeline.

Cite this definition

The AI Engineering Editors. "What is chunking in RAG? Strategies explained." The AI Engineering, updated Nov 06, 2026. https://theaiengineering.com/glossary/chunking

Title
What is chunking in RAG? Strategies explained
Author
The AI Engineering Editors
Publisher
The AI Engineering
Updated
View as Markdown

Part of the RAG and vector search hub · Updated