What is chunking in RAG? Strategies explained
Also called document chunking, text chunking, chunking strategies
Definition
Chunking is the step in a retrieval pipeline that splits documents into smaller passages, called chunks, so each one can be embedded, indexed, and retrieved on its own.
Chunking is the step in a retrieval-augmented generation pipeline that splits source documents into smaller passages before indexing. Each chunk becomes one unit of retrieval: it gets its own embedding, its own entry in the index, and competes on its own for a place in the prompt. Chunk boundaries therefore determine which pieces of information can be retrieved together.
How chunking works#
Chunking works by applying a splitting rule to each document and, optionally, attaching metadata such as the document title, section heading, or page number to every chunk. The main strategies:
| Strategy | How it splits | Trade-off |
|---|---|---|
| Fixed-size | Every N tokens, usually with overlap | Simple, but can cut sentences and ideas in half |
| Recursive | On paragraphs, then sentences, then words, until under a size limit | Keeps natural boundaries; a common default |
| Structure-aware | On headings, sections, code blocks, or table rows | Follows the document's layout; needs a parser per format |
| Semantic | Where embedding similarity between adjacent sentences drops | Groups related sentences; costs an extra embedding pass |
Overlap, which repeats a few sentences between neighboring chunks, lowers the chance that an answer spans a boundary, at the cost of a larger index.
Why chunking matters#
Chunking matters because a retriever can only return what a chunk contains. Chunks that are too small lose context: a sentence such as "coverage begins after 90 days" is useless without the name of the policy it belongs to. Chunks that are too large dilute the embedding with unrelated text and consume more of the context window per retrieved result.
Anthropic's contextual retrieval method targets the lost-context problem. A model writes a short, chunk-specific note (typically 50 to 100 tokens) that situates the chunk within the whole document, and the note is prepended before embedding. In Anthropic's tests, this cut top-20 retrieval failures by 35% with contextual embeddings alone, and by 49% when combined with contextual BM25.
A practical approach is to start with a recursive splitter, build a small set of test questions with known answer passages, and measure retrieval recall while changing chunk size and overlap. Reranking can recover precision when chunks are small, and What is RAG walks through the full pipeline.
Related terms
Guides that cover chunking
Cite this definition
The AI Engineering Editors. "What is chunking in RAG? Strategies explained." The AI Engineering, updated Nov 06, 2026. https://theaiengineering.com/glossary/chunking
- Title
- What is chunking in RAG? Strategies explained
- Author
- The AI Engineering Editors
- Publisher
- The AI Engineering
- Updated