Chunking strategies for RAG: how to choose, tune, and evaluate them

Guide · RAG

Chunking strategies for RAG are the rules a retrieval pipeline uses to split documents into passages, called chunks, before each one is embedded, indexed, and retrieved on its own. The strategy decides what a retriever can return, so it caps answer quality before the embedding model or reranker gets a say. This guide compares the main strategies, shows tested Python for three of them, and explains how to choose one with a retrieval evaluation. It belongs to the RAG hub.

Why chunk size matters in RAG#

Chunk size matters because each chunk becomes one embedding, and one vector can represent only so much. Small chunks match a query precisely but lose the context that makes them meaningful. Large chunks keep that context but blur several topics into one vector and spend more of the prompt per result. The embedding model's input limit sets the ceiling.

A chunk that reads "This limit does not apply to Enterprise plans" is useless without the limit it refers to. A 2,000-token chunk covering refunds, invoices, and taxes has the opposite problem: its single embedding sits between all three topics and may not rank first for any of them. Retrieved chunks also share the context window with instructions and history, so ten 800-token chunks add 8,000 tokens to every request. Budgeting that space is part of context engineering.

The embedding model's maximum input is a hard ceiling, not a target. OpenAI lists a maximum input of 8,192 tokens for text-embedding-3-small and text-embedding-3-large in its embeddings guide, and Google's embeddings documentation lists 2,048 tokens for gemini-embedding-001 and 8,192 for gemini-embedding-2. Small open models are tighter: the all-MiniLM-L6-v2 model card says input longer than 256 word pieces is truncated by default, so the end of a longer chunk never reaches the vector. Measure chunks in the embedding model's own tokens, not characters.

The right size depends on the questions. In NVIDIA's 2025 chunking experiments, datasets of factoid queries did best with 256- to 512-token chunks, while analytical queries benefited from 1,024-token chunks or one chunk per page. Page-level chunking had the highest average accuracy (0.648) and the lowest standard deviation across the five datasets tested.

Chunking strategies compared#

Chunking strategies differ in how they choose boundaries (fixed counts, separators, document structure, or embedding similarity) and in whether the unit that is searched is the unit that is returned. The simple methods are cheap and predictable. The advanced ones add indexing cost or a second lookup to recover context that an isolated chunk loses.

Strategy How it splits Strengths Costs and risks Good fit
Fixed-size with overlap Every N tokens, repeating M tokens between neighbors Predictable sizes, trivial to build Cuts sentences and tables; overlap grows the index Uniform prose, baselines
Recursive Paragraphs, then lines, sentences, words Keeps natural boundaries Ignores headings, so sections merge Most plain text
Document-aware Headings, tags, functions, table rows, page layout Matches how authors organized content Needs a parser per format Docs sites, wikis, code, PDFs
Semantic Where similarity between neighboring sentences drops Topically coherent chunks Extra embedding pass; inconsistent gains Long text without headings
Sentence window Embeds one sentence, returns its neighbors Precise match, readable context Many more vectors Dense factual prose
Parent-child Embeds small children, returns the parent section Precise match, full section context Second store and lookup Manuals, policies, contracts
Late chunking Embeds the whole document, then pools tokens per chunk Document context without LLM calls Needs token-level outputs Text full of cross-references
Contextual retrieval An LLM prepends a short context note to each chunk Large reported recall gains One LLM call per chunk at indexing Corpora where misses are costly

Fixed-size chunking with overlap#

Fixed-size chunking cuts a document every N tokens and repeats the last M tokens of each chunk at the start of the next. It is the easiest strategy to reason about and a useful baseline for any comparison. Its weakness is that boundaries ignore meaning, so sentences, list items, and table rows get cut in half.

OpenAI names the cl100k_base encoding for its third-generation embedding models, and the tiktoken package provides it:

import tiktoken
 
# OpenAI's embeddings guide names cl100k_base for text-embedding-3
enc = tiktoken.get_encoding("cl100k_base")
 
 
def token_chunks(text, size=256, overlap=32):
    """Windows of `size` tokens. Each window starts with the
    last `overlap` tokens of the previous one."""
    if not 0 <= overlap < size:
        raise ValueError("need 0 <= overlap < size")
    ids = enc.encode(text)
    step = size - overlap
    chunks = []
    for start in range(0, len(ids), step):
        window = ids[start:start + size]
        chunks.append({
            "text": enc.decode(window),
            "start_token": start,
            "n_tokens": len(window),
        })
        if start + size >= len(ids):
            break
    return chunks

Each chunk records its starting token, which later supports citations and reassembly. Overlap exists for facts that straddle a boundary, as Figure 1 shows.

Fixed-size 512-token chunks overlap by 64 tokens, so an answer passage that crosses the end of chunk 1 still appears whole inside chunk 2. 040080012001600Chunk 1tokens 0-511Chunk 2starts 64 tokens earlyChunk 3Chunk 4remainderAnswer passagewhole only in chunk 2Chunk 1 ends
Figure 1: Fixed-size chunks of 512 tokens with 64 tokens of overlap, on a token axis. A passage cut by one boundary survives intact in the next chunk.

The evidence on how much overlap to use is mixed. OpenAI's file search defaults are 800-token chunks with 400 tokens of overlap, with overlap capped at half the chunk size. NVIDIA found 15% overlap best among the 10%, 15%, and 20% it tested, and notes that 10% to 20% is common practice. The late chunking paper, covered below, found that overlap generally neither helped nor hurt in its experiments. Treat overlap as a parameter to test, not a rule.

Recursive chunking#

Recursive chunking splits on the coarsest separator that keeps pieces under the size limit (paragraphs, then lines, then sentences, then words) and merges small pieces back up to the limit. It keeps natural boundaries without a format-specific parser, which is why LangChain recommends its recursive splitter for generic text.

The algorithm fits in a short function:

# paragraphs, then lines, sentences, and words
SEPARATORS = ("\n\n", "\n", ". ", " ")
 
 
def recursive_split(text, limit=400, separators=SEPARATORS, length=len):
    """Split on the coarsest separator; recurse on oversized pieces."""
    if length(text) <= limit:
        return [text.strip()] if text.strip() else []
    if not separators:  # nothing left to split on: hard cut
        return [text[i:i + limit] for i in range(0, len(text), limit)]
    sep, finer = separators[0], separators[1:]
    parts = text.split(sep)
    if len(parts) == 1:
        return recursive_split(text, limit, finer, length)
    # keep each separator attached to the piece before it
    pieces = [p + sep for p in parts[:-1]] + [parts[-1]]
 
    chunks, current = [], ""
    for piece in pieces:
        if length(current + piece) <= limit:
            current += piece  # merge small pieces up to the limit
            continue
        if current.strip():
            chunks.append(current.strip())
        if length(piece) > limit:
            chunks.extend(recursive_split(piece, limit, finer, length))
            current = ""
        else:
            current = piece
    if current.strip():
        chunks.append(current.strip())
    return chunks

Pass length=lambda s: len(enc.encode(s)) to measure tokens instead of characters. Running it on a short Markdown handbook with a 300-character limit exposes the blind spot: the headings ## Security and ### Single sign-on land at the bottom of the preceding chunk, and the data-retention paragraph is indexed without its heading. To a generic splitter, a heading is one more line.

Tuned, a recursive splitter is a strong baseline. In Chroma's 2024 technical report, which retrieved five chunks per query with text-embedding-3-large, a recursive splitter capped at 200 tokens with no overlap reached 88.1% recall and an IoU of 6.9, the best IoU among the plain recursive and token splitters. An 800-token window with 400 tokens of overlap, the file search default above, had slightly below-average recall (87.9%) and the lowest scores on every other metric, including an IoU of 1.4. IoU is a token-level score that penalizes irrelevant and redundant retrieved text.

Document-aware chunking for Markdown, HTML, code, tables, and PDFs#

Document-aware chunking uses a format's own structure as boundaries: headings in Markdown and HTML, functions and classes in code, rows in tables, and layout elements in PDFs. Chunks then match how the author organized the content, and each chunk can carry its heading path as metadata. The cost is a parser for every format you ingest.

A common pattern runs two stages: split on structure first, then split any section that is still too long by tokens. With LangChain's langchain-text-splitters package:

from langchain_text_splitters import (
    MarkdownHeaderTextSplitter,
    RecursiveCharacterTextSplitter,
)
 
markdown = open("handbook.md").read()
 
# Stage 1: one section per heading, heading path kept as metadata
by_heading = MarkdownHeaderTextSplitter(
    headers_to_split_on=[("#", "h1"), ("##", "h2"), ("###", "h3")]
)
sections = by_heading.split_text(markdown)
 
# Stage 2: split long sections by tokens, using the embedder's tokenizer
by_size = RecursiveCharacterTextSplitter.from_tiktoken_encoder(
    encoding_name="cl100k_base", chunk_size=200, chunk_overlap=20
)
chunks = by_size.split_documents(sections)
 
for chunk in chunks:
    # prepend the heading path: a contextual chunk header
    breadcrumb = " > ".join(chunk.metadata.values())
    text_to_embed = f"{breadcrumb}\n\n{chunk.page_content}"
    print(text_to_embed.splitlines()[0])
# Acme Cloud handbook > Billing > Refunds
# Acme Cloud handbook > Billing > Invoices
# Acme Cloud handbook > Security > Single sign-on
# Acme Cloud handbook > Security > Data retention

On the same handbook, every chunk now starts with its own heading path. That breadcrumb is a cheap contextual chunk header: it lets the retriever tell apart passages that look alike, such as two "Pricing" sections from different products. Pass encoding_name explicitly, because from_tiktoken_encoder defaults to the older gpt2 encoding. LangChain's token splitting guide notes that the recursive variant enforces the size as a hard limit.

Other formats need their own rules:

  • HTML: split on header tags or semantic sections, and strip navigation, cookie banners, and footers first.
  • Code: split on functions and classes. LangChain's Python separators start with \nclass and \ndef before falling back to blank lines. Prepend the file path and enclosing class.
  • Tables: keep a table whole, or split by rows and repeat the header row in every chunk. Unstructured's chunking isolates tables: a table is never combined with other elements, and an oversized one is split into table chunks.
  • PDFs and scans: run a layout-aware parser from the data ingestion directory so columns, page headers, and footers do not interleave. Unstructured's by_title strategy guarantees that a chunk never contains text from two sections. Keep page numbers for citations.

Semantic chunking#

Semantic chunking places boundaries where the meaning shifts, measured by embedding similarity between neighboring sentences, instead of at a fixed size. It produces topically coherent chunks from text that has no headings. The price is an embedding pass over every sentence at indexing time, and published comparisons find its gains over simpler splitters inconsistent.

The method was popularized by Greg Kamradt's levels of text splitting notebook: embed each sentence together with its neighbors, compute the cosine distance between consecutive windows, and start a new chunk wherever the distance exceeds a percentile threshold (the notebook uses the 95th). LlamaIndex ships an adaptation as SemanticSplitterNodeParser.

Semantic chunking vs fixed-size and recursive chunking#

Qu, Tu, and Bao (2024) compared semantic and fixed-size chunking on document retrieval, evidence retrieval, and answer generation, and concluded that semantic chunking's computational cost was not justified by consistent performance gains. In Chroma's report, the default Kamradt chunker produced chunks of about 660 tokens on average and scored 83.6% recall, below the 200-token recursive splitter. Chroma's own embedding-clustering chunker, which caps chunk size, and its LLM-based chunker did score best on several metrics, so semantic boundaries can pay off when implemented with care. Try semantic chunking on long, unstructured text, set a maximum size, and keep it only if your eval shows a gain.

Sentence-window and parent-child retrieval#

Sentence-window and parent-child retrieval separate the unit you search from the unit you return. A small unit, such as one sentence or a short child chunk, is embedded for precise matching. At query time the system swaps in the surrounding window or the larger parent section, so the model reads full context. The pattern is often called small-to-big retrieval.

LlamaIndex's sentence window example parses documents into one sentence per node, stores the neighboring sentences in metadata (five on each side by default), and replaces each retrieved sentence with its window before generation. LangChain's ParentDocumentRetriever, now in the langchain-classic package, embeds small chunks and returns the parent documents they came from.

Documents are split into parent chunks and smaller child chunks; only children are embedded, and at query time matching children are swapped for their parents. IndexingQuery timeDocumentSource fileParent chunksStored by IDChild chunksTagged parent_idVector indexChild vectors onlyLLM promptParent textSwap in parentsDedupe by IDSearch childrenPrecise matchUser queryfetch by ID
Figure 2: Parent-child retrieval searches small child chunks for precision and sends their parent sections to the model for context.

Deduplicate parents when several children of the same section match, or the prompt fills with copies. Research supports fine-grained matching units: Dense X Retrieval by Chen et al. found that indexing propositions, atomic statements that each express one fact, outperformed passage-level indexing for retrieval. A reranker can then score the parent text the model will actually read.

Late chunking and contextual retrieval#

Late chunking and contextual retrieval attack the same flaw: a chunk embedded in isolation forgets its document, so a chunk that says "the new version fixes this bug" loses which product and which bug it means. Late chunking fixes this inside the embedding model. Contextual retrieval fixes it by having an LLM write context into the chunk text before indexing.

Late chunking#

Late chunking, introduced by Günther et al. at Jina AI in 2024, runs a long-context embedding model over the whole document (or the largest part that fits), then applies chunk boundaries to the token embeddings and mean-pools each span. Every chunk vector is computed with attention over the full text, and the method needs no extra training.

Averaged across three embedding models and four BEIR datasets with 256-token chunks, nDCG@10 rose from 52.2 with naive chunking to 54.0 with late chunking, and a separate experiment showed the largest gains at small chunk sizes. Late chunking needs the model's token-level outputs, so it runs on a self-hosted embedding model or one built for document-aware chunk embeddings, such as Perplexity's pplx-embed-context-v1, which embeds each passage with respect to its surrounding document.

Contextual retrieval#

Anthropic's contextual retrieval sends each chunk with its full document to an LLM, which writes a short context, usually 50 to 100 tokens, that is prepended to the chunk before both embedding and BM25 indexing. In Anthropic's tests, contextual embeddings cut the top-20 retrieval failure rate by 35%, adding contextual BM25 cut it by 49%, and adding reranking cut it by 67%. The cost is one LLM call per chunk at indexing time; prompt caching keeps it manageable because each document is cached once and reused for all of its chunks. BM25 here is the keyword half of hybrid search.

RAG chunking best practices#

RAG chunking best practices come down to a few habits: measure chunks in the embedding model's tokens, split on structure before size, keep tables and code blocks whole, prepend titles and headings, store metadata with every chunk, and version the chunker so you can re-index and compare. Then let an evaluation choose the size.

  • Clean text before chunking: remove navigation, repeated page headers, and boilerplate that would otherwise appear in every chunk.
  • Keep the chunk limit well below the embedding model's maximum input, and check whether your provider truncates or rejects longer input.
  • Re-chunk and re-embed the whole corpus when the strategy changes; an index that mixes strategies is hard to debug.

Metadata to store with each chunk#

Most vector databases store metadata as a filterable payload beside each vector. Store enough to cite, filter, reassemble, and re-index:

Field Why it matters
doc_id, chunk_id, position Reassemble neighbors, deduplicate, cite
Source URL, title, heading path Citations and contextual headers
Page number or token offsets Point readers to the exact location
parent_id Small-to-big retrieval
Content type (prose, table, code) Filter or format differently in the prompt
Last modified date Freshness filters and incremental re-indexing
Access permissions Filter results per user before they reach the model
Chunker and embedding model version Compare strategies and re-index safely

How to evaluate chunking strategies#

Evaluate chunking with a retrieval test set: real queries, each labeled with the evidence text that answers it. Index the same corpus once per candidate strategy, retrieve the top k for every query, and compare recall@k and mean reciprocal rank (MRR). Label evidence text rather than chunk IDs, because chunk IDs change every time the strategy does.

  1. Collect real user queries, or questions written by people who know the corpus, and record the sentence that answers each one.
  2. Index the corpus once per strategy with the same embedding model, retriever, and reranker settings.
  3. Measure recall@k, the share of queries whose evidence appears in the top k, and MRR, the mean of 1/rank for the first hit.
  4. Compare at an equal token budget. Larger chunks raise recall by returning more text, so compare ten 200-token chunks against five 400-token chunks, not five against five.
  5. Confirm the winner end to end with answer faithfulness and correctness.
def normalize(s):
    return " ".join(s.lower().split())
 
 
def evaluate(search, eval_set, k=5):
    """search(query, k) returns ranked chunk texts. Each eval item
    holds a query and the evidence sentence that answers it, so the
    labels survive a change of chunking strategy."""
    hits, reciprocal_ranks = 0, 0.0
    for item in eval_set:
        results = [normalize(c) for c in search(item["query"], k)]
        evidence = normalize(item["evidence"])
        ranks = [i for i, c in enumerate(results, 1) if evidence in c]
        if ranks:
            hits += 1
            reciprocal_ranks += 1 / ranks[0]
    n = len(eval_set)
    return {f"recall@{k}": hits / n, "mrr": reciprocal_ranks / n}
 
 
eval_set = [
    {
        "query": "Can I get my money back on a yearly plan?",
        "evidence": "Annual plans can be refunded within 30 days",
    },
    {
        "query": "How long are audit logs kept on Enterprise?",
        "evidence": "Audit logs are retained for 365 days",
    },
]
# One search function per candidate index, same eval set for all:
# for name, search in strategies.items():
#     print(name, evaluate(search, eval_set, k=5))

A chunk counts as a hit only when it contains the whole evidence sentence, so a boundary that splits the answer shows up as a miss, which is the failure this eval exists to catch. With one evidence passage per query, recall@k equals the hit rate. The LLM evaluation guide covers the judge-based faithfulness metrics for step 5 and how to gate releases on regressions.

Where to go next#

For the full pipeline these chunks feed, read What is RAG. If retrieval is not the right fix for your gap, RAG vs fine-tuning covers the alternative. The embedding and reranking directory lists the models whose input limits and tokenizers set your chunk budget.

Frequently asked questions

What is the best chunk size for RAG?

No single chunk size is best; it depends on your queries, documents, and embedding model. Published tests point to a few hundred tokens as a strong starting range. NVIDIA found 256 to 512 tokens worked best for factoid questions, while analytical questions benefited from 1,024-token or page-level chunks. Start with a structure-aware splitter at a few hundred tokens and tune it against a retrieval eval set.

What is semantic chunking?

Semantic chunking splits a document where its meaning shifts rather than at a fixed length. A typical implementation embeds each sentence with its neighbors, measures the cosine distance between consecutive embeddings, and starts a new chunk wherever the distance crosses a threshold, such as the 95th percentile. It produces topically coherent chunks from unstructured text, at the cost of embedding every sentence during indexing.

Semantic chunking vs recursive chunking: which is better?

Neither wins everywhere. Recursive chunking is cheaper, predictable, and hard to beat when tuned: Chroma found a 200-token recursive splitter performed well across metrics, and Qu, Tu, and Bao found semantic chunking's cost was not justified by consistent gains. Semantic chunking can help on long text without headings, especially with a maximum size cap. Run both against the same retrieval eval set and keep the cheaper one unless the other clearly wins.

How much chunk overlap should I use?

Overlap of 10 to 20 percent of the chunk size is common practice, and NVIDIA found 15 percent best among the values it tested. The evidence that overlap helps is mixed: the late chunking paper found it generally neither helped nor hurt, and Chroma found that less overlap improved token efficiency, though it lowered recall in one small-chunk setting. Overlap matters most for fixed-size chunks; structure-aware chunks need less. Confirm the setting with a retrieval eval.

What is late chunking?

Late chunking is an embedding technique from Jina AI researchers that runs a long-context embedding model over the whole document first, then splits the resulting token embeddings into chunks and averages each span into one vector. Each chunk vector therefore reflects the surrounding document, which helps with references such as pronouns. It needs no extra training, but it needs access to token-level outputs from a long-context embedding model.

How do you chunk documents in LangChain?

Install the langchain-text-splitters package and start with RecursiveCharacterTextSplitter, which LangChain recommends for generic text. Use from_tiktoken_encoder with an explicit encoding_name to measure chunks in tokens, MarkdownHeaderTextSplitter or HTMLHeaderTextSplitter to split on headings first, and from_language for source code. For small-to-big retrieval, ParentDocumentRetriever is in the langchain-classic package.

Terms in this guide

Cite this guide

The AI Engineering Editors. "Chunking strategies for RAG: how to choose, tune, and evaluate them." The AI Engineering, updated Aug 23, 2026. https://theaiengineering.com/articles/chunking-strategies-for-rag

Title
Chunking strategies for RAG: how to choose, tune, and evaluate them
Author
The AI Engineering Editors
Publisher
The AI Engineering
Published
Updated
View as Markdown

The AI Engineering Editors ·