---
title: "Chunking strategies for RAG: how to choose, tune, and evaluate them"
description: "Chunking strategies for RAG compared: fixed-size, recursive, document-aware, semantic, parent-child, and late chunking, with tested Python and an eval method."
summary: "Chunking decides what a retriever can return, so the splitting rule caps RAG quality before the embedding model or reranker gets a say. This guide compares the main strategies, shows tested Python splitters, and explains how to choose one with a recall@k and MRR evaluation on your own corpus."
type: "guide"
canonical_url: "https://theaiengineering.com/articles/chunking-strategies-for-rag"
markdown_url: "https://theaiengineering.com/articles/chunking-strategies-for-rag.md"
topic: "RAG and vector search"
author: "The AI Engineering Editors"
publisher: "The AI Engineering"
published: "2026-08-23"
updated: "2026-08-23"
keywords: ["chunking strategies for rag","rag chunking strategies","rag chunking best practices","semantic chunking","semantic chunking vs fixed-size chunking","semantic chunking vs recursive chunking","chunking strategies for rag langchain","chunking strategies for llm applications"]
citation: "The AI Engineering Editors. \"Chunking strategies for RAG: how to choose, tune, and evaluate them.\" The AI Engineering, updated Aug 23, 2026. https://theaiengineering.com/articles/chunking-strategies-for-rag"
---

# Chunking strategies for RAG: how to choose, tune, and evaluate them

## TL;DR

- Chunk size trades retrieval precision against context: small chunks match queries precisely but lose surrounding meaning, while large chunks keep context but blur several topics into one embedding.
- A recursive or structure-aware splitter measured in the embedding model's tokens is a strong default; in Chroma's 2024 tests, a recursive splitter at 200 tokens with no overlap performed well across metrics.
- Semantic chunking adds an embedding pass at indexing time, and Qu, Tu, and Bao (2024) found its computational cost was not justified by consistent gains over fixed-size chunking.
- Parent-child retrieval, late chunking, and contextual retrieval restore the document context an isolated chunk loses; Anthropic reported that contextual embeddings plus contextual BM25 cut top-20 retrieval failures by 49%.
- Choose a chunking strategy with a retrieval eval set labeled by evidence text rather than chunk IDs, and compare recall@k and MRR at an equal token budget.

Chunking strategies for RAG are the rules a retrieval pipeline uses to split documents into passages, called [chunks](https://theaiengineering.com/glossary/chunking), before each one is embedded, indexed, and retrieved on its own. The strategy decides what a retriever can return, so it caps answer quality before the embedding model or reranker gets a say. This guide compares the main strategies, shows tested Python for three of them, and explains how to choose one with a retrieval evaluation. It belongs to the [RAG hub](https://theaiengineering.com/topics/rag).

## Why chunk size matters in RAG

Chunk size matters because each chunk becomes one embedding, and one vector can represent only so much. Small chunks match a query precisely but lose the context that makes them meaningful. Large chunks keep that context but blur several topics into one vector and spend more of the prompt per result. The embedding model's input limit sets the ceiling.

A chunk that reads "This limit does not apply to Enterprise plans" is useless without the limit it refers to. A 2,000-token chunk covering refunds, invoices, and taxes has the opposite problem: its single [embedding](https://theaiengineering.com/glossary/embeddings) sits between all three topics and may not rank first for any of them. Retrieved chunks also share the context window with instructions and history, so ten 800-token chunks add 8,000 tokens to every request. Budgeting that space is part of [context engineering](https://theaiengineering.com/articles/context-engineering).

The embedding model's maximum input is a hard ceiling, not a target. OpenAI lists a maximum input of 8,192 tokens for `text-embedding-3-small` and `text-embedding-3-large` in its [embeddings guide](https://developers.openai.com/api/docs/guides/embeddings), and Google's [embeddings documentation](https://aistudio.google.com/docs/embeddings) lists 2,048 tokens for `gemini-embedding-001` and 8,192 for `gemini-embedding-2`. Small open models are tighter: the [all-MiniLM-L6-v2 model card](https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2) says input longer than 256 word pieces is truncated by default, so the end of a longer chunk never reaches the vector. Measure chunks in the embedding model's own tokens, not characters.

The right size depends on the questions. In [NVIDIA's 2025 chunking experiments](https://developer.nvidia.com/blog/finding-the-best-chunking-strategy-for-accurate-ai-responses/), datasets of factoid queries did best with 256- to 512-token chunks, while analytical queries benefited from 1,024-token chunks or one chunk per page. Page-level chunking had the highest average accuracy (0.648) and the lowest standard deviation across the five datasets tested.

## Chunking strategies compared

Chunking strategies differ in how they choose boundaries (fixed counts, separators, document structure, or embedding similarity) and in whether the unit that is searched is the unit that is returned. The simple methods are cheap and predictable. The advanced ones add indexing cost or a second lookup to recover context that an isolated chunk loses.

| Strategy | How it splits | Strengths | Costs and risks | Good fit |
|---|---|---|---|---|
| Fixed-size with overlap | Every N tokens, repeating M tokens between neighbors | Predictable sizes, trivial to build | Cuts sentences and tables; overlap grows the index | Uniform prose, baselines |
| Recursive | Paragraphs, then lines, sentences, words | Keeps natural boundaries | Ignores headings, so sections merge | Most plain text |
| Document-aware | Headings, tags, functions, table rows, page layout | Matches how authors organized content | Needs a parser per format | Docs sites, wikis, code, PDFs |
| Semantic | Where similarity between neighboring sentences drops | Topically coherent chunks | Extra embedding pass; inconsistent gains | Long text without headings |
| Sentence window | Embeds one sentence, returns its neighbors | Precise match, readable context | Many more vectors | Dense factual prose |
| Parent-child | Embeds small children, returns the parent section | Precise match, full section context | Second store and lookup | Manuals, policies, contracts |
| Late chunking | Embeds the whole document, then pools tokens per chunk | Document context without LLM calls | Needs token-level outputs | Text full of cross-references |
| Contextual retrieval | An LLM prepends a short context note to each chunk | Large reported recall gains | One LLM call per chunk at indexing | Corpora where misses are costly |

## Fixed-size chunking with overlap

Fixed-size chunking cuts a document every N tokens and repeats the last M tokens of each chunk at the start of the next. It is the easiest strategy to reason about and a useful baseline for any comparison. Its weakness is that boundaries ignore meaning, so sentences, list items, and table rows get cut in half.

OpenAI names the `cl100k_base` encoding for its third-generation embedding models, and the `tiktoken` package provides it:

```python
import tiktoken

# OpenAI's embeddings guide names cl100k_base for text-embedding-3
enc = tiktoken.get_encoding("cl100k_base")


def token_chunks(text, size=256, overlap=32):
    """Windows of `size` tokens. Each window starts with the
    last `overlap` tokens of the previous one."""
    if not 0 <= overlap < size:
        raise ValueError("need 0 <= overlap < size")
    ids = enc.encode(text)
    step = size - overlap
    chunks = []
    for start in range(0, len(ids), step):
        window = ids[start:start + size]
        chunks.append({
            "text": enc.decode(window),
            "start_token": start,
            "n_tokens": len(window),
        })
        if start + size >= len(ids):
            break
    return chunks
```

Each chunk records its starting token, which later supports citations and reassembly. Overlap exists for facts that straddle a boundary, as Figure 1 shows.

**Figure 1: Fixed-size chunks of 512 tokens with 64 tokens of overlap, on a token axis. A passage cut by one boundary survives intact in the next chunk.**

Fixed-size 512-token chunks overlap by 64 tokens, so an answer passage that crosses the end of chunk 1 still appears whole inside chunk 2.

Spans:

- Chunk 1: 0 to 512 (tokens 0-511)
- Chunk 2: 448 to 960 (starts 64 tokens early)
- Chunk 3: 896 to 1408
- Chunk 4: 1344 to 1600 (remainder)
- Answer passage: 470 to 545 (whole only in chunk 2)

Markers:

- 512: chunk 1 ends

The evidence on how much overlap to use is mixed. OpenAI's [file search defaults](https://developers.openai.com/api/docs/guides/retrieval) are 800-token chunks with 400 tokens of overlap, with overlap capped at half the chunk size. NVIDIA found 15% overlap best among the 10%, 15%, and 20% it tested, and notes that 10% to 20% is common practice. The late chunking paper, covered below, found that overlap generally neither helped nor hurt in its experiments. Treat overlap as a parameter to test, not a rule.

## Recursive chunking

Recursive chunking splits on the coarsest separator that keeps pieces under the size limit (paragraphs, then lines, then sentences, then words) and merges small pieces back up to the limit. It keeps natural boundaries without a format-specific parser, which is why LangChain [recommends its recursive splitter](https://docs.langchain.com/oss/python/integrations/splitters/recursive_text_splitter) for generic text.

The algorithm fits in a short function:

```python
# paragraphs, then lines, sentences, and words
SEPARATORS = ("\n\n", "\n", ". ", " ")


def recursive_split(text, limit=400, separators=SEPARATORS, length=len):
    """Split on the coarsest separator; recurse on oversized pieces."""
    if length(text) <= limit:
        return [text.strip()] if text.strip() else []
    if not separators:  # nothing left to split on: hard cut
        return [text[i:i + limit] for i in range(0, len(text), limit)]
    sep, finer = separators[0], separators[1:]
    parts = text.split(sep)
    if len(parts) == 1:
        return recursive_split(text, limit, finer, length)
    # keep each separator attached to the piece before it
    pieces = [p + sep for p in parts[:-1]] + [parts[-1]]

    chunks, current = [], ""
    for piece in pieces:
        if length(current + piece) <= limit:
            current += piece  # merge small pieces up to the limit
            continue
        if current.strip():
            chunks.append(current.strip())
        if length(piece) > limit:
            chunks.extend(recursive_split(piece, limit, finer, length))
            current = ""
        else:
            current = piece
    if current.strip():
        chunks.append(current.strip())
    return chunks
```

Pass `length=lambda s: len(enc.encode(s))` to measure tokens instead of characters. Running it on a short Markdown handbook with a 300-character limit exposes the blind spot: the headings `## Security` and `### Single sign-on` land at the bottom of the preceding chunk, and the data-retention paragraph is indexed without its heading. To a generic splitter, a heading is one more line.

Tuned, a recursive splitter is a strong baseline. In [Chroma's 2024 technical report](https://www.trychroma.com/research/evaluating-chunking), which retrieved five chunks per query with `text-embedding-3-large`, a recursive splitter capped at 200 tokens with no overlap reached 88.1% recall and an IoU of 6.9, the best IoU among the plain recursive and token splitters. An 800-token window with 400 tokens of overlap, the file search default above, had slightly below-average recall (87.9%) and the lowest scores on every other metric, including an IoU of 1.4. IoU is a token-level score that penalizes irrelevant and redundant retrieved text.

## Document-aware chunking for Markdown, HTML, code, tables, and PDFs

Document-aware chunking uses a format's own structure as boundaries: headings in Markdown and HTML, functions and classes in code, rows in tables, and layout elements in PDFs. Chunks then match how the author organized the content, and each chunk can carry its heading path as metadata. The cost is a parser for every format you ingest.

A common pattern runs two stages: split on structure first, then split any section that is still too long by tokens. With LangChain's `langchain-text-splitters` package:

```python
from langchain_text_splitters import (
    MarkdownHeaderTextSplitter,
    RecursiveCharacterTextSplitter,
)

markdown = open("handbook.md").read()

# Stage 1: one section per heading, heading path kept as metadata
by_heading = MarkdownHeaderTextSplitter(
    headers_to_split_on=[("#", "h1"), ("##", "h2"), ("###", "h3")]
)
sections = by_heading.split_text(markdown)

# Stage 2: split long sections by tokens, using the embedder's tokenizer
by_size = RecursiveCharacterTextSplitter.from_tiktoken_encoder(
    encoding_name="cl100k_base", chunk_size=200, chunk_overlap=20
)
chunks = by_size.split_documents(sections)

for chunk in chunks:
    # prepend the heading path: a contextual chunk header
    breadcrumb = " > ".join(chunk.metadata.values())
    text_to_embed = f"{breadcrumb}\n\n{chunk.page_content}"
    print(text_to_embed.splitlines()[0])
# Acme Cloud handbook > Billing > Refunds
# Acme Cloud handbook > Billing > Invoices
# Acme Cloud handbook > Security > Single sign-on
# Acme Cloud handbook > Security > Data retention
```

On the same handbook, every chunk now starts with its own heading path. That breadcrumb is a cheap contextual chunk header: it lets the retriever tell apart passages that look alike, such as two "Pricing" sections from different products. Pass `encoding_name` explicitly, because `from_tiktoken_encoder` defaults to the older `gpt2` encoding. LangChain's [token splitting guide](https://docs.langchain.com/oss/python/integrations/splitters/split_by_token) notes that the recursive variant enforces the size as a hard limit.

Other formats need their own rules:

- **HTML:** split on header tags or semantic sections, and strip navigation, cookie banners, and footers first.
- **Code:** split on functions and classes. LangChain's Python separators start with `\nclass ` and `\ndef ` before falling back to blank lines. Prepend the file path and enclosing class.
- **Tables:** keep a table whole, or split by rows and repeat the header row in every chunk. [Unstructured's chunking](https://docs.unstructured.io/open-source/core-functionality/chunking) isolates tables: a table is never combined with other elements, and an oversized one is split into table chunks.
- **PDFs and scans:** run a layout-aware parser from the [data ingestion directory](https://theaiengineering.com/platforms/data-ingestion) so columns, page headers, and footers do not interleave. Unstructured's `by_title` strategy guarantees that a chunk never contains text from two sections. Keep page numbers for citations.

## Semantic chunking

Semantic chunking places boundaries where the meaning shifts, measured by embedding similarity between neighboring sentences, instead of at a fixed size. It produces topically coherent chunks from text that has no headings. The price is an embedding pass over every sentence at indexing time, and published comparisons find its gains over simpler splitters inconsistent.

The method was popularized by Greg Kamradt's [levels of text splitting notebook](https://github.com/FullStackRetrieval-com/RetrievalTutorials/blob/main/tutorials/LevelsOfTextSplitting/5_Levels_Of_Text_Splitting.ipynb): embed each sentence together with its neighbors, compute the [cosine distance](https://theaiengineering.com/glossary/cosine-similarity) between consecutive windows, and start a new chunk wherever the distance exceeds a percentile threshold (the notebook uses the 95th). LlamaIndex ships an adaptation as `SemanticSplitterNodeParser`.

### Semantic chunking vs fixed-size and recursive chunking

[Qu, Tu, and Bao (2024)](https://arxiv.org/abs/2410.13070) compared semantic and fixed-size chunking on document retrieval, evidence retrieval, and answer generation, and concluded that semantic chunking's computational cost was not justified by consistent performance gains. In Chroma's report, the default Kamradt chunker produced chunks of about 660 tokens on average and scored 83.6% recall, below the 200-token recursive splitter. Chroma's own embedding-clustering chunker, which caps chunk size, and its LLM-based chunker did score best on several metrics, so semantic boundaries can pay off when implemented with care. Try semantic chunking on long, unstructured text, set a maximum size, and keep it only if your eval shows a gain.

## Sentence-window and parent-child retrieval

Sentence-window and parent-child retrieval separate the unit you search from the unit you return. A small unit, such as one sentence or a short child chunk, is embedded for precise matching. At query time the system swaps in the surrounding window or the larger parent section, so the model reads full context. The pattern is often called small-to-big retrieval.

LlamaIndex's [sentence window example](https://developers.llamaindex.ai/python/examples/node_postprocessor/metadatareplacementdemo/) parses documents into one sentence per node, stores the neighboring sentences in metadata (five on each side by default), and replaces each retrieved sentence with its window before generation. LangChain's [ParentDocumentRetriever](https://reference.langchain.com/python/langchain-classic/retrievers/parent_document_retriever/ParentDocumentRetriever), now in the `langchain-classic` package, embeds small chunks and returns the parent documents they came from.

**Figure 2: Parent-child retrieval searches small child chunks for precision and sends their parent sections to the model for context.**

Documents are split into parent chunks and smaller child chunks; only children are embedded, and at query time matching children are swapped for their parents.

Components:

- **Document**: Source file
- **Parent chunks**: Stored by ID
- **Child chunks**: Tagged parent_id
- **Vector index**: Child vectors only
- **LLM prompt**: Parent text
- **Swap in parents**: Dedupe by ID
- **Search children**: Precise match
- **User query**

Groups:

- Indexing: Document, Parent chunks, Child chunks, Vector index
- Query time: LLM prompt, Swap in parents, Search children, User query

Connections:

- Document → Parent chunks
- Parent chunks → Child chunks
- Child chunks → Vector index
- Vector index → Search children
- User query → Search children
- Search children → Swap in parents
- Parent chunks → Swap in parents: fetch by ID
- Swap in parents → LLM prompt

Deduplicate parents when several children of the same section match, or the prompt fills with copies. Research supports fine-grained matching units: [Dense X Retrieval](https://arxiv.org/abs/2312.06648) by Chen et al. found that indexing propositions, atomic statements that each express one fact, outperformed passage-level indexing for retrieval. A [reranker](https://theaiengineering.com/glossary/reranking) can then score the parent text the model will actually read.

## Late chunking and contextual retrieval

[Late chunking](https://theaiengineering.com/glossary/late-chunking) and [contextual retrieval](https://theaiengineering.com/glossary/contextual-retrieval) attack the same flaw: a chunk embedded in isolation forgets its document, so a chunk that says "the new version fixes this bug" loses which product and which bug it means. Late chunking fixes this inside the embedding model. Contextual retrieval fixes it by having an LLM write context into the chunk text before indexing.

### Late chunking

[Late chunking](https://arxiv.org/abs/2409.04701), introduced by Günther et al. at Jina AI in 2024, runs a long-context embedding model over the whole document (or the largest part that fits), then applies chunk boundaries to the token embeddings and mean-pools each span. Every chunk vector is computed with attention over the full text, and the method needs no extra training.

Averaged across three embedding models and four BEIR datasets with 256-token chunks, nDCG@10 rose from 52.2 with naive chunking to 54.0 with late chunking, and a separate experiment showed the largest gains at small chunk sizes. Late chunking needs the model's token-level outputs, so it runs on a self-hosted embedding model or one built for document-aware chunk embeddings, such as Perplexity's [pplx-embed-context-v1](https://theaiengineering.com/releases/2026/02#pplx-embed-v1), which embeds each passage with respect to its surrounding document.

### Contextual retrieval

Anthropic's [contextual retrieval](https://www.anthropic.com/news/contextual-retrieval) sends each chunk with its full document to an LLM, which writes a short context, usually 50 to 100 tokens, that is prepended to the chunk before both embedding and [BM25](https://theaiengineering.com/glossary/bm25) indexing. In Anthropic's tests, contextual embeddings cut the top-20 retrieval failure rate by 35%, adding contextual BM25 cut it by 49%, and adding reranking cut it by 67%. The cost is one LLM call per chunk at indexing time; prompt caching keeps it manageable because each document is cached once and reused for all of its chunks. BM25 here is the keyword half of [hybrid search](https://theaiengineering.com/glossary/hybrid-search).

## RAG chunking best practices

RAG chunking best practices come down to a few habits: measure chunks in the embedding model's tokens, split on structure before size, keep tables and code blocks whole, prepend titles and headings, store metadata with every chunk, and version the chunker so you can re-index and compare. Then let an evaluation choose the size.

- Clean text before chunking: remove navigation, repeated page headers, and boilerplate that would otherwise appear in every chunk.
- Keep the chunk limit well below the embedding model's maximum input, and check whether your provider truncates or rejects longer input.
- Re-chunk and re-embed the whole corpus when the strategy changes; an index that mixes strategies is hard to debug.

### Metadata to store with each chunk

Most [vector databases](https://theaiengineering.com/platforms/vector-databases) store metadata as a filterable payload beside each vector. Store enough to cite, filter, reassemble, and re-index:

| Field | Why it matters |
|---|---|
| `doc_id`, `chunk_id`, position | Reassemble neighbors, deduplicate, cite |
| Source URL, title, heading path | Citations and contextual headers |
| Page number or token offsets | Point readers to the exact location |
| `parent_id` | Small-to-big retrieval |
| Content type (prose, table, code) | Filter or format differently in the prompt |
| Last modified date | Freshness filters and incremental re-indexing |
| Access permissions | Filter results per user before they reach the model |
| Chunker and embedding model version | Compare strategies and re-index safely |

## How to evaluate chunking strategies

Evaluate chunking with a [retrieval test set](https://theaiengineering.com/glossary/golden-dataset): real queries, each labeled with the evidence text that answers it. Index the same corpus once per candidate strategy, retrieve the top k for every query, and compare [recall@k](https://theaiengineering.com/glossary/recall-at-k) and mean reciprocal rank (MRR). Label evidence text rather than chunk IDs, because chunk IDs change every time the strategy does.

1. Collect real user queries, or questions written by people who know the corpus, and record the sentence that answers each one.
2. Index the corpus once per strategy with the same embedding model, retriever, and reranker settings.
3. Measure recall@k, the share of queries whose evidence appears in the top k, and MRR, the mean of 1/rank for the first hit.
4. Compare at an equal token budget. Larger chunks raise recall by returning more text, so compare ten 200-token chunks against five 400-token chunks, not five against five.
5. Confirm the winner end to end with answer [faithfulness](https://theaiengineering.com/glossary/faithfulness) and correctness.

```python
def normalize(s):
    return " ".join(s.lower().split())


def evaluate(search, eval_set, k=5):
    """search(query, k) returns ranked chunk texts. Each eval item
    holds a query and the evidence sentence that answers it, so the
    labels survive a change of chunking strategy."""
    hits, reciprocal_ranks = 0, 0.0
    for item in eval_set:
        results = [normalize(c) for c in search(item["query"], k)]
        evidence = normalize(item["evidence"])
        ranks = [i for i, c in enumerate(results, 1) if evidence in c]
        if ranks:
            hits += 1
            reciprocal_ranks += 1 / ranks[0]
    n = len(eval_set)
    return {f"recall@{k}": hits / n, "mrr": reciprocal_ranks / n}


eval_set = [
    {
        "query": "Can I get my money back on a yearly plan?",
        "evidence": "Annual plans can be refunded within 30 days",
    },
    {
        "query": "How long are audit logs kept on Enterprise?",
        "evidence": "Audit logs are retained for 365 days",
    },
]
# One search function per candidate index, same eval set for all:
# for name, search in strategies.items():
#     print(name, evaluate(search, eval_set, k=5))
```

A chunk counts as a hit only when it contains the whole evidence sentence, so a boundary that splits the answer shows up as a miss, which is the failure this eval exists to catch. With one evidence passage per query, recall@k equals the hit rate. The [LLM evaluation guide](https://theaiengineering.com/articles/llm-evaluation) covers the judge-based faithfulness metrics for step 5 and how to gate releases on regressions.

## Where to go next

For the full pipeline these chunks feed, read [What is RAG](https://theaiengineering.com/articles/what-is-rag). If retrieval is not the right fix for your gap, [RAG vs fine-tuning](https://theaiengineering.com/articles/rag-vs-fine-tuning) covers the alternative. The [embedding and reranking directory](https://theaiengineering.com/platforms/embedding-reranking) lists the models whose input limits and tokenizers set your chunk budget.

## Frequently asked questions

### What is the best chunk size for RAG?

No single chunk size is best; it depends on your queries, documents, and embedding model. Published tests point to a few hundred tokens as a strong starting range. NVIDIA found 256 to 512 tokens worked best for factoid questions, while analytical questions benefited from 1,024-token or page-level chunks. Start with a structure-aware splitter at a few hundred tokens and tune it against a retrieval eval set.

### What is semantic chunking?

Semantic chunking splits a document where its meaning shifts rather than at a fixed length. A typical implementation embeds each sentence with its neighbors, measures the cosine distance between consecutive embeddings, and starts a new chunk wherever the distance crosses a threshold, such as the 95th percentile. It produces topically coherent chunks from unstructured text, at the cost of embedding every sentence during indexing.

### Semantic chunking vs recursive chunking: which is better?

Neither wins everywhere. Recursive chunking is cheaper, predictable, and hard to beat when tuned: Chroma found a 200-token recursive splitter performed well across metrics, and Qu, Tu, and Bao found semantic chunking's cost was not justified by consistent gains. Semantic chunking can help on long text without headings, especially with a maximum size cap. Run both against the same retrieval eval set and keep the cheaper one unless the other clearly wins.

### How much chunk overlap should I use?

Overlap of 10 to 20 percent of the chunk size is common practice, and NVIDIA found 15 percent best among the values it tested. The evidence that overlap helps is mixed: the late chunking paper found it generally neither helped nor hurt, and Chroma found that less overlap improved token efficiency, though it lowered recall in one small-chunk setting. Overlap matters most for fixed-size chunks; structure-aware chunks need less. Confirm the setting with a retrieval eval.

### What is late chunking?

Late chunking is an embedding technique from Jina AI researchers that runs a long-context embedding model over the whole document first, then splits the resulting token embeddings into chunks and averages each span into one vector. Each chunk vector therefore reflects the surrounding document, which helps with references such as pronouns. It needs no extra training, but it needs access to token-level outputs from a long-context embedding model.

### How do you chunk documents in LangChain?

Install the langchain-text-splitters package and start with RecursiveCharacterTextSplitter, which LangChain recommends for generic text. Use from_tiktoken_encoder with an explicit encoding_name to measure chunks in tokens, MarkdownHeaderTextSplitter or HTMLHeaderTextSplitter to split on headings first, and from_language for source code. For small-to-big retrieval, ParentDocumentRetriever is in the langchain-classic package.

## Terms in this guide

- [Chunking](https://theaiengineering.com/glossary/chunking): Chunking is the step in a retrieval pipeline that splits documents into smaller passages, called chunks, so each one can be embedded, indexed, and retrieved on its own.
- [Embeddings](https://theaiengineering.com/glossary/embeddings): Embeddings are fixed-length vectors of numbers that represent data such as text or images so that items with similar meaning sit close together, which lets software compare meaning with arithmetic.
- [Tokens](https://theaiengineering.com/glossary/tokens): Tokens are the units of text a language model processes, typically whole words, word fragments, punctuation, or bytes, each mapped to an integer ID from the model's fixed vocabulary.
- [Context window](https://theaiengineering.com/glossary/context-window): A context window is the maximum number of tokens a language model can take into account in a single request, including the system prompt, conversation history, tool results, and the generated output.
- [Vector database](https://theaiengineering.com/glossary/vector-database): A vector database is a database built to store embedding vectors and quickly return the vectors most similar to a query vector, usually with approximate nearest neighbor indexes such as HNSW.
- [Hybrid search](https://theaiengineering.com/glossary/hybrid-search): Hybrid search is a retrieval method that runs keyword search, typically BM25, and vector similarity search on the same query and merges the two ranked lists into one result set.
- [Reranking](https://theaiengineering.com/glossary/reranking): Reranking is a second retrieval stage that rescores an initial list of candidate documents with a more accurate but slower model, usually a cross-encoder, and keeps only the most relevant ones.
- [Prompt caching](https://theaiengineering.com/glossary/prompt-caching): Prompt caching is an LLM API and inference-server feature that stores the computed attention state for a repeated prompt prefix, so later requests sharing that prefix skip reprocessing it.
- [Retrieval-augmented generation (RAG)](https://theaiengineering.com/glossary/retrieval-augmented-generation): Retrieval-augmented generation (RAG) is a technique that retrieves relevant passages from an external knowledge source at query time and adds them to a language model's prompt so the answer is grounded in that data.
- [Cosine similarity](https://theaiengineering.com/glossary/cosine-similarity): Cosine similarity is a measure of how closely two vectors point in the same direction, computed as their dot product divided by the product of their lengths, with values from -1 to 1.
- [Late chunking](https://theaiengineering.com/glossary/late-chunking): Late chunking is an embedding technique that runs a long-context embedding model over a whole document first, then splits the token embeddings at chunk boundaries and averages each span into one chunk vector.
- [Contextual retrieval](https://theaiengineering.com/glossary/contextual-retrieval): Contextual retrieval is a RAG indexing technique in which a language model writes a short note situating each chunk within its source document, and the note is prepended before embedding and BM25 indexing.
- [BM25](https://theaiengineering.com/glossary/bm25): BM25 is a keyword ranking function that scores a document against a query using how often each query term appears, how rare the term is across the corpus, and how long the document is.
- [Golden dataset](https://theaiengineering.com/glossary/golden-dataset): A golden dataset is a curated, versioned set of representative inputs, each paired with an expected output or grading criteria, that a team treats as ground truth when evaluating an LLM application.
- [Recall@k](https://theaiengineering.com/glossary/recall-at-k): Recall@k is a retrieval metric that measures the fraction of a query's relevant documents that appear in the top k results, averaged over a set of test queries.
- [Faithfulness](https://theaiengineering.com/glossary/faithfulness): Faithfulness is an evaluation metric that measures what share of the claims in a generated answer can be inferred from the context the model was given, usually scored from 0 to 1.

## Related guides

- [What is retrieval-augmented generation (RAG)? How RAG works](https://theaiengineering.com/articles/what-is-rag): Retrieval-augmented generation (RAG) grounds LLM answers in your documents. Learn the RAG pipeline, hybrid search, agentic RAG, evaluation, and failure modes.
- [RAG vs fine-tuning: how to choose for your LLM application](https://theaiengineering.com/articles/rag-vs-fine-tuning): RAG vs fine-tuning: RAG adds knowledge at query time, while fine-tuning changes model behavior. Compare freshness, cost, latency, data needs, LoRA, and hybrids.
- [LLM evaluation: a practical guide to evals, metrics, and judges](https://theaiengineering.com/articles/llm-evaluation): LLM evaluation for engineers: build eval datasets, pick metrics, calibrate LLM-as-a-judge, evaluate RAG and agents, and gate every release in CI.
- [What is context engineering? A practical guide for AI agents](https://theaiengineering.com/articles/context-engineering): Context engineering is the practice of choosing what fills an LLM context window. Learn its components, why context rot happens, and how to measure results.

Part of the [RAG and vector search](https://theaiengineering.com/topics/rag) topic hub.

## Cite this guide

The AI Engineering Editors. "Chunking strategies for RAG: how to choose, tune, and evaluate them." The AI Engineering, updated Aug 23, 2026. https://theaiengineering.com/articles/chunking-strategies-for-rag

- Title: Chunking strategies for RAG: how to choose, tune, and evaluate them
- Author: The AI Engineering Editors
- Publisher: The AI Engineering
- Published: 2026-08-23
- Updated: 2026-08-23
- URL: https://theaiengineering.com/articles/chunking-strategies-for-rag
- Markdown: https://theaiengineering.com/articles/chunking-strategies-for-rag.md

To cite one section, link to its anchor:

- [Why chunk size matters in RAG](https://theaiengineering.com/articles/chunking-strategies-for-rag#why-chunk-size-matters-in-rag)
- [Chunking strategies compared](https://theaiengineering.com/articles/chunking-strategies-for-rag#chunking-strategies-compared)
- [Fixed-size chunking with overlap](https://theaiengineering.com/articles/chunking-strategies-for-rag#fixed-size-chunking-with-overlap)
- [Recursive chunking](https://theaiengineering.com/articles/chunking-strategies-for-rag#recursive-chunking)
- [Document-aware chunking for Markdown, HTML, code, tables, and PDFs](https://theaiengineering.com/articles/chunking-strategies-for-rag#document-aware-chunking-for-markdown-html-code-tables-and-pdfs)
- [Semantic chunking](https://theaiengineering.com/articles/chunking-strategies-for-rag#semantic-chunking)
- [Sentence-window and parent-child retrieval](https://theaiengineering.com/articles/chunking-strategies-for-rag#sentence-window-and-parent-child-retrieval)
- [Late chunking and contextual retrieval](https://theaiengineering.com/articles/chunking-strategies-for-rag#late-chunking-and-contextual-retrieval)
- [RAG chunking best practices](https://theaiengineering.com/articles/chunking-strategies-for-rag#rag-chunking-best-practices)
- [How to evaluate chunking strategies](https://theaiengineering.com/articles/chunking-strategies-for-rag#how-to-evaluate-chunking-strategies)
- [Where to go next](https://theaiengineering.com/articles/chunking-strategies-for-rag#where-to-go-next)
- [Frequently asked questions](https://theaiengineering.com/articles/chunking-strategies-for-rag#faq)
