What is late chunking for embeddings in RAG?

Also called contextual chunk embeddings, late chunking embeddings

Definition

Late chunking is an embedding technique that runs a long-context embedding model over a whole document first, then splits the token embeddings at chunk boundaries and averages each span into one chunk vector.

Late chunking is a way to embed document chunks so that each chunk vector reflects the whole document around it. Instead of splitting text first and embedding each piece alone, it runs a long-context embedding model over the full document, then applies the chunk boundaries to the resulting token embeddings and averages each span. Günther et al. at Jina AI introduced the method in Late Chunking: Contextual Chunk Embeddings Using Long-Context Embedding Models (2024). It matters for chunking because a chunk that says "the new version fixes this bug" otherwise loses which product and which bug it means.

How late chunking works#

Late chunking works by moving the chunking step from before the embedding model to after it. The model reads the whole document, so every token embedding is computed with attention over the full text. Chunk boundaries are then mapped to token ranges, and each range is mean-pooled into one vector. The method needs no extra training.

Naive chunking splits text, then embeds each chunk alone; late chunking embeds the whole document, then splits token embeddings and pools each span. Naive chunkingLate chunkingDocumentSplit into chunksEmbed each chunkSees one chunkChunk vectorsDocumentEmbed all tokensFull contextSplit token vectorsAt chunk boundariesMean-pool spansOne vector per chunk
Figure 1: Naive chunking pools each chunk in isolation; late chunking pools spans of token embeddings computed over the whole document.

Any splitter can supply the boundaries: fixed token counts, sentences, or semantic breaks. For documents longer than the model's context window, the paper describes long late chunking, which splits the text into large macro chunks with some overlapping tokens for context and applies late chunking inside each. The authors also propose a fine-tuning method that trains the model with pooling restricted to spans, which improves results further.

Late chunking vs naive chunking#

Naive chunking embeds each chunk as a separate input, so the model never sees the surrounding text. Late chunking embeds the document once and derives every chunk vector from token embeddings that already carry document context. The chunks and the number of vectors stay the same; only the information inside each vector changes.

The paper tested three embedding models on four BEIR retrieval datasets and reports average nDCG@10 for both approaches:

Chunk boundaries Naive chunking Late chunking
Fixed size, 256 tokens 52.2 54.0
Five sentences 52.4 54.3
Semantic sentence breaks 52.4 53.8

The authors found the largest gains at small chunk sizes, where an isolated chunk has the least context of its own. Late chunking did not help on synthetic needle and passkey tasks, where the surrounding text is irrelevant by design, and some reading comprehension tasks favored naive chunking with large chunks. Adding overlap between chunks showed no clear advantage in their experiments.

Late chunking needs the model's token-level outputs, so an embedding API that returns only one pooled vector per input cannot support it on the client side. Some providers expose the idea directly: Jina's embeddings API has a late_chunking flag that concatenates the inputs and processes them as one sequence before splitting, and Perplexity's pplx-embed-context-v1 produces document-aware chunk embeddings. Contextual retrieval reaches a similar goal with an LLM call per chunk and works with any embedding model.

The chunking strategies guide compares late chunking with the other ways to restore lost context.

Cite this definition

The AI Engineering Editors. "What is late chunking for embeddings in RAG?" The AI Engineering, updated Nov 06, 2026. https://theaiengineering.com/glossary/late-chunking

Title
What is late chunking for embeddings in RAG?
Author
The AI Engineering Editors
Publisher
The AI Engineering
Updated
View as Markdown

Part of the RAG and vector search hub · Updated