What is late chunking for embeddings in RAG?
Also called contextual chunk embeddings, late chunking embeddings
Definition
Late chunking is an embedding technique that runs a long-context embedding model over a whole document first, then splits the token embeddings at chunk boundaries and averages each span into one chunk vector.
Late chunking is a way to embed document chunks so that each chunk vector reflects the whole document around it. Instead of splitting text first and embedding each piece alone, it runs a long-context embedding model over the full document, then applies the chunk boundaries to the resulting token embeddings and averages each span. Günther et al. at Jina AI introduced the method in Late Chunking: Contextual Chunk Embeddings Using Long-Context Embedding Models (2024). It matters for chunking because a chunk that says "the new version fixes this bug" otherwise loses which product and which bug it means.
How late chunking works#
Late chunking works by moving the chunking step from before the embedding model to after it. The model reads the whole document, so every token embedding is computed with attention over the full text. Chunk boundaries are then mapped to token ranges, and each range is mean-pooled into one vector. The method needs no extra training.
Any splitter can supply the boundaries: fixed token counts, sentences, or semantic breaks. For documents longer than the model's context window, the paper describes long late chunking, which splits the text into large macro chunks with some overlapping tokens for context and applies late chunking inside each. The authors also propose a fine-tuning method that trains the model with pooling restricted to spans, which improves results further.
Late chunking vs naive chunking#
Naive chunking embeds each chunk as a separate input, so the model never sees the surrounding text. Late chunking embeds the document once and derives every chunk vector from token embeddings that already carry document context. The chunks and the number of vectors stay the same; only the information inside each vector changes.
The paper tested three embedding models on four BEIR retrieval datasets and reports average nDCG@10 for both approaches:
| Chunk boundaries | Naive chunking | Late chunking |
|---|---|---|
| Fixed size, 256 tokens | 52.2 | 54.0 |
| Five sentences | 52.4 | 54.3 |
| Semantic sentence breaks | 52.4 | 53.8 |
The authors found the largest gains at small chunk sizes, where an isolated chunk has the least context of its own. Late chunking did not help on synthetic needle and passkey tasks, where the surrounding text is irrelevant by design, and some reading comprehension tasks favored naive chunking with large chunks. Adding overlap between chunks showed no clear advantage in their experiments.
Late chunking needs the model's token-level outputs, so an embedding API that returns only one pooled vector per input cannot support it on the client side. Some providers expose the idea directly: Jina's embeddings API has a late_chunking flag that concatenates the inputs and processes them as one sequence before splitting, and Perplexity's pplx-embed-context-v1 produces document-aware chunk embeddings. Contextual retrieval reaches a similar goal with an LLM call per chunk and works with any embedding model.
The chunking strategies guide compares late chunking with the other ways to restore lost context.
Related terms
Guides that cover late chunking
Cite this definition
The AI Engineering Editors. "What is late chunking for embeddings in RAG?" The AI Engineering, updated Nov 06, 2026. https://theaiengineering.com/glossary/late-chunking
- Title
- What is late chunking for embeddings in RAG?
- Author
- The AI Engineering Editors
- Publisher
- The AI Engineering
- Updated