What is reranking in RAG and vector search?
Also called re-ranking, reranker, two-stage retrieval
Definition
Reranking is a second retrieval stage that rescores an initial list of candidate documents with a more accurate but slower model, usually a cross-encoder, and keeps only the most relevant ones.
Reranking is a second stage in a retrieval pipeline that takes the candidates returned by a fast first-stage search and reorders them with a slower, more accurate model. The first stage, such as vector search or hybrid search, returns a broad candidate list, for example the top 100 results. The reranker scores each candidate against the query and passes only the top few to the language model.
How reranking works#
Reranking works by scoring each query-document pair jointly instead of comparing two precomputed vectors. Most rerankers are cross-encoders: the query and the document go into one transformer together, so attention can compare every query token with every document token, and the model outputs a single relevance score. The Sentence Transformers documentation recommends this two-stage pattern: retrieve candidates with a bi-encoder, then re-rank them with a cross-encoder.
Other approaches include late-interaction models such as ColBERT, which compare stored per-token vectors, and LLM-based rerankers that prompt a language model to judge relevance. In Anthropic's contextual retrieval tests, adding a reranking step that scored the top 150 chunks and kept 20 raised the reduction in top-20 retrieval failures from 49% to 67%.
Reranker vs embedding model#
A reranker and an embedding model trade speed for precision in opposite directions.
| Embedding model (bi-encoder) | Reranker (cross-encoder) | |
|---|---|---|
| Input | Query or document, encoded alone | Query and document, encoded together |
| Output | A vector, computed ahead of time | A relevance score for one pair |
| Work per query | One embedding plus an index lookup | One model pass per candidate |
| Practical scale | Millions of documents | Tens to hundreds of candidates |
| Strength | Recall | Precision |
A cross-encoder produces no reusable vector, so its scores cannot be indexed; it only works on a short list that another retriever has already produced. That is why reranking is a second stage rather than a replacement for embedding search.
Reranking adds latency to every query, so measure whether the precision gain is worth it on an evaluation set drawn from real RAG traffic. What is RAG shows where reranking fits in the pipeline.
Related terms
Guides that cover reranking
Cite this definition
The AI Engineering Editors. "What is reranking in RAG and vector search?" The AI Engineering, updated Nov 06, 2026. https://theaiengineering.com/glossary/reranking
- Title
- What is reranking in RAG and vector search?
- Author
- The AI Engineering Editors
- Publisher
- The AI Engineering
- Updated