---
title: "What is retrieval-augmented generation (RAG)? How RAG works"
description: "Retrieval-augmented generation (RAG) grounds LLM answers in your documents. Learn the RAG pipeline, hybrid search, agentic RAG, evaluation, and failure modes."
summary: "Retrieval-augmented generation (RAG) retrieves relevant documents at query time and passes them to a language model as context for its answer. This explainer walks through each pipeline stage, hybrid search, agentic RAG, RAG evaluation, and the failure modes that break RAG in production."
type: "explainer"
canonical_url: "https://theaiengineering.com/articles/what-is-rag"
markdown_url: "https://theaiengineering.com/articles/what-is-rag.md"
topic: "RAG and vector search"
author: "The AI Engineering Editors"
publisher: "The AI Engineering"
published: "2026-07-09"
updated: "2026-07-09"
keywords: ["retrieval augmented generation","what is rag in ai","rag vs agentic rag","vector database for rag","rag pipeline"]
citation: "The AI Engineering Editors. \"What is retrieval-augmented generation (RAG)? How RAG works.\" The AI Engineering, updated Jul 09, 2026. https://theaiengineering.com/articles/what-is-rag"
---

# What is retrieval-augmented generation (RAG)? How RAG works

## TL;DR

- Retrieval-augmented generation (RAG) retrieves relevant passages from an external corpus at query time and adds them to the prompt, so a language model can answer from current or private data without retraining.
- Lewis et al. introduced the term in a 2020 paper that combined a pre-trained seq2seq generator (parametric memory) with a dense vector index of Wikipedia (non-parametric memory).
- A RAG pipeline has an offline path (parsing, chunking, embedding, indexing) and an online path (query rewriting, retrieval, reranking, generation), and many quality problems start in the offline path.
- Hybrid search runs keyword retrieval such as BM25 alongside vector search and merges the results, often with reciprocal rank fusion, because each method finds documents the other misses.
- Agentic RAG puts retrieval inside an agent loop that can rewrite queries, pick sources, and retrieve again, which helps on multi-step questions at the cost of more latency and more model calls.

Retrieval-augmented generation (RAG) is a technique in which an application retrieves relevant documents from an external knowledge source at query time and passes them to a large language model as context for its answer. Retrieval-augmented generation lets a model answer from data it was never trained on, such as last week's policy update or a private codebase, without retraining the model. This explainer covers each stage of a RAG pipeline, hybrid search, agentic RAG, evaluation, and the failure modes behind most production problems.

## What is retrieval-augmented generation?

RAG combines a retriever, which finds relevant passages in an external corpus, with a generator, a language model that writes an answer conditioned on those passages.

The term comes from [Lewis et al., "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks"](https://arxiv.org/abs/2005.11401), published at NeurIPS 2020. The paper paired parametric memory, a pre-trained seq2seq model, with non-parametric memory, a dense vector index of Wikipedia searched by a neural retriever. The authors trained the retriever and generator jointly, keeping the document encoder fixed, and set the state of the art on three open-domain question answering benchmarks.

Today "RAG" usually names a broader pattern. The retriever and the generator are typically separate, off-the-shelf components: a search index on one side and an instruction-tuned LLM on the other, connected by a prompt. That separation is what makes RAG practical, because a team can change the corpus, the index, or the model independently. Deciding what goes into the prompt, and in what order, is a core part of [context engineering](https://theaiengineering.com/articles/context-engineering).

## Why is RAG used in LLMs?

RAG is used because a language model's built-in knowledge is frozen at training time, generic, and impossible to attribute to a source. Retrieval adds current information, private data, and passages the application can cite. Retrieval also lets the application enforce document permissions at query time, which knowledge stored in model weights cannot support.

- **Freshness:** updating an index takes minutes; updating model weights takes a training run.
- **Private data:** internal wikis, tickets, contracts, and code that no public model saw.
- **Attribution:** answers can link to the passages they used, so users can check them.
- **Access control:** the retriever can filter by the user's permissions before anything reaches the model.

RAG reduces [hallucinations](https://theaiengineering.com/glossary/hallucination) when it retrieves the right passages and the prompt tells the model to answer only from them. It does not remove them. When retrieval returns a wrong or partial passage, the model can produce a fluent answer grounded in the wrong source, which is harder to spot than an obvious fabrication.

## How does RAG work? The RAG pipeline step by step

A RAG pipeline has two paths. The offline path prepares the corpus: parse documents, split them into chunks, embed each chunk, and index the results. The online path answers a query: rewrite and embed the question, retrieve candidate chunks, rerank them, and generate an answer with citations from the best few.

**Figure 1: The offline indexing path and the online query path of a RAG pipeline.**

Offline, documents are parsed, chunked, embedded, and indexed; online, a query is rewritten, embedded, searched against the index, reranked, and used to generate a cited answer.

Components:

- **Parse and clean**
- **Chunk**
- **Embed chunks**
- **Index**: Vector and keyword
- **User query**
- **Rewrite query**
- **Embed query**
- **Retrieve**: Top 50-100
- **Rerank**: Keep top 5-10
- **Generate**: From the best few
- **Answer**: With citations

Groups:

- Offline: build the index: Parse and clean, Chunk, Embed chunks, Index
- Online: answer a query: User query, Rewrite query, Embed query, Retrieve, Rerank, Generate, Answer

Connections:

- Parse and clean → Chunk
- Chunk → Embed chunks
- Embed chunks → Index
- Index → Retrieve: search
- User query → Rewrite query
- Rewrite query → Embed query
- Embed query → Retrieve
- Retrieve → Rerank
- Rerank → Generate
- Generate → Answer

The candidate counts in the figure are illustrative; the right values depend on corpus size, chunk size, and the reranker's latency budget.

### Ingestion and parsing

Parsing is the least glamorous stage and the one that most often caps quality. PDFs with columns, scanned pages, tables, and slide decks lose structure when converted to plain text. Keep metadata with every document from the start: source URL, title, section path, last-modified date, and access permissions. Filters, citations, and freshness rules all depend on it later.

### Chunking

[Chunking](https://theaiengineering.com/glossary/chunking) splits documents into passages that a retriever can return individually. Large chunks dilute relevance and spend context on unrelated text; small chunks lose the surrounding meaning. Common strategies are fixed-size windows with overlap, recursive splitting on headings and paragraphs, and semantic splitting where topic shifts occur. Prepending the document title and section heading to each chunk before embedding often helps the retriever disambiguate passages that look alike.

### Embeddings and indexing

An embedding model maps each chunk to a vector so that semantically similar text lands close together. The same model must embed both chunks and queries. The vectors go into an [approximate nearest-neighbor index](https://theaiengineering.com/glossary/approximate-nearest-neighbor), such as HNSW, which trades a small amount of recall for fast search. Most teams also build a keyword index over the same chunks for hybrid search. See [embeddings](https://theaiengineering.com/glossary/embeddings) for how the vectors are produced.

### Retrieval and reranking

At query time, the application may first rewrite the question, for example by resolving pronouns from earlier conversation turns, then retrieves a broad candidate set with metadata filters applied. A [reranker](https://theaiengineering.com/glossary/reranking), usually a cross-encoder that reads the query and each passage together, rescores the candidates. Cross-encoders are slower than vector search but more accurate, so they run only on the shortlist.

### Generation

The final prompt combines instructions, the top passages with source identifiers, and the question. Instructions typically require answering only from the provided context, citing passage IDs, and saying so when the context is insufficient. Position matters: [Liu et al. found](https://arxiv.org/abs/2307.03172) that models used information best when it appeared at the beginning or end of a long input, and performance degraded when it sat in the middle.

## Hybrid search: combining keyword and vector retrieval

Hybrid search runs a keyword retriever, typically [BM25](https://theaiengineering.com/glossary/bm25), and a vector retriever over the same corpus, then merges the two ranked lists. Keyword search catches exact terms such as product codes, error strings, and names. Vector search catches paraphrases and synonyms. Merging them covers both kinds of query in one result set.

The most common merge method is reciprocal rank fusion (RRF), from Cormack, Clarke, and Büttcher at SIGIR 2009. RRF scores each document by summing `1 / (k + rank)` across lists, so it works on ranks and needs no normalization between BM25 scores and cosine similarities. The paper used `k = 60`. See [hybrid search](https://theaiengineering.com/glossary/hybrid-search) for weighted alternatives.

```python
def reciprocal_rank_fusion(result_lists, k=60):
    """Merge ranked lists of document IDs into one ranking."""
    scores = {}
    for results in result_lists:
        for rank, doc_id in enumerate(results, start=1):
            scores[doc_id] = scores.get(doc_id, 0.0) + 1.0 / (k + rank)
    return sorted(scores, key=scores.get, reverse=True)

bm25 = ["doc7", "doc2", "doc9"]
dense = ["doc2", "doc4", "doc7"]
print(reciprocal_rank_fusion([bm25, dense]))
# ['doc2', 'doc7', 'doc4', 'doc9']
```

A document that ranks well in both lists, like `doc2`, beats one that tops a single list.

## Do you need a vector database for RAG?

No. RAG needs a retriever, not a particular database. A small corpus can live in an in-memory index or a vector extension of a database the team already runs, and some corpora work well with keyword search alone. A dedicated [vector database](https://theaiengineering.com/glossary/vector-database) earns its place with large corpora, high query volume, or heavy filtering.

When comparing options in the [vector database category](https://theaiengineering.com/platforms/vector-databases), the criteria that matter for RAG are native hybrid search, metadata filtering that stays fast under restrictive filters, multi-tenancy for per-customer isolation, how quickly new documents become searchable, and the operational model. Retrieval does not have to be vector search at all: an agent can retrieve by calling a SQL query, a text search tool, or a web search API.

## RAG vs agentic RAG

Classic RAG retrieves once per query on a fixed path: embed the question, fetch the top chunks, generate. [Agentic RAG](https://theaiengineering.com/glossary/agentic-rag) puts retrieval inside an agent loop, so the system can break a question into parts, choose among sources, check whether the results are sufficient, and retrieve again. The trade is better coverage on hard questions for higher latency and cost.

A [2025 survey by Singh et al.](https://arxiv.org/abs/2501.09136) defines agentic RAG as embedding autonomous agents into the RAG pipeline, using design patterns such as reflection, planning, tool use, and multi-agent collaboration.

| | Classic RAG | Agentic RAG |
|---|---|---|
| Retrieval calls | One per query | Several, chosen during the run |
| Control flow | Fixed pipeline | Agent loop with tool calls |
| Sources | Usually one index | Indexes, APIs, SQL, web search |
| Latency | Low and predictable | Higher and variable |
| Model calls | One generation call | Several: plan, grade, answer |
| Typical failures | Missed or bad chunks | Loops, tool errors, runaway cost |
| Good fit | Docs Q&A, support answers | Multi-hop research, cross-source analysis |

In agentic RAG, retrieval becomes a tool the model calls through tool calling, and search tools are often exposed to agents as [MCP servers](https://theaiengineering.com/articles/what-is-model-context-protocol). The [AI agents explainer](https://theaiengineering.com/articles/what-are-ai-agents) covers the loop itself.

## How to evaluate a RAG system

Evaluate retrieval and generation separately, then end to end. Retrieval metrics ask whether the right passages reached the model. Generation metrics ask whether the answer is [faithful](https://theaiengineering.com/glossary/faithfulness) to those passages and responsive to the question. Separating the two shows which stage to fix when quality drops, instead of changing prompts at random.

| Stage | Metric | Question it answers |
|---|---|---|
| Retrieval | [Recall@k](https://theaiengineering.com/glossary/recall-at-k) | Did a relevant passage appear in the top k? |
| Retrieval | MRR or nDCG | How high did relevant passages rank? |
| Retrieval | Context precision | How much retrieved text was relevant? |
| Generation | Faithfulness | Is each claim supported by the retrieved context? |
| Generation | Answer relevance | Does the answer address the question asked? |
| End to end | Correctness | Does the answer match a reference answer? |
| End to end | Citation accuracy | Do cited passages support the claims? |

Build the test set from real user queries, label the passages that should be retrieved, and keep it versioned. Faithfulness and relevance are usually scored with LLM-as-a-judge prompts, and frameworks such as [Ragas](https://arxiv.org/abs/2309.15217) score RAG pipelines without human-written reference answers. The [LLM evaluation guide](https://theaiengineering.com/articles/llm-evaluation) covers judge calibration and regression testing.

## Common RAG failure modes

Most RAG failures trace back to retrieval rather than generation: the right passage was never indexed, was split across chunks, ranked below the cutoff, or was filtered out. The remaining failures happen in generation, where the model received the right context and then ignored it, misread it, or contradicted it with its built-in knowledge.

| Symptom | Likely cause | Fix |
|---|---|---|
| "Not found" for content that exists | Parser dropped tables or scanned pages | Better parsing, OCR, ingest audits |
| Right document, wrong detail | Chunk boundary split the answer | Structure-aware chunks, overlap, parent retrieval |
| Misses product codes and names | Vector-only retrieval | Add keyword search |
| Relevant passage ignored | Too many chunks, buried mid-context | Rerank, send fewer chunks, order by score |
| Outdated answers | Stale index, no date metadata | Incremental re-indexing, recency filters |
| Users see restricted content | No permission filter in retrieval | Apply access filters in the retriever |
| Output follows text inside a document | Indirect prompt injection | Treat retrieved text as data, add guardrails |

The last row is a security issue, not a quality issue. Retrieved documents are untrusted input, and an instruction hidden in a web page or a ticket can steer the model. See [prompt injection](https://theaiengineering.com/glossary/prompt-injection) for mitigations.

## Is RAG still relevant with long context windows?

Yes. A long context window lets some applications skip retrieval for a small, stable corpus, but most corpora are larger than any context window, change often, or carry permissions. Long prompts also raise per-request cost and latency, and models do not use every position in a long input equally well.

For small, static knowledge bases, preloading the corpus and reusing it with prompt caching is a real alternative, often called cache-augmented generation. The RAG vs fine-tuning comparison covers that option alongside fine-tuning and explains when each one fits.

## Where to go next

To decide between retrieval and training, read [RAG vs fine-tuning](https://theaiengineering.com/articles/rag-vs-fine-tuning). To measure a pipeline you already run, start with the [LLM evaluation guide](https://theaiengineering.com/articles/llm-evaluation). The [RAG topic hub](https://theaiengineering.com/topics/rag) collects every RAG guide and glossary entry on the site.

## Frequently asked questions

### What is RAG in AI?

In AI, RAG stands for retrieval-augmented generation, a technique that connects a large language model to an external knowledge source. At query time, the application searches that source for passages relevant to the question and includes them in the prompt, so the model answers from those passages instead of relying only on what it learned during training.

### Does RAG reduce hallucinations?

RAG reduces hallucinations when retrieval finds the right passages and the prompt instructs the model to answer only from them. RAG does not eliminate hallucinations: if retrieval returns wrong or partial passages, the model can produce a fluent answer grounded in the wrong source. Measuring faithfulness and retrieval recall separately shows which of the two failed.

### What is chunking in RAG?

Chunking is the step in a RAG pipeline that splits documents into smaller passages before embedding and indexing them. Chunk size and boundaries decide what a retriever can return: chunks that are too large dilute relevance, and chunks that are too small lose context. Common strategies are fixed-size with overlap, recursive splitting on structure, and semantic splitting.

### What is reranking in RAG?

Reranking is a second retrieval stage that rescores the top candidates from a fast first-stage search using a more accurate model, usually a cross-encoder that reads the query and each passage together. The retriever might return 50 to 100 candidates, and the reranker keeps the few most relevant, which improves precision before the passages reach the language model.

### Is RAG dead?

No. Long context windows and agentic search changed how RAG is built, but they did not remove the need for retrieval. Most knowledge bases are larger than any context window, change often, and carry access permissions, and sending an entire corpus with every request is slow and expensive. Retrieval still selects which information reaches the model.

## Terms in this guide

- [Retrieval-augmented generation (RAG)](https://theaiengineering.com/glossary/retrieval-augmented-generation): Retrieval-augmented generation (RAG) is a technique that retrieves relevant passages from an external knowledge source at query time and adds them to a language model's prompt so the answer is grounded in that data.
- [Embeddings](https://theaiengineering.com/glossary/embeddings): Embeddings are fixed-length vectors of numbers that represent data such as text or images so that items with similar meaning sit close together, which lets software compare meaning with arithmetic.
- [Vector database](https://theaiengineering.com/glossary/vector-database): A vector database is a database built to store embedding vectors and quickly return the vectors most similar to a query vector, usually with approximate nearest neighbor indexes such as HNSW.
- [Chunking](https://theaiengineering.com/glossary/chunking): Chunking is the step in a retrieval pipeline that splits documents into smaller passages, called chunks, so each one can be embedded, indexed, and retrieved on its own.
- [Reranking](https://theaiengineering.com/glossary/reranking): Reranking is a second retrieval stage that rescores an initial list of candidate documents with a more accurate but slower model, usually a cross-encoder, and keeps only the most relevant ones.
- [Hybrid search](https://theaiengineering.com/glossary/hybrid-search): Hybrid search is a retrieval method that runs keyword search, typically BM25, and vector similarity search on the same query and merges the two ranked lists into one result set.
- [Hallucination](https://theaiengineering.com/glossary/hallucination): A hallucination is output from a language model that is fluent and plausible but factually false, unsupported by the provided sources, or invented outright, such as a fabricated citation or package name.
- [Approximate nearest neighbor search](https://theaiengineering.com/glossary/approximate-nearest-neighbor): Approximate nearest neighbor (ANN) search is a family of indexing methods that find vectors close to a query vector without comparing it to every stored vector, accepting that some true nearest neighbors are missed.
- [Context rot](https://theaiengineering.com/glossary/context-rot): Context rot is the decline in a language model's accuracy and recall as the number of tokens in its input grows, even when the information a task needs is present in the context.
- [BM25](https://theaiengineering.com/glossary/bm25): BM25 is a keyword ranking function that scores a document against a query using how often each query term appears, how rare the term is across the corpus, and how long the document is.
- [Cosine similarity](https://theaiengineering.com/glossary/cosine-similarity): Cosine similarity is a measure of how closely two vectors point in the same direction, computed as their dot product divided by the product of their lengths, with values from -1 to 1.
- [Agentic RAG](https://theaiengineering.com/glossary/agentic-rag): Agentic RAG is a retrieval-augmented generation design in which an AI agent treats retrieval as a tool, deciding when to search, what query to send, which source to use, and whether to search again.
- [Faithfulness](https://theaiengineering.com/glossary/faithfulness): Faithfulness is an evaluation metric that measures what share of the claims in a generated answer can be inferred from the context the model was given, usually scored from 0 to 1.
- [Recall@k](https://theaiengineering.com/glossary/recall-at-k): Recall@k is a retrieval metric that measures the fraction of a query's relevant documents that appear in the top k results, averaged over a set of test queries.
- [Golden dataset](https://theaiengineering.com/glossary/golden-dataset): A golden dataset is a curated, versioned set of representative inputs, each paired with an expected output or grading criteria, that a team treats as ground truth when evaluating an LLM application.
- [Large language model (LLM)](https://theaiengineering.com/glossary/large-language-model): A large language model (LLM) is a neural network, usually a transformer with billions of parameters, trained on web-scale text to predict the next token and then adapted to follow instructions.
- [Tool calling](https://theaiengineering.com/glossary/tool-calling): Tool calling is an LLM capability in which the model, given a list of tool definitions, outputs a structured request naming a tool and its arguments, which the application executes and returns.
- [Context window](https://theaiengineering.com/glossary/context-window): A context window is the maximum number of tokens a language model can take into account in a single request, including the system prompt, conversation history, tool results, and the generated output.
- [Prompt caching](https://theaiengineering.com/glossary/prompt-caching): Prompt caching is an LLM API and inference-server feature that stores the computed attention state for a repeated prompt prefix, so later requests sharing that prefix skip reprocessing it.
- [LLM-as-a-judge](https://theaiengineering.com/glossary/llm-as-a-judge): LLM-as-a-judge is an evaluation method in which a language model scores or compares the outputs of an LLM application against a rubric, a reference answer, or another response, replacing or supplementing human graders.

## Related guides

- [RAG vs fine-tuning: how to choose for your LLM application](https://theaiengineering.com/articles/rag-vs-fine-tuning): RAG vs fine-tuning: RAG adds knowledge at query time, while fine-tuning changes model behavior. Compare freshness, cost, latency, data needs, LoRA, and hybrids.
- [Chunking strategies for RAG: how to choose, tune, and evaluate them](https://theaiengineering.com/articles/chunking-strategies-for-rag): Chunking strategies for RAG compared: fixed-size, recursive, document-aware, semantic, parent-child, and late chunking, with tested Python and an eval method.
- [What is context engineering? A practical guide for AI agents](https://theaiengineering.com/articles/context-engineering): Context engineering is the practice of choosing what fills an LLM context window. Learn its components, why context rot happens, and how to measure results.
- [LLM evaluation: a practical guide to evals, metrics, and judges](https://theaiengineering.com/articles/llm-evaluation): LLM evaluation for engineers: build eval datasets, pick metrics, calibrate LLM-as-a-judge, evaluate RAG and agents, and gate every release in CI.

Part of the [RAG and vector search](https://theaiengineering.com/topics/rag) topic hub.

## Cite this guide

The AI Engineering Editors. "What is retrieval-augmented generation (RAG)? How RAG works." The AI Engineering, updated Jul 09, 2026. https://theaiengineering.com/articles/what-is-rag

- Title: What is retrieval-augmented generation (RAG)? How RAG works
- Author: The AI Engineering Editors
- Publisher: The AI Engineering
- Published: 2026-07-09
- Updated: 2026-07-09
- URL: https://theaiengineering.com/articles/what-is-rag
- Markdown: https://theaiengineering.com/articles/what-is-rag.md

To cite one section, link to its anchor:

- [What is retrieval-augmented generation?](https://theaiengineering.com/articles/what-is-rag#what-is-retrieval-augmented-generation)
- [Why is RAG used in LLMs?](https://theaiengineering.com/articles/what-is-rag#why-is-rag-used-in-llms)
- [How does RAG work? The RAG pipeline step by step](https://theaiengineering.com/articles/what-is-rag#how-does-rag-work-the-rag-pipeline-step-by-step)
- [Hybrid search: combining keyword and vector retrieval](https://theaiengineering.com/articles/what-is-rag#hybrid-search-combining-keyword-and-vector-retrieval)
- [Do you need a vector database for RAG?](https://theaiengineering.com/articles/what-is-rag#do-you-need-a-vector-database-for-rag)
- [RAG vs agentic RAG](https://theaiengineering.com/articles/what-is-rag#rag-vs-agentic-rag)
- [How to evaluate a RAG system](https://theaiengineering.com/articles/what-is-rag#how-to-evaluate-a-rag-system)
- [Common RAG failure modes](https://theaiengineering.com/articles/what-is-rag#common-rag-failure-modes)
- [Is RAG still relevant with long context windows?](https://theaiengineering.com/articles/what-is-rag#is-rag-still-relevant-with-long-context-windows)
- [Where to go next](https://theaiengineering.com/articles/what-is-rag#where-to-go-next)
- [Frequently asked questions](https://theaiengineering.com/articles/what-is-rag#faq)
