What is retrieval-augmented generation (RAG)? How RAG works
Retrieval-augmented generation (RAG) is a technique in which an application retrieves relevant documents from an external knowledge source at query time and passes them to a large language model as context for its answer. Retrieval-augmented generation lets a model answer from data it was never trained on, such as last week's policy update or a private codebase, without retraining the model. This explainer covers each stage of a RAG pipeline, hybrid search, agentic RAG, evaluation, and the failure modes behind most production problems.
What is retrieval-augmented generation?#
RAG combines a retriever, which finds relevant passages in an external corpus, with a generator, a language model that writes an answer conditioned on those passages.
The term comes from Lewis et al., "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks", published at NeurIPS 2020. The paper paired parametric memory, a pre-trained seq2seq model, with non-parametric memory, a dense vector index of Wikipedia searched by a neural retriever. The authors trained the retriever and generator jointly, keeping the document encoder fixed, and set the state of the art on three open-domain question answering benchmarks.
Today "RAG" usually names a broader pattern. The retriever and the generator are typically separate, off-the-shelf components: a search index on one side and an instruction-tuned LLM on the other, connected by a prompt. That separation is what makes RAG practical, because a team can change the corpus, the index, or the model independently. Deciding what goes into the prompt, and in what order, is a core part of context engineering.
Why is RAG used in LLMs?#
RAG is used because a language model's built-in knowledge is frozen at training time, generic, and impossible to attribute to a source. Retrieval adds current information, private data, and passages the application can cite. Retrieval also lets the application enforce document permissions at query time, which knowledge stored in model weights cannot support.
- Freshness: updating an index takes minutes; updating model weights takes a training run.
- Private data: internal wikis, tickets, contracts, and code that no public model saw.
- Attribution: answers can link to the passages they used, so users can check them.
- Access control: the retriever can filter by the user's permissions before anything reaches the model.
RAG reduces hallucinations when it retrieves the right passages and the prompt tells the model to answer only from them. It does not remove them. When retrieval returns a wrong or partial passage, the model can produce a fluent answer grounded in the wrong source, which is harder to spot than an obvious fabrication.
How does RAG work? The RAG pipeline step by step#
A RAG pipeline has two paths. The offline path prepares the corpus: parse documents, split them into chunks, embed each chunk, and index the results. The online path answers a query: rewrite and embed the question, retrieve candidate chunks, rerank them, and generate an answer with citations from the best few.
The candidate counts in the figure are illustrative; the right values depend on corpus size, chunk size, and the reranker's latency budget.
Ingestion and parsing#
Parsing is the least glamorous stage and the one that most often caps quality. PDFs with columns, scanned pages, tables, and slide decks lose structure when converted to plain text. Keep metadata with every document from the start: source URL, title, section path, last-modified date, and access permissions. Filters, citations, and freshness rules all depend on it later.
Chunking#
Chunking splits documents into passages that a retriever can return individually. Large chunks dilute relevance and spend context on unrelated text; small chunks lose the surrounding meaning. Common strategies are fixed-size windows with overlap, recursive splitting on headings and paragraphs, and semantic splitting where topic shifts occur. Prepending the document title and section heading to each chunk before embedding often helps the retriever disambiguate passages that look alike.
Embeddings and indexing#
An embedding model maps each chunk to a vector so that semantically similar text lands close together. The same model must embed both chunks and queries. The vectors go into an approximate nearest-neighbor index, such as HNSW, which trades a small amount of recall for fast search. Most teams also build a keyword index over the same chunks for hybrid search. See embeddings for how the vectors are produced.
Retrieval and reranking#
At query time, the application may first rewrite the question, for example by resolving pronouns from earlier conversation turns, then retrieves a broad candidate set with metadata filters applied. A reranker, usually a cross-encoder that reads the query and each passage together, rescores the candidates. Cross-encoders are slower than vector search but more accurate, so they run only on the shortlist.
Generation#
The final prompt combines instructions, the top passages with source identifiers, and the question. Instructions typically require answering only from the provided context, citing passage IDs, and saying so when the context is insufficient. Position matters: Liu et al. found that models used information best when it appeared at the beginning or end of a long input, and performance degraded when it sat in the middle.
Hybrid search: combining keyword and vector retrieval#
Hybrid search runs a keyword retriever, typically BM25, and a vector retriever over the same corpus, then merges the two ranked lists. Keyword search catches exact terms such as product codes, error strings, and names. Vector search catches paraphrases and synonyms. Merging them covers both kinds of query in one result set.
The most common merge method is reciprocal rank fusion (RRF), from Cormack, Clarke, and Büttcher at SIGIR 2009. RRF scores each document by summing 1 / (k + rank) across lists, so it works on ranks and needs no normalization between BM25 scores and cosine similarities. The paper used k = 60. See hybrid search for weighted alternatives.
def reciprocal_rank_fusion(result_lists, k=60):
"""Merge ranked lists of document IDs into one ranking."""
scores = {}
for results in result_lists:
for rank, doc_id in enumerate(results, start=1):
scores[doc_id] = scores.get(doc_id, 0.0) + 1.0 / (k + rank)
return sorted(scores, key=scores.get, reverse=True)
bm25 = ["doc7", "doc2", "doc9"]
dense = ["doc2", "doc4", "doc7"]
print(reciprocal_rank_fusion([bm25, dense]))
# ['doc2', 'doc7', 'doc4', 'doc9']A document that ranks well in both lists, like doc2, beats one that tops a single list.
Do you need a vector database for RAG?#
No. RAG needs a retriever, not a particular database. A small corpus can live in an in-memory index or a vector extension of a database the team already runs, and some corpora work well with keyword search alone. A dedicated vector database earns its place with large corpora, high query volume, or heavy filtering.
When comparing options in the vector database category, the criteria that matter for RAG are native hybrid search, metadata filtering that stays fast under restrictive filters, multi-tenancy for per-customer isolation, how quickly new documents become searchable, and the operational model. Retrieval does not have to be vector search at all: an agent can retrieve by calling a SQL query, a text search tool, or a web search API.
RAG vs agentic RAG#
Classic RAG retrieves once per query on a fixed path: embed the question, fetch the top chunks, generate. Agentic RAG puts retrieval inside an agent loop, so the system can break a question into parts, choose among sources, check whether the results are sufficient, and retrieve again. The trade is better coverage on hard questions for higher latency and cost.
A 2025 survey by Singh et al. defines agentic RAG as embedding autonomous agents into the RAG pipeline, using design patterns such as reflection, planning, tool use, and multi-agent collaboration.
| Classic RAG | Agentic RAG | |
|---|---|---|
| Retrieval calls | One per query | Several, chosen during the run |
| Control flow | Fixed pipeline | Agent loop with tool calls |
| Sources | Usually one index | Indexes, APIs, SQL, web search |
| Latency | Low and predictable | Higher and variable |
| Model calls | One generation call | Several: plan, grade, answer |
| Typical failures | Missed or bad chunks | Loops, tool errors, runaway cost |
| Good fit | Docs Q&A, support answers | Multi-hop research, cross-source analysis |
In agentic RAG, retrieval becomes a tool the model calls through tool calling, and search tools are often exposed to agents as MCP servers. The AI agents explainer covers the loop itself.
How to evaluate a RAG system#
Evaluate retrieval and generation separately, then end to end. Retrieval metrics ask whether the right passages reached the model. Generation metrics ask whether the answer is faithful to those passages and responsive to the question. Separating the two shows which stage to fix when quality drops, instead of changing prompts at random.
| Stage | Metric | Question it answers |
|---|---|---|
| Retrieval | Recall@k | Did a relevant passage appear in the top k? |
| Retrieval | MRR or nDCG | How high did relevant passages rank? |
| Retrieval | Context precision | How much retrieved text was relevant? |
| Generation | Faithfulness | Is each claim supported by the retrieved context? |
| Generation | Answer relevance | Does the answer address the question asked? |
| End to end | Correctness | Does the answer match a reference answer? |
| End to end | Citation accuracy | Do cited passages support the claims? |
Build the test set from real user queries, label the passages that should be retrieved, and keep it versioned. Faithfulness and relevance are usually scored with LLM-as-a-judge prompts, and frameworks such as Ragas score RAG pipelines without human-written reference answers. The LLM evaluation guide covers judge calibration and regression testing.
Common RAG failure modes#
Most RAG failures trace back to retrieval rather than generation: the right passage was never indexed, was split across chunks, ranked below the cutoff, or was filtered out. The remaining failures happen in generation, where the model received the right context and then ignored it, misread it, or contradicted it with its built-in knowledge.
| Symptom | Likely cause | Fix |
|---|---|---|
| "Not found" for content that exists | Parser dropped tables or scanned pages | Better parsing, OCR, ingest audits |
| Right document, wrong detail | Chunk boundary split the answer | Structure-aware chunks, overlap, parent retrieval |
| Misses product codes and names | Vector-only retrieval | Add keyword search |
| Relevant passage ignored | Too many chunks, buried mid-context | Rerank, send fewer chunks, order by score |
| Outdated answers | Stale index, no date metadata | Incremental re-indexing, recency filters |
| Users see restricted content | No permission filter in retrieval | Apply access filters in the retriever |
| Output follows text inside a document | Indirect prompt injection | Treat retrieved text as data, add guardrails |
The last row is a security issue, not a quality issue. Retrieved documents are untrusted input, and an instruction hidden in a web page or a ticket can steer the model. See prompt injection for mitigations.
Is RAG still relevant with long context windows?#
Yes. A long context window lets some applications skip retrieval for a small, stable corpus, but most corpora are larger than any context window, change often, or carry permissions. Long prompts also raise per-request cost and latency, and models do not use every position in a long input equally well.
For small, static knowledge bases, preloading the corpus and reusing it with prompt caching is a real alternative, often called cache-augmented generation. The RAG vs fine-tuning comparison covers that option alongside fine-tuning and explains when each one fits.
Where to go next#
To decide between retrieval and training, read RAG vs fine-tuning. To measure a pipeline you already run, start with the LLM evaluation guide. The RAG topic hub collects every RAG guide and glossary entry on the site.
Frequently asked questions
What is RAG in AI?
- In AI, RAG stands for retrieval-augmented generation, a technique that connects a large language model to an external knowledge source. At query time, the application searches that source for passages relevant to the question and includes them in the prompt, so the model answers from those passages instead of relying only on what it learned during training.
Does RAG reduce hallucinations?
- RAG reduces hallucinations when retrieval finds the right passages and the prompt instructs the model to answer only from them. RAG does not eliminate hallucinations: if retrieval returns wrong or partial passages, the model can produce a fluent answer grounded in the wrong source. Measuring faithfulness and retrieval recall separately shows which of the two failed.
What is chunking in RAG?
- Chunking is the step in a RAG pipeline that splits documents into smaller passages before embedding and indexing them. Chunk size and boundaries decide what a retriever can return: chunks that are too large dilute relevance, and chunks that are too small lose context. Common strategies are fixed-size with overlap, recursive splitting on structure, and semantic splitting.
What is reranking in RAG?
- Reranking is a second retrieval stage that rescores the top candidates from a fast first-stage search using a more accurate model, usually a cross-encoder that reads the query and each passage together. The retriever might return 50 to 100 candidates, and the reranker keeps the few most relevant, which improves precision before the passages reach the language model.
Is RAG dead?
- No. Long context windows and agentic search changed how RAG is built, but they did not remove the need for retrieval. Most knowledge bases are larger than any context window, change often, and carry access permissions, and sending an entire corpus with every request is slow and expensive. Retrieval still selects which information reaches the model.
Terms in this guide
Cite this guide
The AI Engineering Editors. "What is retrieval-augmented generation (RAG)? How RAG works." The AI Engineering, updated Jul 09, 2026. https://theaiengineering.com/articles/what-is-rag
- Title
- What is retrieval-augmented generation (RAG)? How RAG works
- Author
- The AI Engineering Editors
- Publisher
- The AI Engineering
- Published
- Updated