What is BM25? Keyword ranking for search and RAG
Also called Okapi BM25, keyword search, BM25 ranking
Definition
BM25 is a keyword ranking function that scores a document against a query using how often each query term appears, how rare the term is across the corpus, and how long the document is.
BM25 is a ranking function that scores how well a document matches a keyword query. It combines three signals: how often each query term appears in the document, how rare that term is across the whole corpus, and how long the document is compared with the average. BM25 needs no training and no model, only an inverted index of terms, and it is the default relevance ranking in Elasticsearch. In RAG systems it is the keyword half of hybrid search, the retriever that catches exact strings that embeddings blur.
How BM25 scores a document#
BM25 scores a document by adding up one weight per query term. Each weight multiplies the term's rarity in the corpus (its inverse document frequency) by a term frequency component that saturates as the term repeats and is normalized for document length. Two parameters, k1 and b, control the saturation and the length normalization.
Robertson and Zaragoza's review, The Probabilistic Relevance Framework: BM25 and Beyond (2009), derives the formula. Per query term, the classic form is:
weight = IDF × tf / (k1 × ((1 − b) + b × dl / avgdl) + tf)
Here tf is the term's count in the document, dl is the document's length, and avgdl is the average length across the corpus.
- Term frequency saturation. The first mention of "refund" in a document is strong evidence of relevance; the tenth adds little. The tf / (k1 + tf) shape rises quickly and then flattens, so no single term's contribution can pass a ceiling however often it repeats. A larger k1 delays the flattening.
- Length normalization. A long document mentions more terms by chance, so BM25 scales term frequency by the document's length relative to the average. Setting b to 1 applies full length normalization, and setting b to 0 switches it off.
The Elasticsearch similarity documentation lists BM25 as the default similarity, with k1 = 1.2 and b = 0.75. Lucene's BM25Similarity uses the same defaults.
BM25 vs vector search#
BM25 matches the exact tokens in a query, while vector search matches meaning through embeddings. BM25 finds rare identifiers, product codes, and names that embeddings can miss, but it cannot connect a query to a document that uses different words for the same idea. Each finds documents the other misses, which is why production RAG pipelines often run both.
| BM25 | Vector search | |
|---|---|---|
| Matches | Exact terms after tokenization | Similar meaning |
| Synonyms and paraphrases | Missed unless the words overlap | Handled |
| Rare identifiers and codes | Strong | Often weak |
| Model required | None | An embedding model |
| Index | Inverted index of terms | Approximate nearest neighbor index |
| Score | Not normalized; depends on the corpus | Usually cosine similarity |
BM25 also holds up on unfamiliar domains. The BEIR benchmark (Thakur et al., 2021) evaluated retrievers zero-shot across 18 datasets and found BM25 a robust baseline, with reranking and late-interaction models scoring best on average at a much higher computational cost. In Anthropic's contextual retrieval experiments, adding a BM25 index to contextual embeddings raised the cut in top-20 retrieval failures from 35% to 49%, and the write-up notes that BM25 is especially effective on queries with unique identifiers or technical terms.
The two scores sit on incompatible scales, so hybrid systems merge the ranked lists by position and often pass the result to a reranker. What is RAG shows where BM25 sits alongside vector retrieval in a full pipeline.
Related terms
Guides that cover bm25
Cite this definition
The AI Engineering Editors. "What is BM25? Keyword ranking for search and RAG." The AI Engineering, updated Nov 06, 2026. https://theaiengineering.com/glossary/bm25
- Title
- What is BM25? Keyword ranking for search and RAG
- Author
- The AI Engineering Editors
- Publisher
- The AI Engineering
- Updated