---
title: "Semantic caching for LLMs: how it works and when it pays off"
description: "Semantic caching returns a stored LLM response for prompts with similar meaning. Learn how a semantic cache works, how to tune it, and what never to cache."
summary: "A semantic cache embeds each prompt, searches earlier prompts by vector similarity, and replays the stored response when the closest match clears a threshold. This guide covers cache keys, thresholds, expiry, failure modes, measurement, and where semantic caching runs, with a Bifrost example."
type: "guide"
canonical_url: "https://theaiengineering.com/articles/semantic-caching"
markdown_url: "https://theaiengineering.com/articles/semantic-caching.md"
topic: "AI gateways and LLM routing"
author: "The AI Engineering Editors"
publisher: "The AI Engineering"
published: "2026-11-16"
updated: "2026-11-16"
keywords: ["semantic caching","semantic cache","llm caching"]
citation: "The AI Engineering Editors. \"Semantic caching for LLMs: how it works and when it pays off.\" The AI Engineering, updated Nov 16, 2026. https://theaiengineering.com/articles/semantic-caching"
---

# Semantic caching for LLMs: how it works and when it pays off

## TL;DR

- Semantic caching embeds each prompt, searches a vector index of earlier prompts, and returns the stored response when the closest match clears a similarity threshold, so the model is never called on a hit.
- Semantic caching differs from exact-match response caching, which needs an identical request, and from provider prompt caching, which still calls the model and reuses only the computed state of a repeated prompt prefix.
- A safe semantic cache matches tenant, model, generation parameters, and system prompt exactly, compares only the user's wording by similarity, and puts an expiry on every entry.
- Responses that contain tool calls, personalized or time-sensitive answers, and multi-turn agent traffic are the main things to keep out of a semantic cache.
- Tune the threshold on labeled prompt pairs from your own traffic and sample cached answers for review, because a false hit looks like a fast, successful request in every dashboard.

Semantic caching is a technique that stores LLM responses and returns a stored response when a new prompt is close enough in meaning to an earlier one, measured by embedding similarity. A hit skips the model call and saves its full cost and most of its latency; a false hit returns a fluent answer to a different question. This guide explains how semantic caching differs from exact-match and provider prompt caching, how to design the cache key, threshold, and expiry, what not to cache, how to measure hit rate and answer quality, and where a semantic cache runs, with Bifrost as a worked example. It is part of our [AI gateways and LLM routing](https://theaiengineering.com/topics/ai-gateways) hub.

## What is semantic caching?

[Semantic caching](https://theaiengineering.com/glossary/semantic-caching) is a response cache for LLM applications that matches requests by meaning, serving a stored answer when a past prompt's embedding is similar enough to the new one.

A conventional cache matches keys byte for byte, so "How do I reset my password?" and "password reset steps" are two separate misses. A semantic cache converts each prompt into an [embedding](https://theaiengineering.com/glossary/embeddings), a vector that places texts with similar meaning close together, and can serve both prompts from one entry.

A semantic cache has three parts:

- **An embedding model** that turns the prompt text into a vector.
- **A vector index**, usually in a [vector database](https://theaiengineering.com/glossary/vector-database) or a search-enabled key-value store such as Redis, that finds the nearest stored prompt.
- **A response store** that keeps each response with its model, tenant, and expiry.

The design trades certainty for reach. An exact-match cache never answers a question it was not asked; a semantic cache does so by design, and the similarity threshold decides how far it may stretch.

## LLM caching types: exact-match, semantic, and prompt caching

LLM caching comes in three forms that work at different layers. Exact-match response caching replays a stored response for an identical request. Semantic caching replays a stored response for a request with similar meaning. Provider prompt caching still calls the model but reuses the computed state of an identical prompt prefix, so every answer is generated fresh.

| | Exact-match response caching | Semantic caching | Provider prompt caching |
|---|---|---|---|
| What it stores | A full response, keyed by a request hash | A full response plus the prompt's embedding | Attention state (KV cache) for a prefix |
| Match rule | Identical request after normalization | Embedding similarity above a threshold | Identical prefix, token for token |
| Model call on a hit | Skipped | Skipped | Still made; output generated fresh |
| Where it runs | Application, gateway, or proxy | Library, caching service, or gateway | Model provider or inference server |
| Correctness risk | Stale answers | False hits on similar prompts | None to the output |
| Typical hits | Retries, repeated batch jobs | Paraphrased FAQs and support questions | Long system prompts, tools, documents |

[Prompt caching](https://theaiengineering.com/glossary/prompt-caching) is not a response cache. The server keeps the [KV cache](https://theaiengineering.com/glossary/kv-cache) computed for the start of a prompt and reuses it when a later request begins with the same tokens. Anthropic's [prompt caching documentation](https://platform.claude.com/docs/en/build-with-claude/prompt-caching) states that hits require 100 percent identical prompt segments and that caching has no effect on the generated output. [OpenAI](https://developers.openai.com/api/docs/guides/prompt-caching) enables prompt caching by default on supported models, requires the rendered prefix to match, and reports reused tokens in `usage.input_tokens_details.cached_tokens`.

The three layers stack. An exact-match lookup is one key-value read with no false hits, so it runs first; a semantic lookup runs on an exact miss; requests that miss both reach the provider, where a stable prompt prefix can still earn a prompt caching discount.

## How a semantic cache works

A semantic cache sits on the request path before the model call. It scopes the lookup by tenant, model, and parameters, tries an exact match, embeds the prompt on a miss, searches for the nearest cached prompt in that scope, and returns the stored response when the similarity clears the threshold. Otherwise it calls the model and stores the new response.

**Figure 1: A semantic cache tries an exact match, then a similarity search within the request's scope, and calls the model only when both miss.**

A scoped request is checked by exact hash, then embedded and searched; a close match returns the cached response, otherwise the model is called and its response stored.

Components:

- **Request**: prompt + params
- **Build scope**: tenant · model params · prompt
- **Exact lookup**: request hash
- **Cached response**: no model call
- **Embed prompt**: embedding model
- **Vector search**: nearest in scope
- **Call the model**: full cost
- **Store entry**: vector + response TTL, async write

Connections:

- Request → Build scope
- Build scope → Exact lookup
- Exact lookup → Cached response: exact hit
- Exact lookup → Embed prompt: miss
- Embed prompt → Vector search
- Vector search → Cached response: above threshold
- Vector search → Call the model: below threshold
- Call the model → Store entry: write back

A lookup moves through six steps:

1. **Scope the request.** Build a partition key from the fields that must match exactly: tenant or user, model, provider, generation parameters, and a system prompt version.
2. **Try an exact match.** Hash the normalized request and look it up; an exact hit needs no embedding.
3. **Embed the prompt.** On a miss, embed the text you chose to compare, often the latest user message.
4. **Search the index.** Find the nearest stored prompt inside the partition. Most vector stores use [approximate nearest neighbor](https://theaiengineering.com/glossary/approximate-nearest-neighbor) indexes such as HNSW, which accept a small chance of missing the true nearest entry in exchange for fast search.
5. **Compare with the threshold.** If the best match clears it, return the stored response and log a semantic hit with its score.
6. **Call the model and store.** On a miss, forward the request, return the response, and write the embedding, response, and expiry asynchronously so the write adds no latency.

The lookup is not free. [Bifrost's semantic caching documentation](https://docs.getbifrost.ai/features/semantic-caching) puts an exact lookup at one vector store round trip, sub-millisecond to a few milliseconds with a local Redis or Valkey, and a semantic lookup at an embedding API call, typically tens to a few hundred milliseconds, plus a similarity search. A semantic hit therefore costs about one embedding round trip, and a semantic miss is slower than no cache at all, because it pays for the embedding and then the full model call.

## Designing a semantic cache: embeddings, thresholds, keys, and expiry

Four decisions determine whether a semantic cache helps or hurts: which text to embed, which embedding model and threshold to use, which fields must match exactly, and how long entries live. Some tools express the threshold as a similarity and others as a distance, so check each setting's units before copying a value from one tool to another.

### What text to embed

The embedded text decides what "similar" means. Embedding only the latest user message gives focused vectors and the most hits, but the system prompt and earlier turns must then be covered by the exact-match key. Embedding the whole conversation captures context, but a long shared history dominates the vector. Embedding the last few messages, with system or tool messages dropped, sits in between. Gateways default differently:

| Tool | Text compared by similarity | Long conversations | System prompt |
|---|---|---|---|
| Bifrost | Not listed in the docs; the exact hash covers normalized input, parameters, and the stream flag | Skipped above `conversation_history_threshold` messages (default 3) | In the cache key unless `exclude_system_prompt` is set |
| Kong AI Semantic Cache | The last `message_countback` messages (default 1) | Set by `message_countback` | Dropped with `ignore_system_prompts` |
| Azure API Management | The chat prompt | Skipped past `max-message-count` | Dropped with `ignore-system-messages`, which Microsoft recommends |
| LiteLLM proxy | The whole `messages` array | Docs advise single-shot prompts only | Included |

### Choosing an embedding model

A cache compares questions with questions, a symmetric task, while many embedding models are trained for retrieval, matching short queries to longer passages. Models trained on paraphrase and duplicate-question data fit the cache task more closely. RedisVL's `SemanticCache` defaults to `redis/langcache-embed-v1`, which Redis fine-tuned from `Alibaba-NLP/gte-modernbert-base` on the Quora question pairs dataset, and the accompanying [paper by Gill et al. (2025)](https://arxiv.org/abs/2504.02268) reports that compact models fine-tuned for one epoch on domain data outperformed open-source and proprietary baselines on cache precision and recall.

Latency and index compatibility matter as much as accuracy. A hosted embedding API adds a network round trip to every semantic lookup. And vectors from different models are not comparable, so a new model means a new index; Bifrost refuses to reuse a namespace after the embedding model or dimension changes. Our [embedding and reranking directory](https://theaiengineering.com/platforms/embedding-reranking) lists candidates to test.

### Similarity metric and threshold

[Cosine similarity](https://theaiengineering.com/glossary/cosine-similarity) is the usual metric: it measures the angle between two vectors, from -1 to 1, and cosine distance is 1 minus cosine similarity. For unit-length vectors, cosine similarity, dot product, and Euclidean distance rank neighbors in the same order, so the threshold matters far more than the metric.

Threshold conventions differ, and the same number can mean opposite things:

| Tool | Setting | Scale | Stricter when | Default |
|---|---|---|---|---|
| Bifrost | `threshold` | Cosine similarity, 0 to 1 | Higher | 0.8 |
| LiteLLM proxy | `similarity_threshold` | Cosine similarity | Higher | Set in config; the docs example uses 0.8 |
| RedisVL `SemanticCache` | `distance_threshold` | Redis cosine distance, 0 to 2 | Lower | 0.1 |
| Azure API Management | `score-threshold` | Score, 0.0 to 1.0 | Lower | Required; the docs suggest starting at 0.05 |

A RedisVL distance of 0.1 corresponds to a cosine similarity of 0.9, which is stricter than Bifrost's default of 0.8.

The threshold trades precision against recall. Raising a similarity threshold cuts false hits and lowers the hit rate. No default transfers between embedding models, because each model spreads its scores differently. One global threshold is also a compromise across topics. The [vCache paper](https://arxiv.org/abs/2502.03771) (Schroeder et al., 2025) argues that static thresholds give no correctness guarantee and learns a separate threshold for each cached prompt, reporting up to 12.5 times higher hit rates and 26 times lower error rates than static-threshold baselines on its benchmarks.

### Cache keys and scope

Similarity should decide only between entries that are otherwise interchangeable. Everything else that changes the right answer belongs in an exact-match partition key: the tenant or user; the model and provider; generation parameters such as temperature, maximum tokens, tools, and response format; the system prompt or template version; and the stream flag when streamed and buffered responses are stored differently.

**Figure 2: Fields that change the right answer form an exact-match partition key; only the user's wording is compared by similarity.**

Six layers of a semantic cache key: tenant, model, parameters, and system prompt must match exactly, history is capped or hashed, and only the latest message is compared by similarity.

Layers, top to bottom:

1. **Tenant and user**: Virtual key, customer, or user ID
2. **Model and provider**: Different models give different answers
3. **Generation parameters**: Temperature, max tokens, tools, response format
4. **System prompt version**: Hash or template version ID
5. **Conversation history**: Hash it, cap its length, or skip caching
6. **Latest user message**: Embedded and compared by similarity

Exact-match partition spans Tenant and user through System prompt version. Similarity search spans Latest user message through Latest user message.

Gateways implement scope differently. Bifrost keys on model and provider by default, partitions entries by the calling [virtual key](https://theaiengineering.com/glossary/virtual-key), and serves an entry only to the identity that wrote it. LiteLLM's [semantic cache documentation](https://docs.litellm.ai/docs/proxy/caching_semantic) warns that every end user behind one virtual key shares a bucket unless `semantic_cache_scope` is set to `end_user`. Azure API Management partitions entries with a `vary-by` expression. A coarse partition, such as one per feature, maximizes sharing; a fine one, such as one per user, keeps answers private and hits less often.

### TTL and invalidation

Every entry needs an expiry, and defaults vary widely. Bifrost and Kong both default to five minutes (`ttl: "5m"` and `cache_ttl: 300`). RedisVL's `SemanticCache` sets no TTL by default, so entries never expire unless you set one, and by default each hit refreshes the entry's TTL, so a popular entry lives as long as it keeps getting hits.

Expiry alone is not invalidation. Put model and prompt template versions in the key so old entries stop matching at deploy time, clear a partition when the documents or policies its answers quote change, and delete an entry when someone reports a wrong answer. Bifrost returns a `cache_id` with each response and exposes `DELETE /api/cache/clear/{cache_id}` for one entry and `DELETE /api/cache/clear-by-key/{cacheKey}` for a partition.

## Failure modes and what not to cache

Semantic caching fails quietly. A false hit returns a fluent answer to a different question with a success status and low latency, so error dashboards never show it. The main failure modes are near-identical prompts with different meanings, personalized or time-sensitive answers, tool calls with side effects, streamed responses, and multi-turn context, and each has a known guard.

### False hits on near-identical prompts

Embeddings summarize meaning, and some small tokens change the answer far more than they move the vector. Pairs such as "cancel my order" and "cancel my subscription", or "convert 5 km to miles" and "convert 50 km to miles", can score close together and still need different answers. Negations, numbers, version strings, product names, and identifiers are the usual causes.

Raise the threshold, or allow only exact matches, for topics where a false hit is expensive and for prompts that contain numbers, IDs, dates, or code. Add a second check on candidate hits, such as a cross-encoder reranker or a rule that key entities must match; [GPTCache](https://github.com/zilliztech/GPTCache) builds this in as a separate similarity evaluator module. Then sample hits for review, as described below.

### Personalized and time-sensitive answers

An answer built from a user's account, permissions, or private documents is correct for that user only, and an answer about prices, stock levels, or service status is correct only for a period. Scope personalized answers per user or keep them out of the cache, and give time-sensitive answers short TTLs. In retrieval-augmented applications, two users can ask the same question and receive different retrieved context because of document permissions; a cache keyed only on the question erases that difference.

### Tool calls with side effects

A response that contains a [tool call](https://theaiengineering.com/glossary/tool-calling) is an instruction to act. Replaying it makes the agent run the action again, with arguments the model chose for someone else's prompt: sending an email, issuing a refund, or writing to a database. Bifrost does not cache responses that carry tool calls by default for this reason, and LiteLLM's documentation warns that on agentic traffic a semantic cache often makes an agent repeat the same tool call. Cache tool-call responses only when the tools are read-only and idempotent.

### Streaming responses

A [streamed response](https://theaiengineering.com/glossary/llm-streaming) can be stored only after its final chunk arrives, so the cache keeps a copy while forwarding chunks and must discard streams that were canceled, cut off, or ended in an error. Bifrost replays cached streams chunk by chunk and sends its cache headers before the first chunk. Hits also distort time to first token and throughput metrics, so tag cached responses in logs.

### Multi-turn context

"What about the second option?" means nothing without the conversation before it, so embedding only the last message matches unrelated conversations. Embedding the whole conversation fails differently: consecutive turns share almost all their text, and LiteLLM's documentation reports that their embeddings score about 0.99 similar, so each turn matches the cached previous turn and a higher threshold does not reliably fix it. Bifrost and Azure API Management therefore limit semantic caching to short conversations: Bifrost skips caching above three messages by default, and Azure's `max-message-count` sets a configurable cut-off.

### What not to cache

Keep these out of a semantic cache, or cache them only by exact match with a tight scope:

- Tool-call responses and requests made inside an agent loop.
- Answers that depend on the user's identity or permissions, unless scoped to that user.
- Prices, inventory, status, news, and anything tied to the current date.
- Prompts where one character changes the answer: IDs, quantities, versions, and code.
- Creative outputs where users expect variety.
- Truncated, failed, refused, or guardrail-filtered responses.
- Secrets or personal data the cache store is not approved to hold.

## How to measure a semantic cache

Measure a semantic cache on four things: hit rate, latency saved, cost saved, and the quality of cached answers. The first three come from gateway or application logs. The fourth needs sampled evaluation, because a false hit looks like a fast, successful request in every dashboard and no error rate will reveal it.

| Metric | Definition | Watch for |
|---|---|---|
| Hit rate | Hits per cacheable lookup, split into exact and semantic | Rising hits while quality falls |
| Latency saved | Time saved on hits minus lookup overhead on misses | Semantic misses slower than no cache |
| Cost saved | Avoided model spend minus embedding and storage cost | Cheap models, where embedding cost is relatively large |
| False-hit rate | Sampled semantic hits whose answer is wrong for the new prompt | Clusters in one topic or language |
| Missed-hit rate | Misses whose nearest entry would have been correct | A threshold stricter than it needs to be |

Estimate net savings for a period as:

```text
net savings = (exact hits + semantic hits) x average cost of an avoided model call
            - semantic lookups x cost of one embedding call
            - vector store cost for the period
```

Repeat rates depend heavily on the product, so published hit rates say little about yours. Where your cache or application allows it, start in shadow mode: look up every request and log the would-be hit and its score, but still call the model, then compare cached and fresh answers offline. That yields an estimated hit rate and a labeled set before any user sees a cached answer.

### Sampled evaluation of cached answers

Sample a fixed share of semantic hits every day, keeping the new prompt, the matched prompt, the score, and the cached response. Grade whether the cached response correctly answers the new prompt, with human reviewers for high-stakes domains and an [LLM-as-a-judge](https://theaiengineering.com/glossary/llm-as-a-judge) for volume, and track the false-hit rate by topic and by similarity band. Regeneration clicks and negative ratings on cached responses, compared with fresh ones, add a cheap second signal. Our guide to [LLM evaluation](https://theaiengineering.com/articles/llm-evaluation) covers grader design.

### Tuning the threshold with a labeled set

Tune the threshold on pairs drawn from real traffic:

1. **Collect pairs.** For each sampled request, record the nearest cached prompt and its score, on misses as well as hits. If your gateway reports scores only on hits, embed the pairs offline with the same model.
2. **Label them.** Mark a pair acceptable when the cached answer would be correct for the new prompt, and include hard negatives such as the order and subscription example.
3. **Sweep thresholds.** For each candidate, compute the hit rate, false-hit rate, and recall.
4. **Pick and verify.** Choose the loosest threshold whose false-hit rate fits your budget, confirm it on a held-out set, and keep the pairs as a [golden dataset](https://theaiengineering.com/glossary/golden-dataset) to re-run when the embedding model, prompts, or traffic change.

```python
def sweep(pairs, thresholds):
    """pairs: (similarity, acceptable) for each new prompt and its nearest
    cached prompt. acceptable is True when the cached answer is correct
    for the new prompt."""
    positives = sum(ok for _, ok in pairs)
    rows = []
    for t in thresholds:
        hits = [ok for score, ok in pairs if score >= t]
        correct = sum(hits)
        rows.append({
            "threshold": t,
            "hit_rate": len(hits) / len(pairs),
            "false_hit_rate": 1 - correct / len(hits) if hits else 0.0,
            "recall": correct / positives if positives else 0.0,
        })
    return rows


def loosest_threshold(pairs, max_false_hit_rate):
    """Lowest similarity threshold whose false-hit rate fits the budget."""
    for row in sweep(pairs, [t / 100 for t in range(50, 100)]):
        if row["false_hit_rate"] <= max_false_hit_rate:
            return row
    return None
```

The sketch assumes a similarity scale, where higher is stricter. For a distance-based setting such as RedisVL's `distance_threshold`, flip the comparison. If one threshold cannot meet the budget across all traffic, split it by route or topic before giving up on semantic matching.

## When does semantic caching pay off?

Semantic caching pays off when many users ask the same questions in different words, the answers are expensive to generate and stay correct for hours or days, and the product can tolerate a small, measured false-hit rate. It rarely pays off for agent loops, personalized answers, or traffic where few prompts repeat, because overhead and risk outweigh the hits.

| Pays off | Rarely pays off |
|---|---|
| Support and documentation assistants | Agent loops and multi-step tool use |
| Long, expensive answers to standalone questions | Short answers from cheap models |
| Answers that stay valid for hours or days | Prices, status, and other fast-changing data |
| Answers that are the same for every user | Answers built from each user's data |
| Traffic you can sample and grade | Domains where any wrong answer is unacceptable |

A safe rollout order is exact-match caching first, which has no false hits, then semantic caching on the routes where shadow-mode data shows enough paraphrased repeats.

## Where semantic caching runs

Semantic caching runs in three places: as a library inside the application, as a managed caching service the application calls, or as a feature of an AI gateway that every application shares. Libraries give the most control over what is embedded; gateways apply one cache policy, scope, and set of metrics to all traffic without application changes.

| Option | Type | Matching | Storage | Notes, November 2026 |
|---|---|---|---|---|
| Bifrost | Open source gateway (Apache 2.0) | Exact, then semantic | Redis or Valkey, Weaviate, Qdrant, Pinecone | Scoped per virtual key and identity |
| LiteLLM proxy | Open source gateway (MIT outside its enterprise directory) | Exact or semantic | Redis, Valkey, or Qdrant for semantic | Scoped per key, optionally per end user |
| Kong AI Semantic Cache | Kong AI Gateway plugin | Semantic, optional exact check | Redis or Valkey, pgvector | Enterprise only |
| Azure API Management | Managed gateway policies | Semantic | Azure Managed Redis with RediSearch | Lookup and store policies |
| Cloudflare AI Gateway | Managed gateway | Exact only | Managed | Semantic listed as planned |
| RedisVL `SemanticCache` | Python library (MIT) | Semantic | Redis | 0.32.0 released November 11, 2026 |
| Redis LangCache | Managed caching service | Semantic, optional exact search | Redis | Public preview on Redis Cloud |
| GPTCache | Python library (MIT) | Exact or semantic | SQLite plus Milvus, FAISS, and others | Last PyPI release 0.1.44, August 2024 |

### Libraries and caching services

[RedisVL](https://redis.io/docs/latest/develop/ai/redisvl/api/cache/), the Redis vector library for Python, provides `SemanticCache` with `check` and `store` methods, a `distance_threshold`, optional TTLs, and `filterable_fields` for scoping lookups by tenant. The same module includes a client for [Redis LangCache](https://redis.io/docs/latest/develop/ai/langcache/), a hosted service that generates embeddings itself; Redis lists it as a public preview on Redis Cloud, with a self-managed Kubernetes deployment in private preview. GPTCache splits a semantic cache into modules (embedding generator, vector store, cache storage, and similarity evaluator), but its README now says it no longer adds support for new model APIs. Our [vector databases directory](https://theaiengineering.com/platforms/vector-databases) lists the stores these libraries run on.

### Gateways

A gateway cache serves every application behind it, and scoping follows the gateway's own keys and teams. The trade-off is that the gateway cannot tell whether an answer is personal or time-sensitive, so applications signal that through headers or routes. Kong's [AI Semantic Cache plugin](https://developer.konghq.com/plugins/ai-semantic-cache/) reports `X-Cache-Status` as Hit, Miss, Bypass, or Refresh. [Cloudflare AI Gateway](https://developers.cloudflare.com/ai-gateway/features/caching/) caches only identical requests and lists semantic search as planned. Our [AI gateway explainer](https://theaiengineering.com/articles/what-is-an-ai-gateway) shows where the cache sits among a gateway's other policies.

## Worked example: semantic caching in Bifrost

Bifrost, the open source AI gateway from Maxim AI, implements [semantic caching](https://docs.getbifrost.ai/features/semantic-caching) as its `semantic_cache` plugin with two lookup paths: an exact hash match first, then an embedding similarity search on a miss. Entries live in Redis or Valkey, Weaviate, Qdrant, or Pinecone, are scoped to the calling virtual key, and can be tuned per request with headers.

A vector store must be configured first, even for exact-only caching. This `config.json` enables Redis and the plugin in exact-plus-semantic mode with the documented defaults:

```json
{
  "vector_store": {
    "enabled": true,
    "type": "redis",
    "config": { "addr": "localhost:6379" }
  },
  "plugins": [
    {
      "enabled": true,
      "name": "semantic_cache",
      "config": {
        "provider": "openai",
        "embedding_model": "text-embedding-3-small",
        "dimension": 1536,
        "ttl": "5m",
        "threshold": 0.8,
        "conversation_history_threshold": 3,
        "exclude_system_prompt": false,
        "cache_by_model": true,
        "cache_by_provider": true,
        "cache_tool_call_responses": false
      }
    }
  ]
}
```

Setting `dimension` to `1` and omitting `provider` and `embedding_model` switches to direct-only mode, which needs no embedding provider. The defaults implement several guards from this guide: model and provider are part of the key, responses with tool calls are not stored, conversations longer than three messages are skipped, and entries expire after five minutes.

Caching is opt-in per request. Bifrost caches a request only when it carries an `x-bf-cache-key` header or the deployment sets `default_cache_key`. The key names the partition, and Bifrost adds the [virtual key](https://docs.getbifrost.ai/features/governance/virtual-keys) and, where the request is attributed to one, the user on top of it. Other headers adjust a single request:

```bash
curl -X POST http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "x-bf-vk: <VIRTUAL_KEY>" \
  -H "x-bf-cache-key: support-faq" \
  -H "x-bf-cache-ttl: 1h" \
  -H "x-bf-cache-threshold: 0.9" \
  -d '{
    "model": "openai/gpt-4o-mini",
    "messages": [{"role": "user", "content": "How do I rotate an API key?"}]
  }'
```

- `x-bf-cache-ttl` overrides the expiry, as a duration string or seconds.
- `x-bf-cache-threshold` can only make matching stricter; values below the configured threshold are raised to it.
- `x-bf-cache-type` limits the lookup to `direct` or `semantic`.
- `x-bf-cache-no-store: true` serves hits but does not write the response to the cache.

Each response reports what happened. The `extra_fields.cache_debug` object, on the final chunk for streams, carries `cache_hit`, `cache_id`, and `hit_type`, plus `similarity` and `threshold` on a semantic hit, and `x-bifrost-cache-*` response headers carry the same facts on every route, including SDK routes whose bodies have no `extra_fields`. Logging them gives the hit rate by path and the score distribution needed to tune the threshold, and the `cache_id` feeds the invalidation endpoints described above.

Where to go next: our guide to [LLM routing](https://theaiengineering.com/articles/llm-routing) covers how caching interacts with routing and fallbacks, and the [AI gateways directory](https://theaiengineering.com/platforms/ai-gateways) lists more gateways with caching features. The [AI gateway glossary entry](https://theaiengineering.com/glossary/ai-gateway) defines the proxy layer where a shared semantic cache runs.

## Frequently asked questions

### What is semantic caching in LLM applications?

Semantic caching is a response cache that matches LLM requests by meaning instead of exact text. It converts each prompt into an embedding, searches stored prompt embeddings for the nearest match, and returns the cached response when the similarity clears a threshold. A hit skips the model call, which saves its cost and most of its latency. The risk is a false hit, where a similar prompt needed a different answer.

### What is the difference between semantic caching and prompt caching?

Semantic caching replays a complete stored response when a new prompt is similar in meaning, so the model is not called at all. Prompt caching is a provider or inference-server feature that still calls the model but reuses the computed state for an identical prompt prefix, which cuts prefill cost and time to first token. Prompt caching never changes the output; semantic caching can return an answer that does not fit the new prompt.

### What similarity threshold should a semantic cache use?

No threshold transfers between embedding models or workloads, so tune it on your own traffic. Collect pairs of new prompts and their nearest cached prompts, label whether the cached answer would be correct, and pick the loosest threshold that keeps false hits within your budget. Check the units first: Bifrost and LiteLLM use cosine similarity, where higher is stricter, while RedisVL and Azure API Management use distance-style scores, where lower is stricter.

### Is GPTCache still maintained?

GPTCache, the MIT-licensed semantic cache library from Zilliz, was not archived as of November 2026, and its repository received commits in September 2026. Its most recent PyPI release, however, was version 0.1.44 in August 2024, and its README says the project no longer adds support for new model APIs and recommends its generic get and set API. Check recent activity before adopting it for new work.

### Can you use semantic caching with AI agents?

Usually not on the agent loop itself. Consecutive agent turns share most of their text, so their embeddings look nearly identical, and a cache can replay a stale response or an earlier tool call. LiteLLM's documentation recommends semantic caching only for single-shot prompts, and Bifrost does not cache responses containing tool calls by default. Agents can still benefit from caching read-only lookups, such as answers to standalone questions, inside their tools.

### What are the types of LLM caching?

LLM caching has three common forms. Exact-match response caching replays a stored response when a request is identical. Semantic caching replays a stored response when a request is similar in meaning, measured by embedding similarity. Prompt caching, offered by model providers and inference servers, reuses the computed attention state for a repeated prompt prefix while still generating a fresh response. The three can run together, in that order.

## Terms in this guide

- [Semantic caching](https://theaiengineering.com/glossary/semantic-caching): Semantic caching is a technique that returns a stored LLM response when a new query is close in meaning to an earlier one, measured by embedding similarity instead of exact text match.
- [Prompt caching](https://theaiengineering.com/glossary/prompt-caching): Prompt caching is an LLM API and inference-server feature that stores the computed attention state for a repeated prompt prefix, so later requests sharing that prefix skip reprocessing it.
- [Embeddings](https://theaiengineering.com/glossary/embeddings): Embeddings are fixed-length vectors of numbers that represent data such as text or images so that items with similar meaning sit close together, which lets software compare meaning with arithmetic.
- [Cosine similarity](https://theaiengineering.com/glossary/cosine-similarity): Cosine similarity is a measure of how closely two vectors point in the same direction, computed as their dot product divided by the product of their lengths, with values from -1 to 1.
- [Vector database](https://theaiengineering.com/glossary/vector-database): A vector database is a database built to store embedding vectors and quickly return the vectors most similar to a query vector, usually with approximate nearest neighbor indexes such as HNSW.
- [Approximate nearest neighbor search](https://theaiengineering.com/glossary/approximate-nearest-neighbor): Approximate nearest neighbor (ANN) search is a family of indexing methods that find vectors close to a query vector without comparing it to every stored vector, accepting that some true nearest neighbors are missed.
- [AI gateway](https://theaiengineering.com/glossary/ai-gateway): An AI gateway is a proxy layer between applications and model providers that exposes one API for many models and centralizes routing, failover, access control, cost tracking, caching, and logging.
- [Virtual key](https://theaiengineering.com/glossary/virtual-key): A virtual key is an API key issued by an AI gateway that maps a caller to its allowed models, budget, and rate limits, while the gateway keeps the real provider credentials.
- [KV cache](https://theaiengineering.com/glossary/kv-cache): The KV cache is GPU memory that stores the attention key and value vectors already computed for previous tokens, so a transformer generating text computes them once instead of at every step.
- [Tool calling](https://theaiengineering.com/glossary/tool-calling): Tool calling is an LLM capability in which the model, given a list of tool definitions, outputs a structured request naming a tool and its arguments, which the application executes and returns.
- [LLM streaming](https://theaiengineering.com/glossary/llm-streaming): LLM streaming is the delivery of a language model's output in small chunks as it is generated, usually as server-sent events over HTTP, so clients can show text before the response is complete.
- [Golden dataset](https://theaiengineering.com/glossary/golden-dataset): A golden dataset is a curated, versioned set of representative inputs, each paired with an expected output or grading criteria, that a team treats as ground truth when evaluating an LLM application.
- [LLM-as-a-judge](https://theaiengineering.com/glossary/llm-as-a-judge): LLM-as-a-judge is an evaluation method in which a language model scores or compares the outputs of an LLM application against a rubric, a reference answer, or another response, replacing or supplementing human graders.

## Related guides

- [What is an AI gateway? How it works and when you need one](https://theaiengineering.com/articles/what-is-an-ai-gateway): An AI gateway is a proxy between your apps and model providers. Learn how it works, its core features, AI gateway vs API gateway, and open source options.
- [LLM routing: strategies, fallbacks, and learned routers](https://theaiengineering.com/articles/llm-routing): LLM routing guide: rule-based and cost-aware strategies, fallbacks, load balancing, semantic and learned routers like RouteLLM, router evals, and pitfalls.
- [LLM inference optimization: a practical guide for engineers](https://theaiengineering.com/articles/llm-inference-optimization): LLM inference optimization guide: prefill vs decode, TTFT, KV cache and PagedAttention, continuous batching, quantization, speculative decoding, and GPUs.
- [Bifrost vs LiteLLM: architecture, features, and performance compared](https://theaiengineering.com/articles/bifrost-vs-litellm): Bifrost vs LiteLLM compared: a Go single binary against a Python proxy, with deployment, routing, virtual keys, caching, MCP, benchmarks, and migration.

Part of the [AI gateways and LLM routing](https://theaiengineering.com/topics/ai-gateways) topic hub.

## Cite this guide

The AI Engineering Editors. "Semantic caching for LLMs: how it works and when it pays off." The AI Engineering, updated Nov 16, 2026. https://theaiengineering.com/articles/semantic-caching

- Title: Semantic caching for LLMs: how it works and when it pays off
- Author: The AI Engineering Editors
- Publisher: The AI Engineering
- Published: 2026-11-16
- Updated: 2026-11-16
- URL: https://theaiengineering.com/articles/semantic-caching
- Markdown: https://theaiengineering.com/articles/semantic-caching.md

To cite one section, link to its anchor:

- [What is semantic caching?](https://theaiengineering.com/articles/semantic-caching#what-is-semantic-caching)
- [LLM caching types: exact-match, semantic, and prompt caching](https://theaiengineering.com/articles/semantic-caching#llm-caching-types-exact-match-semantic-and-prompt-caching)
- [How a semantic cache works](https://theaiengineering.com/articles/semantic-caching#how-a-semantic-cache-works)
- [Designing a semantic cache: embeddings, thresholds, keys, and expiry](https://theaiengineering.com/articles/semantic-caching#designing-a-semantic-cache-embeddings-thresholds-keys-and-expiry)
- [Failure modes and what not to cache](https://theaiengineering.com/articles/semantic-caching#failure-modes-and-what-not-to-cache)
- [How to measure a semantic cache](https://theaiengineering.com/articles/semantic-caching#how-to-measure-a-semantic-cache)
- [When does semantic caching pay off?](https://theaiengineering.com/articles/semantic-caching#when-does-semantic-caching-pay-off)
- [Where semantic caching runs](https://theaiengineering.com/articles/semantic-caching#where-semantic-caching-runs)
- [Worked example: semantic caching in Bifrost](https://theaiengineering.com/articles/semantic-caching#worked-example-semantic-caching-in-bifrost)
- [Frequently asked questions](https://theaiengineering.com/articles/semantic-caching#faq)
