What is semantic caching for LLMs?
Also called semantic cache, LLM semantic caching
Definition
Semantic caching is a technique that returns a stored LLM response when a new query is close in meaning to an earlier one, measured by embedding similarity instead of exact text match.
Semantic caching is a technique that stores LLM responses and returns a stored response when a new query means roughly the same thing as an earlier one. A traditional cache matches only identical keys, so "How do I reset my password?" and "password reset steps" are two separate misses. A semantic cache compares embeddings of the two queries and can serve both from one entry.
How semantic caching works#
Semantic caching works in four steps:
- Embed the incoming query with an embedding model.
- Search an index of previously cached query embeddings, often held in a vector database.
- If the nearest match passes a similarity threshold, return its stored response without calling the model.
- On a miss, call the model, then store the new query embedding and its response.
Bifrost, an open source AI gateway, implements this design with an exact-match check first and an embedding search on a miss, a default cosine similarity threshold of 0.8, a default expiry of five minutes, and Redis or Valkey, Weaviate, Qdrant, or Pinecone as the vector store. Libraries such as GPTCache offer the same design with pluggable embedding models, vector stores, and similarity evaluators. Semantic caches often run inside an AI gateway so that every application behind it shares the cache.
The similarity threshold is the main tuning control. GPTCache's documentation notes that semantic caches can produce false positives on hits and false negatives on misses. A loose threshold returns wrong answers for queries that look alike but differ in one key detail, such as "cancel my order" and "cancel my subscription"; a strict threshold rarely hits. Scope caches per user or tenant when responses depend on private data, and set expiry times for answers that go stale. Bifrost's defaults, for example, partition entries by virtual key, skip conversations longer than three messages, and leave tool-call responses out of the cache.
Semantic caching vs prompt caching#
Semantic caching and prompt caching save cost at different points. Semantic caching skips the model call entirely by reusing a whole response. Prompt caching, offered by model providers, still calls the model but reuses the processed state of an identical prompt prefix, which cuts the cost and latency of handling that prefix. Anthropic's prompt caching documentation requires a 100% identical prefix for a cache hit.
| Semantic caching | Prompt caching | |
|---|---|---|
| Match rule | Similar meaning | Identical prefix |
| Runs in | Application or gateway | Model provider |
| What it skips | The whole model call | Reprocessing the cached prefix |
| Correctness risk | Wrong answer on a false hit | None; the model still generates |
The two combine well: prompt caching speeds up long shared system prompts, and semantic caching absorbs repeated questions. What is an AI gateway shows where a semantic cache sits in the request path.
Related terms
Guides that cover semantic caching
Cite this definition
The AI Engineering Editors. "What is semantic caching for LLMs?" The AI Engineering, updated Nov 11, 2026. https://theaiengineering.com/glossary/semantic-caching
- Title
- What is semantic caching for LLMs?
- Author
- The AI Engineering Editors
- Publisher
- The AI Engineering
- Updated