What is semantic caching for LLMs?

Also called semantic cache, LLM semantic caching

Definition

Semantic caching is a technique that returns a stored LLM response when a new query is close in meaning to an earlier one, measured by embedding similarity instead of exact text match.

Semantic caching is a technique that stores LLM responses and returns a stored response when a new query means roughly the same thing as an earlier one. A traditional cache matches only identical keys, so "How do I reset my password?" and "password reset steps" are two separate misses. A semantic cache compares embeddings of the two queries and can serve both from one entry.

How semantic caching works#

Semantic caching works in four steps:

  1. Embed the incoming query with an embedding model.
  2. Search an index of previously cached query embeddings, often held in a vector database.
  3. If the nearest match passes a similarity threshold, return its stored response without calling the model.
  4. On a miss, call the model, then store the new query embedding and its response.

Bifrost, an open source AI gateway, implements this design with an exact-match check first and an embedding search on a miss, a default cosine similarity threshold of 0.8, a default expiry of five minutes, and Redis or Valkey, Weaviate, Qdrant, or Pinecone as the vector store. Libraries such as GPTCache offer the same design with pluggable embedding models, vector stores, and similarity evaluators. Semantic caches often run inside an AI gateway so that every application behind it shares the cache.

The similarity threshold is the main tuning control. GPTCache's documentation notes that semantic caches can produce false positives on hits and false negatives on misses. A loose threshold returns wrong answers for queries that look alike but differ in one key detail, such as "cancel my order" and "cancel my subscription"; a strict threshold rarely hits. Scope caches per user or tenant when responses depend on private data, and set expiry times for answers that go stale. Bifrost's defaults, for example, partition entries by virtual key, skip conversations longer than three messages, and leave tool-call responses out of the cache.

Semantic caching vs prompt caching#

Semantic caching and prompt caching save cost at different points. Semantic caching skips the model call entirely by reusing a whole response. Prompt caching, offered by model providers, still calls the model but reuses the processed state of an identical prompt prefix, which cuts the cost and latency of handling that prefix. Anthropic's prompt caching documentation requires a 100% identical prefix for a cache hit.

Semantic caching Prompt caching
Match rule Similar meaning Identical prefix
Runs in Application or gateway Model provider
What it skips The whole model call Reprocessing the cached prefix
Correctness risk Wrong answer on a false hit None; the model still generates

The two combine well: prompt caching speeds up long shared system prompts, and semantic caching absorbs repeated questions. What is an AI gateway shows where a semantic cache sits in the request path.

Cite this definition

The AI Engineering Editors. "What is semantic caching for LLMs?" The AI Engineering, updated Nov 11, 2026. https://theaiengineering.com/glossary/semantic-caching

Title
What is semantic caching for LLMs?
Author
The AI Engineering Editors
Publisher
The AI Engineering
Updated
View as Markdown

Part of the AI gateways and LLM routing hub · Updated