---
title: "What is semantic caching for LLMs?"
description: "Semantic caching reuses stored LLM responses for queries with similar meaning, matched by embeddings. See how it works and how it differs from prompt caching."
type: "glossary-term"
canonical_url: "https://theaiengineering.com/glossary/semantic-caching"
markdown_url: "https://theaiengineering.com/glossary/semantic-caching.md"
topic: "AI gateways and LLM routing"
author: "The AI Engineering Editors"
publisher: "The AI Engineering"
updated: "2026-11-11"
keywords: ["Semantic caching","semantic cache","LLM semantic caching"]
citation: "The AI Engineering Editors. \"What is semantic caching for LLMs?\" The AI Engineering, updated Nov 11, 2026. https://theaiengineering.com/glossary/semantic-caching"
---

# What is semantic caching for LLMs?

Also called semantic cache, LLM semantic caching.

> **Definition of Semantic caching:** Semantic caching is a technique that returns a stored LLM response when a new query is close in meaning to an earlier one, measured by embedding similarity instead of exact text match.

Semantic caching is a technique that stores LLM responses and returns a stored response when a new query means roughly the same thing as an earlier one. A traditional cache matches only identical keys, so "How do I reset my password?" and "password reset steps" are two separate misses. A semantic cache compares [embeddings](https://theaiengineering.com/glossary/embeddings) of the two queries and can serve both from one entry.

## How semantic caching works

Semantic caching works in four steps:

1. Embed the incoming query with an embedding model.
2. Search an index of previously cached query embeddings, often held in a [vector database](https://theaiengineering.com/glossary/vector-database).
3. If the nearest match passes a similarity threshold, return its stored response without calling the model.
4. On a miss, call the model, then store the new query embedding and its response.

[Bifrost](https://docs.getbifrost.ai/features/semantic-caching), an open source AI gateway, implements this design with an exact-match check first and an embedding search on a miss, a default cosine similarity threshold of 0.8, a default expiry of five minutes, and Redis or Valkey, Weaviate, Qdrant, or Pinecone as the vector store. Libraries such as [GPTCache](https://github.com/zilliztech/GPTCache) offer the same design with pluggable embedding models, vector stores, and similarity evaluators. Semantic caches often run inside an [AI gateway](https://theaiengineering.com/glossary/ai-gateway) so that every application behind it shares the cache.

The similarity threshold is the main tuning control. GPTCache's documentation notes that semantic caches can produce false positives on hits and false negatives on misses. A loose threshold returns wrong answers for queries that look alike but differ in one key detail, such as "cancel my order" and "cancel my subscription"; a strict threshold rarely hits. Scope caches per user or tenant when responses depend on private data, and set expiry times for answers that go stale. Bifrost's defaults, for example, partition entries by virtual key, skip conversations longer than three messages, and leave tool-call responses out of the cache.

## Semantic caching vs prompt caching

Semantic caching and [prompt caching](https://theaiengineering.com/glossary/prompt-caching) save cost at different points. Semantic caching skips the model call entirely by reusing a whole response. Prompt caching, offered by model providers, still calls the model but reuses the processed state of an identical prompt prefix, which cuts the cost and latency of handling that prefix. Anthropic's [prompt caching documentation](https://platform.claude.com/docs/en/build-with-claude/prompt-caching) requires a 100% identical prefix for a cache hit.

| | Semantic caching | Prompt caching |
|---|---|---|
| Match rule | Similar meaning | Identical prefix |
| Runs in | Application or gateway | Model provider |
| What it skips | The whole model call | Reprocessing the cached prefix |
| Correctness risk | Wrong answer on a false hit | None; the model still generates |

The two combine well: prompt caching speeds up long shared system prompts, and semantic caching absorbs repeated questions. [What is an AI gateway](https://theaiengineering.com/articles/what-is-an-ai-gateway) shows where a semantic cache sits in the request path.

## Related terms

- [Prompt caching](https://theaiengineering.com/glossary/prompt-caching): Prompt caching is an LLM API and inference-server feature that stores the computed attention state for a repeated prompt prefix, so later requests sharing that prefix skip reprocessing it.
- [Embeddings](https://theaiengineering.com/glossary/embeddings): Embeddings are fixed-length vectors of numbers that represent data such as text or images so that items with similar meaning sit close together, which lets software compare meaning with arithmetic.
- [AI gateway](https://theaiengineering.com/glossary/ai-gateway): An AI gateway is a proxy layer between applications and model providers that exposes one API for many models and centralizes routing, failover, access control, cost tracking, caching, and logging.
- [Vector database](https://theaiengineering.com/glossary/vector-database): A vector database is a database built to store embedding vectors and quickly return the vectors most similar to a query vector, usually with approximate nearest neighbor indexes such as HNSW.
- [LLM routing](https://theaiengineering.com/glossary/llm-routing): LLM routing is the practice of directing each request to one of several language models or providers, based on rules or a learned classifier, to balance cost, quality, latency, and availability.
- [Cosine similarity](https://theaiengineering.com/glossary/cosine-similarity): Cosine similarity is a measure of how closely two vectors point in the same direction, computed as their dot product divided by the product of their lengths, with values from -1 to 1.

## Guides that cover semantic caching

- [What is an AI gateway? How it works and when you need one](https://theaiengineering.com/articles/what-is-an-ai-gateway): An AI gateway is a proxy between your apps and model providers. Learn how it works, its core features, AI gateway vs API gateway, and open source options.
- [LLM inference optimization: a practical guide for engineers](https://theaiengineering.com/articles/llm-inference-optimization): LLM inference optimization guide: prefill vs decode, TTFT, KV cache and PagedAttention, continuous batching, quantization, speculative decoding, and GPUs.
- [Bifrost vs LiteLLM: architecture, features, and performance compared](https://theaiengineering.com/articles/bifrost-vs-litellm): Bifrost vs LiteLLM compared: a Go single binary against a Python proxy, with deployment, routing, virtual keys, caching, MCP, benchmarks, and migration.
- [Semantic caching for LLMs: how it works and when it pays off](https://theaiengineering.com/articles/semantic-caching): Semantic caching returns a stored LLM response for prompts with similar meaning. Learn how a semantic cache works, how to tune it, and what never to cache.
- [LLM routing: strategies, fallbacks, and learned routers](https://theaiengineering.com/articles/llm-routing): LLM routing guide: rule-based and cost-aware strategies, fallbacks, load balancing, semantic and learned routers like RouteLLM, router evals, and pitfalls.

Part of the [AI gateways and LLM routing](https://theaiengineering.com/topics/ai-gateways) topic hub.

## Cite this definition

The AI Engineering Editors. "What is semantic caching for LLMs?" The AI Engineering, updated Nov 11, 2026. https://theaiengineering.com/glossary/semantic-caching

- Title: What is semantic caching for LLMs?
- Author: The AI Engineering Editors
- Publisher: The AI Engineering
- Updated: 2026-11-11
- URL: https://theaiengineering.com/glossary/semantic-caching
- Markdown: https://theaiengineering.com/glossary/semantic-caching.md
