---
title: "AI gateways and LLM routing"
description: "AI gateways explained: how an LLM gateway routes, caches, secures and governs model traffic, with guides to LLM routing, semantic caching and guardrails."
type: "topic-hub"
canonical_url: "https://theaiengineering.com/topics/ai-gateways"
markdown_url: "https://theaiengineering.com/topics/ai-gateways.md"
author: "The AI Engineering Editors"
publisher: "The AI Engineering"
keywords: ["ai gateway","llm gateway","what is an ai gateway","llm routing","model routing","semantic caching","ai guardrails"]
citation: "The AI Engineering Editors. \"AI gateways and LLM routing.\" The AI Engineering. https://theaiengineering.com/topics/ai-gateways"
---

# AI gateways and LLM routing

The control layer between applications and models: routing, keys, caching and policy.

An AI gateway is a proxy layer that sits between applications and model providers. It gives every application one API while centralizing routing, fallbacks, rate limits, budgets, caching, guardrails and logging.

The guides here explain what gateways do, how routing strategies and semantic caching work, and how to evaluate open source and managed options.

## AI gateways and LLM routing guides

- [Bifrost vs LiteLLM: architecture, features, and performance compared](https://theaiengineering.com/articles/bifrost-vs-litellm): Bifrost vs LiteLLM compared: a Go single binary against a Python proxy, with deployment, routing, virtual keys, caching, MCP, benchmarks, and migration.
- [Semantic caching for LLMs: how it works and when it pays off](https://theaiengineering.com/articles/semantic-caching): Semantic caching returns a stored LLM response for prompts with similar meaning. Learn how a semantic cache works, how to tune it, and what never to cache.
- [LLM routing: strategies, fallbacks, and learned routers](https://theaiengineering.com/articles/llm-routing): LLM routing guide: rule-based and cost-aware strategies, fallbacks, load balancing, semantic and learned routers like RouteLLM, router evals, and pitfalls.
- [What is an AI gateway? How it works and when you need one](https://theaiengineering.com/articles/what-is-an-ai-gateway): An AI gateway is a proxy between your apps and model providers. Learn how it works, its core features, AI gateway vs API gateway, and open source options.

## Key terms

- [AI gateway](https://theaiengineering.com/glossary/ai-gateway): An AI gateway is a proxy layer between applications and model providers that exposes one API for many models and centralizes routing, failover, access control, cost tracking, caching, and logging.
- [Guardrails](https://theaiengineering.com/glossary/guardrails): Guardrails are programmatic checks placed around a large language model that validate, filter, or block inputs, outputs, and tool calls so an AI application stays within its safety, security, and policy limits.
- [LLM load balancing](https://theaiengineering.com/glossary/llm-load-balancing): LLM load balancing is the distribution of requests for one model across several interchangeable deployments, such as API keys, regions, providers, or replicas, by weight, latency, load, or remaining quota.
- [LLM routing](https://theaiengineering.com/glossary/llm-routing): LLM routing is the practice of directing each request to one of several language models or providers, based on rules or a learned classifier, to balance cost, quality, latency, and availability.
- [Model cascade](https://theaiengineering.com/glossary/model-cascade): A model cascade is an LLM serving pattern that calls models in order from cheapest to most capable and returns the first answer that passes a quality check, escalating only when it fails.
- [Model failover](https://theaiengineering.com/glossary/model-failover): Model failover is the automatic rerouting of an LLM request to a backup model, deployment, or provider when the primary target returns an error, times out, or hits a rate limit.
- [OpenAI-compatible API](https://theaiengineering.com/glossary/openai-compatible-api): An OpenAI-compatible API is an HTTP API that accepts OpenAI's request and response formats, so code written for OpenAI's SDKs can call another server by changing the base URL, API key, and model name.
- [Prompt injection](https://theaiengineering.com/glossary/prompt-injection): Prompt injection is an attack in which text supplied to a large language model, typed by a user or hidden in content the model reads, alters the application's intended behavior or output.
- [Rate limiting](https://theaiengineering.com/glossary/rate-limiting): Rate limiting is the enforcement of a maximum number of requests or tokens a client may send to an API within a time window, with excess requests rejected, usually with HTTP 429.
- [Semantic caching](https://theaiengineering.com/glossary/semantic-caching): Semantic caching is a technique that returns a stored LLM response when a new query is close in meaning to an earlier one, measured by embedding similarity instead of exact text match.
- [Virtual key](https://theaiengineering.com/glossary/virtual-key): A virtual key is an API key issued by an AI gateway that maps a caller to its allowed models, budget, and rate limits, while the gateway keeps the real provider credentials.

## Tools and platforms

- [AI gateways](https://theaiengineering.com/platforms/ai-gateways): 32 platforms. An AI gateway is a proxy between applications and model providers that exposes one API while handling routing, failover, authentication, rate limits, caching, and usage tracking.
- [Guardrails and AI security](https://theaiengineering.com/platforms/guardrails-security): 31 platforms. LLM guardrails are checks that inspect the inputs and outputs of language model applications and block, rewrite, or flag content that breaks a policy, such as prompt injection, leaked personal data, harmful text, or unsafe tool calls.

## Cite this page

The AI Engineering Editors. "AI gateways and LLM routing." The AI Engineering. https://theaiengineering.com/topics/ai-gateways

- Title: AI gateways and LLM routing
- Author: The AI Engineering Editors
- Publisher: The AI Engineering
- URL: https://theaiengineering.com/topics/ai-gateways
- Markdown: https://theaiengineering.com/topics/ai-gateways.md
