What is an AI gateway? How LLM gateways work
Also called LLM gateway, AI proxy, model gateway
Definition
An AI gateway is a proxy layer between applications and model providers that exposes one API for many models and centralizes routing, failover, access control, cost tracking, caching, and logging.
An AI gateway is a proxy that sits between applications and the model providers they call. Applications send every model request to the gateway, which forwards it to the chosen provider and returns the response. Because all model traffic passes through one point, the gateway can apply routing, failover, access control, budgets, caching, and logging consistently across teams and providers.
How an AI gateway works#
An AI gateway works by exposing one endpoint, often in an OpenAI-compatible format, and translating each request into the target provider's API.
Open source gateways such as Bifrost and LiteLLM and managed services such as Cloudflare AI Gateway document a similar set of functions:
- Unified API and keys: one client format, one place to store provider credentials, and per-team virtual keys with their own budgets and rate limits.
- Routing and failover: LLM routing by model alias, cost, or latency, with retries on another provider when one fails.
- Caching: exact-match or semantic caching to skip repeated calls.
- Guardrails: input and output guardrails for sensitive data, prompt injection, or policy violations.
- Observability: request logs, token counts, latency, and cost per key, which feed LLM observability tools.
Some gateways also proxy MCP server and agent-to-agent traffic. Bifrost, for example, runs an MCP gateway and an Agent Gateway for A2A agents next to its model routing, and LiteLLM documents support for MCP tools and A2A agents alongside model calls.
AI gateway vs API gateway#
An AI gateway is a specialized API gateway for model traffic. A general API gateway handles routing, authentication, and request-count rate limits for any HTTP service. An AI gateway adds model-aware features: limits measured in tokens rather than requests, cost tracking per model, translation between provider formats, streamed responses, retries across providers, and inspection of prompt and completion content.
The two categories overlap in practice. Microsoft's documentation for AI gateway capabilities in Azure API Management describes them as an extension of the product's existing API gateway, adding token limit policies, token metrics, and semantic caching rather than shipping a separate product.
What is an AI gateway covers architecture and deployment choices, and the AI gateways directory lists available options.
Related terms
Guides that cover ai gateway
- What is an AI gateway? How it works and when you need one
- LLM routing: strategies, fallbacks, and learned routers
- LLM observability: what to capture, which metrics, and tools
- Bifrost vs LiteLLM: architecture, features, and performance compared
- Semantic caching for LLMs: how it works and when it pays off
- How to build an MCP server: a Python and TypeScript tutorial
- A2A protocol vs MCP: differences and when to use each
- What is the Model Context Protocol (MCP)? How it works
- What is AI engineering? The discipline, the stack, and the work
Cite this definition
The AI Engineering Editors. "What is an AI gateway? How LLM gateways work." The AI Engineering, updated Nov 11, 2026. https://theaiengineering.com/glossary/ai-gateway
- Title
- What is an AI gateway? How LLM gateways work
- Author
- The AI Engineering Editors
- Publisher
- The AI Engineering
- Updated