What is an AI gateway? How LLM gateways work

Also called LLM gateway, AI proxy, model gateway

Definition

An AI gateway is a proxy layer between applications and model providers that exposes one API for many models and centralizes routing, failover, access control, cost tracking, caching, and logging.

An AI gateway is a proxy that sits between applications and the model providers they call. Applications send every model request to the gateway, which forwards it to the chosen provider and returns the response. Because all model traffic passes through one point, the gateway can apply routing, failover, access control, budgets, caching, and logging consistently across teams and providers.

How an AI gateway works#

An AI gateway works by exposing one endpoint, often in an OpenAI-compatible format, and translating each request into the target provider's API.

Open source gateways such as Bifrost and LiteLLM and managed services such as Cloudflare AI Gateway document a similar set of functions:

  • Unified API and keys: one client format, one place to store provider credentials, and per-team virtual keys with their own budgets and rate limits.
  • Routing and failover: LLM routing by model alias, cost, or latency, with retries on another provider when one fails.
  • Caching: exact-match or semantic caching to skip repeated calls.
  • Guardrails: input and output guardrails for sensitive data, prompt injection, or policy violations.
  • Observability: request logs, token counts, latency, and cost per key, which feed LLM observability tools.

Some gateways also proxy MCP server and agent-to-agent traffic. Bifrost, for example, runs an MCP gateway and an Agent Gateway for A2A agents next to its model routing, and LiteLLM documents support for MCP tools and A2A agents alongside model calls.

AI gateway vs API gateway#

An AI gateway is a specialized API gateway for model traffic. A general API gateway handles routing, authentication, and request-count rate limits for any HTTP service. An AI gateway adds model-aware features: limits measured in tokens rather than requests, cost tracking per model, translation between provider formats, streamed responses, retries across providers, and inspection of prompt and completion content.

The two categories overlap in practice. Microsoft's documentation for AI gateway capabilities in Azure API Management describes them as an extension of the product's existing API gateway, adding token limit policies, token metrics, and semantic caching rather than shipping a separate product.

What is an AI gateway covers architecture and deployment choices, and the AI gateways directory lists available options.

Cite this definition

The AI Engineering Editors. "What is an AI gateway? How LLM gateways work." The AI Engineering, updated Nov 11, 2026. https://theaiengineering.com/glossary/ai-gateway

Title
What is an AI gateway? How LLM gateways work
Author
The AI Engineering Editors
Publisher
The AI Engineering
Updated
View as Markdown

Part of the AI gateways and LLM routing hub · Updated