---
title: "What is an AI gateway? How it works and when you need one"
description: "An AI gateway is a proxy between your apps and model providers. Learn how it works, its core features, AI gateway vs API gateway, and open source options."
summary: "An AI gateway is a proxy that sits between applications and model providers, giving every team one API with shared routing, budgets, keys, caching, guardrails, and logs. This explainer covers how a gateway works, how it differs from an API gateway, MCP gateways, and open source and managed options."
type: "explainer"
canonical_url: "https://theaiengineering.com/articles/what-is-an-ai-gateway"
markdown_url: "https://theaiengineering.com/articles/what-is-an-ai-gateway.md"
topic: "AI gateways and LLM routing"
author: "The AI Engineering Editors"
publisher: "The AI Engineering"
published: "2026-10-01"
updated: "2026-11-11"
keywords: ["ai gateway","llm gateway","what is an ai gateway","ai gateway open source","mcp gateway"]
citation: "The AI Engineering Editors. \"What is an AI gateway? How it works and when you need one.\" The AI Engineering, updated Nov 11, 2026. https://theaiengineering.com/articles/what-is-an-ai-gateway"
---

# What is an AI gateway? How it works and when you need one

## TL;DR

- An AI gateway is a proxy between applications and model providers that exposes one API and applies routing, fallbacks, budgets, key management, caching, guardrails, and logging in one place.
- An AI gateway differs from a traditional API gateway because it meters tokens instead of requests, parses prompts and responses, and handles streamed model output.
- An MCP gateway applies the same idea to agent tool traffic, centralizing access control and audit logs for Model Context Protocol servers.
- Open source gateways such as Bifrost, LiteLLM, and Agent Router (formerly Envoy AI Gateway) run in your own infrastructure, while managed gateways such as Cloudflare AI Gateway and Vercel AI Gateway remove the operations work.
- A gateway adds a network hop and becomes a critical dependency, so it needs replicas, health checks, and latency budgets like any other production proxy.

An AI gateway is a proxy that sits between your applications and the model providers they call, exposing one API and enforcing shared policy on every request. An AI gateway centralizes the concerns that otherwise get reimplemented in each service: provider credentials, routing and [fallbacks](https://theaiengineering.com/glossary/model-failover), rate limits and budgets, caching, guardrails, and request logs. This explainer covers where a gateway sits, what it does, how it differs from an API gateway, what an MCP gateway adds, and which open source and managed options exist. For related guides, see our [AI gateways, routing, and caching](https://theaiengineering.com/topics/ai-gateways) hub.

## What is an AI gateway?

An [AI gateway](https://theaiengineering.com/glossary/ai-gateway) is a reverse proxy for model traffic that gives applications one endpoint for many models and applies authentication, routing, budgets, caching, guardrails, and logging centrally. Applications send requests in one format, usually the [OpenAI-compatible](https://theaiengineering.com/glossary/openai-compatible-api) chat format, and the gateway translates each request for the selected provider and normalizes the response.

The term "LLM gateway" usually means the same thing. Some vendors reserve "AI gateway" for products that also handle embeddings, images, audio, and agent tool traffic. Teams typically adopt a gateway when one of these becomes true:

- More than one team calls models, and each team manages its own provider keys.
- The organization uses more than one provider from our [model providers](https://theaiengineering.com/platforms/model-providers) and [inference providers](https://theaiengineering.com/platforms/inference-providers) directories, or a self-hosted model.
- Finance needs spend attributed to teams, features, or customers.
- Security or compliance needs one place to enforce data policies and keep audit logs.
- An outage at one provider has already taken a product down.

## How does an AI gateway work?

An AI gateway works as a policy pipeline on the request path. Each call passes through authentication, budget and [rate-limit checks](https://theaiengineering.com/glossary/rate-limiting), input guardrails, a cache lookup, and model selection before it reaches a provider. On the way back, the gateway applies output guardrails, records tokens, latency, and cost, and returns a response in a consistent format.

**Figure 1: An AI gateway gives every application one endpoint and applies shared policy before any request reaches a provider.**

Three client applications send requests to one AI gateway, which applies shared policy and routes to two provider APIs and a self-hosted model.

Components:

- **Web app**
- **Agent**
- **Batch job**
- **AI gateway**: auth · budgets · routing · cache · guardrails logs · cost tracking · MCP tool access
- **Provider A**: Hosted API
- **Provider B**: Hosted API
- **Self-hosted model**: vLLM or SGLang

Connections:

- Web app → AI gateway
- Agent → AI gateway: one endpoint
- Batch job → AI gateway
- AI gateway → Provider A
- AI gateway → Provider B: fallback
- AI gateway → Self-hosted model

A typical request moves through seven steps:

1. The application sends an OpenAI-compatible request with a gateway-issued key and a model name or alias.
2. The gateway maps the key to a team, project, or customer and checks its rate limits and remaining budget.
3. Input guardrails scan the prompt for policy violations, such as personal data or injection patterns.
4. The gateway checks its cache and returns a stored response on a hit.
5. On a miss, routing rules select a provider and model, and the gateway attaches that provider's credential.
6. If the provider returns an error or times out, the gateway retries or falls back to the next configured target.
7. The response passes output guardrails, the gateway logs usage and cost, and the result streams back to the caller.

## Core capabilities of an AI gateway

The core capabilities of an AI gateway are a unified API, routing with fallbacks, rate limits and budgets, centralized key management, caching, guardrails, and logging with cost tracking. Products differ mainly in depth: how many providers they translate, how fine-grained their budgets are, and whether guardrails and MCP support are built in or delegated.

### Unified API

A unified API lets applications switch models by changing a model name instead of rewriting integration code. The gateway translates request fields, tool definitions, and streaming formats between providers. Translation is never perfect: provider-specific features such as cache-control parameters or reasoning settings may not pass through every gateway, so test the features you depend on.

### Routing and fallbacks

Routing selects which model and provider serve each request, based on rules, cost, latency, or provider health. Fallbacks send a failed request to a backup target after an error, timeout, or rate-limit response. Our guide to [LLM routing](https://theaiengineering.com/articles/llm-routing) covers [routing](https://theaiengineering.com/glossary/llm-routing) strategies and load balancing in depth.

### Rate limits and budgets

Model traffic is metered in tokens, so useful limits are tokens per minute and spend per day or month, applied per key, team, or customer. A budget stops a runaway agent loop or a leaked key from turning into a large invoice. Request-per-second limits alone miss the difference between a ten-token request and a hundred-thousand-token one.

### Key management

The gateway holds provider credentials, and applications receive gateway-issued [virtual keys](https://theaiengineering.com/glossary/virtual-key) instead. Revoking a team's access, rotating a provider key, or moving traffic to a new account then requires one change in one place. The gateway becomes a high-value target as a result, so its own credentials belong in a secret manager.

### Caching

Exact-match caching returns a stored response for an identical request. [Semantic caching](https://theaiengineering.com/glossary/semantic-caching) returns a stored response for a request with similar meaning, measured by embedding similarity, which raises hit rates and also the risk of returning a wrong answer. Gateway caching is separate from provider-side prompt caching, which reuses computation for a repeated prompt prefix but still generates a fresh response.

### Guardrails

[Guardrails](https://theaiengineering.com/glossary/guardrails) inspect prompts and responses against policy: detecting personal data, blocking [prompt injection](https://theaiengineering.com/glossary/prompt-injection) patterns, filtering unsafe output, or validating response formats. A gateway can run checks itself or call external classifiers. Output guardrails on streamed responses are harder, because the response reaches the gateway in chunks, and the gateway must either buffer it or check it incrementally.

### Logging and cost tracking

Because every request passes through it, the gateway can record model, provider, tokens, latency, status, and cost for all traffic without changes to application code. Gateway logs show per-request facts; they do not show what happened inside an application's retrieval or agent loop. Our guide to [LLM observability](https://theaiengineering.com/articles/llm-observability) covers application-level tracing.

## AI gateway vs API gateway

An AI gateway differs from an API gateway in how much of the traffic it inspects. An API gateway routes HTTP requests by path and header and treats the body as opaque. An AI gateway parses prompts and responses, meters tokens and spend, routes by model and provider health, caches completions, and handles streamed output token by token.

| | API gateway | AI gateway |
|---|---|---|
| Unit of traffic | HTTP request | Model call, measured in tokens |
| Rate limiting | Requests per second | Tokens per minute and spend per period |
| Routing basis | Path, header, service health | Model name, cost, latency, provider health, content |
| Payload awareness | Body usually opaque | Parses prompts, tool calls, and responses |
| Caching | Exact response caching | Exact and semantic caching of completions |
| Streaming | Pass-through | Counts tokens and applies policy on streamed output |
| Security | Authentication, authorization, WAF rules | Adds injection checks, PII redaction, output filters |
| Telemetry | Status codes and latency | Adds tokens, cost, model, and finish reasons |

The categories are converging. Kong describes its open source project as "the API and AI gateway," and Azure API Management documents AI gateway capabilities, such as a token-limit policy and semantic caching policies, as extensions of its existing gateway. If an organization already runs an API gateway, check its AI features before adding a second proxy.

## LLM gateway vs MCP gateway

An LLM gateway governs traffic from applications to models. An [MCP gateway](https://theaiengineering.com/glossary/mcp-gateway) governs traffic from agents to tools exposed through the [Model Context Protocol](https://theaiengineering.com/glossary/model-context-protocol). Both centralize authentication, policy, and audit logs, but an MCP gateway controls which tools each agent may discover and call, rather than which models it may use.

An agent connected directly to many MCP servers holds a credential for each server and exposes every tool to the model. An MCP gateway gives the agent one endpoint, filters the tool catalog per user, handles authorization with each server, and logs every tool call. Open source examples include [Bifrost](https://docs.getbifrost.ai/mcp/overview), which exposes the MCP servers it connects to through one `/mcp` endpoint and filters tools per virtual key; Docker MCP Gateway; Microsoft MCP Gateway, a reverse proxy and management layer for MCP servers on Kubernetes; IBM ContextForge; and agentgateway, a Linux Foundation project that also proxies A2A agent-to-agent traffic.

Several AI gateways, including Bifrost, LiteLLM, Kong AI Gateway, and Agent Router, now include MCP features, so one proxy can govern both model calls and tool calls. Our explainer on [what the Model Context Protocol is](https://theaiengineering.com/articles/what-is-model-context-protocol) covers the protocol these gateways proxy.

## AI gateway open source and managed options

AI gateway options split into open source gateways you run yourself and managed gateways a vendor operates. Open source gateways keep prompts inside your network and allow custom plugins, but you operate a critical proxy. Managed gateways remove that work and add a third party to the data path, along with data processing terms and region choices.

| Gateway | Model | What it is |
|---|---|---|
| Bifrost | Open source (Apache 2.0), self-hosted | Go gateway from Maxim AI with an OpenAI-compatible API, failover, load balancing, semantic caching, virtual keys, budgets, and MCP |
| Agent Router | Open source (Apache 2.0), self-hosted | Formerly Envoy AI Gateway; built on Envoy Gateway with an OpenAI-compatible API, provider fallback, token limits, and MCP routing |
| Kong AI Gateway | Built on Kong Gateway (Apache 2.0) | AI policies for semantic caching, prompt guards, token rate limiting, and MCP |
| LiteLLM | Open source (MIT outside its enterprise directory), self-hosted | Python SDK and OpenAI-compatible proxy for 100+ providers with keys, budgets, fallbacks, caching, guardrails, and MCP |
| Portkey Gateway | Open source (MIT) | TypeScript gateway with integrated guardrails |
| Cloudflare AI Gateway | Managed | Analytics, logging, caching, rate limiting, retries, model fallback, and dynamic routing |
| Vercel AI Gateway | Managed | One endpoint across providers with budgets, bring-your-own-key, provider and model fallbacks, and request logs |
| OpenRouter | Managed model aggregator | Hundreds of models behind one OpenAI-compatible endpoint with automatic fallbacks |

Two entries changed recently. Envoy AI Gateway became Agent Router and [joined the Agentic AI Foundation](https://aaif.io/blog/agent-router-joins-aaif) in September 2026; the announcement states that the license remains Apache 2.0 and that existing users do not need to migrate. The [LiteLLM documentation](https://docs.litellm.ai/docs/) distinguishes its SDK, for calling providers from code, from its proxy server, which platform teams run as a shared gateway.

Managed gateways document their scope clearly. [Cloudflare AI Gateway](https://developers.cloudflare.com/ai-gateway/) lists analytics, logging, caching, rate limiting, and retries with model fallback. [Vercel AI Gateway](https://vercel.com/docs/ai-gateway) accepts OpenAI Chat Completions, OpenAI Responses, and Anthropic Messages formats and works from infrastructure outside Vercel. Our [AI gateways directory](https://theaiengineering.com/platforms/ai-gateways) lists more options by deployment model.

## Do you need an AI gateway? Build vs buy

You need an AI gateway when several teams or products share model access, when you use more than one provider, or when budgets, audit logs, and failover must be enforced centrally. A single application calling one provider can usually rely on the provider SDK's retries and application tracing. Building a thin proxy is easy; building a dependable gateway is not.

A first version of an in-house proxy is a small project. The ongoing work is what pushes teams toward an existing gateway:

- **Provider drift:** providers add parameters, tool formats, and streaming event types, and each change needs translation work.
- **Token accounting:** providers report usage in different shapes, including cached and reasoning tokens, and streamed responses report usage differently from buffered ones.
- **Streaming:** the proxy must forward server-sent events without buffering, or users lose the time to first token they expect.
- **Multi-tenancy:** keys, budgets, and logs must be isolated per team or customer.
- **Availability:** the proxy needs replicas, health checks, and graceful restarts, because every model call depends on it.

| Option | Choose when | Watch for |
|---|---|---|
| Provider SDK only | One app, one provider, simple needs | Duplicated logic as apps multiply |
| Self-hosted open source gateway | Data must stay in your network, or you need custom plugins | You operate a critical proxy |
| Managed gateway | You want governance without running infrastructure | Data processing terms, region, extra vendor |
| Existing API gateway with AI features | You already run Kong, Azure API Management, or similar | Depth of model-specific features |

## Risks and trade-offs of running an AI gateway

An AI gateway concentrates risk as well as control. It adds a network hop to every model call, becomes a single point of failure if deployed carelessly, holds every provider credential, and can hide provider-specific features behind a common API. Each risk has a standard mitigation that should be in place before production traffic moves.

- **Availability:** run several replicas behind a load balancer, and decide whether critical flows can bypass the gateway and call a provider directly when the gateway is down.
- **Latency:** measure the gateway's added p99 latency under realistic load, and deploy it close to both applications and providers.
- **Credential exposure:** store provider keys in a secret manager, restrict admin access, and audit configuration changes.
- **Feature lag:** keep a passthrough route for provider features the gateway does not yet translate.
- **Cache correctness:** scope caches per tenant and set conservative similarity thresholds for semantic caching.

Where to go next: our guide to [LLM routing](https://theaiengineering.com/articles/llm-routing) covers how a gateway selects models and handles fallbacks, and our guide to [LLM observability](https://theaiengineering.com/articles/llm-observability) covers the tracing that complements gateway logs.

## Frequently asked questions

### What does an LLM gateway do?

An LLM gateway receives every model request from your applications, authenticates the caller, checks budgets and rate limits, applies guardrails, selects a model and provider, and forwards the call. On the way back it records tokens, latency, cost, and errors, optionally caches the response, and returns it in a consistent format. Applications use one endpoint instead of integrating each provider separately.

### Is OpenRouter an AI gateway?

OpenRouter is a hosted model aggregator that provides several AI gateway functions: one OpenAI-compatible endpoint for hundreds of models and automatic fallbacks between providers. Unlike a self-hosted gateway, OpenRouter runs outside your infrastructure, and your requests pass through its service. Teams that need internal budgets, custom guardrails, or self-hosted logging often place their own gateway in front of OpenRouter and direct providers.

### Is LiteLLM open source?

Yes. LiteLLM's repository is MIT-licensed, except for content in its enterprise directory, which has a separate license. The project includes a Python SDK that calls more than 100 providers in a unified format and a self-hosted, OpenAI-compatible proxy server that adds virtual keys, budgets, rate limits, load balancing, fallbacks, caching, guardrails, logging, and MCP gateway features.

### What is an MCP gateway?

An MCP gateway is a proxy between AI agents and Model Context Protocol servers. It gives agents one endpoint for tool discovery, authenticates users, filters which tools each user or agent may call, and logs every tool invocation for audit. Examples include Bifrost, which governs model and tool traffic in one gateway, Docker MCP Gateway, Microsoft MCP Gateway, IBM ContextForge, and agentgateway.

### What is the difference between an AI gateway and an API gateway?

An API gateway routes and secures HTTP requests by path, header, and service health, and usually treats the request body as opaque. An AI gateway is built for model traffic: it parses prompts and responses, rate limits by tokens and spend, routes by model, cost, and provider health, caches completions, and handles streamed output. Some API gateways, such as Kong and Azure API Management, now add AI features.

### Is an LLM proxy the same as an LLM gateway?

The terms overlap, and many tools use them interchangeably. An LLM proxy usually describes a thinner layer that translates and forwards requests to providers, sometimes with retries and logging. An LLM gateway usually adds governance features on top: per-team keys, budgets, rate limits, guardrails, caching, and routing policies. Check the feature list rather than the label.

## Terms in this guide

- [AI gateway](https://theaiengineering.com/glossary/ai-gateway): An AI gateway is a proxy layer between applications and model providers that exposes one API for many models and centralizes routing, failover, access control, cost tracking, caching, and logging.
- [LLM routing](https://theaiengineering.com/glossary/llm-routing): LLM routing is the practice of directing each request to one of several language models or providers, based on rules or a learned classifier, to balance cost, quality, latency, and availability.
- [Semantic caching](https://theaiengineering.com/glossary/semantic-caching): Semantic caching is a technique that returns a stored LLM response when a new query is close in meaning to an earlier one, measured by embedding similarity instead of exact text match.
- [Guardrails](https://theaiengineering.com/glossary/guardrails): Guardrails are programmatic checks placed around a large language model that validate, filter, or block inputs, outputs, and tool calls so an AI application stays within its safety, security, and policy limits.
- [Model Context Protocol (MCP)](https://theaiengineering.com/glossary/model-context-protocol): Model Context Protocol (MCP) is an open protocol, introduced by Anthropic in November 2024, that standardizes how AI applications connect to external tools, data sources, and prompts through MCP servers.
- [Tokens](https://theaiengineering.com/glossary/tokens): Tokens are the units of text a language model processes, typically whole words, word fragments, punctuation, or bytes, each mapped to an integer ID from the model's fixed vocabulary.
- [Model failover](https://theaiengineering.com/glossary/model-failover): Model failover is the automatic rerouting of an LLM request to a backup model, deployment, or provider when the primary target returns an error, times out, or hits a rate limit.
- [Rate limiting](https://theaiengineering.com/glossary/rate-limiting): Rate limiting is the enforcement of a maximum number of requests or tokens a client may send to an API within a time window, with excess requests rejected, usually with HTTP 429.
- [OpenAI-compatible API](https://theaiengineering.com/glossary/openai-compatible-api): An OpenAI-compatible API is an HTTP API that accepts OpenAI's request and response formats, so code written for OpenAI's SDKs can call another server by changing the base URL, API key, and model name.
- [Virtual key](https://theaiengineering.com/glossary/virtual-key): A virtual key is an API key issued by an AI gateway that maps a caller to its allowed models, budget, and rate limits, while the gateway keeps the real provider credentials.
- [LLM load balancing](https://theaiengineering.com/glossary/llm-load-balancing): LLM load balancing is the distribution of requests for one model across several interchangeable deployments, such as API keys, regions, providers, or replicas, by weight, latency, load, or remaining quota.
- [LLM streaming](https://theaiengineering.com/glossary/llm-streaming): LLM streaming is the delivery of a language model's output in small chunks as it is generated, usually as server-sent events over HTTP, so clients can show text before the response is complete.
- [MCP gateway](https://theaiengineering.com/glossary/mcp-gateway): An MCP gateway is a proxy that sits between MCP hosts and many MCP servers, exposing them through one endpoint and centralizing authentication, tool access policy, and audit logging.
- [Prompt caching](https://theaiengineering.com/glossary/prompt-caching): Prompt caching is an LLM API and inference-server feature that stores the computed attention state for a repeated prompt prefix, so later requests sharing that prefix skip reprocessing it.
- [MCP server](https://theaiengineering.com/glossary/mcp-server): An MCP server is a program that exposes tools, resources, and prompts to AI applications through the Model Context Protocol, running locally over stdio or remotely over Streamable HTTP.
- [A2A protocol](https://theaiengineering.com/glossary/a2a-protocol): The A2A (Agent2Agent) protocol is an open standard, introduced by Google in April 2025, that lets AI agents built on different frameworks discover each other, exchange messages, and delegate tasks.
- [Time to first token](https://theaiengineering.com/glossary/time-to-first-token): Time to first token (TTFT) is the elapsed time from submitting a request to an LLM until the first token of the response arrives, covering request queuing, network transit, and prompt prefill.

## Related guides

- [LLM routing: strategies, fallbacks, and learned routers](https://theaiengineering.com/articles/llm-routing): LLM routing guide: rule-based and cost-aware strategies, fallbacks, load balancing, semantic and learned routers like RouteLLM, router evals, and pitfalls.
- [LLM observability: what to capture, which metrics, and tools](https://theaiengineering.com/articles/llm-observability): LLM observability guide: what to capture in traces and spans, metrics to track, OpenTelemetry GenAI conventions, tool categories, and keeping prompts private.
- [What is the Model Context Protocol (MCP)? How it works](https://theaiengineering.com/articles/what-is-model-context-protocol): The Model Context Protocol (MCP) is an open standard for connecting AI apps to tools and data. Learn its architecture, transports, tool calls, and security.
- [Bifrost vs LiteLLM: architecture, features, and performance compared](https://theaiengineering.com/articles/bifrost-vs-litellm): Bifrost vs LiteLLM compared: a Go single binary against a Python proxy, with deployment, routing, virtual keys, caching, MCP, benchmarks, and migration.

Part of the [AI gateways and LLM routing](https://theaiengineering.com/topics/ai-gateways) topic hub.

## Cite this guide

The AI Engineering Editors. "What is an AI gateway? How it works and when you need one." The AI Engineering, updated Nov 11, 2026. https://theaiengineering.com/articles/what-is-an-ai-gateway

- Title: What is an AI gateway? How it works and when you need one
- Author: The AI Engineering Editors
- Publisher: The AI Engineering
- Published: 2026-10-01
- Updated: 2026-11-11
- URL: https://theaiengineering.com/articles/what-is-an-ai-gateway
- Markdown: https://theaiengineering.com/articles/what-is-an-ai-gateway.md

To cite one section, link to its anchor:

- [What is an AI gateway?](https://theaiengineering.com/articles/what-is-an-ai-gateway#what-is-an-ai-gateway)
- [How does an AI gateway work?](https://theaiengineering.com/articles/what-is-an-ai-gateway#how-does-an-ai-gateway-work)
- [Core capabilities of an AI gateway](https://theaiengineering.com/articles/what-is-an-ai-gateway#core-capabilities-of-an-ai-gateway)
- [AI gateway vs API gateway](https://theaiengineering.com/articles/what-is-an-ai-gateway#ai-gateway-vs-api-gateway)
- [LLM gateway vs MCP gateway](https://theaiengineering.com/articles/what-is-an-ai-gateway#llm-gateway-vs-mcp-gateway)
- [AI gateway open source and managed options](https://theaiengineering.com/articles/what-is-an-ai-gateway#ai-gateway-open-source-and-managed-options)
- [Do you need an AI gateway? Build vs buy](https://theaiengineering.com/articles/what-is-an-ai-gateway#do-you-need-an-ai-gateway-build-vs-buy)
- [Risks and trade-offs of running an AI gateway](https://theaiengineering.com/articles/what-is-an-ai-gateway#risks-and-trade-offs-of-running-an-ai-gateway)
- [Frequently asked questions](https://theaiengineering.com/articles/what-is-an-ai-gateway#faq)
