---
title: "What is an AI gateway? How LLM gateways work"
description: "An AI gateway is a proxy between apps and LLM providers that handles routing, failover, caching, and logging. Learn how it differs from an API gateway."
type: "glossary-term"
canonical_url: "https://theaiengineering.com/glossary/ai-gateway"
markdown_url: "https://theaiengineering.com/glossary/ai-gateway.md"
topic: "AI gateways and LLM routing"
author: "The AI Engineering Editors"
publisher: "The AI Engineering"
updated: "2026-11-11"
keywords: ["AI gateway","LLM gateway","AI proxy","model gateway"]
citation: "The AI Engineering Editors. \"What is an AI gateway? How LLM gateways work.\" The AI Engineering, updated Nov 11, 2026. https://theaiengineering.com/glossary/ai-gateway"
---

# What is an AI gateway? How LLM gateways work

Also called LLM gateway, AI proxy, model gateway.

> **Definition of AI gateway:** An AI gateway is a proxy layer between applications and model providers that exposes one API for many models and centralizes routing, failover, access control, cost tracking, caching, and logging.

An AI gateway is a proxy that sits between applications and the model providers they call. Applications send every model request to the gateway, which forwards it to the chosen provider and returns the response. Because all model traffic passes through one point, the gateway can apply routing, [failover](https://theaiengineering.com/glossary/model-failover), access control, budgets, caching, and logging consistently across teams and providers.

## How an AI gateway works

An AI gateway works by exposing one endpoint, often in an [OpenAI-compatible format](https://theaiengineering.com/glossary/openai-compatible-api), and translating each request into the target provider's API.

Open source gateways such as [Bifrost](https://docs.getbifrost.ai/features/drop-in-replacement) and [LiteLLM](https://docs.litellm.ai/docs/simple_proxy) and managed services such as [Cloudflare AI Gateway](https://developers.cloudflare.com/ai-gateway/) document a similar set of functions:

- **Unified API and keys**: one client format, one place to store provider credentials, and per-team virtual keys with their own budgets and rate limits.
- **Routing and failover**: [LLM routing](https://theaiengineering.com/glossary/llm-routing) by model alias, cost, or latency, with retries on another provider when one fails.
- **Caching**: exact-match or [semantic caching](https://theaiengineering.com/glossary/semantic-caching) to skip repeated calls.
- **Guardrails**: input and output [guardrails](https://theaiengineering.com/glossary/guardrails) for sensitive data, prompt injection, or policy violations.
- **Observability**: request logs, token counts, latency, and cost per key, which feed [LLM observability](https://theaiengineering.com/glossary/llm-observability) tools.

Some gateways also proxy [MCP server](https://theaiengineering.com/glossary/mcp-server) and agent-to-agent traffic. Bifrost, for example, runs an [MCP gateway](https://docs.getbifrost.ai/mcp/overview) and an [Agent Gateway](https://docs.getbifrost.ai/features/agent-gateway/overview) for A2A agents next to its model routing, and LiteLLM documents support for MCP tools and A2A agents alongside model calls.

## AI gateway vs API gateway

An AI gateway is a specialized API gateway for model traffic. A general API gateway handles routing, authentication, and request-count [rate limits](https://theaiengineering.com/glossary/rate-limiting) for any HTTP service. An AI gateway adds model-aware features: limits measured in tokens rather than requests, cost tracking per model, translation between provider formats, streamed responses, retries across providers, and inspection of prompt and completion content.

The two categories overlap in practice. Microsoft's documentation for [AI gateway capabilities in Azure API Management](https://learn.microsoft.com/en-us/azure/api-management/genai-gateway-capabilities) describes them as an extension of the product's existing API gateway, adding token limit policies, token metrics, and semantic caching rather than shipping a separate product.

[What is an AI gateway](https://theaiengineering.com/articles/what-is-an-ai-gateway) covers architecture and deployment choices, and the [AI gateways directory](https://theaiengineering.com/platforms/ai-gateways) lists available options.

## Related terms

- [LLM routing](https://theaiengineering.com/glossary/llm-routing): LLM routing is the practice of directing each request to one of several language models or providers, based on rules or a learned classifier, to balance cost, quality, latency, and availability.
- [Semantic caching](https://theaiengineering.com/glossary/semantic-caching): Semantic caching is a technique that returns a stored LLM response when a new query is close in meaning to an earlier one, measured by embedding similarity instead of exact text match.
- [Guardrails](https://theaiengineering.com/glossary/guardrails): Guardrails are programmatic checks placed around a large language model that validate, filter, or block inputs, outputs, and tool calls so an AI application stays within its safety, security, and policy limits.
- [LLM observability](https://theaiengineering.com/glossary/llm-observability): LLM observability is the practice of instrumenting LLM applications to record each request's prompts, responses, tool calls, token usage, latency, cost, and quality scores as traces that teams can search and analyze.
- [MCP server](https://theaiengineering.com/glossary/mcp-server): An MCP server is a program that exposes tools, resources, and prompts to AI applications through the Model Context Protocol, running locally over stdio or remotely over Streamable HTTP.
- [Model failover](https://theaiengineering.com/glossary/model-failover): Model failover is the automatic rerouting of an LLM request to a backup model, deployment, or provider when the primary target returns an error, times out, or hits a rate limit.
- [Rate limiting](https://theaiengineering.com/glossary/rate-limiting): Rate limiting is the enforcement of a maximum number of requests or tokens a client may send to an API within a time window, with excess requests rejected, usually with HTTP 429.
- [Virtual key](https://theaiengineering.com/glossary/virtual-key): A virtual key is an API key issued by an AI gateway that maps a caller to its allowed models, budget, and rate limits, while the gateway keeps the real provider credentials.
- [OpenAI-compatible API](https://theaiengineering.com/glossary/openai-compatible-api): An OpenAI-compatible API is an HTTP API that accepts OpenAI's request and response formats, so code written for OpenAI's SDKs can call another server by changing the base URL, API key, and model name.
- [MCP gateway](https://theaiengineering.com/glossary/mcp-gateway): An MCP gateway is a proxy that sits between MCP hosts and many MCP servers, exposing them through one endpoint and centralizing authentication, tool access policy, and audit logging.

## Guides that cover ai gateway

- [What is an AI gateway? How it works and when you need one](https://theaiengineering.com/articles/what-is-an-ai-gateway): An AI gateway is a proxy between your apps and model providers. Learn how it works, its core features, AI gateway vs API gateway, and open source options.
- [LLM routing: strategies, fallbacks, and learned routers](https://theaiengineering.com/articles/llm-routing): LLM routing guide: rule-based and cost-aware strategies, fallbacks, load balancing, semantic and learned routers like RouteLLM, router evals, and pitfalls.
- [LLM observability: what to capture, which metrics, and tools](https://theaiengineering.com/articles/llm-observability): LLM observability guide: what to capture in traces and spans, metrics to track, OpenTelemetry GenAI conventions, tool categories, and keeping prompts private.
- [Bifrost vs LiteLLM: architecture, features, and performance compared](https://theaiengineering.com/articles/bifrost-vs-litellm): Bifrost vs LiteLLM compared: a Go single binary against a Python proxy, with deployment, routing, virtual keys, caching, MCP, benchmarks, and migration.
- [Semantic caching for LLMs: how it works and when it pays off](https://theaiengineering.com/articles/semantic-caching): Semantic caching returns a stored LLM response for prompts with similar meaning. Learn how a semantic cache works, how to tune it, and what never to cache.
- [How to build an MCP server: a Python and TypeScript tutorial](https://theaiengineering.com/articles/how-to-build-an-mcp-server): Learn how to build an MCP server with the official Python and TypeScript SDKs: tools, resources, transports, testing, client config, and production security.
- [A2A protocol vs MCP: differences and when to use each](https://theaiengineering.com/articles/a2a-vs-mcp): A2A protocol vs MCP: A2A connects AI agents to other agents, and MCP connects an agent to tools and data. Compare agent cards, tasks, transports, and security.
- [What is the Model Context Protocol (MCP)? How it works](https://theaiengineering.com/articles/what-is-model-context-protocol): The Model Context Protocol (MCP) is an open standard for connecting AI apps to tools and data. Learn its architecture, transports, tool calls, and security.
- [What is AI engineering? The discipline, the stack, and the work](https://theaiengineering.com/articles/what-is-ai-engineering): What is AI engineering? A plain definition, how it differs from ML and software engineering, the layers of the stack, and what AI engineers do day to day.

Part of the [AI gateways and LLM routing](https://theaiengineering.com/topics/ai-gateways) topic hub.

## Cite this definition

The AI Engineering Editors. "What is an AI gateway? How LLM gateways work." The AI Engineering, updated Nov 11, 2026. https://theaiengineering.com/glossary/ai-gateway

- Title: What is an AI gateway? How LLM gateways work
- Author: The AI Engineering Editors
- Publisher: The AI Engineering
- Updated: 2026-11-11
- URL: https://theaiengineering.com/glossary/ai-gateway
- Markdown: https://theaiengineering.com/glossary/ai-gateway.md
