AI gateways
An AI gateway is a proxy between applications and model providers that exposes one API while handling routing, failover, authentication, rate limits, caching, and usage tracking. Teams adopt one when several applications or providers need to share credentials, budgets, and compliance rules.
The options fall into four groups. Open-source gateways such as Bifrost, LiteLLM, Agent Router, and agentgateway run inside your own network. Managed services such as Cloudflare AI Gateway, Vercel AI Gateway, and OpenRouter remove operations work and, in some cases, consolidate billing. API management vendors, including Kong, Google Apigee, and Azure API Management, extend gateways that already carry REST traffic, while AWS and Databricks govern model traffic inside their own platforms.
Gateways now govern more than model calls: many also proxy Model Context Protocol servers and agent-to-agent traffic. The market is consolidating as well. Palo Alto Networks acquired Portkey in 2026, and the TensorZero repository was archived the same year.
AI gateways compared
| Platform | Open source | Deployment | Best for |
|---|---|---|---|
| Agent Router (formerly Envoy AI Gateway) | Yes (Apache-2.0) | Self-hosted | Kubernetes platform teams standardized on Envoy |
| agentgateway | Yes (Apache-2.0) | Self-hosted | One proxy for LLM, MCP, and A2A traffic |
| Amazon Bedrock AgentCore Gateway | No | Managed cloud | AWS teams connecting agents to tools and models |
| Apache APISIX | Yes (Apache-2.0) | Self-hosted | Teams already running APISIX for API traffic |
| Apigee AI gateway | No | Managed cloud, Self-hosted | Google Cloud teams with existing Apigee deployments |
| Azure API Management AI gateway | No | Managed cloud, Self-hosted | Azure and Microsoft Foundry customers |
| Bifrost | Yes (Apache-2.0) | Self-hosted | Low-overhead self-hosted gateway for LLM and MCP traffic |
| Cloudflare AI Gateway | No | Managed cloud | Teams already building on Cloudflare Workers |
| Databricks Unity Gateway | No | Managed cloud | Databricks customers governing AI under Unity Catalog |
| Eden AI | No | Managed cloud, API | European teams consolidating model access |
| F5 AI Gateway | No | Self-hosted | Enterprises standardized on F5 application delivery |
| Gravitee AI Gateway | No | Self-hosted, Managed cloud | Gravitee customers governing AI and API traffic together |
| Helicone AI Gateway | Yes (GPL-3.0) | Self-hosted, Managed cloud | Existing Helicone users who want a self-hosted router |
| Higress | Yes (Apache-2.0) | Self-hosted, Managed cloud | Istio users and Alibaba Cloud customers |
| IBM DataPower Interact Gateway | No | Self-hosted, Managed cloud | IBM API Connect and webMethods customers |
| Kong AI Gateway | Yes (Apache-2.0) | Self-hosted, Managed cloud | Teams already using Kong for API management |
| LiteLLM | Yes (MIT) | Self-hosted, Library | Self-hosted multi-provider proxy for platform teams |
| LLM Gateway | Yes (AGPL-3.0) | Self-hosted, Managed cloud | Small teams wanting a self-hostable gateway with analytics |
| MLflow AI Gateway | Yes (Apache-2.0) | Self-hosted | MLflow users adding a governed model endpoint |
| Multi-Provider Generative AI Gateway on AWS | Yes (MIT-0) | Self-hosted | Self-hosting LiteLLM on AWS with reference infrastructure |
| Netlify AI Gateway | No | Managed cloud | Projects already deployed on Netlify |
| New API | Yes (AGPL-3.0) | Self-hosted | Self-hosted key distribution and usage metering |
| Not Diamond | No | Managed cloud, API | Routing coding-agent requests across frontier models |
| OpenRouter | No | Managed cloud, API | Trying many models through a single account |
| Plano (formerly Arch Gateway) | Yes (Apache-2.0) | Self-hosted | Moving routing and guardrails out of agent code |
| Prisma AIRS AI Gateway (formerly Portkey) | Yes (MIT) | Managed cloud, Self-hosted | Enterprises pairing an AI gateway with Palo Alto security |
| Requesty | No | Managed cloud, API | Teams that need EU data residency for model traffic |
| Traefik AI Gateway | No | Self-hosted | Kubernetes teams already running Traefik |
| TrueFoundry AI Gateway | No | Managed cloud, Self-hosted | Enterprises mixing hosted APIs and self-hosted models |
| Tyk AI Studio | Yes (AGPL-3.0) | Self-hosted | Tyk users adding AI governance |
| Vercel AI Gateway | No | Managed cloud | TypeScript apps using the AI SDK |
| Zuplo AI Gateway | No | Managed cloud | API teams already using Zuplo |
Profiles

Agent Router (formerly Envoy AI Gateway)
theagentrouter.ai
Agent Router, formerly Envoy AI Gateway, is an Agentic AI Foundation project built on Envoy Gateway that gives applications one OpenAI-compatible API for hosted and self-hosted models and MCP servers, configured through Kubernetes resources.
Envoy AI Gateway was renamed Agent Router and moved to the Agentic AI Foundation, keeping its Apache-2.0 license, CRDs, container images, and aigw CLI.
agentgateway
agentgateway.dev
agentgateway is an open-source proxy written in Rust and hosted by the Linux Foundation that handles LLM, MCP, agent-to-agent, and HTTP traffic in one data plane, with budgets, failover, guardrails, and inference routing to self-hosted models; Solo.io sells an enterprise edition.

Amazon Bedrock AgentCore Gateway
Amazon Web Services · aws.amazon.com
Amazon Bedrock AgentCore Gateway is a managed AWS service that turns APIs, Lambda functions, and existing services into MCP tools, fronts other agents, and routes inference across model providers through one endpoint, with inbound and outbound authentication and semantic tool search.

Apache APISIX
Apache Software Foundation · apisix.apache.org
Apache APISIX is an open-source API gateway from the Apache Software Foundation whose AI plugins proxy and load-balance requests across model providers, with retries, token-based rate limits, prompt controls, RAG support, and gateway-level observability.

Apigee AI gateway
Google Cloud · cloud.google.com
Apigee, Google Cloud's API management platform, adds AI gateway capabilities for model and agent traffic, including cross-provider model routing, LLM token limit policies, semantic caching, Model Armor prompt screening, and authorization for MCP proxies.
Azure API Management AI gateway
Microsoft · learn.microsoft.com
The AI gateway in Azure API Management is a set of policies that govern model APIs in OpenAI, Anthropic, and Vertex AI formats, MCP servers, and A2A agents, with token limits, semantic caching, load balancing, circuit breakers, and content safety checks.

Bifrost
Maxim AI · getmaxim.ai
Bifrost is an open-source AI gateway from Maxim AI, written in Go, that unifies many model providers behind one OpenAI-compatible API and also serves as an MCP gateway, with fallbacks, load balancing, semantic caching, virtual keys, budgets, and guardrails.
Cloudflare AI Gateway
Cloudflare · cloudflare.com
Cloudflare AI Gateway is a managed gateway on Cloudflare's network that proxies requests to model providers and adds analytics, logging, caching, rate limiting, dynamic routing with fallbacks, and unified billing for AI workloads.
Databricks Unity Gateway
Databricks · databricks.com
Databricks Unity Gateway governs model, agent, and MCP traffic inside Databricks through Unity Catalog, with identity-based policies, request and token limits, PII and PHI guardrails, budgets, fallbacks, governed inference tables, and a CLI for managing coding agents.
Unity Gateway replaces Databricks' earlier AI Gateway for serving endpoints, which the documentation now describes as the legacy version.
Eden AI
edenai.co
Eden AI is an AI gateway built in France that provides one OpenAI-compatible API to a large catalog of models, with cost-based routing, budgets, and per-team model access, aimed at European teams with GDPR requirements.
F5 AI Gateway
F5 · f5.com
F5 AI Gateway combines a model gateway, an MCP gateway, and AI guardrails under one control plane, with spend tracking by provider, team, and user, semantic caching, smart and GPU-aware routing, and audited agent-to-tool access.
Gravitee AI Gateway
Gravitee · gravitee.io
Gravitee AI Gateway extends the Gravitee API management platform to LLM, MCP server, and agent-to-agent traffic, applying shared authentication, policy enforcement, observability, and cost visibility to each.

Helicone AI Gateway
Helicone · helicone.ai
Helicone AI Gateway is an open-source gateway written in Rust and licensed GPL-3.0 that routes OpenAI-format requests across model providers with load balancing, rate limits, and response caching, and sends traces to Helicone's observability platform.
Mintlify acquired Helicone in March 2026 and moved it to maintenance mode; Helicone's site now states that it is no longer part of Mintlify.
Higress
higress.ai
Higress is an open-source AI gateway and CNCF Sandbox project built on Istio and Envoy that gives LLM APIs, agents, and MCP servers unified access, traffic control, security, and observability, with a managed edition sold as Alibaba Cloud AI Gateway.

IBM DataPower Interact Gateway
IBM · ibm.com
IBM DataPower Interact Gateway is a Kubernetes-native gateway built on the DataPower Nano Gateway framework that mediates LLM and MCP traffic, turns existing APIs into agent tools, and adds token-based rate limiting, response caching, and analytics.
Kong AI Gateway
Kong Inc. · konghq.com
Kong AI Gateway adds AI plugins to Kong Gateway for multi-provider routing, load balancing, semantic caching, semantic prompt guards, PII sanitization, and token quotas, and also governs MCP traffic; core AI plugins ship in the open-source edition.

LiteLLM
BerriAI · litellm.ai
LiteLLM is an open-source AI gateway and Python SDK, MIT-licensed outside its enterprise directory, that exposes more than 100 model providers through an OpenAI-compatible API with virtual keys, spend tracking, budgets, guardrails, and load balancing.

LLM Gateway
llmgateway.io
LLM Gateway is an AI gateway whose core is open source under AGPL-3.0, with source-available enterprise code, that routes requests to many models through one OpenAI-compatible API and adds key management and cost analytics, self-hosted or as a managed cloud.
MLflow AI Gateway
mlflow.org
MLflow AI Gateway is the open-source gateway built into MLflow that manages OpenAI, Anthropic, and other providers through one secure endpoint, with centralized credentials, access control, cost tracking, and rate limiting.

Multi-Provider Generative AI Gateway on AWS
Amazon Web Services · aws.amazon.com
This AWS Solutions Library guidance, released under MIT-0, deploys LiteLLM on Amazon ECS or EKS with AWS WAF, ElastiCache, RDS, and Secrets Manager, putting Amazon Bedrock and external providers behind one OpenAI-compatible API with virtual keys and usage logs.
Netlify AI Gateway
Netlify · netlify.com
Netlify AI Gateway gives sites, functions, and agents deployed on Netlify access to commercial models from OpenAI, Anthropic, and Google and open models such as Kimi and DeepSeek without separate provider accounts or API keys to manage.

New API
newapi.ai
New API is a self-hosted AI gateway written in Go and licensed AGPL-3.0, built on the earlier One API project, that exposes OpenAI-, Claude-, and Gemini-compatible endpoints over many upstream providers with API key management, model access control, and usage tracking.

Not Diamond
notdiamond.ai
Not Diamond is a model-routing service that chooses which LLM should handle each request; its current product targets coding agents, routing their calls across frontier models to balance output quality against cost.
OpenRouter
openrouter.ai
OpenRouter is a hosted service that provides hundreds of models from many providers through one OpenAI-compatible endpoint and one account, with automatic provider fallbacks and per-request routing preferences.
Plano (formerly Arch Gateway)
Katanemo · planoai.dev
Plano, formerly Arch Gateway, is an open-source proxy and data plane from Katanemo, written in Rust, that sits in front of agents as an AI-native proxy and handles agent orchestration, LLM routing, guardrails, and observability.
Katanemo renamed its Arch gateway project Plano; the former katanemo/archgw repository now redirects to katanemo/plano.

Prisma AIRS AI Gateway (formerly Portkey)
Palo Alto Networks · portkey.ai
Prisma AIRS AI Gateway, previously Portkey, routes requests to many model providers and adds guardrails, observability, governance, prompt management, and an MCP gateway; its core gateway is MIT-licensed open source, and Palo Alto Networks connects it to Prisma AIRS security.
Palo Alto Networks completed its acquisition of Portkey on May 29, 2026, and now presents the product as Prisma AIRS AI Gateway.

Requesty
requesty.ai
Requesty is a managed AI gateway and LLM router that exposes models from OpenAI, Anthropic, Google, and other providers through one OpenAI-compatible API, with routing, caching, failover, observability, EU data residency, and governance controls.
Traefik AI Gateway
Traefik Labs · traefik.io
Traefik AI Gateway, part of Traefik Hub, turns AI endpoints into managed APIs on infrastructure the customer runs, with connections to major LLMs, semantic caching, Presidio PII protection, NVIDIA safety NIM guardrails, and an MCP gateway, configured through GitOps.

TrueFoundry AI Gateway
TrueFoundry · truefoundry.com
TrueFoundry AI Gateway is an enterprise gateway for hosted and self-hosted models, with routing and fallbacks, governance policies, real-time monitoring, and deployment in a VPC, on premises, or in air-gapped environments; it can also serve open models through vLLM or SGLang.
Tyk AI Studio
Tyk · tyk.io
Tyk AI Studio is an AGPL-3.0 AI gateway and governance layer built on the Tyk API Gateway that acts as one control plane for LLM, agent, MCP tool, and RAG traffic, enforcing policies and attributing cost for every request.

Vercel AI Gateway
Vercel · vercel.com
Vercel AI Gateway is a managed endpoint for hundreds of text, image, video, and embedding models that accepts OpenAI- and Anthropic-format requests, supports provider fallbacks and bring-your-own-key, and integrates with Vercel's AI SDK.
Zuplo AI Gateway
Zuplo · zuplo.com
Zuplo AI Gateway runs LLM traffic through the same programmable API gateway as a team's other APIs, with multi-provider routing, semantic caching, prompt-injection protection, data loss prevention, and budget pools for teams and agents.
How to choose
- 01Deployment
- Self-hosted in your VPC or Kubernetes cluster, managed SaaS, or hybrid with a hosted control plane.
- 02Provider coverage and API format
- The number of supported providers and whether the gateway accepts OpenAI, Anthropic, or other native request formats.
- 03Routing and reliability
- Fallbacks, retries, load balancing across keys and regions, and latency-based or cost-based routing.
- 04Governance
- Virtual keys, per-team budgets, rate limits, audit logs, and single sign-on.
- 05Guardrails and MCP
- PII redaction, prompt-injection checks, and support for proxying MCP servers and tool calls.
- 06Overhead and observability
- Latency added per request, plus OpenTelemetry export and usage dashboards.
Frequently asked questions
What is an AI gateway?
- An AI gateway is a proxy that sits between applications and AI model providers. Applications send every request to the gateway through one API, and the gateway authenticates the caller, applies rate limits and budgets, routes the request to a provider, retries or fails over on errors, and records tokens, cost, and latency for each call.
AI gateway vs API gateway: what is the difference?
- A traditional API gateway routes and secures HTTP traffic to internal services based on paths, methods, and request counts. An AI gateway understands model traffic: it counts tokens, translates between provider API formats, streams responses, caches semantically similar prompts, and applies AI-specific guardrails. Some API gateways, such as Kong, Apache APISIX, and Azure API Management, add AI gateway features through plugins or policies.
Is LiteLLM open source?
- Yes. LiteLLM's repository is MIT-licensed except for content under its enterprise directory, which has a separate commercial license. Teams can self-host the proxy server, including in air-gapped environments, with their own provider keys, or call providers directly through the Python SDK. BerriAI, the company behind LiteLLM, also sells an enterprise tier.
LiteLLM vs OpenRouter: what is the difference?
- LiteLLM is open-source software that you run yourself, with your own provider API keys, to put many providers behind one OpenAI-compatible endpoint with budgets and virtual keys. OpenRouter is a hosted service that provides hundreds of models through one account and one endpoint. LiteLLM keeps traffic inside your infrastructure, while OpenRouter removes the need for separate provider accounts.
What is an LLM router?
- An LLM router is software that chooses which model handles each request, using rules such as cost, latency, or task type, or a trained classifier that predicts which model will answer a prompt well. Gateways such as Bifrost and LiteLLM route by configured rules and fallbacks, while services such as Not Diamond select a model for each prompt automatically.
AI gateway vs MCP gateway: what is the difference?
- An AI gateway governs requests to language models, while an MCP gateway governs connections between agents and Model Context Protocol servers that expose tools and data. An MCP gateway centralizes server authentication, tool allowlists, and audit logs. Several products, including Bifrost, Kong AI Gateway, Agent Router, agentgateway, and Amazon Bedrock AgentCore Gateway, handle both kinds of traffic.
Related guides
AI gateways and LLM routing hubOther categories
- Model providers
- Inference providers
- GPU clouds
- Serving engines
- Vector databases
- Embedding and reranking models
- Document parsing and data ingestion
- Fine-tuning and post-training
- Agent frameworks
- Agent infrastructure
- Web search APIs
- MCP platforms
- Voice AI
- LLM observability and evaluation
- Guardrails and AI security
- AI coding tools
Product details change quickly; check each vendor’s site for current features, licensing and pricing.