AI gateways

An AI gateway is a proxy between applications and model providers that exposes one API while handling routing, failover, authentication, rate limits, caching, and usage tracking. Teams adopt one when several applications or providers need to share credentials, budgets, and compliance rules.

The options fall into four groups. Open-source gateways such as Bifrost, LiteLLM, Agent Router, and agentgateway run inside your own network. Managed services such as Cloudflare AI Gateway, Vercel AI Gateway, and OpenRouter remove operations work and, in some cases, consolidate billing. API management vendors, including Kong, Google Apigee, and Azure API Management, extend gateways that already carry REST traffic, while AWS and Databricks govern model traffic inside their own platforms.

Gateways now govern more than model calls: many also proxy Model Context Protocol servers and agent-to-agent traffic. The market is consolidating as well. Palo Alto Networks acquired Portkey in 2026, and the TensorZero repository was archived the same year.

AI gateways compared

PlatformOpen sourceDeploymentBest for
Agent Router (formerly Envoy AI Gateway)Yes (Apache-2.0)Self-hostedKubernetes platform teams standardized on Envoy
agentgatewayYes (Apache-2.0)Self-hostedOne proxy for LLM, MCP, and A2A traffic
Amazon Bedrock AgentCore GatewayNoManaged cloudAWS teams connecting agents to tools and models
Apache APISIXYes (Apache-2.0)Self-hostedTeams already running APISIX for API traffic
Apigee AI gatewayNoManaged cloud, Self-hostedGoogle Cloud teams with existing Apigee deployments
Azure API Management AI gatewayNoManaged cloud, Self-hostedAzure and Microsoft Foundry customers
BifrostYes (Apache-2.0)Self-hostedLow-overhead self-hosted gateway for LLM and MCP traffic
Cloudflare AI GatewayNoManaged cloudTeams already building on Cloudflare Workers
Databricks Unity GatewayNoManaged cloudDatabricks customers governing AI under Unity Catalog
Eden AINoManaged cloud, APIEuropean teams consolidating model access
F5 AI GatewayNoSelf-hostedEnterprises standardized on F5 application delivery
Gravitee AI GatewayNoSelf-hosted, Managed cloudGravitee customers governing AI and API traffic together
Helicone AI GatewayYes (GPL-3.0)Self-hosted, Managed cloudExisting Helicone users who want a self-hosted router
HigressYes (Apache-2.0)Self-hosted, Managed cloudIstio users and Alibaba Cloud customers
IBM DataPower Interact GatewayNoSelf-hosted, Managed cloudIBM API Connect and webMethods customers
Kong AI GatewayYes (Apache-2.0)Self-hosted, Managed cloudTeams already using Kong for API management
LiteLLMYes (MIT)Self-hosted, LibrarySelf-hosted multi-provider proxy for platform teams
LLM GatewayYes (AGPL-3.0)Self-hosted, Managed cloudSmall teams wanting a self-hostable gateway with analytics
MLflow AI GatewayYes (Apache-2.0)Self-hostedMLflow users adding a governed model endpoint
Multi-Provider Generative AI Gateway on AWSYes (MIT-0)Self-hostedSelf-hosting LiteLLM on AWS with reference infrastructure
Netlify AI GatewayNoManaged cloudProjects already deployed on Netlify
New APIYes (AGPL-3.0)Self-hostedSelf-hosted key distribution and usage metering
Not DiamondNoManaged cloud, APIRouting coding-agent requests across frontier models
OpenRouterNoManaged cloud, APITrying many models through a single account
Plano (formerly Arch Gateway)Yes (Apache-2.0)Self-hostedMoving routing and guardrails out of agent code
Prisma AIRS AI Gateway (formerly Portkey)Yes (MIT)Managed cloud, Self-hostedEnterprises pairing an AI gateway with Palo Alto security
RequestyNoManaged cloud, APITeams that need EU data residency for model traffic
Traefik AI GatewayNoSelf-hostedKubernetes teams already running Traefik
TrueFoundry AI GatewayNoManaged cloud, Self-hostedEnterprises mixing hosted APIs and self-hosted models
Tyk AI StudioYes (AGPL-3.0)Self-hostedTyk users adding AI governance
Vercel AI GatewayNoManaged cloudTypeScript apps using the AI SDK
Zuplo AI GatewayNoManaged cloudAPI teams already using Zuplo

Profiles

  • Agent Router (formerly Envoy AI Gateway) logo

    Agent Router (formerly Envoy AI Gateway)

    theagentrouter.ai

    Agent Router, formerly Envoy AI Gateway, is an Agentic AI Foundation project built on Envoy Gateway that gives applications one OpenAI-compatible API for hosted and self-hosted models and MCP servers, configured through Kubernetes resources.

    Envoy AI Gateway was renamed Agent Router and moved to the Agentic AI Foundation, keeping its Apache-2.0 license, CRDs, container images, and aigw CLI.

    • Open source · Apache-2.0
    • Self-hosted
    • Go
  • agentgateway logo

    agentgateway

    agentgateway.dev

    agentgateway is an open-source proxy written in Rust and hosted by the Linux Foundation that handles LLM, MCP, agent-to-agent, and HTTP traffic in one data plane, with budgets, failover, guardrails, and inference routing to self-hosted models; Solo.io sells an enterprise edition.

    • Open source · Apache-2.0
    • Self-hosted
    • Rust
  • Amazon Bedrock AgentCore Gateway logo

    Amazon Bedrock AgentCore Gateway

    Amazon Web Services · aws.amazon.com

    Amazon Bedrock AgentCore Gateway is a managed AWS service that turns APIs, Lambda functions, and existing services into MCP tools, fronts other agents, and routes inference across model providers through one endpoint, with inbound and outbound authentication and semantic tool search.

    • Managed cloud
  • Apache APISIX logo

    Apache APISIX

    Apache Software Foundation · apisix.apache.org

    Apache APISIX is an open-source API gateway from the Apache Software Foundation whose AI plugins proxy and load-balance requests across model providers, with retries, token-based rate limits, prompt controls, RAG support, and gateway-level observability.

    • Open source · Apache-2.0
    • Self-hosted
    • Lua
  • Apigee AI gateway logo

    Apigee AI gateway

    Google Cloud · cloud.google.com

    Apigee, Google Cloud's API management platform, adds AI gateway capabilities for model and agent traffic, including cross-provider model routing, LLM token limit policies, semantic caching, Model Armor prompt screening, and authorization for MCP proxies.

    • Managed cloud
    • Self-hosted
  • Azure API Management AI gateway logo

    Azure API Management AI gateway

    Microsoft · learn.microsoft.com

    The AI gateway in Azure API Management is a set of policies that govern model APIs in OpenAI, Anthropic, and Vertex AI formats, MCP servers, and A2A agents, with token limits, semantic caching, load balancing, circuit breakers, and content safety checks.

    • Managed cloud
    • Self-hosted
  • Bifrost logo

    Bifrost

    Maxim AI · getmaxim.ai

    Bifrost is an open-source AI gateway from Maxim AI, written in Go, that unifies many model providers behind one OpenAI-compatible API and also serves as an MCP gateway, with fallbacks, load balancing, semantic caching, virtual keys, budgets, and guardrails.

    • Open source · Apache-2.0
    • Self-hosted
    • Go
  • Cloudflare AI Gateway

    Cloudflare · cloudflare.com

    Cloudflare AI Gateway is a managed gateway on Cloudflare's network that proxies requests to model providers and adds analytics, logging, caching, rate limiting, dynamic routing with fallbacks, and unified billing for AI workloads.

    • Managed cloud
  • Databricks Unity Gateway

    Databricks · databricks.com

    Databricks Unity Gateway governs model, agent, and MCP traffic inside Databricks through Unity Catalog, with identity-based policies, request and token limits, PII and PHI guardrails, budgets, fallbacks, governed inference tables, and a CLI for managing coding agents.

    Unity Gateway replaces Databricks' earlier AI Gateway for serving endpoints, which the documentation now describes as the legacy version.

    • Managed cloud
  • Eden AI

    edenai.co

    Eden AI is an AI gateway built in France that provides one OpenAI-compatible API to a large catalog of models, with cost-based routing, budgets, and per-team model access, aimed at European teams with GDPR requirements.

    • Managed cloud
    • API
  • F5 AI Gateway logo

    F5 AI Gateway

    F5 · f5.com

    F5 AI Gateway combines a model gateway, an MCP gateway, and AI guardrails under one control plane, with spend tracking by provider, team, and user, semantic caching, smart and GPU-aware routing, and audited agent-to-tool access.

    • Self-hosted
  • Gravitee AI Gateway logo

    Gravitee AI Gateway

    Gravitee · gravitee.io

    Gravitee AI Gateway extends the Gravitee API management platform to LLM, MCP server, and agent-to-agent traffic, applying shared authentication, policy enforcement, observability, and cost visibility to each.

    • Self-hosted
    • Managed cloud
  • Helicone AI Gateway logo

    Helicone AI Gateway

    Helicone · helicone.ai

    Helicone AI Gateway is an open-source gateway written in Rust and licensed GPL-3.0 that routes OpenAI-format requests across model providers with load balancing, rate limits, and response caching, and sends traces to Helicone's observability platform.

    Mintlify acquired Helicone in March 2026 and moved it to maintenance mode; Helicone's site now states that it is no longer part of Mintlify.

    • Open source · GPL-3.0
    • Self-hosted
    • Managed cloud
    • Rust
  • Higress logo

    Higress

    higress.ai

    Higress is an open-source AI gateway and CNCF Sandbox project built on Istio and Envoy that gives LLM APIs, agents, and MCP servers unified access, traffic control, security, and observability, with a managed edition sold as Alibaba Cloud AI Gateway.

    • Open source · Apache-2.0
    • Self-hosted
    • Managed cloud
    • Go
  • IBM DataPower Interact Gateway logo

    IBM DataPower Interact Gateway

    IBM · ibm.com

    IBM DataPower Interact Gateway is a Kubernetes-native gateway built on the DataPower Nano Gateway framework that mediates LLM and MCP traffic, turns existing APIs into agent tools, and adds token-based rate limiting, response caching, and analytics.

    • Self-hosted
    • Managed cloud
  • Kong AI Gateway

    Kong Inc. · konghq.com

    Kong AI Gateway adds AI plugins to Kong Gateway for multi-provider routing, load balancing, semantic caching, semantic prompt guards, PII sanitization, and token quotas, and also governs MCP traffic; core AI plugins ship in the open-source edition.

    • Open source · Apache-2.0
    • Self-hosted
    • Managed cloud
    • Lua
  • LiteLLM logo

    LiteLLM

    BerriAI · litellm.ai

    LiteLLM is an open-source AI gateway and Python SDK, MIT-licensed outside its enterprise directory, that exposes more than 100 model providers through an OpenAI-compatible API with virtual keys, spend tracking, budgets, guardrails, and load balancing.

    • Open source · MIT
    • Self-hosted
    • Library
    • Python
  • LLM Gateway logo

    LLM Gateway

    llmgateway.io

    LLM Gateway is an AI gateway whose core is open source under AGPL-3.0, with source-available enterprise code, that routes requests to many models through one OpenAI-compatible API and adds key management and cost analytics, self-hosted or as a managed cloud.

    • Open source · AGPL-3.0
    • Self-hosted
    • Managed cloud
    • TypeScript
  • MLflow AI Gateway logo

    MLflow AI Gateway

    mlflow.org

    MLflow AI Gateway is the open-source gateway built into MLflow that manages OpenAI, Anthropic, and other providers through one secure endpoint, with centralized credentials, access control, cost tracking, and rate limiting.

    • Open source · Apache-2.0
    • Self-hosted
    • Python
  • Multi-Provider Generative AI Gateway on AWS logo

    Multi-Provider Generative AI Gateway on AWS

    Amazon Web Services · aws.amazon.com

    This AWS Solutions Library guidance, released under MIT-0, deploys LiteLLM on Amazon ECS or EKS with AWS WAF, ElastiCache, RDS, and Secrets Manager, putting Amazon Bedrock and external providers behind one OpenAI-compatible API with virtual keys and usage logs.

    • Open source · MIT-0
    • Self-hosted
  • Netlify AI Gateway

    Netlify · netlify.com

    Netlify AI Gateway gives sites, functions, and agents deployed on Netlify access to commercial models from OpenAI, Anthropic, and Google and open models such as Kimi and DeepSeek without separate provider accounts or API keys to manage.

    • Managed cloud
  • New API logo

    New API

    newapi.ai

    New API is a self-hosted AI gateway written in Go and licensed AGPL-3.0, built on the earlier One API project, that exposes OpenAI-, Claude-, and Gemini-compatible endpoints over many upstream providers with API key management, model access control, and usage tracking.

    • Open source · AGPL-3.0
    • Self-hosted
    • Go
  • Not Diamond logo

    Not Diamond

    notdiamond.ai

    Not Diamond is a model-routing service that chooses which LLM should handle each request; its current product targets coding agents, routing their calls across frontier models to balance output quality against cost.

    • Managed cloud
    • API
  • OpenRouter

    openrouter.ai

    OpenRouter is a hosted service that provides hundreds of models from many providers through one OpenAI-compatible endpoint and one account, with automatic provider fallbacks and per-request routing preferences.

    • Managed cloud
    • API
  • Plano (formerly Arch Gateway) logo

    Plano (formerly Arch Gateway)

    Katanemo · planoai.dev

    Plano, formerly Arch Gateway, is an open-source proxy and data plane from Katanemo, written in Rust, that sits in front of agents as an AI-native proxy and handles agent orchestration, LLM routing, guardrails, and observability.

    Katanemo renamed its Arch gateway project Plano; the former katanemo/archgw repository now redirects to katanemo/plano.

    • Open source · Apache-2.0
    • Self-hosted
    • Rust
  • Prisma AIRS AI Gateway (formerly Portkey) logo

    Prisma AIRS AI Gateway (formerly Portkey)

    Palo Alto Networks · portkey.ai

    Prisma AIRS AI Gateway, previously Portkey, routes requests to many model providers and adds guardrails, observability, governance, prompt management, and an MCP gateway; its core gateway is MIT-licensed open source, and Palo Alto Networks connects it to Prisma AIRS security.

    Palo Alto Networks completed its acquisition of Portkey on May 29, 2026, and now presents the product as Prisma AIRS AI Gateway.

    • Open source · MIT
    • Managed cloud
    • Self-hosted
    • TypeScript
  • Requesty logo

    Requesty

    requesty.ai

    Requesty is a managed AI gateway and LLM router that exposes models from OpenAI, Anthropic, Google, and other providers through one OpenAI-compatible API, with routing, caching, failover, observability, EU data residency, and governance controls.

    • Managed cloud
    • API
  • Traefik AI Gateway

    Traefik Labs · traefik.io

    Traefik AI Gateway, part of Traefik Hub, turns AI endpoints into managed APIs on infrastructure the customer runs, with connections to major LLMs, semantic caching, Presidio PII protection, NVIDIA safety NIM guardrails, and an MCP gateway, configured through GitOps.

    • Self-hosted
  • TrueFoundry AI Gateway logo

    TrueFoundry AI Gateway

    TrueFoundry · truefoundry.com

    TrueFoundry AI Gateway is an enterprise gateway for hosted and self-hosted models, with routing and fallbacks, governance policies, real-time monitoring, and deployment in a VPC, on premises, or in air-gapped environments; it can also serve open models through vLLM or SGLang.

    • Managed cloud
    • Self-hosted
  • Tyk AI Studio logo

    Tyk AI Studio

    Tyk · tyk.io

    Tyk AI Studio is an AGPL-3.0 AI gateway and governance layer built on the Tyk API Gateway that acts as one control plane for LLM, agent, MCP tool, and RAG traffic, enforcing policies and attributing cost for every request.

    • Open source · AGPL-3.0
    • Self-hosted
    • Go
  • Vercel AI Gateway logo

    Vercel AI Gateway

    Vercel · vercel.com

    Vercel AI Gateway is a managed endpoint for hundreds of text, image, video, and embedding models that accepts OpenAI- and Anthropic-format requests, supports provider fallbacks and bring-your-own-key, and integrates with Vercel's AI SDK.

    • Managed cloud
    • TypeScript
  • Zuplo AI Gateway logo

    Zuplo AI Gateway

    Zuplo · zuplo.com

    Zuplo AI Gateway runs LLM traffic through the same programmable API gateway as a team's other APIs, with multi-provider routing, semantic caching, prompt-injection protection, data loss prevention, and budget pools for teams and agents.

    • Managed cloud

How to choose

01Deployment
Self-hosted in your VPC or Kubernetes cluster, managed SaaS, or hybrid with a hosted control plane.
02Provider coverage and API format
The number of supported providers and whether the gateway accepts OpenAI, Anthropic, or other native request formats.
03Routing and reliability
Fallbacks, retries, load balancing across keys and regions, and latency-based or cost-based routing.
04Governance
Virtual keys, per-team budgets, rate limits, audit logs, and single sign-on.
05Guardrails and MCP
PII redaction, prompt-injection checks, and support for proxying MCP servers and tool calls.
06Overhead and observability
Latency added per request, plus OpenTelemetry export and usage dashboards.

Frequently asked questions

What is an AI gateway?

An AI gateway is a proxy that sits between applications and AI model providers. Applications send every request to the gateway through one API, and the gateway authenticates the caller, applies rate limits and budgets, routes the request to a provider, retries or fails over on errors, and records tokens, cost, and latency for each call.

AI gateway vs API gateway: what is the difference?

A traditional API gateway routes and secures HTTP traffic to internal services based on paths, methods, and request counts. An AI gateway understands model traffic: it counts tokens, translates between provider API formats, streams responses, caches semantically similar prompts, and applies AI-specific guardrails. Some API gateways, such as Kong, Apache APISIX, and Azure API Management, add AI gateway features through plugins or policies.

Is LiteLLM open source?

Yes. LiteLLM's repository is MIT-licensed except for content under its enterprise directory, which has a separate commercial license. Teams can self-host the proxy server, including in air-gapped environments, with their own provider keys, or call providers directly through the Python SDK. BerriAI, the company behind LiteLLM, also sells an enterprise tier.

LiteLLM vs OpenRouter: what is the difference?

LiteLLM is open-source software that you run yourself, with your own provider API keys, to put many providers behind one OpenAI-compatible endpoint with budgets and virtual keys. OpenRouter is a hosted service that provides hundreds of models through one account and one endpoint. LiteLLM keeps traffic inside your infrastructure, while OpenRouter removes the need for separate provider accounts.

What is an LLM router?

An LLM router is software that chooses which model handles each request, using rules such as cost, latency, or task type, or a trained classifier that predicts which model will answer a prompt well. Gateways such as Bifrost and LiteLLM route by configured rules and fallbacks, while services such as Not Diamond select a model for each prompt automatically.

AI gateway vs MCP gateway: what is the difference?

An AI gateway governs requests to language models, while an MCP gateway governs connections between agents and Model Context Protocol servers that expose tools and data. An MCP gateway centralizes server authentication, tool allowlists, and audit logs. Several products, including Bifrost, Kong AI Gateway, Agent Router, agentgateway, and Amazon Bedrock AgentCore Gateway, handle both kinds of traffic.

Other categories

Product details change quickly; check each vendor’s site for current features, licensing and pricing.