LiteLLM alternatives: 9 open-source and managed AI gateways compared
LiteLLM alternatives are other AI gateways that do LiteLLM's job, one API in front of many model providers with keys, budgets, routing, and logs, using a different runtime, license, or operating model. LiteLLM itself is an open-source Python SDK and LLM proxy from BerriAI. This comparison explains why teams evaluate replacements, states the selection criteria, compares nine self-hosted and managed options, and closes with a migration plan. It is part of our AI gateways and LLM routing hub.
What is LiteLLM?#
LiteLLM is an open-source Python SDK and self-hosted proxy server that puts more than 100 model providers behind one OpenAI-compatible API, with cost tracking, budgets, and routing.
The project has two parts. The SDK is a library that application code calls directly. The proxy server is a shared AI gateway that platform teams run for many applications, and it is the part the alternatives in this comparison replace. The proxy issues virtual keys, tracks spend per key and team, enforces budgets and rate limits, load balances across deployments, falls back between models, and includes an MCP gateway with permissions by key, team, or organization.
The repository is MIT-licensed except for the enterprise directory, which carries a separate license. In June 2026, BerriAI announced a migration of the gateway to Rust and promised that config, database schema, client API, and the Python SDK interface would stay the same. Our explainer on what an AI gateway is covers the general architecture these tools share.
Why teams look for LiteLLM alternatives#
Teams look for LiteLLM alternatives for six documented reasons: per-request overhead and memory on the Python proxy path, the Postgres and Redis footprint of a production deployment, features that require an Enterprise license, two supply chain incidents in 2026, the absence of a hosted LiteLLM service, and how model and agent traffic should share one gateway.
Proxy overhead under concurrency#
LiteLLM's own Rust announcement measured about 7.5 ms of added latency per request and a peak of about 359 MB of memory on the Python forwarding path, and described CPU and memory rising with concurrency under load. The Rust migration is LiteLLM's answer, and it is partly shipped: the v1.110.0 release candidates run the router on the Rust core, and the migration status page targets running the common /v1/messages path end to end in Rust by December 1, 2026. As of November 2026, that page still listed chat completions translation for most providers on the Python path.
Published benchmarks disagree, and each was run by a vendor. In Maxim AI's published benchmark (t3.medium instances on AWS EC2 in us-east-1, 500 virtual users for 60 seconds), Bifrost served 424 requests per second against 44.84 for LiteLLM's Python proxy, with P99 latency of 1.68 seconds versus 90.72 seconds, peak memory of 120 MB versus 372 MB, and a success rate of 100% versus 88.78%. LiteLLM's July 2026 forwarding-only benchmark, with logging, spend tracking, and persistence disabled, reported 0.7 ms of added P99 latency for its Rust gateway, 2.3 ms for Portkey, 4.5 ms for Bifrost, and 257.7 ms for its Python v1 path. Neither setup is your workload, so the migration section below describes how to run your own test.
Operational footprint#
LiteLLM's virtual keys require a Postgres database, and its production guide says to run Redis 7.0 or newer as soon as there is more than one proxy instance. The same guide sets 1 vCPU and 4 GiB of memory per worker as a floor and a default of 10 database connections per worker, which multiply across instances. LiteLLM's own 3,000-requests-per-second benchmark ran 33 gateway pods with 4 workers each, plus Postgres and Redis.
Features in the Enterprise tier#
LiteLLM's Enterprise page lists the features its license key enables. They include SSO for the admin UI beyond 5 users, audit logs, role-based access control, secret manager integrations, IP allowlists, tag-based budgets, model-specific budgets per virtual key, team-based logging controls, and guardrails scoped per key or team. Several built-in guardrail integrations, including Lakera prompt injection and OpenAI moderation, also require the license. Enterprise has no published price.
Two supply chain incidents in 2026#
LiteLLM published advisories for two compromised releases in 2026. On March 24, versions 1.82.7 and 1.82.8 on PyPI contained a credential stealer; LiteLLM said the official Proxy Docker image was not affected and that it believed the compromise originated from the Trivy dependency in its CI/CD scanning workflow. On October 21, version 1.107.0 reached PyPI and the v1.107.0, latest, main-latest, and main-stable tags on Docker Hub and GHCR, built from an attacker-controlled repository; LiteLLM said customers on its private enterprise release were not affected. Both advisories list remediation steps.
No hosted option, and agent traffic#
LiteLLM's pricing page, checked on November 16, 2026, lists only self-hosted tiers, so teams that want no proxy to operate must look elsewhere. Agent traffic raises a different question. LiteLLM already proxies MCP servers, so the comparison point is how each gateway handles Model Context Protocol tool filtering, per-user OAuth, and A2A agent traffic alongside model calls.
How we chose and compared the alternatives#
Each alternative was checked against six criteria from our AI gateways directory, using its official documentation, repository, and announcements as of November 16, 2026. Projects that are archived or in maintenance mode were left out, and every entry's current name, license, and ownership was confirmed at the source.
- Deployment: self-hosted in your VPC or Kubernetes cluster, managed SaaS, or hybrid with a hosted control plane.
- Provider coverage and API format: the providers supported and whether the gateway accepts OpenAI, Anthropic, or other native request formats.
- Routing and reliability: fallbacks, retries, load balancing across keys and regions, and rule-based routing.
- Governance: virtual keys, per-team budgets, rate limits, audit logs, and single sign-on.
- Guardrails and MCP: PII redaction, prompt-injection checks, and support for proxying MCP servers and tool calls.
- Overhead and observability: latency added per request, plus OpenTelemetry export and usage dashboards.
Each entry covers what it is, deployment and license, strengths, trade-offs, best fit, and migration from LiteLLM.
LiteLLM alternatives at a glance#
The table summarizes license, deployment model, implementation language, MCP support, and governance for each alternative, based on official documentation and repositories as of November 16, 2026. Open-source gateways run where you deploy them; managed services run on the vendor's network; Kong and Apache APISIX extend existing API gateways with AI plugins.
| Gateway | License | Deployment | Language | MCP support | Governance |
|---|---|---|---|---|---|
| Bifrost | Apache 2.0 | Self-hosted: npx, Docker, Kubernetes | Go | MCP gateway with agent mode and Code Mode; A2A agent gateway | Virtual keys, budgets, rate limits; SSO and RBAC in Enterprise |
| Agent Router | Apache 2.0 | Self-hosted on Kubernetes with Envoy Gateway, or the aigw CLI |
Go control plane, Envoy data plane | One router for MCP tools, filtered by requester | Token limits per team, app, or model |
| agentgateway | Apache 2.0 | Self-hosted binary or Kubernetes | Rust | MCP gateway with tool federation and OAuth; A2A | Budgets, rate limiting, RBAC |
| Kong AI Gateway | Kong Gateway OSS is Apache 2.0; AI Gateway Enterprise is commercial | Self-hosted, Konnect, or Kong-hosted | Lua | AI MCP Proxy plugin (Enterprise, 3.12+) | Consumers, token rate limits (Enterprise) |
| Apache APISIX | Apache 2.0 | Self-hosted | Lua | openapi-to-mcp turns HTTP APIs into MCP tools |
Consumers, token quotas |
| Prisma AIRS AI Gateway (formerly Portkey) | Core gateway MIT; platform commercial | SaaS, hybrid data plane in your VPC, or open-source gateway | TypeScript | MCP Gateway | RBAC, budgets, SSO |
| Cloudflare AI Gateway | Proprietary | Managed | Not applicable | Not listed among AI Gateway features | Spend limits, rate limits, gateway auth |
| Vercel AI Gateway | Proprietary | Managed | Not applicable | Not an MCP gateway | Budgets per team, project, key, or member |
| OpenRouter | Proprietary | Managed | Not applicable | Not an MCP gateway | Guardrails with budgets and model allowlists |
Self-hosted open-source gateways#
Self-hosted open-source gateways are the closest replacements for a LiteLLM proxy, because they run in your own network with your own provider keys. Bifrost, Agent Router, and agentgateway all accept OpenAI-format requests, govern MCP tool traffic as well as model calls, and differ mainly in runtime, configuration model, and how much ships outside an enterprise edition.
1. Bifrost#
Bifrost is an open-source AI gateway from Maxim AI, written in Go, that acts as an LLM gateway, an MCP gateway, and an agent gateway behind one OpenAI-compatible API. Its latest HTTP release on November 16, 2026 was v2.5.1, published November 10.
Deployment and license: Apache 2.0. Start it with npx -y @maximhq/bifrost or docker run -p 8080:8080 maximhq/bifrost, or deploy it to Kubernetes with Helm; Maxim describes it as a single binary. Configuration lives in SQLite by default or in PostgreSQL. Clustering, SSO, guardrails, and adaptive load balancing are in the paid Enterprise tier.
Strengths:
- Drop-in SDK endpoints for the OpenAI, Anthropic, and LiteLLM SDKs, and a support matrix of more than 30 providers, including vLLM, SGLang, and Ollama.
- Retries and fallbacks with exponential backoff, plus weighted load balancing across keys.
- Virtual keys with budgets, rate limits, allowed models and providers, and MCP tool filtering, grouped under teams and customers.
- An MCP gateway that connects to servers over STDIO, HTTP, or SSE and exposes them at one
/mcpendpoint, with agent mode and Code Mode, for which Maxim reports up to 92.8% fewer input tokens. - Semantic caching with exact-match and similarity lookups on Redis or Valkey, Qdrant, Weaviate, or Pinecone.
- Maxim's benchmarking docs report 11 µs of added overhead at 5,000 requests per second on a t3.xlarge with mocked provider calls; these and the LiteLLM comparison above are Maxim's own numbers.
Trade-offs: Guardrails, cluster mode, SSO, and secret manager integrations require Enterprise. The semantic cache needs a vector store, and caching activates only when a request carries a cache key or a default key is configured. Bifrost's native endpoints take provider/model strings, so check how LiteLLM model_name aliases map after import.
Best for: Teams that want to self-host one gateway for model, MCP, and agent traffic, with a direct import path from an existing LiteLLM proxy.
Migrating from LiteLLM: Bifrost's LiteLLM migration guide documents a CLI that reads a LiteLLM proxy and recreates its providers, keys, budgets, rate limits, organizations (as customers), teams, users, and virtual keys. New sk-bf-* key values are generated, partial wildcards such as openai/gpt-4* are skipped and reported, and the tool never writes to LiteLLM. Code that uses the LiteLLM SDK can point at Bifrost's /litellm endpoint instead.
2. Agent Router (formerly Envoy AI Gateway)#
Agent Router is the open-source control plane for AI and agent traffic that was previously named Envoy AI Gateway, now an Agentic AI Foundation project. It gives applications one OpenAI-compatible API for hosted models, self-hosted inference, and MCP servers, enforced by Envoy and Envoy Gateway.
Deployment and license: Apache 2.0, written in Go. Run it locally with aigw run, which serves an OpenAI-compatible endpoint on port 1975, or deploy it on Kubernetes with Envoy Gateway. The Agentic AI Foundation announcement from September 2026 says existing deployments do not need to migrate, and the repository README confirms the CRDs, aigw CLI, images, and Helm charts kept their names. The latest release was v1.2.1 on November 6, 2026.
Strengths: Routing to hosted providers and self-hosted vLLM, token limits per team, app, or model, provider fallback, model name virtualization, one MCP tool catalog filtered per requester, and telemetry that follows OpenTelemetry GenAI conventions. A two-tier pattern adds an endpoint picker for self-hosted model clusters.
Trade-offs: Production configuration is Kubernetes custom resources (AIGatewayRoute, AIServiceBackend, BackendSecurityPolicy) on top of Envoy Gateway, which suits platform teams but is more to operate than a single config file for teams not already on Kubernetes.
Best for: Kubernetes platform teams that already run Envoy or Envoy Gateway.
Migrating from LiteLLM: Each LiteLLM model_list entry becomes an AIServiceBackend and a route rule, provider credentials move into BackendSecurityPolicy resources, and token limits replace LiteLLM's tpm_limit values. Applications change only their base URL.
3. agentgateway#
agentgateway is an open-source proxy hosted by the Linux Foundation that handles LLM, MCP, agent-to-agent, and ordinary HTTP and gRPC traffic in one data plane. Solo.io contributed the project and sells Solo Enterprise for agentgateway as a commercial edition.
Deployment and license: Apache 2.0, written in Rust. It runs as a standalone binary with flat YAML configuration or on Kubernetes with its controller and Gateway API support. The latest release was v1.7.0 on October 27, 2026.
Strengths: The project README lists an OpenAI-compatible LLM gateway with budget controls, load balancing, and failover; an MCP gateway with tool federation, several transports, and OAuth; an A2A gateway; inference routing to self-hosted models by GPU utilization, KV cache, LoRA adapters, and queue depth; and guardrails, RBAC, rate limiting, and OpenTelemetry. Its site also lists semantic caching and per-key or per-team spend caps.
Trade-offs: agentgateway is a general-purpose proxy, so teams that only need model routing take on a broader configuration surface. Solo's commercial edition adds a hardened distribution, further MCP security and governance features, and 24x7 support.
Best for: Teams that want one proxy for LLM, MCP, and A2A traffic, especially on Kubernetes.
Migrating from LiteLLM: Providers and models move into agentgateway's YAML or Gateway API resources, LiteLLM virtual keys are replaced by agentgateway's authentication and RBAC policies, and clients point at its OpenAI-compatible route.
API gateways with AI plugins#
An API gateway with AI plugins is the shortest path when the organization already runs one for REST traffic. Kong and Apache APISIX add model proxying, token-based rate limits, prompt guards, and caching as plugins, so AI traffic shares the authentication, consumers, and logging already in place, at the cost of fewer model-specific features than a dedicated gateway.
4. Kong AI Gateway#
Kong AI Gateway is Kong's set of AI policies on Kong Gateway that secures, governs, and observes LLM, MCP, and A2A traffic. Kong now also offers AI Gateway 2.x, a dedicated control plane in Konnect that declares AI models, providers, policies, and MCP servers as resources.
Deployment and license: Kong Gateway's open-source repository is Apache 2.0, and its latest tag on November 16, 2026 was 3.9.4. That repository includes six AI plugins: AI Proxy, AI Prompt Guard, AI Prompt Decorator, AI Prompt Template, and AI request and response transformers. Kong runs self-hosted, with a Konnect control plane, or fully hosted on Dedicated Cloud Gateways.
Strengths: The AI Gateway documentation lists semantic caching, prompt compression, AI rate limiting, semantic prompt and response guards, a PII sanitizer, and AWS Bedrock Guardrails, Azure Content Safety, and Google Model Armor integrations. The AI MCP Proxy plugin fronts existing MCP servers or converts REST APIs into MCP tools.
Trade-offs: The AI MCP Proxy plugin is available only in Kong's AI Gateway Enterprise offering, from version 3.12, and AI Proxy Advanced, AI Semantic Cache, and AI Rate Limiting Advanced are not in the open-source repository, so open-source Kong gets basic proxying and prompt controls.
Best for: Organizations that already use Kong for API management and want AI traffic under the same policies.
Migrating from LiteLLM: Each LiteLLM model becomes an AI Proxy route or, in AI Gateway 2.x, an AI Model entity; LiteLLM keys become Kong consumers with key authentication; and token budgets move to AI Rate Limiting Advanced policies (Enterprise).
5. Apache APISIX#
Apache APISIX is an open-source API gateway from the Apache Software Foundation whose AI plugins proxy, load balance, rate limit, and guard requests to model providers.
Deployment and license: Apache 2.0, written in Lua, self-hosted. The latest release was 3.20.0 on November 11, 2026.
Strengths: ai-proxy-multi adds load balancing, retries, fallbacks, and health checks across OpenAI, Anthropic, Azure, Gemini, Vertex AI, Amazon Bedrock, DeepSeek, OpenRouter, and other OpenAI-compatible APIs. ai-rate-limiting enforces token quotas with local or Redis-backed counters, ai-cache provides exact and optional semantic caching, and ai-prompt-guard and ai-lakera-guard inspect content. The openapi-to-mcp plugin turns an HTTP API's OpenAPI operations into MCP tools served from the gateway.
Trade-offs: APISIX turns APIs into MCP tools, but the 3.20.0 plugin list has no plugin for proxying third-party MCP servers. Token quotas rely on the usage each provider reports, and the AI plugins document token quotas rather than dollar budgets per team.
Best for: Teams that already run APISIX for API traffic.
Migrating from LiteLLM: Providers become ai-proxy-multi instances with weights on a route, LiteLLM keys become APISIX consumers, and tpm_limit values move to ai-rate-limiting quotas.
Managed and hybrid gateways#
Managed gateways replace the LiteLLM proxy with a service the vendor operates, so there is no proxy, database, or cache to run. In exchange, prompts pass through a third party, governance follows the vendor's account model, and only Prisma AIRS AI Gateway, of the four here, documents an MCP gateway and a self-hosted data plane.
6. Prisma AIRS AI Gateway (formerly Portkey)#
Prisma AIRS AI Gateway is the product name Palo Alto Networks now uses for Portkey's AI gateway, which routes requests to many model providers and adds guardrails, observability, governance, and an MCP gateway. Palo Alto Networks completed its acquisition of Portkey on May 29, 2026.
Deployment and license: The core open-source gateway is MIT-licensed TypeScript and runs with npx @portkey-ai/gateway. The commercial platform runs as SaaS or as a hybrid deployment, with the data plane in your VPC on Kubernetes and the control plane hosted.
Strengths: Fallbacks, retries with exponential backoff, weighted load balancing, simple caching, and dozens of built-in guardrails in the open-source gateway; RBAC, budgets, SSO, semantic caching, PII redaction, and an MCP Gateway with OAuth 2.1 in the commercial platform; and integration with Palo Alto's Prisma AIRS security products.
Trade-offs: The open-source repository's latest release was v1.15.2 in January 2026, and on November 16, 2026 its latest commit dated from May 2026. The hybrid data plane depends on a hosted control plane.
Best for: Enterprises that want a hosted control plane, an MCP gateway, and Palo Alto security integration.
Migrating from LiteLLM: Point the OpenAI SDK at the gateway's /v1 endpoint, recreate fallback and load-balancing rules in the gateway's own configuration format, and recreate team keys and budgets in the platform.
7. Cloudflare AI Gateway#
Cloudflare AI Gateway is a managed gateway on Cloudflare's network that proxies requests to model providers and adds analytics, logging, caching, rate limiting, spend limits, routing, guardrails, and data loss prevention.
Deployment and license: Proprietary and managed. Requests go to provider-specific endpoints (https://gateway.ai.cloudflare.com/v1/{account_id}/{gateway_id}/{provider}) that keep each provider's native schema, or to Cloudflare's unified REST API. According to Cloudflare's pricing page on November 16, 2026, core features were free on all plans, and Unified Billing credits carried a 5% fee with provider inference passed through at no markup.
Strengths: The feature list covers caching, spend limits by model, provider, or custom metadata, sliding or fixed-window rate limiting, model aliases with dynamic routes and fallbacks, guardrails, DLP scanning, stored provider keys (BYOK), custom metadata, custom costs, and Logpush export.
Trade-offs: Spend limits apply by model, provider, or custom metadata rather than to per-team virtual keys. Cloudflare's feature list does not include MCP proxying. Log storage limits differ by Workers plan.
Best for: Teams that already build on Cloudflare Workers or want a free managed layer for analytics, caching, and spend limits.
Migrating from LiteLLM: Replace the LiteLLM base URL with a provider-specific gateway URL, store provider keys with BYOK, tag requests with custom metadata for team attribution, and recreate budgets as spend limits.
8. Vercel AI Gateway#
Vercel AI Gateway is a managed endpoint for models across providers that accepts OpenAI Chat Completions, OpenAI Responses, and Anthropic Messages requests and works from any cloud, not only from Vercel deployments.
Deployment and license: Proprietary and managed, at https://ai-gateway.vercel.sh. A Vercel account is needed for keys and billing, but no Vercel project. Vercel's documentation states that the gateway adds zero markup to provider token prices, including with BYOK.
Strengths: The AI Gateway docs describe ordered provider and model fallbacks, budgets for a team, project, API key, or team member, BYOK, request logs with every routing attempt, cross-provider prompt caching, and modalities from text to video, embeddings, and realtime. Its Python integrations list includes LiteLLM.
Trade-offs: Budgets cover spend billed through Vercel's keys; BYOK spend is metered separately. Vercel documents AI Gateway as the model layer and MCP as a separate tool layer, so it is not an MCP gateway. Response caching stays in your application.
Best for: TypeScript teams using the AI SDK, and teams that want managed failover without operating a proxy.
Migrating from LiteLLM: Set the base URL to https://ai-gateway.vercel.sh/v1, switch model names to provider/model strings, and issue an AI Gateway API key per team or project with its own budget.
9. OpenRouter#
OpenRouter is a hosted service that provides hundreds of models from many providers through one OpenAI-compatible endpoint and one account, with automatic provider fallbacks.
Deployment and license: Proprietary and managed, at https://openrouter.ai/api/v1. According to OpenRouter's FAQ on November 16, 2026, it charged a 5.5% fee ($0.80 minimum) on credit purchases, passed provider inference rates through without markup, and on pay-as-you-go included $25,000 a month of BYOK usage before a 5% BYOK fee.
Strengths: One account instead of many provider contracts, automatic fallback to the next provider on errors, model fallbacks through a models array, guardrails with budgets and model or provider allowlists, and zero data retention routing options. Prompts and completions are not logged by default.
Trade-offs: Every request passes through OpenRouter's service. Guardrails cover budgets and allowlists, not the per-team virtual keys, custom guardrails, and self-hosted logs of a gateway you run. OpenRouter documents an MCP client in its Agent SDK, not an MCP gateway.
Best for: Trying many models through one account, and as one upstream behind a self-hosted gateway.
Migrating from LiteLLM: Change the base URL and key, and map model names to OpenRouter slugs. Alternatively, keep a self-hosted gateway and add OpenRouter as one of its providers; LiteLLM and Bifrost both support it.
Other options and why they were left out#
Several other gateways fit narrower cases. Higress is an Apache 2.0 AI gateway and CNCF Sandbox project with an MCP gateway and a managed edition, Alibaba Cloud AI Gateway. TrueFoundry AI Gateway is a commercial gateway for VPC, on-premises, and air-gapped deployment. Azure API Management, Apigee, and Amazon Bedrock AgentCore Gateway suit teams standardized on those clouds.
Helicone AI Gateway was left out because Mintlify, which acquired Helicone, said on March 3, 2026 that Helicone would run in maintenance mode, its gateway repository had no commits after November 2025, and Helicone's site said on November 16, 2026 that it is now part of Experiential Labs. The AI gateways directory lists these and other options with their current status.
Which LiteLLM alternative should you choose?#
Choose by where traffic must run, then by what you already operate. If prompts and keys can leave your network, a managed gateway removes the proxy entirely. If they cannot, extend the API gateway you already run, or adopt a standalone AI gateway when you need MCP and agent governance next to model routing.
The branches overlap in practice. Bifrost and agentgateway also suit teams that run an API gateway but want a separate layer for AI traffic, and Prisma AIRS AI Gateway's open-source core can be self-hosted.
| If your main reason for leaving LiteLLM is | Start by evaluating | Why |
|---|---|---|
| Proxy overhead or memory | Bifrost, agentgateway | Compiled Go and Rust gateways; test against LiteLLM's Rust path too |
| Postgres and Redis to operate | Bifrost, a managed gateway | Bifrost stores config in SQLite by default; managed services run their own state |
| Enterprise-only controls | Each vendor's open-source and paid tiers | Bifrost, Kong, agentgateway, and Prisma AIRS also keep some controls in paid editions |
| Supply chain exposure | Any option, with pinned versions and verified images | Version pinning and artifact verification apply to every gateway |
| No proxy to run | Cloudflare, Vercel, OpenRouter | The vendor operates the gateway |
| MCP and agent traffic | Bifrost, agentgateway, Agent Router, Prisma AIRS | Each documents MCP gateway features alongside model routing |
| An existing API gateway | Kong, APISIX, Agent Router | AI traffic shares existing consumers, policies, and logs |
How to migrate off LiteLLM#
Migrating off LiteLLM takes five steps: inventory what the proxy does today, stand up the new gateway with the same providers, change only the base URL and key in applications, run both gateways side by side with a small traffic share, and load test both before cutting over. Application code rarely changes, because every option here accepts OpenAI-format requests.
1. Inventory the LiteLLM configuration. List each model_list entry and its model_name alias, fallbacks, virtual keys with their budgets and tpm_limit and rpm_limit values, teams, logging callbacks, and guardrails. Aliases matter most: clients send LiteLLM's model_name values, and most alternatives expect provider/model strings or their own aliases.
2. Recreate the configuration in the new gateway. For Bifrost, the migration CLI does this from a running proxy; run it in dry-run mode first to review what will be skipped:
export LITELLM_URL="http://localhost:4000"
export LITELLM_MASTER_KEY="sk-1234"
export LITELLM_CONFIG="/path/to/litellm/config.yaml"
export BIFROST_URL="http://localhost:8080"
export BIFROST_API_KEY="your-bifrost-api-key"
DRY_RUN=1 npx @maximhq/bifrost-migration-cli # report only, writes nothing
npx @maximhq/bifrost-migration-cli # import into BifrostFor other gateways, use the mappings in each entry above. Budgets and rate limits rarely translate one to one, so check reset windows and whether limits apply per key, team, or model.
3. Make the gateway a configuration value in applications. Read the base URL and key from the environment, so moving a service between gateways needs only a deploy:
import os
from openai import OpenAI
client = OpenAI(
base_url=os.environ["LLM_GATEWAY_URL"], # e.g. http://localhost:4000 or http://localhost:8080/openai
api_key=os.environ["LLM_GATEWAY_KEY"], # the gateway's virtual key, not a provider key
)4. Run both gateways side by side. Put a weighted split in the load balancer or ingress in front of both, or move one team's keys at a time. Both gateways call the same provider accounts, so watch shared provider rate limits during the overlap.
5. Load test both gateways the same way. Point both at the same mock upstream with a fixed response time, so the test measures gateway overhead rather than provider variance. Ramp concurrency from 1 to above your expected peak, and record added P50 and P99 latency, error rate, memory and CPU per instance, and time to first token for streamed responses. Enable the features you will run in production, such as logging, budgets, and caching, because forwarding-only numbers understate their cost. LiteLLM reports its own overhead in the x-litellm-overhead-duration-ms response header, which helps separate gateway time from provider time.
Cut over when the new gateway meets your latency target and its spend records reconcile with provider invoices, and keep LiteLLM running, unrouted, for a full billing cycle. Our guide to LLM observability covers the tracing that should run across both gateways during the overlap.
Where to go next: our guide to LLM routing covers fallback and load-balancing design in any gateway, semantic caching explains the cache trade-offs several of these gateways share, and the AI gateways directory lists more than 30 gateways by deployment model.
Frequently asked questions
What is the best alternative to LiteLLM?
- The right LiteLLM alternative depends on what pushed you to look. For a self-hosted gateway with a direct import path from a LiteLLM proxy, Bifrost has a migration CLI. For Kubernetes teams on Envoy, Agent Router fits. For one proxy across LLM, MCP, and A2A traffic, agentgateway is an option. To stop operating a proxy, Cloudflare AI Gateway, Vercel AI Gateway, and OpenRouter are managed.
Is LiteLLM safe to use after the 2026 supply chain attacks?
- LiteLLM disclosed two compromised releases in 2026: PyPI versions 1.82.7 and 1.82.8 on March 24, and version 1.107.0 on PyPI and its Docker registries on October 21. Its advisories list the affected versions, the indicators to check for, and the credentials to rotate. Teams that keep LiteLLM should pin exact versions, avoid floating image tags such as latest, and verify artifacts before deploying them.
What is an LLM proxy?
- An LLM proxy is a server that sits between applications and model providers, accepts requests in one format, usually the OpenAI chat format, and forwards them to the selected provider. Basic proxies translate requests and retry failures. Fuller proxies, often called AI gateways, add virtual keys, budgets, rate limits, caching, guardrails, and logging. LiteLLM, Bifrost, and the other tools in this comparison all work as LLM proxies.
Is Bifrost compatible with LiteLLM?
- Yes, in two ways. Bifrost exposes a /litellm endpoint, so code that uses the LiteLLM Python SDK can point its base URL at Bifrost and keep its existing model strings. For LiteLLM proxy users, Bifrost publishes a migration CLI that reads providers, budgets, rate limits, organizations, teams, users, and virtual keys from LiteLLM and recreates them in Bifrost. Virtual key values are regenerated, so callers need new keys.
Is LiteLLM written in Python or Rust?
- LiteLLM began as a Python SDK and Python proxy server. In June 2026, BerriAI announced it was moving the gateway's request path to a Rust core while keeping its config, database schema, and APIs unchanged. As of November 2026, its v1.110.0 release candidates run the router on Rust, and the migration page targets an end-to-end Rust path for /v1/messages by December 1, 2026.
Can OpenRouter replace LiteLLM?
- OpenRouter can replace LiteLLM when the main need is one OpenAI-compatible endpoint for many models without separate provider accounts. It is a hosted service, so prompts pass through OpenRouter, and it offers guardrails with budgets and model allowlists rather than LiteLLM-style virtual keys per team. Teams that need self-hosted logs or internal budgets can run their own gateway and add OpenRouter as one upstream provider.
Terms in this guide
Cite this guide
The AI Engineering Editors. "LiteLLM alternatives: 9 open-source and managed AI gateways compared." The AI Engineering, updated Nov 17, 2026. https://theaiengineering.com/articles/litellm-alternatives
- Title
- LiteLLM alternatives: 9 open-source and managed AI gateways compared
- Author
- The AI Engineering Editors
- Publisher
- The AI Engineering
- Published
- Updated