---
title: "Bifrost vs LiteLLM: architecture, features, and performance compared"
description: "Bifrost vs LiteLLM compared: a Go single binary against a Python proxy, with deployment, routing, virtual keys, caching, MCP, benchmarks, and migration."
summary: "Bifrost and LiteLLM are open-source AI gateways with overlapping features and different architectures: one Go process against a Python proxy backed by Postgres and Redis. This comparison covers deployment, a feature table, both vendors' published benchmarks, LiteLLM's move to Rust, and migration."
type: "comparison"
canonical_url: "https://theaiengineering.com/articles/bifrost-vs-litellm"
markdown_url: "https://theaiengineering.com/articles/bifrost-vs-litellm.md"
topic: "AI gateways and LLM routing"
author: "The AI Engineering Editors"
publisher: "The AI Engineering"
published: "2026-11-16"
updated: "2026-11-16"
keywords: ["bifrost vs litellm","bifrost ai gateway","bifrost llm gateway","bifrost gateway","litellm proxy"]
citation: "The AI Engineering Editors. \"Bifrost vs LiteLLM: architecture, features, and performance compared.\" The AI Engineering, updated Nov 16, 2026. https://theaiengineering.com/articles/bifrost-vs-litellm"
---

# Bifrost vs LiteLLM: architecture, features, and performance compared

## TL;DR

- Bifrost is an Apache 2.0 gateway written in Go that runs as one process with a built-in web UI and SQLite storage by default, while LiteLLM pairs a Python SDK with a proxy server that needs Postgres for virtual keys and Redis once it runs more than one worker.
- Both gateways cover the same core features: an OpenAI-compatible API, fallbacks, load balancing, virtual keys with budgets, exact and semantic caching, Prometheus metrics, OpenTelemetry traces, an MCP gateway, and an A2A agent gateway.
- Guardrails, SSO, RBAC, audit logs, and multi-node state sync are Enterprise features in Bifrost; in LiteLLM, the guardrail framework and multi-instance scaling are open source, SSO is free for up to 5 users, and audit logs and RBAC need an Enterprise license.
- Benchmarks published by Maxim AI and by LiteLLM both measured far lower gateway overhead for Bifrost than for LiteLLM's Python proxy, and LiteLLM's own tests measured its unreleased Rust branch lowest of all.
- Bifrost serves a /litellm endpoint for the LiteLLM Python SDK and offers a migration CLI that copies a LiteLLM proxy's providers, teams, users, budgets, and virtual keys, though callers receive new virtual key tokens.

Bifrost and LiteLLM are open-source AI gateways that put many model providers behind one API: Bifrost is a Go gateway from Maxim AI that ships as a single binary under the Apache 2.0 license, and LiteLLM is a Python proxy server plus a Python SDK from BerriAI, MIT licensed outside its enterprise directory. The Bifrost vs LiteLLM decision turns more on architecture than on feature lists, because the two now overlap on routing, keys, caching, MCP, and agent traffic. This comparison checks both projects' documentation as of November 16, 2026, and covers deployment, features, published benchmarks, and migration. It belongs to our [AI gateways and LLM routing](https://theaiengineering.com/topics/ai-gateways) hub.

## Bifrost vs LiteLLM at a glance

Bifrost is a single Go process that keeps its state in local SQLite files by default, while the LiteLLM proxy is a Python service that needs Postgres for virtual keys and spend tracking, plus Redis once it runs more than one worker. Both expose an OpenAI-compatible API and sell an Enterprise tier on top of the open-source core.

| | Bifrost | LiteLLM |
|---|---|---|
| Maintainer | Maxim AI | BerriAI |
| Language | Go | Python for the SDK and proxy; a Rust gateway is in release candidates |
| License | Apache 2.0 | MIT, except the `litellm/enterprise/` directory |
| Latest release (November 16, 2026) | Gateway v2.5.1, released November 10, 2026 | v1.108.9 stable, released November 14, 2026; v1.110.0 release candidates |
| What you run | Gateway binary with a web UI, or the Go SDK inside a Go app | Proxy server with an admin UI, or the Python SDK inside a Python app |
| Install options | npx, Docker, Helm, Go module | Docker, pip or uv, Helm, Terraform |
| Default port | 8080 | 4000 |
| State stores | SQLite config and log stores by default; Postgres optional | Postgres for keys, budgets, and spend; Redis for more than one worker |
| Providers | 35 in the docs support matrix | 100+ |
| Paid tier | Bifrost Enterprise | LiteLLM Enterprise |

LiteLLM is partway through a move from Python to Rust, covered in the architecture section.

## What are Bifrost and LiteLLM?

Bifrost and LiteLLM are self-hosted [AI gateways](https://theaiengineering.com/glossary/ai-gateway): proxies that give applications one [OpenAI-compatible API](https://theaiengineering.com/glossary/openai-compatible-api) to many model providers and apply shared routing, keys, budgets, and logging.

Our explainer on [what an AI gateway is](https://theaiengineering.com/articles/what-is-an-ai-gateway) covers the category.

### What is the Bifrost AI gateway?

The Bifrost AI gateway is an open-source gateway from Maxim AI, written in Go, that handles model, MCP, and agent traffic through one process. Its [supported providers page](https://docs.getbifrost.ai/providers/supported-providers/overview) lists 35 providers, including OpenAI, Anthropic, Bedrock, Vertex AI, Azure, Gemini, Groq, Mistral, and xAI, plus self-hosted vLLM, SGLang, and Ollama.

As a Bifrost LLM gateway, it accepts OpenAI-format requests and also serves provider-shaped routes, so existing OpenAI, Anthropic, Google GenAI, Bedrock, Cohere, and LiteLLM SDK code can point at it by changing a base URL. The same process runs an MCP gateway and an agent gateway for A2A agents. Teams that want the gateway inside a Go program can import the Go SDK (`github.com/maximhq/bifrost/core`) instead of running a server. Bifrost Enterprise adds guardrails, SSO, RBAC, audit logs, clustering, and adaptive routing.

### What is the LiteLLM proxy?

The LiteLLM proxy is a self-hosted, OpenAI-compatible gateway server that BerriAI's [documentation](https://docs.litellm.ai/docs/) describes as a gateway for platform teams managing model access across an organization. It sits in the same repository as the LiteLLM Python SDK, a library that calls more than 100 providers from Python code with one interface.

The SDK and the proxy share a router for retries, fallbacks, and load balancing. The proxy adds virtual keys, budgets, rate limits, spend tracking, an admin UI, guardrails, an MCP gateway, and an A2A agent gateway. Its supported endpoints include `/chat/completions`, `/responses`, `/messages`, `/embeddings`, `/images`, `/audio`, `/batches`, `/rerank`, and `/a2a`. LiteLLM Enterprise is a license key on the same image that adds SSO beyond 5 users, SCIM, audit logs, RBAC, secret managers, and per-key guardrails.

## Architecture and deployment

Bifrost runs as one Go process that holds configuration, keys, budgets, and usage in memory and persists them to SQLite or Postgres. The LiteLLM proxy runs as one or more Python worker processes that share keys, budgets, rate limits, and cooldowns through Postgres and Redis.

**Figure 1: Bifrost keeps gateway state inside one process; the LiteLLM proxy shares state across workers through Postgres and Redis.**

Side-by-side request paths: an application calls one Bifrost Go process backed by local stores, and another calls LiteLLM proxy workers backed by Postgres and Redis.

Components:

- **Application**: OpenAI-format call
- **Bifrost**: One Go process API and web UI
- **Local stores**: SQLite by default Postgres optional
- **Model providers**
- **Vector store**: For semantic cache optional
- **Application**: OpenAI-format call
- **LiteLLM proxy**: Python workers on one or more pods
- **Postgres**: Keys, budgets, spend logs
- **Model providers**
- **Redis**: Shared limits, cooldowns, cache

Groups:

- Bifrost: Application, Bifrost, Local stores, Model providers, Vector store
- LiteLLM: Application, LiteLLM proxy, Postgres, Model providers, Redis

Connections:

- Application → Bifrost
- Bifrost → Model providers
- Bifrost → Local stores
- Bifrost → Vector store
- Application → LiteLLM proxy
- LiteLLM proxy → Model providers
- LiteLLM proxy → Postgres
- LiteLLM proxy → Redis

### Deploying Bifrost

Bifrost's [gateway setup guide](https://docs.getbifrost.ai/quickstart/gateway/setting-up) starts the gateway with one command. The API and the web UI share port 8080:

```bash
# Run locally with npx
npx -y @maximhq/bifrost

# Or run the Docker image, persisting data to ./data
docker run -p 8080:8080 -v $(pwd)/data:/app/data maximhq/bifrost
```

The app directory holds an optional `config.json`, a `config.db` SQLite database for configuration made in the UI, and a `logs.db` SQLite database for request logs. Postgres 16 or later can replace either store, and a vector store is needed only for semantic caching. For Kubernetes, Bifrost publishes a Helm chart and guides for EKS, GKE, and AKS. For production, the setup guide advises pinning an image version, enabling authentication, using TLS, and choosing durable storage.

### Deploying the LiteLLM proxy

The LiteLLM proxy runs from a Docker image or a pip or uv install and listens on port 4000, with the admin UI at `/ui`. The [Docker quickstart](https://docs.litellm.ai/docs/proxy/docker_quick_start) runs it without a database from a config file:

```bash
docker run \
  -v "$PWD/litellm_config.yaml:/app/config.yaml:ro" \
  -e OPENAI_API_KEY \
  -e LITELLM_MASTER_KEY \
  -p 127.0.0.1:4000:4000 \
  docker.litellm.ai/berriai/litellm:latest \
  --config /app/config.yaml
```

That mode has no virtual keys, database-backed model management, or spend tracking: the quickstart states that virtual keys require a database, and LiteLLM's [production guide](https://docs.litellm.ai/docs/proxy/prod) calls PostgreSQL its primary supported database for production. `LITELLM_MASTER_KEY` authorizes management calls and is the default admin UI password, and `LITELLM_SALT_KEY` encrypts provider keys added in the UI. The production guide asks for Redis 7.0 or later once there is more than one proxy instance, sizes each Kubernetes pod at 1 vCPU and 4 GiB of memory as a floor with one Uvicorn worker per pod, and sets `NUM_WORKERS` to the vCPU count on a single VM. LiteLLM publishes Helm and Terraform deployments for Kubernetes, AWS, and GCP.

The LiteLLM Python SDK needs none of this, because it runs inside the application process.

### LiteLLM's move to Rust

LiteLLM is rewriting its gateway in Rust, and the work was in release candidates on November 16, 2026. The project's [migration tracker](https://docs.litellm.ai/rust-migration) reported 39 percent of its major APIs built in v1.110.0-rc.1, including a gateway binary in the Docker image that forwards requests it cannot serve to Python. In the planned design, an Axum server receives every request, serves supported ones in Rust, and forwards the rest to a Python worker on the same host, with both sides using the same Redis and Postgres protocols.

The tracker names December 1, 2026 as the target for running the common `/v1/messages` path end to end in Rust, initially with simple-shuffle routing, Anthropic models on Anthropic, Bedrock, or Vertex AI, and supported callbacks such as S3 logs and spend logging. Custom Python callbacks and other Python-only features stay on Python for now. Until that path ships and covers your configuration, plan capacity around the Python proxy.

### Running more than one instance

Open-source Bifrost keeps gateway state in each node's memory, while LiteLLM's open-source proxy shares state across workers and replicas through Redis. Bifrost's [multinode guide](https://docs.getbifrost.ai/deployment-guides/how-to/multinode) states that running multiple OSS Bifrost nodes with a Postgres backend is not supported, because each node loads provider configs, keys, budgets, and usage into memory and does not read them back. Its recommended OSS pattern shares one `config.json` across nodes with no config store. The same page gives Maxim's view that one OSS instance can handle around 3,000 to 5,000 requests per second, and real-time state sync across nodes comes from Enterprise cluster mode, which uses gossip-based peer-to-peer clustering.

LiteLLM's [Redis requirements page](https://docs.litellm.ai/docs/proxy/redis_requirements) says to use Redis whenever more than one worker process runs, whether from `--num_workers` or replicas. Without Redis, a limit of 100 requests per minute across 4 workers lets through 400, budgets can overshoot, a cooled-down deployment stays in rotation on other workers, and cache hit rates drop roughly by the worker count.

## Feature comparison

Bifrost and LiteLLM cover the same feature categories, so the useful differences are in depth and in where the open-source line sits. Bifrost keeps virtual keys, caching, MCP, and observability in open source and puts guardrails, SSO, and clustering in Enterprise. LiteLLM keeps guardrails and multi-instance scaling in open source and gates SSO beyond 5 users, audit logs, and RBAC.

The table reflects each project's documentation on November 16, 2026.

| Capability | Bifrost | LiteLLM |
|---|---|---|
| Provider coverage | 35 providers in the support matrix, hosted and self-hosted | 100+ providers through the SDK and proxy |
| API formats | OpenAI-compatible endpoints plus Anthropic, Google GenAI, Bedrock, Cohere, and LiteLLM SDK routes | OpenAI format plus `/messages`, `/responses`, `/rerank`, `/batches`, and `/a2a` |
| Routing and fallbacks | Weighted key load balancing, retries with backoff and jitter, per-request fallback chains, CEL routing rules, model aliases, complexity router; adaptive routing in Enterprise | Router strategies (simple-shuffle default, usage-based, latency-based, least-busy, cost-based), fallbacks, context window fallbacks, cooldowns |
| Virtual keys and budgets | Budgets and token or request limits per customer, team, virtual key, and provider config | Postgres-backed keys with budgets and RPM or TPM limits per key, user, team, and end user; model-specific and tag budgets in Enterprise |
| Caching | Exact hash and semantic matching on Redis or Valkey, Qdrant, Weaviate, or Pinecone | Exact match on Redis, S3, GCS, memory, or disk; semantic on Redis, Valkey, or Qdrant |
| Guardrails | Enterprise: request and response rules, secrets detection, redaction modes, third-party integrations | Open-source framework with custom guardrails and Presidio; per-key guardrails and several built-in integrations need Enterprise |
| MCP gateway | STDIO, HTTP, and SSE servers behind one `/mcp` endpoint; tool filtering per virtual key; agent mode; Code Mode | Streamable HTTP, SSE, and stdio (off by default); access by key, team, and organization; tool permissions per entity; OAuth 2.0 |
| A2A agent gateway | Agent Gateway over JSON-RPC, REST, or gRPC; virtual keys grant agent access | `/a2a/{agent_id}` endpoint; key and team checks; per-agent spend |
| Observability | Built-in log store and dashboard, Prometheus `/metrics`, OpenTelemetry traces with GenAI conventions | Prometheus `/metrics`, OpenTelemetry callback with opt-in GenAI conventions, many logging callbacks |
| Plugins | Go plugins compiled to `.so` files, with LLM and HTTP transport hooks | Python `CustomLogger` callbacks and `CustomGuardrail` classes |
| Admin UI | Built-in web UI on port 8080 | Admin UI at `/ui` on port 4000 |
| SSO, RBAC, audit logs | Enterprise: OIDC SSO, SCIM, RBAC, signed audit logs | SSO free for up to 5 users; SCIM, RBAC, and audit logs in Enterprise |
| Multi-node state sync | Enterprise cluster mode | Open source, through Redis and Postgres |
| License | Apache 2.0 | MIT outside `litellm/enterprise/` |

### Routing, keys, and budgets

Both gateways implement [model failover](https://theaiengineering.com/glossary/model-failover) and [load balancing](https://theaiengineering.com/glossary/llm-load-balancing) across keys and deployments. [Bifrost's retry and fallback rules](https://docs.getbifrost.ai/features/retries-and-fallbacks) retry network errors and 5xx responses on the same key with exponential backoff and jitter, rotate to another key on a 429 or a rejected credential, and give each entry in a request's `fallbacks` list its own retry budget. LiteLLM's router recommends `simple-shuffle` for production and puts a failing deployment into a cooldown, 5 seconds by default. Our [LLM routing guide](https://theaiengineering.com/articles/llm-routing) compares these strategies in depth.

Bifrost attaches budgets and [rate limits](https://theaiengineering.com/glossary/rate-limiting) to customers, teams, virtual keys, and individual providers within a key, and a request must pass every applicable budget; keys owned by users and budgets from access profiles are Enterprise features. LiteLLM attaches budgets and RPM or TPM limits to keys, users, teams, and customers, and adds a per-key `max_parallel_requests` cap. Open-source Bifrost marks a failing key dead for the rest of that request, while holding refused keys out of rotation across requests is part of Enterprise adaptive routing.

### Caching and guardrails

Both gateways offer exact-match and [semantic caching](https://theaiengineering.com/glossary/semantic-caching). [Bifrost's semantic cache](https://docs.getbifrost.ai/features/semantic-caching) runs a direct hash lookup first, falls back to similarity matching at a default threshold of 0.8, and scopes entries to the virtual key that wrote them. LiteLLM's caching documentation warns that semantic caching suits single-shot prompts and works badly on agentic traffic.

[Guardrails](https://theaiengineering.com/glossary/guardrails) mark the clearest licensing split. Bifrost's guardrails, which check prompts, responses, and MCP tool arguments and results, ship in Enterprise, with integrations for AWS Bedrock Guardrails, Azure Content Safety, Google Model Armor, Presidio, and others. LiteLLM's open-source guardrail framework runs checks before, during, or after a call and includes custom guardrails and Presidio; per-key guardrail control and integrations such as Lakera prompt injection need an Enterprise license.

### MCP, agents, and observability

Both projects run an [MCP gateway](https://theaiengineering.com/glossary/mcp-gateway). [Bifrost's MCP gateway](https://docs.getbifrost.ai/mcp/overview) scopes every `/mcp` request to the tools its virtual key allows, runs an agent mode that can auto-execute approved tools, and offers Code Mode, where the model writes a short script in a sandboxed Starlark interpreter instead of receiving every tool definition. In Maxim AI's own Code Mode benchmark with 508 tools across 16 servers, input tokens fell 92.8 percent at the same 100 percent pass rate. LiteLLM's [MCP gateway](https://docs.litellm.ai/docs/mcp) controls access per key, team, and organization, filters tools with `mcp_tool_permissions`, and supports OAuth 2.0 for upstream servers.

For [A2A](https://theaiengineering.com/glossary/a2a-protocol) agents, Bifrost's agent gateway lets a client and an agent use different transports (JSON-RPC, REST, or gRPC). LiteLLM's agent gateway connects to agents hosted on LangGraph, Vertex AI Agent Engine, Azure AI Foundry, Bedrock AgentCore, and Pydantic AI. Our [A2A vs MCP comparison](https://theaiengineering.com/articles/a2a-vs-mcp) explains how the two protocols divide the work.

Both export Prometheus metrics and [OpenTelemetry](https://theaiengineering.com/glossary/opentelemetry) traces using the GenAI semantic conventions; LiteLLM turns those conventions on with an opt-in environment variable. Bifrost also keeps a request log with a per-request overhead breakdown in its UI. Our guide to [LLM observability](https://theaiengineering.com/articles/llm-observability) covers what gateway telemetry misses.

## Performance and published benchmarks

The Bifrost vs LiteLLM benchmarks that Maxim AI and LiteLLM publish are vendor-run. Maxim AI's tests and LiteLLM's own November 2026 tests both measured much lower overhead and tail latency for Bifrost than for LiteLLM's Python proxy, and LiteLLM's tests measured its unreleased Rust branch below Bifrost. Setups differ widely, so treat every figure as a starting hypothesis.

| Benchmark | Setup | Bifrost | LiteLLM Python proxy |
|---|---|---|---|
| Maxim AI, 500 RPS load test | One t3.medium (2 vCPUs, 4 GB) in AWS us-east-1, 500 virtual users, 60 seconds; versions not stated | 424 req/s, P50 804 ms, P99 1.68 s, 120 MB peak memory, 100% success | 44.84 req/s, P50 38.65 s, P99 90.72 s, 372 MB peak memory, 88.78% success |
| Maxim AI, overhead test | 60 ms mock OpenAI upstream | 0.99 ms overhead | 40 ms overhead |
| LiteLLM, short-prompt chat | 4 instances with 4 vCPUs, 8 GB, and 4 workers each; mock OpenAI endpoint; Postgres on, Redis off; 1,000 users | Not tested | 1,170 req/s, p95 overhead 8 ms, p99 overhead 13 ms |
| LiteLLM, full-features run (November 13, 2026) | One replica each with 4 cores and 16 GiB, 50 req/s for 5 minutes, 10k to 50k token requests, 1,000 virtual keys, single run | v2.3.0: median overhead 1.2 ms, p99 4.2 ms, 2,809 MiB peak | 4 workers: median overhead 79.3 ms, p99 1,587.8 ms, 9,553 MiB peak |

Sources: [Maxim AI's benchmark page](https://www.getmaxim.ai/bifrost/resources/benchmarks), with code in [maximhq/bifrost-benchmarking](https://github.com/maximhq/bifrost-benchmarking); [LiteLLM's benchmarks page](https://docs.litellm.ai/docs/benchmarks); and LiteLLM's [full-features benchmark post](https://docs.litellm.ai/blog/better-benchmarks), with code and data in [BerriAI/ai-gateway-bench](https://github.com/BerriAI/ai-gateway-bench).

The Python proxy scales by adding processes. LiteLLM's high-throughput profile reached 3,000 requests per second with a p99 gateway latency of 91.6 ms on 33 pods of 4 workers each, with Postgres, Redis, budgets, and spend tracking on. In LiteLLM's full-features run, the Python proxy served only 78.9 percent of requests at 75 requests per second and 29.3 percent at 100, while its Rust branch measured a median overhead of 0.9 ms, a p99 of 1.8 ms, and 357 MiB of peak memory. That branch is not a stable release, and LiteLLM's post expects comparable fast-path performance in its December 1 release. Bifrost's peak memory was 120 MB in Maxim's test and 2,809 MiB in LiteLLM's full-features run, which used 10k to 50k token requests with S3 request logs and Prometheus metrics on. LiteLLM's post also notes that Bifrost's Bedrock calls reached the mock over TLS in that run while the other gateways' did not.

### Why a Go process and a Python proxy behave differently under concurrency

A Go gateway can spread one process across every CPU core, while a CPython gateway runs Python code on one core at a time per process and must add processes to use more cores. Under high concurrency, that difference shows up as queueing inside each Python worker and as shared state moving out to Redis.

**Figure 2: Bifrost scales across cores inside one process; LiteLLM scales by adding worker processes that coordinate through Redis and Postgres.**

Bifrost runs goroutines from one Go process across all CPU cores with state in memory, while LiteLLM runs several Python worker processes that share state through Redis and Postgres.

Components:

- **Bifrost process**: Goroutines run on every core
- **In-memory state**: Keys, budgets, usage, routing
- **Worker 1**: asyncio event loop Python on one core
- **Worker N**: asyncio event loop Python on one core
- **Redis and Postgres**: Shared limits, budgets, cooldowns, keys, spend

Groups:

- Bifrost: Bifrost process, In-memory state
- LiteLLM proxy: Worker 1, Worker N, Redis and Postgres

Connections:

- Bifrost process → In-memory state: same memory
- Worker 1 → Redis and Postgres
- Worker N → Redis and Postgres: network hop

**Bifrost uses goroutines inside one process.** The Go runtime schedules lightweight goroutines across all available cores. [Bifrost's concurrency architecture](https://docs.getbifrost.ai/architecture/core/concurrency) adds an isolated worker pool per provider, channel-based handoffs between stages, and object pools that reuse request and response structures to limit allocations. Keys, budgets, and usage counters sit in the same memory, so checking a budget needs no network call.

**LiteLLM uses an event loop per worker process.** In CPython's default build, the [global interpreter lock](https://docs.python.org/3/glossary.html#term-global-interpreter-lock) lets only one thread execute Python bytecode at a time; Python 3.13 added an optional build that disables it. Each Uvicorn worker runs an asyncio event loop that interleaves many requests while they wait on the network. The CPU work around each request (parsing, auth lookup, routing, cost calculation, and callbacks) still runs on that one loop, so once that work exceeds what one core can do, requests queue and tail latency climbs. LiteLLM's November 2026 profiling, with all callbacks on and one pinned core per gateway, measured 33.7 ms of gateway time for one non-streamed request with a cached key on the Python proxy, against 1.7 ms on its Rust branch.

More workers add cores but also separate memory spaces. LiteLLM's benchmark post notes that in-process auth caches are harder to share in Python because the workers are forked processes, and its Redis page shows what happens to limits, budgets, and caches without shared state. Bifrost's single-process design avoids that coordination, but open-source Bifrost has no built-in way to share its in-memory state across nodes.

### How to load-test both on your own traffic

Both vendors publish their test code, so a comparison on your own traffic can reuse it with your own settings:

1. **Match production features.** Turn on virtual keys, budgets, request logging, and the callbacks or plugins you plan to use. Overhead with logging on differs from forwarding-only overhead.
2. **Use realistic payloads.** Replay your prompt sizes, streaming ratio, and tool-call patterns. LiteLLM's profiling found gateways holding several copies of a large request body in memory while it streams.
3. **Subtract a baseline.** Run the same load against a mock upstream or the provider directly, then subtract it, as both benchmark kits do.
4. **Sweep concurrency.** Step from low load to past your expected peak and record p50, p99, error rate, CPU, and memory at each step, not only peak throughput.
5. **Size each gateway the way you would run it.** Compare one Bifrost node against the LiteLLM pod count, workers, and Redis you would deploy, and test failover by injecting 429 and 5xx errors.
6. **Read the gateway's own timing.** LiteLLM returns its overhead in the `x-litellm-overhead-duration-ms` response header, and Bifrost can export a per-component overhead histogram named `bifrost_overhead_component_microseconds`.

## Migrating from LiteLLM to Bifrost

Bifrost documents two migration paths from LiteLLM: keep the LiteLLM Python SDK and point it at Bifrost's `/litellm` endpoint, or copy a LiteLLM proxy's providers, teams, users, and virtual keys into Bifrost with a migration CLI. Either way, callers need Bifrost virtual keys, and code that calls the proxy changes its base URL and model names.

### Keep the LiteLLM SDK

[Bifrost's LiteLLM SDK integration](https://docs.getbifrost.ai/integrations/litellm-sdk) accepts LiteLLM SDK calls at `/litellm`, so an application changes one argument. A virtual key goes in the `x-bf-vk` header:

```python
from litellm import completion

response = completion(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": "Hello!"}],
    base_url="http://localhost:8080/litellm",           # point the SDK at Bifrost
    extra_headers={"x-bf-vk": "<BIFROST_VIRTUAL_KEY>"},  # Bifrost governance
)
```

The integration only works for providers that both LiteLLM and Bifrost support; the docs warn that requests to a provider only one of them supports will fail. Bifrost's compatibility plugin, described in its docs as built for LiteLLM proxy and SDK compatibility, can also convert text completions to chat, drop parameters a model does not support, and route chat requests to the Responses API for models that need it.

### Move a LiteLLM proxy's data

[Bifrost's LiteLLM migration guide](https://docs.getbifrost.ai/migration-guides/litellm) describes a one-time tool that reads a running LiteLLM proxy and recreates its entities through Bifrost's management API. It only reads from LiteLLM. A dry run reports what would change:

```bash
export LITELLM_URL="http://localhost:4000"
export LITELLM_MASTER_KEY="<LITELLM_MASTER_KEY>"
export LITELLM_CONFIG="/path/to/litellm/config.yaml"
export BIFROST_URL="http://localhost:8080"
export BIFROST_API_KEY="<BIFROST_API_KEY>"
# Needed only when models live in the LiteLLM database:
# export LITELLM_DB_URL="postgresql://..." LITELLM_SALT_KEY="..."

DRY_RUN=1 npx @maximhq/bifrost-migration-cli   # preview
npx @maximhq/bifrost-migration-cli             # migrate
```

The tool maps model deployments to Bifrost providers, keys, and model configs; organizations to customers; teams to teams; internal users to users; and virtual keys to virtual keys, carrying over budgets and RPM and TPM limits. Environment references such as `os.environ/OPENAI_API_KEY` become `env.OPENAI_API_KEY`, so those variables must exist where Bifrost runs. The guide lists the gaps: partial wildcards such as `openai/gpt-4*` are skipped, users without an email are skipped, organization-level model restrictions fold into virtual keys, and virtual key token values are not migrated.

### What changes in application code

Applications that call the LiteLLM proxy with the OpenAI SDK change three things: the base URL, the key, and possibly the model name. Bifrost generates new virtual keys in the `sk-bf-` format, which must be reissued to every caller, and its OpenAI-compatible route takes `provider/model` names:

```python
from openai import OpenAI

# Before: LiteLLM proxy on its default port, with a LiteLLM virtual key
# client = OpenAI(base_url="http://localhost:4000", api_key="<LITELLM_VIRTUAL_KEY>")

# After: Bifrost's OpenAI-compatible route, with a Bifrost virtual key
client = OpenAI(base_url="http://localhost:8080/openai", api_key="<BIFROST_VIRTUAL_KEY>")

response = client.chat.completions.create(
    model="openai/gpt-4o-mini",  # provider/model; unprefixed names default to OpenAI
    messages=[{"role": "user", "content": "Hello!"}],
)
```

The LiteLLM proxy serves the `model_name` values defined in its `config.yaml`, so confirm how each one resolves after migration and recreate any you still need with Bifrost's model aliasing. Some provider names also differ: the migration guide maps `vertex_ai` to `vertex`, `hosted_vllm` to `vllm`, `ollama_chat` to `ollama`, and `azure_ai` to `azure`. Run both gateways side by side, shift a share of traffic, and compare logs, costs, and error rates before cutting over.

## When to choose Bifrost or LiteLLM

Choose Bifrost for a single-binary gateway that handles high request rates on one node, MCP and agent traffic, and drop-in routes for several provider SDKs. Choose LiteLLM for a Python library inside your application, the widest provider list of the two, or open-source horizontal scaling with shared budgets. Then confirm the choice with a load test on your own traffic.

| Scenario | Start with | Why |
|---|---|---|
| One shared gateway with minimal infrastructure | Bifrost | Single binary with a web UI and SQLite stores; no Postgres or Redis required |
| High request rates on one node | Bifrost | One process uses every core; lower overhead than the Python proxy in both vendors' benchmarks |
| Agents that use many MCP tools | Bifrost | Tool filtering per virtual key, agent mode, and Code Mode |
| A Go service that wants an embedded gateway | Bifrost Go SDK | Provider routing in-process, no separate server |
| A Python application that wants a library, not a service | LiteLLM SDK | In-process calls to 100+ providers with router fallbacks |
| A provider only one gateway supports | Whichever lists it | Check both provider lists; LiteLLM's is longer |
| Several open-source replicas with shared limits and budgets | LiteLLM | Redis and Postgres share state across workers without an Enterprise license |
| Guardrails or a few SSO users without an Enterprise license | LiteLLM | Open-source guardrail framework; SSO free for up to 5 users |
| Custom request logic written in Python | LiteLLM | `CustomLogger` hooks and custom guardrails |

The two can also be combined, because the LiteLLM SDK can call Bifrost's `/litellm` endpoint. For other open-source and managed gateways, see our [AI gateways directory](https://theaiengineering.com/platforms/ai-gateways). For how a gateway fits into a production stack, start with our explainer on [what an AI gateway is](https://theaiengineering.com/articles/what-is-an-ai-gateway), and for routing strategies behind both gateways, read the [LLM routing guide](https://theaiengineering.com/articles/llm-routing).

## Frequently asked questions

### Is Bifrost faster than LiteLLM?

In Maxim AI's published benchmark on one t3.medium instance at a 500 requests-per-second target, Bifrost served 424 requests per second against 44.84 for LiteLLM, with a P99 latency of 1.68 seconds against 90.72 seconds. LiteLLM's own November 2026 benchmark also measured lower overhead for Bifrost than for its Python proxy, and lower still for its unreleased Rust branch. Both tests are vendor-run, so test on your own traffic.

### Is Bifrost open source?

Yes. The Bifrost gateway, Go SDK, and plugins are published in the maximhq/bifrost repository under the Apache 2.0 license, and self-hosting the open-source gateway is free. Bifrost Enterprise adds guardrails, OIDC single sign-on, SCIM, role-based access control, signed audit logs, cluster mode, adaptive routing, and secret manager integrations. LiteLLM is MIT licensed outside its enterprise directory and also sells an Enterprise license.

### Does LiteLLM proxy require a database?

The LiteLLM proxy can run from a config file without a database, but virtual keys, database-backed model management, spend tracking, and global budgets require Postgres, which LiteLLM calls its primary supported production database. LiteLLM's documentation also says to add Redis once you run more than one worker process, because rate limits, budgets, cooldowns, and caches are otherwise counted separately in each worker.

### How do I migrate from LiteLLM to Bifrost?

Bifrost documents two paths. Applications built on the LiteLLM Python SDK can set base_url to Bifrost's /litellm endpoint and keep their code. Teams running a LiteLLM proxy can run npx @maximhq/bifrost-migration-cli, which reads the proxy's config and database and recreates providers, keys, customers, teams, users, and virtual keys in Bifrost. Virtual key tokens are not copied, so callers need new sk-bf- keys.

### Is LiteLLM written in Rust now?

Partly, and the work is in progress. On November 16, 2026, LiteLLM's migration tracker reported 39 percent of its major APIs built in v1.110.0-rc.1, with a Rust gateway binary in the Docker image that forwards requests it cannot serve to a Python worker. Its next milestone, targeted for December 1, 2026, is running the common /v1/messages path end to end in Rust.

### Do Bifrost and LiteLLM support MCP?

Yes, both act as MCP gateways. Bifrost connects to MCP servers over STDIO, HTTP, or SSE, exposes them at one /mcp endpoint, filters tools per virtual key, and adds agent mode and Code Mode. LiteLLM supports Streamable HTTP, SSE, and stdio servers, with stdio off by default, controls access by key, team, and organization, sets tool permissions per entity, and supports OAuth 2.0 for MCP servers.

## Terms in this guide

- [AI gateway](https://theaiengineering.com/glossary/ai-gateway): An AI gateway is a proxy layer between applications and model providers that exposes one API for many models and centralizes routing, failover, access control, cost tracking, caching, and logging.
- [OpenAI-compatible API](https://theaiengineering.com/glossary/openai-compatible-api): An OpenAI-compatible API is an HTTP API that accepts OpenAI's request and response formats, so code written for OpenAI's SDKs can call another server by changing the base URL, API key, and model name.
- [Virtual key](https://theaiengineering.com/glossary/virtual-key): A virtual key is an API key issued by an AI gateway that maps a caller to its allowed models, budget, and rate limits, while the gateway keeps the real provider credentials.
- [Model failover](https://theaiengineering.com/glossary/model-failover): Model failover is the automatic rerouting of an LLM request to a backup model, deployment, or provider when the primary target returns an error, times out, or hits a rate limit.
- [LLM load balancing](https://theaiengineering.com/glossary/llm-load-balancing): LLM load balancing is the distribution of requests for one model across several interchangeable deployments, such as API keys, regions, providers, or replicas, by weight, latency, load, or remaining quota.
- [Rate limiting](https://theaiengineering.com/glossary/rate-limiting): Rate limiting is the enforcement of a maximum number of requests or tokens a client may send to an API within a time window, with excess requests rejected, usually with HTTP 429.
- [Semantic caching](https://theaiengineering.com/glossary/semantic-caching): Semantic caching is a technique that returns a stored LLM response when a new query is close in meaning to an earlier one, measured by embedding similarity instead of exact text match.
- [Guardrails](https://theaiengineering.com/glossary/guardrails): Guardrails are programmatic checks placed around a large language model that validate, filter, or block inputs, outputs, and tool calls so an AI application stays within its safety, security, and policy limits.
- [MCP gateway](https://theaiengineering.com/glossary/mcp-gateway): An MCP gateway is a proxy that sits between MCP hosts and many MCP servers, exposing them through one endpoint and centralizing authentication, tool access policy, and audit logging.
- [A2A protocol](https://theaiengineering.com/glossary/a2a-protocol): The A2A (Agent2Agent) protocol is an open standard, introduced by Google in April 2025, that lets AI agents built on different frameworks discover each other, exchange messages, and delegate tasks.
- [OpenTelemetry](https://theaiengineering.com/glossary/opentelemetry): OpenTelemetry is an open source, vendor-neutral observability framework of APIs, SDKs, a wire protocol, and naming conventions for traces, metrics, and logs, including conventions for LLM calls and agent runs.
- [LLM routing](https://theaiengineering.com/glossary/llm-routing): LLM routing is the practice of directing each request to one of several language models or providers, based on rules or a learned classifier, to balance cost, quality, latency, and availability.

## Related guides

- [What is an AI gateway? How it works and when you need one](https://theaiengineering.com/articles/what-is-an-ai-gateway): An AI gateway is a proxy between your apps and model providers. Learn how it works, its core features, AI gateway vs API gateway, and open source options.
- [LLM routing: strategies, fallbacks, and learned routers](https://theaiengineering.com/articles/llm-routing): LLM routing guide: rule-based and cost-aware strategies, fallbacks, load balancing, semantic and learned routers like RouteLLM, router evals, and pitfalls.
- [LLM observability: what to capture, which metrics, and tools](https://theaiengineering.com/articles/llm-observability): LLM observability guide: what to capture in traces and spans, metrics to track, OpenTelemetry GenAI conventions, tool categories, and keeping prompts private.
- [A2A protocol vs MCP: differences and when to use each](https://theaiengineering.com/articles/a2a-vs-mcp): A2A protocol vs MCP: A2A connects AI agents to other agents, and MCP connects an agent to tools and data. Compare agent cards, tasks, transports, and security.

Part of the [AI gateways and LLM routing](https://theaiengineering.com/topics/ai-gateways) topic hub.

## Cite this guide

The AI Engineering Editors. "Bifrost vs LiteLLM: architecture, features, and performance compared." The AI Engineering, updated Nov 16, 2026. https://theaiengineering.com/articles/bifrost-vs-litellm

- Title: Bifrost vs LiteLLM: architecture, features, and performance compared
- Author: The AI Engineering Editors
- Publisher: The AI Engineering
- Published: 2026-11-16
- Updated: 2026-11-16
- URL: https://theaiengineering.com/articles/bifrost-vs-litellm
- Markdown: https://theaiengineering.com/articles/bifrost-vs-litellm.md

To cite one section, link to its anchor:

- [Bifrost vs LiteLLM at a glance](https://theaiengineering.com/articles/bifrost-vs-litellm#bifrost-vs-litellm-at-a-glance)
- [What are Bifrost and LiteLLM?](https://theaiengineering.com/articles/bifrost-vs-litellm#what-are-bifrost-and-litellm)
- [Architecture and deployment](https://theaiengineering.com/articles/bifrost-vs-litellm#architecture-and-deployment)
- [Feature comparison](https://theaiengineering.com/articles/bifrost-vs-litellm#feature-comparison)
- [Performance and published benchmarks](https://theaiengineering.com/articles/bifrost-vs-litellm#performance-and-published-benchmarks)
- [Migrating from LiteLLM to Bifrost](https://theaiengineering.com/articles/bifrost-vs-litellm#migrating-from-litellm-to-bifrost)
- [When to choose Bifrost or LiteLLM](https://theaiengineering.com/articles/bifrost-vs-litellm#when-to-choose-bifrost-or-litellm)
- [Frequently asked questions](https://theaiengineering.com/articles/bifrost-vs-litellm#faq)
