---
title: "What is rate limiting for LLM APIs? RPM, TPM, and 429"
description: "Rate limiting caps the requests and tokens a client can send to an LLM API per minute. How RPM and TPM limits work, what HTTP 429 means, and how to limit users."
type: "glossary-term"
canonical_url: "https://theaiengineering.com/glossary/rate-limiting"
markdown_url: "https://theaiengineering.com/glossary/rate-limiting.md"
topic: "AI gateways and LLM routing"
author: "The AI Engineering Editors"
publisher: "The AI Engineering"
updated: "2026-11-06"
keywords: ["Rate limiting","rate limits","RPM","TPM"]
citation: "The AI Engineering Editors. \"What is rate limiting for LLM APIs? RPM, TPM, and 429.\" The AI Engineering, updated Nov 06, 2026. https://theaiengineering.com/glossary/rate-limiting"
---

# What is rate limiting for LLM APIs? RPM, TPM, and 429

Also called rate limits, RPM, TPM.

> **Definition of Rate limiting:** Rate limiting is the enforcement of a maximum number of requests or tokens a client may send to an API within a time window, with excess requests rejected, usually with HTTP 429.

Rate limiting is a cap on how much traffic a client can send to an API in a period of time. LLM APIs limit both requests and [tokens](https://theaiengineering.com/glossary/tokens), commonly expressed as requests per minute (RPM) and tokens per minute (TPM), and reject traffic above the limit with HTTP 429 Too Many Requests. AI engineers deal with rate limits in two directions: the limits a model provider places on their organization, and the limits they place on their own users, teams, agents, and tools.

## How provider rate limits work

Provider rate limits work by tracking each organization's usage per model on several dimensions and rejecting a request once any one limit is reached. OpenAI and Anthropic both limit requests and tokens per minute, raise limits with usage tiers, and return headers that report remaining capacity, so clients can slow down before a 429.

The details differ:

- OpenAI's [rate limits guide](https://developers.openai.com/api/docs/guides/rate-limits) measures RPM, requests per day, TPM, tokens per day, and images per minute at the organization and project level. It counts the larger of `max_tokens` and the estimated request size toward the token limit, so an oversized `max_tokens` uses up capacity.
- Anthropic's [rate limits documentation](https://platform.claude.com/docs/en/api/rate-limits) measures RPM, input tokens per minute, and output tokens per minute for each model class. It uses the token bucket algorithm, so capacity refills continuously instead of resetting at fixed intervals, and a 60 RPM limit might be enforced as 1 request per second. Input tokens read from the [prompt cache](https://theaiengineering.com/glossary/prompt-caching) do not count toward the default input limit.

[RFC 6585](https://www.rfc-editor.org/rfc/rfc6585) defines status code 429 for a user who has sent too many requests in a given amount of time. The response may include a `Retry-After` header, and the RFC leaves it to each server to decide how it identifies the user and counts requests. OpenAI's guidance is to wait at least the `Retry-After` value when it is present and otherwise retry with exponential backoff, jitter, and a cap on attempts.

A 429 can also signal a spend cap rather than a rate limit. Anthropic's spend-cap 429 has no `retry-after` header and keeps failing until access resumes, so retrying wastes time; [model failover](https://theaiengineering.com/glossary/model-failover) to another provider or an alert is the useful response.

## Rate limiting your own users and tools

Rate limiting your own traffic means placing per-user, per-key, per-team, or per-tool limits in front of the provider, so one tenant, a runaway agent loop, or a leaked key cannot drain the organization's shared quota or budget. Token-based limits fit model calls better than request counts, because one request can carry ten tokens or a hundred thousand.

AI gateways enforce these limits per [virtual key](https://theaiengineering.com/glossary/virtual-key) or team, often next to spend budgets, and Anthropic also lets an organization set lower limits for each workspace. Tool servers need limits as well: the [MCP specification](https://modelcontextprotocol.io/specification/2026-07-28/server/tools) (version 2026-07-28) lists rate limiting tool invocations among the things an [MCP server](https://theaiengineering.com/glossary/mcp-server) must do. A single model response can request several tool calls, so size tool limits for agent loops as well as for people.

| | Provider limit | Your own limit |
|---|---|---|
| Set by | Model provider, by usage tier | Your gateway, application, or tool server |
| Scope | Organization, project, or workspace | User, key, team, agent, or tool |
| Units | Requests and tokens per minute or day | Requests, tokens, tool calls, or spend |
| When exceeded | 429 from the provider | 429 or a clear error from your service |

Return errors that name the limit and when to retry, and log them as their own class in [LLM observability](https://theaiengineering.com/glossary/llm-observability) data, separate from timeouts and server errors. The [AI gateway explainer](https://theaiengineering.com/articles/what-is-an-ai-gateway) covers token limits and budgets at the proxy layer.

## Related terms

- [AI gateway](https://theaiengineering.com/glossary/ai-gateway): An AI gateway is a proxy layer between applications and model providers that exposes one API for many models and centralizes routing, failover, access control, cost tracking, caching, and logging.
- [Virtual key](https://theaiengineering.com/glossary/virtual-key): A virtual key is an API key issued by an AI gateway that maps a caller to its allowed models, budget, and rate limits, while the gateway keeps the real provider credentials.
- [Tokens](https://theaiengineering.com/glossary/tokens): Tokens are the units of text a language model processes, typically whole words, word fragments, punctuation, or bytes, each mapped to an integer ID from the model's fixed vocabulary.
- [Model failover](https://theaiengineering.com/glossary/model-failover): Model failover is the automatic rerouting of an LLM request to a backup model, deployment, or provider when the primary target returns an error, times out, or hits a rate limit.
- [LLM load balancing](https://theaiengineering.com/glossary/llm-load-balancing): LLM load balancing is the distribution of requests for one model across several interchangeable deployments, such as API keys, regions, providers, or replicas, by weight, latency, load, or remaining quota.
- [MCP server](https://theaiengineering.com/glossary/mcp-server): An MCP server is a program that exposes tools, resources, and prompts to AI applications through the Model Context Protocol, running locally over stdio or remotely over Streamable HTTP.

## Guides that cover rate limiting

- [What is an AI gateway? How it works and when you need one](https://theaiengineering.com/articles/what-is-an-ai-gateway): An AI gateway is a proxy between your apps and model providers. Learn how it works, its core features, AI gateway vs API gateway, and open source options.
- [AI engineer roadmap: skills, projects, and resources by stage](https://theaiengineering.com/articles/ai-engineer-roadmap): A practical AI engineer roadmap in seven stages, each with the skills to learn, a project to ship, and resources worth the time, from Python to production.
- [How to build an MCP server: a Python and TypeScript tutorial](https://theaiengineering.com/articles/how-to-build-an-mcp-server): Learn how to build an MCP server with the official Python and TypeScript SDKs: tools, resources, transports, testing, client config, and production security.
- [LLM observability: what to capture, which metrics, and tools](https://theaiengineering.com/articles/llm-observability): LLM observability guide: what to capture in traces and spans, metrics to track, OpenTelemetry GenAI conventions, tool categories, and keeping prompts private.
- [What is tool calling? How LLM function calling works](https://theaiengineering.com/articles/tool-calling): Tool calling lets an LLM request actions from your code through structured calls. Learn the request loop, parallel calls, structured outputs, security, and MCP.
- [What is AI engineering? The discipline, the stack, and the work](https://theaiengineering.com/articles/what-is-ai-engineering): What is AI engineering? A plain definition, how it differs from ML and software engineering, the layers of the stack, and what AI engineers do day to day.
- [What is the Model Context Protocol (MCP)? How it works](https://theaiengineering.com/articles/what-is-model-context-protocol): The Model Context Protocol (MCP) is an open standard for connecting AI apps to tools and data. Learn its architecture, transports, tool calls, and security.
- [Bifrost vs LiteLLM: architecture, features, and performance compared](https://theaiengineering.com/articles/bifrost-vs-litellm): Bifrost vs LiteLLM compared: a Go single binary against a Python proxy, with deployment, routing, virtual keys, caching, MCP, benchmarks, and migration.
- [LLM routing: strategies, fallbacks, and learned routers](https://theaiengineering.com/articles/llm-routing): LLM routing guide: rule-based and cost-aware strategies, fallbacks, load balancing, semantic and learned routers like RouteLLM, router evals, and pitfalls.

Part of the [AI gateways and LLM routing](https://theaiengineering.com/topics/ai-gateways) topic hub.

## Cite this definition

The AI Engineering Editors. "What is rate limiting for LLM APIs? RPM, TPM, and 429." The AI Engineering, updated Nov 06, 2026. https://theaiengineering.com/glossary/rate-limiting

- Title: What is rate limiting for LLM APIs? RPM, TPM, and 429
- Author: The AI Engineering Editors
- Publisher: The AI Engineering
- Updated: 2026-11-06
- URL: https://theaiengineering.com/glossary/rate-limiting
- Markdown: https://theaiengineering.com/glossary/rate-limiting.md
