What is rate limiting for LLM APIs? RPM, TPM, and 429

Also called rate limits, RPM, TPM

Definition

Rate limiting is the enforcement of a maximum number of requests or tokens a client may send to an API within a time window, with excess requests rejected, usually with HTTP 429.

Rate limiting is a cap on how much traffic a client can send to an API in a period of time. LLM APIs limit both requests and tokens, commonly expressed as requests per minute (RPM) and tokens per minute (TPM), and reject traffic above the limit with HTTP 429 Too Many Requests. AI engineers deal with rate limits in two directions: the limits a model provider places on their organization, and the limits they place on their own users, teams, agents, and tools.

How provider rate limits work#

Provider rate limits work by tracking each organization's usage per model on several dimensions and rejecting a request once any one limit is reached. OpenAI and Anthropic both limit requests and tokens per minute, raise limits with usage tiers, and return headers that report remaining capacity, so clients can slow down before a 429.

The details differ:

  • OpenAI's rate limits guide measures RPM, requests per day, TPM, tokens per day, and images per minute at the organization and project level. It counts the larger of max_tokens and the estimated request size toward the token limit, so an oversized max_tokens uses up capacity.
  • Anthropic's rate limits documentation measures RPM, input tokens per minute, and output tokens per minute for each model class. It uses the token bucket algorithm, so capacity refills continuously instead of resetting at fixed intervals, and a 60 RPM limit might be enforced as 1 request per second. Input tokens read from the prompt cache do not count toward the default input limit.

RFC 6585 defines status code 429 for a user who has sent too many requests in a given amount of time. The response may include a Retry-After header, and the RFC leaves it to each server to decide how it identifies the user and counts requests. OpenAI's guidance is to wait at least the Retry-After value when it is present and otherwise retry with exponential backoff, jitter, and a cap on attempts.

A 429 can also signal a spend cap rather than a rate limit. Anthropic's spend-cap 429 has no retry-after header and keeps failing until access resumes, so retrying wastes time; model failover to another provider or an alert is the useful response.

Rate limiting your own users and tools#

Rate limiting your own traffic means placing per-user, per-key, per-team, or per-tool limits in front of the provider, so one tenant, a runaway agent loop, or a leaked key cannot drain the organization's shared quota or budget. Token-based limits fit model calls better than request counts, because one request can carry ten tokens or a hundred thousand.

AI gateways enforce these limits per virtual key or team, often next to spend budgets, and Anthropic also lets an organization set lower limits for each workspace. Tool servers need limits as well: the MCP specification (version 2026-07-28) lists rate limiting tool invocations among the things an MCP server must do. A single model response can request several tool calls, so size tool limits for agent loops as well as for people.

Provider limit Your own limit
Set by Model provider, by usage tier Your gateway, application, or tool server
Scope Organization, project, or workspace User, key, team, agent, or tool
Units Requests and tokens per minute or day Requests, tokens, tool calls, or spend
When exceeded 429 from the provider 429 or a clear error from your service

Return errors that name the limit and when to retry, and log them as their own class in LLM observability data, separate from timeouts and server errors. The AI gateway explainer covers token limits and budgets at the proxy layer.

Cite this definition

The AI Engineering Editors. "What is rate limiting for LLM APIs? RPM, TPM, and 429." The AI Engineering, updated Nov 06, 2026. https://theaiengineering.com/glossary/rate-limiting

Title
What is rate limiting for LLM APIs? RPM, TPM, and 429
Author
The AI Engineering Editors
Publisher
The AI Engineering
Updated
View as Markdown

Part of the AI gateways and LLM routing hub · Updated