What is test-time compute? Inference scaling explained
Also called inference-time compute, test-time scaling, inference scaling, inference-time scaling
Definition
Test-time compute is the computation a language model spends while answering a query, through longer reasoning, multiple samples, or search, as opposed to the train-time compute spent building the model.
Test-time compute is the computation a large language model spends at inference, while it answers a query, as opposed to train-time compute, the computation spent pretraining and post-training it. A model raises its test-time compute by writing a longer chain of reasoning, by sampling several answers and combining them, or by searching over partial solutions with a verifier. Reasoning models are trained to spend it, and model APIs expose settings that steer how much they spend. For an AI engineer, it is a per-request control over accuracy, cost, and latency.
How test-time compute scales#
Test-time compute scales in two directions. Sequential scaling makes one response longer as the model reasons, revises, and checks its work. Parallel scaling generates several responses and selects one by majority vote, a verifier, or a reward model. Search methods combine the two by expanding only the most promising partial solutions, scored step by step.
- Sequential. The model writes a longer chain of thought. OpenAI's o1 announcement reported that o1's performance consistently improved with more reinforcement learning (train-time compute) and with more time spent thinking (test-time compute).
- Parallel. Self-consistency samples several reasoning paths and keeps the most common final answer; best-of-N keeps the sample a verifier scores highest. Metrics such as pass@k describe what repeated sampling can reach.
Snell et al. (2024) found that the best method depends on how hard the prompt is. Allocating test-time compute per prompt improved efficiency by more than 4 times over a best-of-N baseline. In a FLOPs-matched comparison, a smaller model with extra test-time compute outperformed a 14 times larger model on problems where the smaller model already had a non-trivial success rate.
Test-time compute vs train-time compute#
Train-time compute is spent before release to pretrain and post-train a model, and its cost is shared across every later request. Test-time compute is spent again on every request, so it grows with traffic and appears directly in each response's token count, bill, and latency.
| Train-time compute | Test-time compute | |
|---|---|---|
| When it is spent | Before release | On every request |
| Who pays | The model developer | Whoever serves or calls the model |
| How it grows | Larger models, more data, more reinforcement learning | Longer reasoning, more samples, search |
| Where it shows up | Model capability | Output tokens, cost, and latency |
Controlling test-time compute in APIs#
Model APIs expose test-time compute as a request setting. OpenAI's API takes a reasoning effort level, Anthropic's API takes an effort setting with adaptive thinking, and Google's Gemini API takes a thinking level. Higher settings let the model generate more reasoning tokens, which providers bill as output tokens even when the response hides them.
- OpenAI. The reasoning guide names the parameter
reasoning.effortin the Responses API andreasoning_effortin Chat Completions, with supported values and defaults that vary by model. Reasoning tokens are hidden from the response but occupy context window space and are billed as output tokens. - Anthropic. Current Claude models use
thinking: {type: "adaptive"}withoutput_config.effort. The older manual mode set a fixed thinking budget withbudget_tokens, minimum 1,024; Anthropic's extended thinking documentation marks it deprecated, and Claude 4.7 and later models reject it. - Google. The Gemini API sets
thinking_levelin the generation config.
Because reasoning precedes the visible answer, higher settings also raise time to first token. Compare benchmark scores only at matched effort settings, and use LLM routing to send only hard prompts to high effort. The reasoning models explainer covers when the extra compute pays off.
Related terms
Guides that cover test-time compute
Cite this definition
The AI Engineering Editors. "What is test-time compute? Inference scaling explained." The AI Engineering, updated Nov 06, 2026. https://theaiengineering.com/glossary/test-time-compute
- Title
- What is test-time compute? Inference scaling explained
- Author
- The AI Engineering Editors
- Publisher
- The AI Engineering
- Updated