---
title: "What is test-time compute? Inference scaling explained"
description: "Test-time compute is the processing an LLM spends while answering a query. How reasoning, sampling, and search scale it, and how effort settings control it."
type: "glossary-term"
canonical_url: "https://theaiengineering.com/glossary/test-time-compute"
markdown_url: "https://theaiengineering.com/glossary/test-time-compute.md"
topic: "LLM models and benchmarks"
author: "The AI Engineering Editors"
publisher: "The AI Engineering"
updated: "2026-11-06"
keywords: ["Test-time compute","inference-time compute","test-time scaling","inference scaling","inference-time scaling"]
citation: "The AI Engineering Editors. \"What is test-time compute? Inference scaling explained.\" The AI Engineering, updated Nov 06, 2026. https://theaiengineering.com/glossary/test-time-compute"
---

# What is test-time compute? Inference scaling explained

Also called inference-time compute, test-time scaling, inference scaling, inference-time scaling.

> **Definition of Test-time compute:** Test-time compute is the computation a language model spends while answering a query, through longer reasoning, multiple samples, or search, as opposed to the train-time compute spent building the model.

Test-time compute is the computation a [large language model](https://theaiengineering.com/glossary/large-language-model) spends at inference, while it answers a query, as opposed to train-time compute, the computation spent pretraining and post-training it. A model raises its test-time compute by writing a longer chain of reasoning, by sampling several answers and combining them, or by searching over partial solutions with a verifier. [Reasoning models](https://theaiengineering.com/glossary/reasoning-model) are trained to spend it, and model APIs expose settings that steer how much they spend. For an AI engineer, it is a per-request control over accuracy, cost, and latency.

## How test-time compute scales

Test-time compute scales in two directions. Sequential scaling makes one response longer as the model reasons, revises, and checks its work. Parallel scaling generates several responses and selects one by majority vote, a verifier, or a reward model. Search methods combine the two by expanding only the most promising partial solutions, scored step by step.

- **Sequential.** The model writes a longer [chain of thought](https://theaiengineering.com/glossary/chain-of-thought). OpenAI's [o1 announcement](https://openai.com/index/learning-to-reason-with-llms/) reported that o1's performance consistently improved with more reinforcement learning (train-time compute) and with more time spent thinking (test-time compute).
- **Parallel.** Self-consistency samples several reasoning paths and keeps the most common final answer; best-of-N keeps the sample a verifier scores highest. Metrics such as [pass@k](https://theaiengineering.com/glossary/pass-at-k) describe what repeated sampling can reach.

[Snell et al. (2024)](https://arxiv.org/abs/2408.03314) found that the best method depends on how hard the prompt is. Allocating test-time compute per prompt improved efficiency by more than 4 times over a best-of-N baseline. In a FLOPs-matched comparison, a smaller model with extra test-time compute outperformed a 14 times larger model on problems where the smaller model already had a non-trivial success rate.

## Test-time compute vs train-time compute

Train-time compute is spent before release to pretrain and post-train a model, and its cost is shared across every later request. Test-time compute is spent again on every request, so it grows with traffic and appears directly in each response's token count, bill, and latency.

| | Train-time compute | Test-time compute |
|---|---|---|
| When it is spent | Before release | On every request |
| Who pays | The model developer | Whoever serves or calls the model |
| How it grows | Larger models, more data, more reinforcement learning | Longer reasoning, more samples, search |
| Where it shows up | Model capability | Output [tokens](https://theaiengineering.com/glossary/tokens), cost, and latency |

## Controlling test-time compute in APIs

Model APIs expose test-time compute as a request setting. OpenAI's API takes a reasoning effort level, Anthropic's API takes an effort setting with adaptive thinking, and Google's Gemini API takes a thinking level. Higher settings let the model generate more reasoning tokens, which providers bill as output tokens even when the response hides them.

- **OpenAI.** The [reasoning guide](https://developers.openai.com/api/docs/guides/reasoning) names the parameter `reasoning.effort` in the Responses API and `reasoning_effort` in Chat Completions, with supported values and defaults that vary by model. Reasoning tokens are hidden from the response but occupy context window space and are billed as output tokens.
- **Anthropic.** Current Claude models use `thinking: {type: "adaptive"}` with `output_config.effort`. The older manual mode set a fixed thinking budget with `budget_tokens`, minimum 1,024; Anthropic's [extended thinking documentation](https://platform.claude.com/docs/en/build-with-claude/extended-thinking) marks it deprecated, and Claude 4.7 and later models reject it.
- **Google.** The Gemini API sets `thinking_level` in the generation config.

Because reasoning precedes the visible answer, higher settings also raise [time to first token](https://theaiengineering.com/glossary/time-to-first-token). Compare benchmark scores only at matched effort settings, and use [LLM routing](https://theaiengineering.com/glossary/llm-routing) to send only hard prompts to high effort. The [reasoning models explainer](https://theaiengineering.com/articles/reasoning-models) covers when the extra compute pays off.

## Related terms

- [Reasoning model](https://theaiengineering.com/glossary/reasoning-model): A reasoning model is a large language model trained, usually with reinforcement learning, to generate intermediate reasoning tokens before its final answer, spending more inference compute to solve multi-step problems.
- [Chain-of-thought prompting](https://theaiengineering.com/glossary/chain-of-thought): Chain-of-thought prompting is a technique that has a language model write intermediate reasoning steps before its final answer, which improves large models' accuracy on arithmetic, commonsense, and symbolic reasoning tasks.
- [Tokens](https://theaiengineering.com/glossary/tokens): Tokens are the units of text a language model processes, typically whole words, word fragments, punctuation, or bytes, each mapped to an integer ID from the model's fixed vocabulary.
- [pass@k](https://theaiengineering.com/glossary/pass-at-k): pass@k is an evaluation metric that estimates the probability that at least one of k sampled outputs for a task passes its tests, used mainly in code generation and agent benchmarks.
- [Time to first token](https://theaiengineering.com/glossary/time-to-first-token): Time to first token (TTFT) is the elapsed time from submitting a request to an LLM until the first token of the response arrives, covering request queuing, network transit, and prompt prefill.
- [LLM routing](https://theaiengineering.com/glossary/llm-routing): LLM routing is the practice of directing each request to one of several language models or providers, based on rules or a learned classifier, to balance cost, quality, latency, and availability.

## Guides that cover test-time compute

- [Reasoning models: how they work and when to use them](https://theaiengineering.com/articles/reasoning-models): Reasoning models explained: how test-time compute and reinforcement learning produce chain-of-thought reasoning, when to use them, and their cost and latency.
- [LLM benchmarks explained: what they measure and how to read them](https://theaiengineering.com/articles/llm-benchmarks): LLM benchmarks explained: what MMLU-Pro, GPQA, SWE-bench, LiveCodeBench, ARC-AGI, and LMArena measure, why scores saturate, and how to read a leaderboard.
- [LLM inference optimization: a practical guide for engineers](https://theaiengineering.com/articles/llm-inference-optimization): LLM inference optimization guide: prefill vs decode, TTFT, KV cache and PagedAttention, continuous batching, quantization, speculative decoding, and GPUs.

Part of the [LLM models and benchmarks](https://theaiengineering.com/topics/models) topic hub.

## Cite this definition

The AI Engineering Editors. "What is test-time compute? Inference scaling explained." The AI Engineering, updated Nov 06, 2026. https://theaiengineering.com/glossary/test-time-compute

- Title: What is test-time compute? Inference scaling explained
- Author: The AI Engineering Editors
- Publisher: The AI Engineering
- Updated: 2026-11-06
- URL: https://theaiengineering.com/glossary/test-time-compute
- Markdown: https://theaiengineering.com/glossary/test-time-compute.md
