What is temperature in LLMs? Settings explained
Also called LLM temperature, sampling temperature
Definition
Temperature is a sampling parameter that rescales a language model's next-token probabilities: low values concentrate probability on the likeliest tokens, and high values flatten the distribution for more varied output.
Temperature is a sampling parameter that controls how random a language model's output is. At each step, a large language model produces a score, called a logit, for every token in its vocabulary, and temperature rescales those scores before they become probabilities. Lower temperature makes output more consistent across runs; higher temperature makes it more varied.
How temperature works#
Temperature works by dividing each logit by the temperature value T before the softmax function converts the logits into probabilities.
p(token_i) = exp(logit_i / T) / Σ_j exp(logit_j / T)Figure 1: Softmax with temperature T.
- T below 1 sharpens the distribution, so the top tokens receive even more probability. As T approaches 0, sampling approaches greedy decoding, which always picks the single likeliest token.
- T equal to 1 leaves the model's distribution unchanged.
- T above 1 flattens the distribution, so unlikely tokens are sampled more often; very high values produce incoherent text.
Temperature 0 does not guarantee identical outputs. Anthropic's migration guide for Claude Opus 5.5 states that temperature = 0 never guaranteed identical outputs on earlier models, and it tells developers to omit temperature, top_p, and top_k on that model and guide behavior through prompting instead. Check each model's documentation before relying on the parameter.
Temperature vs top-p#
Temperature and top-p both shape sampling, but they act on different parts of the distribution. Temperature reshapes the whole distribution. Top-p, or nucleus sampling, introduced by Holtzman et al. (2019), keeps the smallest set of tokens whose cumulative probability reaches p and samples only from that set, cutting off the unreliable tail. Top-k is a simpler variant that keeps a fixed number of top tokens. The settings interact, so changing one at a time makes the effect easier to measure.
Example#
For extraction, classification, and code generation, a low temperature is the usual choice because repeated runs give similar output, and the output can then be validated with structured outputs or tests. For brainstorming or generating varied synthetic data, a higher temperature adds diversity. Because sampling adds variance at any setting above zero, LLM evaluation should run each test case several times rather than once.
Related terms
Guides that cover temperature
Cite this definition
The AI Engineering Editors. "What is temperature in LLMs? Settings explained." The AI Engineering, updated Nov 06, 2026. https://theaiengineering.com/glossary/temperature
- Title
- What is temperature in LLMs? Settings explained
- Author
- The AI Engineering Editors
- Publisher
- The AI Engineering
- Updated