What is chain-of-thought prompting in LLMs?

Also called CoT, chain of thought, CoT prompting

Definition

Chain-of-thought prompting is a technique that has a language model write intermediate reasoning steps before its final answer, which improves large models' accuracy on arithmetic, commonsense, and symbolic reasoning tasks.

Chain-of-thought (CoT) prompting is a prompt engineering technique that has a language model write out intermediate reasoning steps before it gives a final answer. Wei et al. introduced it in 2022 by adding worked examples with step-by-step solutions to the prompt. Writing the steps gives the model more generated tokens to work through a problem and makes the path to the answer readable, which helps on arithmetic, commonsense, and symbolic reasoning problems. Reasoning models generate a chain of thought without being asked, which changes when the technique is worth using.

How chain-of-thought prompting works#

Chain-of-thought prompting works by showing or telling the model to reason before it answers. Few-shot CoT places worked examples in the prompt, each with a question, the reasoning steps, and the answer, so the model imitates the format. Zero-shot CoT adds a single instruction, such as "Let's think step by step," with no examples.

A few-shot exemplar looks like this:

Q: A warehouse has 3 shelves with 12 boxes each. 7 boxes ship. How many boxes remain?
A: 3 shelves times 12 boxes is 36 boxes. 36 minus 7 is 29. The answer is 29.

The research behind the technique:

  • Few-shot CoT. Wei et al. (2022) reported that eight chain-of-thought exemplars let a 540B-parameter model reach the highest reported accuracy at the time on the GSM8K math word problem benchmark, beating a fine-tuned GPT-3 with a verifier. The authors describe the ability as emerging naturally in sufficiently large models.
  • Zero-shot CoT. Kojima et al. (2022) found that appending "Let's think step by step" raised InstructGPT (text-davinci-002) accuracy from 17.7% to 78.7% on MultiArith and from 10.4% to 40.7% on GSM8K.
  • Self-consistency. Wang et al. (2022) replaced a single greedy answer with several sampled reasoning paths and kept the most consistent final answer, reporting a 17.9% gain on GSM8K over standard chain-of-thought prompting. Sampling several paths is one form of test-time compute.

CoT also affects how benchmarks are run. Chain-of-thought instructions are one reason two published scores for the same model differ, and the MMLU-Pro authors report a benefit from CoT that the original MMLU did not show. Fine-tuning recipes use it too: RAFT trains models to answer from retrieved documents in a chain-of-thought style that quotes the relevant passage verbatim.

Chain-of-thought prompting vs reasoning models#

Chain-of-thought prompting asks a standard model to reason in its visible output. A reasoning model is trained, usually with reinforcement learning, to generate its own chain of thought before answering. With a reasoning model, the reasoning happens whether or not the prompt asks for it, so step-by-step instructions add little and can sometimes hurt performance.

Chain-of-thought prompting Reasoning model
Source of the reasoning Examples or an instruction in the prompt Training, usually reinforcement learning
Where the reasoning appears In the visible response In reasoning tokens, often hidden or summarized
How to control it Prompt wording and exemplars Effort or thinking settings
Works with Standard LLMs Models trained to reason

OpenAI's reasoning best practices guide says that prompting its reasoning models to think step by step or explain their reasoning is unnecessary. Manual chain-of-thought prompting remains a fallback for standard models and for requests where thinking is turned off. For how reasoning models generate their chain of thought, see reasoning model and the reasoning models explainer.

Cite this definition

The AI Engineering Editors. "What is chain-of-thought prompting in LLMs?" The AI Engineering, updated Nov 06, 2026. https://theaiengineering.com/glossary/chain-of-thought

Title
What is chain-of-thought prompting in LLMs?
Author
The AI Engineering Editors
Publisher
The AI Engineering
Updated
View as Markdown

Part of the Prompt and context engineering hub · Updated