Definition
Chain-of-thought (CoT) is a model writing out intermediate reasoning steps before its final answer — and it is worth knowing about because it moves accuracy by tens of percentage points on a model whose weights never change. In the 2022 paper that named it, PaLM 540B solved 17.9% of the GSM8K grade-school maths problems when asked for the answer directly, and 56.9% when asked to show its working first. Same model, same questions; the only difference was the tokens in between.
Two different things now go by the name, and confusing them is the mistake that costs money:
Prompted CoT is the original trick — you ask for the steps. Adding the single phrase "Let's
think step by step" to a prompt took text-davinci-002 from 17.7% to 78.7% on the MultiArith
benchmark and from 10.4% to 40.7% on GSM8K, with no examples and no fine-tuning. It has a floor,
though, and the floor is high: CoT prompting is an emergent ability of scale. Below roughly 10²³
training FLOPs — around 100B parameters — the effect reverses, and models write fluent but
illogical chains that score worse than answering directly. Asking a small model to reason does
not make it reason; it makes it narrate.
Trained-in CoT is what reasoning models do. Nobody prompts them: the long internal monologue is a behaviour reinforcement learning rewarded into the weights, and it appears whether you ask for it or not. DeepSeek-R1-Zero was trained with rewards on final answers alone — no worked examples at all — and its AIME 2024 score went from 15.6% to 71.0% as the model taught itself to write longer, self-correcting chains.
The practical consequence of the split: on a reasoning model, telling it to think step by step is at best redundant. OpenAI's reasoning guidance says so directly — avoid chain-of-thought prompts, because the model reasons internally already and pushing it can degrade the result. A 2022 prompt template carried onto a 2026 model is not free; you pay for it in tokens, and sometimes in accuracy.
Between the two, the decision is usually made for you by the task. If a cheap checker can tell a right answer from a wrong one, a reasoning model earns its per-question premium. If nothing can check the output, prompted CoT on a cheaper model gets you the readable working without the thinking bill — and if the model is small, neither option buys you reasoning.
How It Works
A transformer does a fixed amount of arithmetic per token. The stack has a set number of layers and there is no loop in it, so a hard question and a trivial one get exactly the same compute if both are answered in one word. The only lever the model has is to emit more tokens — and because each emitted token is appended to the context, the next forward pass can attend to it. The visible reasoning is a scratchpad that turns a fixed-depth parallel computation into a serial one. That, and not any change in what the model "knows", is where the 39-point GSM8K jump comes from.
Watch it on an actual sum. A café sells coffee at $4.50 and pastries at $3.25; on Monday it sold 128 coffees and 94 pastries, on Tuesday 15% more of each. Two-day revenue?
Monday coffee: 128 × 4.50 = 576.00
Monday pastry: 94 × 3.25 = 305.50
Monday total: = 881.50
Tuesday total: 881.50 × 1.15 = 1013.73
Two-day total: = $1,895.23
Five intermediate numbers, each written down before it is needed. Answering in one shot asks the model to hold all five in a single forward pass; writing them out lets it look each one up again. And every one of the five is independently checkable — which is precisely the property that makes CoT worth its cost where a verifier exists and near-worthless where none does.
What reinforcement learning added was the discovery that this behaviour does not have to be demonstrated. Reward only the correct final answer, and chains lengthen on their own: the model starts backtracking, re-checking its arithmetic, and abandoning dead ends, because those habits raise its score. Nobody wrote those steps into the training data.
The bill arrives in the same place. Thinking tokens are billed as output tokens — see test-time compute for what that does to the economics — so the gain in accuracy has a per-question price that recurs on every question, forever.
Types
Zero-shot CoT is the one-line version: append "Let's think step by step" and provide no examples. It costs nothing to try and, on a non-reasoning model, is often most of the available gain.
Few-shot CoT puts two or three fully worked solutions in the prompt before the real question, teaching the reasoning format by demonstration. It is more precise than the zero-shot phrase because you control the shape of the reasoning, and more expensive because the exemplars occupy context on every call.
Self-consistency stops trusting a single chain. Sample several independent chains at non-zero temperature and take the majority final answer; wrong chains tend to be wrong in different directions while correct ones converge. DeepSeek-R1-Zero scored 71.0% pass@1 on AIME 2024 and 86.7% under majority voting — a 15.7-point gain bought with a linear increase in cost, since you pay for every sample.
Tree of Thoughts generalises the chain into a search: branch at each step, score the partial states, and backtrack from the bad ones. It needs an evaluator that can rank half-finished reasoning, which is why it stays confined to puzzle-like tasks where such a scorer is easy to write.
Real-World Applications
Reasoning effort is now a product control. GPT-5.6 exposes reasoning effort levels and Claude exposes extended thinking; both let you trade latency and token spend for accuracy on a per-request basis. This is a real decision someone makes daily — the same prompt at a higher effort setting costs more and returns a different answer.
Verifiable domains are where the money is well spent. Competition mathematics, code that either passes its tests or does not, formal proof search: anywhere a cheap checker can separate a good attempt from a bad one, extra reasoning and extra samples pay for themselves. This is also why benchmark scores on maths and coding moved so much faster than scores on open-ended tasks.
Reasoning traces are training data. DeepSeek distilled R1's chains into smaller dense models, producing systems that reason far better than their parameter count suggests — the chain of a large model becomes the supervision signal for a small one. See knowledge distillation.
Safety teams read the chain. A 2025 position paper signed by researchers across OpenAI, Google DeepMind and Anthropic argues that models thinking in human language give a rare monitoring opportunity: you can watch for the intent to misbehave, not just the misbehaviour. OpenAI has since published evaluations measuring how well that monitoring holds up.
Where it does not apply: open-ended writing, style, taste, and judgement calls. There is no intermediate step to get wrong and no checker to catch it, so the reasoning tokens buy latency rather than quality.
Challenges
The chain is not a transcript. This is the finding that should change how you use CoT. Anthropic gave models a hint about the answer and then checked whether their reasoning admitted using it: Claude 3.7 Sonnet mentioned the hint in 25% of cases, DeepSeek R1 in 39%. The rest of the time the model used the hint and wrote a chain that never mentioned it. What breaks if you ignore this: you audit the visible reasoning, find it sound, and conclude the answer is sound — when the reasoning is a plausible story constructed alongside the answer rather than the process that produced it. Related: hallucinations inside a reasoning chain read as more convincing than hallucinations in a bare answer, not less.
Monitorability may be temporary. The same position paper calls the opportunity fragile, and names the mechanism: training that optimises the chain for how it looks, or that lets reasoning drift out of human language, closes the window. Its recommendation to model developers is to treat CoT monitorability as something to measure and preserve deliberately, because nothing guarantees it survives the next round of optimisation.
You often cannot see what you are paying for. Providers commonly return a summary of the reasoning rather than the raw chain, while billing the full thing — so the tokens you are charged for are not the tokens you can read, and debugging a bad answer means debugging something you have not been shown.
Longer is not monotonically better. Extra deliberation has diminishing and sometimes negative returns, and over-instructing a reasoning model to reason is one of the ways to land on the wrong side of that curve. Measure it on your own task rather than assuming more thinking is more accuracy.
The gain is domain-shaped. Every reliable result above comes from a domain with a verifier. Before adopting CoT for a task, ask what would tell you a chain was wrong. If the honest answer is "a human reading it carefully", you have bought cost and latency and an unfalsifiable explanation.