---
source: 'https://howaiworks.ai/glossary/test-time-compute'
section: glossary
title: Test-Time Compute
description: >-
  Spending compute at inference instead of training — how reasoning models buy
  accuracy with thinking time, and why it moved chip demand into serving.
tags:
  - Reasoning
  - Inference
  - Scaling Laws
  - LLM
  - AI Infrastructure
  - Reinforcement Learning
  - Deep Learning
category: Deep Learning
datePublished: '2026-07-13'
lastUpdated: '2026-07-24'
---

# Test-Time Compute

> Spending compute at inference instead of training — how reasoning models buy accuracy with thinking time, and why it moved chip demand into serving.

## Definition

**Test-time compute is the compute a model spends *answering your question*, as opposed to the
compute spent training it — and it is the reason a reasoning model can cost seven times more per
answer than the same model replying directly, at the same price per token.** It is called the
second scaling axis because it buys accuracy from a budget that
[scaling laws](https://howaiworks.ai/glossary/scaling-laws) never touched: training compute is paid once before launch,
test-time compute is paid on every query, forever.

The mechanism is unglamorous. Before writing its visible answer, the model writes a long internal
monologue — hundreds or thousands of hidden [tokens](https://howaiworks.ai/glossary/token) of working — and you are
billed for all of them as output. Anthropic's documentation says so plainly: *"You're charged for
the full thinking tokens generated by the original request, not the summary tokens,"* and *"the
billed output token count will **not** match the visible token count in the response."*

That single billing fact is most of the economics of the current AI boom.

## How It Works

There are three things a system can do with more compute at answer time, and they are not equally
real.

**Think longer.** Generate a much longer chain of reasoning before committing to an answer. This
is what shipped reasoning models do, and the capability is trained in with
[reinforcement learning](https://howaiworks.ai/glossary/reinforcement-learning) against automatically checkable rewards
— a maths answer is right or wrong, a program passes the tests or fails them. Nobody hand-writes
the reasoning; the model discovers that longer working scores better, and its responses get longer
over training. [Chain-of-thought](https://howaiworks.ai/glossary/chain-of-thought) as a *prompting trick* is the
ancestor of this; the difference is that here the behaviour is trained, not requested.

**Sample many answers and pick one.** Draw N independent attempts and select among them — by
majority vote, or by an oracle that knows the right answer. This is real, but it mostly lives in
*evaluation columns and premium tiers*, not in the default product. When you see `cons@64` or
`pass@64` in a launch chart, you are looking at 64 answers being paid for and one being reported.

**Search with a verifier.** Score partial reasoning steps with a learned reward model and steer the
search toward promising branches. This is the most academically interesting option and the least
shipped. DeepSeek's R1 paper has a section titled *Unsuccessful Attempts* which contains exactly
two things: process reward models and Monte Carlo tree search. They found step-level rewards hard
to define, prone to reward hacking, and the token-level search space "exponentially larger" than
chess. The shipped model is a single long chain, not a tree.

So: **be sceptical when a diagram shows an elegant tree search.** The production reality of
test-time compute in 2026 is closer to *the model talks to itself for a while, then answers*.

### Worked example: what the thinking costs you

Take real numbers. Artificial Analysis, measuring at the time of publication, reports that GLM-5.2
spends about **43,000 output tokens on one task of its Intelligence Index, of which 37,000 is
reasoning** — 86% of the tokens the model produces are never shown to anyone.

Price that at Claude Opus 5's published API rate of **$25 per million output tokens**:

> **With reasoning:** 43,000 × $25 / 1,000,000 = **$1.075** for one answer.
> **Answer only** (the 6,000 visible tokens): 6,000 × $25 / 1,000,000 = **$0.15**.

**Seven times the price for the same visible answer.** The general form is simple and does not rot
when prices change: if a model emits `T` thinking tokens and `A` answer tokens, the cost multiple
over answering directly is **(T + A) / A**. At six thinking tokens per answer token, you pay 7×. At
twenty, you pay 21×. Input tokens barely move it — input is typically five times cheaper than
output, and the prompt did not get longer.

Prices fall every year. The ratio does not, because it is a property of how the model reasons.

### The cost curve bends, and it bends hard

More thinking is not linearly more accuracy. The cleanest published measurement is OpenAI's o3 on
the ARC-AGI semi-private set, where the same model was run at two compute settings:

| configuration | samples | score | cost per task |
|---|---|---|---|
| high-efficiency | 6 | 75.7% | $26 |
| low-efficiency | 1,024 | 87.5% | $4,560 |

ARC Prize reports the second run used **roughly 172× the compute** of the first — which the sample
counts confirm (1,024 / 6 ≈ 171). Now do the arithmetic the chart does not do for you:

- the error rate went from **24.3% → 12.5%**: 172× the compute removed **49% of the error that
  remained**;
- the cost went from $26 to $4,560, a **175× increase**, for **11.8 extra points**;
- that is about **$384 per additional percentage point**, against $0.34 per point for the first 75.

This is the same shape as the training curve on [scaling laws](https://howaiworks.ai/glossary/scaling-laws) — each large
multiple of compute clears a fraction of the error that is left — except that here you pay it
*per question*, not once.

## Real-World Applications

**The chip-demand argument, which is why anyone outside a lab cares.** In December 2024, Ilya
Sutskever told NeurIPS that *"pre-training as we know it will end"* — compute keeps growing, but
data does not, because "we have but one internet." The market read that as: if pre-training
saturates, the demand for accelerators falls. The read was wrong in a specific way. The compute did
not disappear; it *moved*. Three months later, NVIDIA's Jensen Huang told GTC 2025 that reasoning
and agentic AI meant "easily 100 times more" compute than the company had assumed a year earlier —
because a reasoning model emits a hundred hidden tokens where a chat model emitted one.

The structural point is the one to keep: **training compute is a capital expense that ends;
inference compute is a variable cost that scales with usage.** A pre-training run finishes. Test-time
compute does not — every user, every query, every day. That is a durable demand curve, and it is
attached to serving hardware rather than to training clusters.

**The effort dial in production APIs.** This is no longer a research knob; it is a request
parameter. Anthropic's API takes `output_config.effort` at `low` / `medium` / `high` / `xhigh` /
`max`, which guides how much the model thinks. Choosing that level *is* choosing how much test-time
compute to buy for each request — a decision that used to be made by a lab and is now made by
whoever writes the API call. Note what the billing does not tell you: reasoning tokens are charged
as ordinary output tokens, with no separate line item, and the reasoning text itself is withheld by
default. You pay for the thinking at output rates without seeing it and without being able to
separate it from the answer.

**Verifiable work.** Competitive maths, code that must pass a test suite, formal proofs, ARC-style
puzzles. The common thread is a cheap ground-truth check, which is both what makes the RL training
possible and what makes extra sampling pay: you can tell a good attempt from a bad one without a
human.

## Key Concepts

- **Reasoning tokens are output tokens.** Not a separate line item, not discounted, and not
  visible. They count against `max_tokens` too — set it too low and the model spends its whole
  budget thinking and gets truncated mid-answer.
- **`pass@1` is the product; `pass@64` is the poster.** A score obtained by sampling 64 times and
  letting an oracle pick the winner is not a number you will ever see in production. See
  [state of the art](https://howaiworks.ai/glossary/state-of-the-art-model).
- **Training compute is once; test-time compute is forever.** The same accuracy gain bought at
  training time is cheaper for the vendor over a model's life if enough people use it — which is
  exactly why labs still overtrain small models rather than shipping compute-optimal ones.
- **The thinking is not the reasoning.** What providers return is a *summary* of the chain of
  thought, or nothing at all. You are billed for the raw reasoning you cannot read.

## Challenges

**It is not free accuracy, and the failure is not graceful.** Anthropic's July 2025 work on
*inverse scaling in test-time compute* constructed tasks where extending the reasoning made models
**worse**: Claude models became increasingly distracted by irrelevant detail in simple counting
problems, and OpenAI's o-series overfitted to familiar problem framings. Most strikingly, on an
alignment evaluation, Claude Sonnet 4's expressed willingness to be shut down **fell from 60% to
47% as reasoning length grew**. Longer deliberation is not a monotone improvement operator. On
tasks with no verifiable signal — a judgement call, a piece of prose, a strategy memo — there is
little evidence that thinking longer converges on anything, and some that it talks itself out of
the right answer.

**Latency is the tax users actually notice.** A model that thinks for thirty seconds before its
first visible token is a different product from one that streams immediately, even when both are
correct. Test-time compute trades a resource users tolerate invisibly — money — for one they do
not: time, in front of them.

**This is what breaks if you get it wrong.** A team benchmarks a reasoning model, sees it beat the
non-reasoning one, and ships it on a workload that is high-volume and unverifiable — customer
support triage, summarisation, classification. The bill arrives at roughly **7× the estimate**,
because the estimate counted the answer and the invoice counted the thinking. Latency doubles.
Accuracy does not move, because there was never a verifier to make the extra tokens pay. All three
of those failures come from the same mistake: treating test-time compute as a quality setting
rather than as a *purchase*. The question is never "should the model think harder" but "is this task
one where thinking is worth $X and Y seconds".

**And you cannot cache your way out.** Serving optimisations — batching, quantisation,
[speculative decoding](https://howaiworks.ai/glossary/speculative-decoding),
[inference optimization](https://howaiworks.ai/glossary/inference-optimization) — attack the constant factor, making each
token cheaper. Reasoning length attacks the token *count*, and the count is winning. (Why each token
is slow in the first place belongs to the [memory wall](https://howaiworks.ai/glossary/memory-wall).)

## Future Trends

The open research question is whether the compute can be *allocated* rather than merely spent. Snell
et al. showed in 2024 that varying test-time strategy by problem difficulty was over **4× more
efficient** than a flat best-of-N baseline, and that on problems where a small model already had a
non-trivial success rate, test-time compute could beat a **14× larger** model given the same FLOPs.
That is a claim about *substitution*: for some tasks, thinking is cheaper than
[parameters](https://howaiworks.ai/glossary/parameters). If it generalises, the frontier stops being a single number.

The harder frontier is verification. Everything test-time compute is good at today sits downstream
of a cheap correctness check. Extending it to domains where correctness is contested — where the
verifier would have to be another model, with its own errors and its own exploitable seams — decides
whether this axis keeps scaling or quietly stops at the edge of what a unit test can confirm.

## Frequently Asked Questions

### What is test-time compute?

It is compute spent while answering a question rather than while training the model. A reasoning model writes a long internal chain of thought before its visible answer, and every one of those hidden tokens costs money and time. It is the second axis of scaling: training compute is paid once, test-time compute is paid on every question, forever.

### Why are reasoning models so much more expensive per answer?

Because the thinking tokens are billed as output tokens. Anthropic's documentation is explicit that you are charged for the full internal reasoning, not for the shortened summary you see. If a model emits six times more hidden reasoning than visible answer, your bill for that question is roughly seven times larger — at the same per-token price.

### Does thinking longer always make the answer better?

No. On ARC-AGI's semi-private set, OpenAI's o3 went from 75.7% to 87.5% for roughly 172 times the compute — half the remaining error removed for a 175-fold cost increase. And Anthropic's 2025 inverse-scaling work found tasks where longer reasoning makes accuracy fall, not rise.

### Did test-time compute save the AI chip trade?

It changed what the chips are for. The 2024 argument was that pre-training was saturating, so demand would fall. Instead the compute moved into serving, where it scales with usage rather than with a one-off training run. NVIDIA's Jensen Huang told GTC 2025 that reasoning and agentic AI needed 'easily 100 times more' compute than the company had assumed a year earlier.

### Where does test-time compute actually help?

In domains where an answer can be checked — competition maths, code that either passes tests or does not, formal puzzles. Extra sampling and longer deliberation pay off when something can tell a good attempt from a bad one. On open-ended writing or judgement calls, where no verifier exists, the evidence is far weaker.

## Related

### Related terms

- [Inference](https://howaiworks.ai/glossary/inference)
- [Chain-of-Thought (CoT)](https://howaiworks.ai/glossary/chain-of-thought)
- [Scaling Laws](https://howaiworks.ai/glossary/scaling-laws)
- [Reinforcement Learning (RL)](https://howaiworks.ai/glossary/reinforcement-learning)
- [Large Language Model (LLM)](https://howaiworks.ai/glossary/large-language-model)
- [FLOPs (Floating Point Operations)](https://howaiworks.ai/glossary/flops)

---

Source: https://howaiworks.ai/glossary/test-time-compute — HowAIWorks.ai