Test-Time Compute

Definition

Test-time compute is the compute a model spends answering your question, as opposed to the compute spent training it — and it is the reason a reasoning model can cost seven times more per answer than the same model replying directly, at the same price per token. It is called the second scaling axis because it buys accuracy from a budget that scaling laws never touched: training compute is paid once before launch, test-time compute is paid on every query, forever.

The mechanism is unglamorous. Before writing its visible answer, the model writes a long internal monologue — hundreds or thousands of hidden tokens of working — and you are billed for all of them as output. Anthropic's documentation says so plainly: "You're charged for the full thinking tokens generated by the original request, not the summary tokens," and "the billed output token count will not match the visible token count in the response."

That single billing fact is most of the economics of the current AI boom.

How It Works

There are three things a system can do with more compute at answer time, and they are not equally real.

Think longer. Generate a much longer chain of reasoning before committing to an answer. This is what shipped reasoning models do, and the capability is trained in with reinforcement learning against automatically checkable rewards — a maths answer is right or wrong, a program passes the tests or fails them. Nobody hand-writes the reasoning; the model discovers that longer working scores better, and its responses get longer over training. Chain-of-thought as a prompting trick is the ancestor of this; the difference is that here the behaviour is trained, not requested.

Sample many answers and pick one. Draw N independent attempts and select among them — by majority vote, or by an oracle that knows the right answer. This is real, but it mostly lives in evaluation columns and premium tiers, not in the default product. When you see cons@64 or pass@64 in a launch chart, you are looking at 64 answers being paid for and one being reported.

Search with a verifier. Score partial reasoning steps with a learned reward model and steer the search toward promising branches. This is the most academically interesting option and the least shipped. DeepSeek's R1 paper has a section titled Unsuccessful Attempts which contains exactly two things: process reward models and Monte Carlo tree search. They found step-level rewards hard to define, prone to reward hacking, and the token-level search space "exponentially larger" than chess. The shipped model is a single long chain, not a tree.

So: be sceptical when a diagram shows an elegant tree search. The production reality of test-time compute in 2026 is closer to the model talks to itself for a while, then answers.

Worked example: what the thinking costs you

Take real numbers. Artificial Analysis, measuring at the time of publication, reports that GLM-5.2 spends about 43,000 output tokens on one task of its Intelligence Index, of which 37,000 is reasoning — 86% of the tokens the model produces are never shown to anyone.

Price that at Claude Opus 4.8's published API rate of $25 per million output tokens:

With reasoning: 43,000 × $25 / 1,000,000 = $1.075 for one answer. Answer only (the 6,000 visible tokens): 6,000 × $25 / 1,000,000 = $0.15.

Seven times the price for the same visible answer. The general form is simple and does not rot when prices change: if a model emits T thinking tokens and A answer tokens, the cost multiple over answering directly is (T + A) / A. At six thinking tokens per answer token, you pay 7×. At twenty, you pay 21×. Input tokens barely move it — input is typically five times cheaper than output, and the prompt did not get longer.

Prices fall every year. The ratio does not, because it is a property of how the model reasons.

The cost curve bends, and it bends hard

More thinking is not linearly more accuracy. The cleanest published measurement is OpenAI's o3 on the ARC-AGI semi-private set, where the same model was run at two compute settings:

ARC Prize reports the second run used roughly 172× the compute of the first — which the sample counts confirm (1,024 / 6 ≈ 171). Now do the arithmetic the chart does not do for you:

• the error rate went from 24.3% → 12.5%: 172× the compute removed 49% of the error that remained;
• the cost went from $26 to $4,560, a 175× increase, for 11.8 extra points;
• that is about $384 per additional percentage point, against $0.34 per point for the first 75.

This is the same shape as the training curve on scaling laws — each large multiple of compute clears a fraction of the error that is left — except that here you pay it per question, not once.

Real-World Applications

The chip-demand argument, which is why anyone outside a lab cares. In December 2024, Ilya Sutskever told NeurIPS that "pre-training as we know it will end" — compute keeps growing, but data does not, because "we have but one internet." The market read that as: if pre-training saturates, the demand for accelerators falls. The read was wrong in a specific way. The compute did not disappear; it moved. Three months later, NVIDIA's Jensen Huang told GTC 2025 that reasoning and agentic AI meant "easily 100 times more" compute than the company had assumed a year earlier — because a reasoning model emits a hundred hidden tokens where a chat model emitted one.

The structural point is the one to keep: training compute is a capital expense that ends; inference compute is a variable cost that scales with usage. A pre-training run finishes. Test-time compute does not — every user, every query, every day. That is a durable demand curve, and it is attached to serving hardware rather than to training clusters.

The effort dial in production APIs. This is no longer a research knob; it is a request parameter. Anthropic's API exposes effort at low / medium / high / xhigh / max, which guides how much the model thinks, and returns usage.output_tokens_details.thinking_tokens so you can see exactly how many billed tokens went into reasoning you never saw. Choosing that level is choosing how much test-time compute to buy for each request — a decision that used to be made by a lab and is now made by whoever writes the API call.

Verifiable work. Competitive maths, code that must pass a test suite, formal proofs, ARC-style puzzles. The common thread is a cheap ground-truth check, which is both what makes the RL training possible and what makes extra sampling pay: you can tell a good attempt from a bad one without a human.

Key Concepts

• Reasoning tokens are output tokens. Not a separate line item, not discounted, and not visible. They count against max_tokens too — set it too low and the model spends its whole budget thinking and gets truncated mid-answer.
• pass@1 is the product; pass@64 is the poster. A score obtained by sampling 64 times and letting an oracle pick the winner is not a number you will ever see in production. See state of the art.
• Training compute is once; test-time compute is forever. The same accuracy gain bought at training time is cheaper for the vendor over a model's life if enough people use it — which is exactly why labs still overtrain small models rather than shipping compute-optimal ones.
• The thinking is not the reasoning. What providers return is a summary of the chain of thought, or nothing at all. You are billed for the raw reasoning you cannot read.

Challenges

It is not free accuracy, and the failure is not graceful. Anthropic's July 2025 work on inverse scaling in test-time compute constructed tasks where extending the reasoning made models worse: Claude models became increasingly distracted by irrelevant detail in simple counting problems, and OpenAI's o-series overfitted to familiar problem framings. Most strikingly, on an alignment evaluation, Claude Sonnet 4's expressed willingness to be shut down fell from 60% to 47% as reasoning length grew. Longer deliberation is not a monotone improvement operator. On tasks with no verifiable signal — a judgement call, a piece of prose, a strategy memo — there is little evidence that thinking longer converges on anything, and some that it talks itself out of the right answer.

Latency is the tax users actually notice. A model that thinks for thirty seconds before its first visible token is a different product from one that streams immediately, even when both are correct. Test-time compute trades a resource users tolerate invisibly — money — for one they do not: time, in front of them.

This is what breaks if you get it wrong. A team benchmarks a reasoning model, sees it beat the non-reasoning one, and ships it on a workload that is high-volume and unverifiable — customer support triage, summarisation, classification. The bill arrives at roughly 7× the estimate, because the estimate counted the answer and the invoice counted the thinking. Latency doubles. Accuracy does not move, because there was never a verifier to make the extra tokens pay. All three of those failures come from the same mistake: treating test-time compute as a quality setting rather than as a purchase. The question is never "should the model think harder" but "is this task one where thinking is worth $X and Y seconds".

And you cannot cache your way out. Serving optimisations — batching, quantisation, speculative decoding, inference optimization — attack the constant factor, making each token cheaper. Reasoning length attacks the token count, and the count is winning. (Why each token is slow in the first place belongs to the memory wall.)

Future Trends

The open research question is whether the compute can be allocated rather than merely spent. Snell et al. showed in 2024 that varying test-time strategy by problem difficulty was over 4× more efficient than a flat best-of-N baseline, and that on problems where a small model already had a non-trivial success rate, test-time compute could beat a 14× larger model given the same FLOPs. That is a claim about substitution: for some tasks, thinking is cheaper than parameters. If it generalises, the frontier stops being a single number.

The harder frontier is verification. Everything test-time compute is good at today sits downstream of a cheap correctness check. Extending it to domains where correctness is contested — where the verifier would have to be another model, with its own errors and its own exploitable seams — decides whether this axis keeps scaling or quietly stops at the edge of what a unit test can confirm.