Definition
A benchmark is a fixed set of test items plus a fixed rule for scoring the answers, so two models can be compared on identical work. The word doing the work is fixed: a benchmark is worth something only while everybody runs the same questions the same way, and nearly every problem with benchmarks is that guarantee quietly breaking.
A benchmark score is also a proxy. Nobody wants a model that can answer 198 hand-written graduate questions in biology, physics and chemistry — GPQA Diamond's entire contents — they want one that can reason about science. The benchmark is a sample standing in for a capability, and everything that goes wrong lives in the gap between the two. One number to take away: with 198 items, a single question is worth 0.5 percentage points, so a three-point lead in a launch chart is six questions.
How It Works
A benchmark run has four moving parts, and only the first is the dataset. There is the item set; the harness, which turns each item into a prompt — system prompt, number of worked examples shown first, whether the model may use tools or write a scratchpad; the decoding settings, chiefly temperature; and the scoring function — exact match, log-probability ranking over the options, execution against unit tests, or a model acting as judge. Change any of the four and the number changes, which is why a score is meaningful only alongside its harness.
Work a concrete one on the kind of suite the frontier is now scored on. SWE-bench Verified is 500 real GitHub issues; the model is dropped into the repository, edits the code to resolve the issue, and the harness runs the repository's own hidden test suite — the headline figure is the fraction of issues where every test passes afterwards. Solve 340 and you report 68.0%. Solve one more and you report 68.2%. That is the resolution of the instrument: on a 500-item set a single issue is worth 0.2 points, so differences under that are not differences. Nothing about that measurement rots — the tasks change, but "edit a real repo until the hidden tests pass" is what an agentic coding score has meant since the format appeared, and it is a description no leaderboard number can make stale.
The deeper limit is statistical. A score is a proportion estimated from a sample, so its standard error is the square root of p(1-p)/N. On GPQA Diamond, a model near 70% carries a standard error of roughly 3.3 percentage points; on MMLU, whose test split holds roughly 14,000 questions across 57 subjects, a model near 80% carries about 0.34 points. Seventy times the items buys about eight times the precision, because noise falls with the square root of N and nothing else. That is why the harder a benchmark gets, the less it can tell you: hard items are expensive, so hard sets are small, and small sets cannot resolve small gaps.
Types
Benchmarks divide by where their ground truth comes from, and the three families fail differently.
Static academic datasets carry an answer key written in advance: MMLU across 57 subjects, GSM8K with its 1,319 grade-school word problems, HumanEval's 164 coding tasks. They are cheap to rerun and perfectly reproducible, which is also their weakness — a fixed public answer key is the thing that leaks into training data. The frontier replacements for a saturated MMLU keep the same answer-key structure but raise the difficulty: GPQA Diamond's 198 graduate questions in biology, physics and chemistry; Humanity's Last Exam (HLE), closed-ended expert questions across more than a hundred subjects, each with a verifiable answer that a web search will not quickly surface; and competition mathematics such as AIME, where the answer is a single integer that either matches or does not. Harder items, same failure mode — a public key still leaks.
Human-preference arenas have no answer key. LMSYS Chatbot Arena (now LMArena) shows a user two anonymous responses to their own prompt, collects a vote, and fits an Elo-style rating. A 100-point gap corresponds to an expected win rate of 64%, a 10-point gap to 51.4% — which needs roughly 5,000 paired votes to separate from a coin flip at 95% confidence. Two models ten points apart on a rating table are tied.
Execution-based and agentic suites define correctness by running something. SWE-bench draws 2,294 real GitHub issues from 12 Python repositories; the model produces a patch and the score is whether the repository's own tests pass afterwards. A test suite cannot be bluffed, which is the appeal, and this family now spans most of what a frontier model is actually sold to do — each task scored by execution rather than a key. For coding in a repository there is SWE-bench Verified and the contamination-resistant SWE-bench Pro, which draws on repositories held out of training precisely so the answer cannot be memorised, alongside Terminal-Bench, whose tasks are complete command-line workflows checked by a test script inside a container. For computer and GUI use there is OSWorld, where the model drives a real desktop with mouse and keyboard and a script inspects the resulting system state. For web browsing there is BrowseComp, built on questions whose answer is hard to find but easy to verify. Sudoku-Bench, introduced in Sakana AI's announcement, applies the same execution-checked scoring to creative reasoning.
Real-World Applications
The most valuable use of benchmarks is the least discussed: internal regression evaluation, increasingly scored by LLM-as-a-judge rather than by a fixed answer key. If you ship a product on top of an LLM, every prompt edit and model upgrade is a silent risk, and the only way to know you did not break something is a suite of your own cases with known-good outputs. A 200-case suite at roughly 1,500 input and 500 output tokens per case costs 300,000 input and 100,000 output tokens per run — a couple of dollars at frontier API prices at the time of publication, cheap enough to run on every pull request. Hold decoding deterministic so you measure the change and not the sampler, and compare the versions item by item rather than headline against headline: only the cases where they disagree carry information.
Public benchmarks drive spending decisions too. Teams choosing a coding agent read SWE-bench Verified — the 500-instance subset OpenAI had professional developers screen for broken tests and under-specified issue text — rather than the full 2,294, because a task nobody could solve from its description tells you nothing. Hardware buyers use MLPerf, MLCommons' training and inference suite, because it fixes the model, dataset and accuracy target so vendor throughput claims become comparable. Safety teams run refusal and dangerous-capability evaluations whose output is not a leaderboard position but a release decision.
Challenges
Contamination is the failure everyone names first. A public test set ends up in a web crawl, then in a pretraining corpus, and the model recalls the answer rather than deriving it — overfitting to the test instead of learning the skill. Defences are partial: BIG-bench popularised embedding a unique canary string so crawlers can filter the set out, and labs screen with n-gram overlap against training text, but neither catches paraphrase, reasoning traces posted in blog write-ups, or synthetic data from a model that saw the items. Scores inflate exactly where they matter most — the famous benchmarks in launch charts.
Saturation is what happens when the screening works and the models improve anyway. The MMLU authors estimated a 95th-percentile human expert would score about 89.8%; once frontier models sit near that line, MMLU can no longer rank them, and the field answers with smaller, harder sets. That is a real fix for saturation and a fresh dose of the problem above — GPQA Diamond's 198 items cannot resolve a two-point difference however carefully they were written.
Construct validity is the quietest failure and the worst: the score measures something adjacent to what you wanted. Multiple-choice sets reward eliminating distractors, a skill with no counterpart when a user asks an open question, which is why high scores coexist comfortably with hallucinations in production.
Harness variance breaks cross-vendor comparison in a way no care about the dataset can fix. Hugging Face pinned its Open LLM Leaderboard to one implementation of the EleutherAI harness with a fixed five-shot MMLU prompt because independent implementations of "MMLU" disagreed about the same model by more than the models on the board differed from each other. Add bespoke scaffolds — extended thinking, several samples with the best one chosen, retries on a failed patch — and self-reported numbers stop sharing an axis.
Future Trends
The clearest direction is benchmarks that cannot be memorised because they did not exist yet. Date-gated sets such as LiveCodeBench draw problems published after a model's training cutoff, converting contamination from a permanent property of a dataset into a dated one. The related move is the private holdout: ARC Prize scores on a set that is never published, trading convenience for a number that means what it says.
Preference arenas are addressing their own confound: LMArena added style controls once it was clear that longer, heavily formatted answers win votes independently of being better — an admission that raw Elo was partly measuring markdown. And as agentic workflows become the thing measured, cost is reported beside accuracy: a score that took 40 tool calls is not the score it looks like.
Code Example
Resolving power is a two-line calculation worth doing before you trust a comparison chart.
from math import sqrt
def standard_error(accuracy, n_items):
"""Binomial standard error of a benchmark score."""
return sqrt(accuracy * (1 - accuracy) / n_items)
for name, n, acc in [("GPQA Diamond", 198, 0.70),
("HumanEval", 164, 0.80),
("SWE-bench Verified", 500, 0.65),
("MMLU", 14000, 0.80)]:
se = standard_error(acc, n)
# 1.96 * sqrt(2) * SE: the gap two independent scores must show to beat noise.
detectable = 2.77 * se
print(f"{name:20} 1 item = {100 / n:.2f} pp | "
f"SE = {se * 100:.2f} pp | needs a {detectable * 100:.1f} pp gap")
The thresholds land near 1 point for MMLU, 6 for SWE-bench Verified and 9 for GPQA Diamond — so the same three-point lead is decisive on the first, short of the second and invisible on the third. Ask for the item count before believing a claim about a state-of-the-art model.