Definition
There is no single state-of-the-art model, and the number in the launch chart is smaller than it looks. "SOTA" means one thing only: on one benchmark, on one day, under one evaluation setup, a model posted the best published score. It is a claim about a test set, not a verdict about a model — and the margin that earns the label is often a handful of questions.
GPQA Diamond, the graduate-level science benchmark that appears in most frontier-model launch charts, contains 198 questions. One question is therefore worth 0.51 percentage points. A model that beats another 88% to 85% — a three-point lead, the kind that gets a headline — has answered six more questions correctly. Whether it wins at all depends on which six questions happened to make it into the set.
How It Works
A SOTA claim is produced by running a model against a fixed set of questions and comparing the score with the best previously published one. Nothing about that process is dishonest. It is simply much weaker evidence than the presentation implies, and it bends in four specific ways.
The margin is usually inside the noise
A benchmark score is a proportion measured on a sample, so it carries sampling error like any other poll. Read the two runs as independent samples — the conservative reading, and the only one the published data lets you compute — and for two models scoring around 85–88% on GPQA Diamond's 198 questions, the standard error of the difference between them is about 3.4 points, which puts the 95% confidence interval at roughly ±6.7 percentage points. That is more than twice the three-point gap.
The real comparison is tighter than that, because both models answered the same 198 questions. That makes it paired data, and the appropriate test is McNemar's, which throws away every question the two models got the same and looks only at the ones they disagree on. Pairing does help — but not enough to rescue the headline. With a net lead of six questions, the gap clears significance only if the two models disagree on roughly ten questions or fewer in total, meaning they agree on 95% of the set. And you cannot check whether they do: labs publish the percentage, never the per-question results, so the data the sharper test needs is not in the chart.
Sampling error is not the only noise, either. At any temperature above zero the same model does not answer the same question the same way twice, so a published score is one draw from a distribution rather than a property of the model. Launch charts almost never report the spread across reruns, and a benchmark item the model gets right half the time contributes a coin flip to the total.
The honest reading of "88% vs 85%" is we cannot distinguish these two models, and the same arithmetic applies to every small benchmark: noise scales with 1/√n, so a 200-question test is inherently about three times noisier than a 2,000-question one. To resolve a three-point difference at all — an interval that merely excludes zero — takes roughly 1,000 questions, and about twice that for a comparison anyone would call adequately powered. A leaderboard sorted to two decimal places is sorting mostly on luck.
The test set may be in the training data
Benchmarks are published on the web, and training corpora are scraped from the web. When the questions and answers leak into pretraining, the model recalls rather than reasons, and the score stops estimating the only thing a held-out test set was ever there to estimate — generalization to questions the model has not seen. This is called contamination, and it is overfitting at the scale of a whole field: a test everyone has been optimising against for two years is no longer a test.
Catching it is harder than it sounds, and there are three usual attempts. Benchmark authors embed a canary string, a unique identifier that a careful lab filters out of its corpus — which demonstrates good faith rather than absence, since nothing forces anyone to look for it. Analysts measure n-gram overlap between test items and training data, which fails precisely where it matters, because frontier training corpora are not published. The strongest evidence is a rebuild: write a fresh test at matched difficulty and see whether the scores hold. Scale AI did this to GSM8K in 2024, regenerating the same grade-school arithmetic task as GSM1k; several model families dropped by close to 13 points on the new set while others were unchanged — which is what contamination looks like on the rare occasion you can see it at all. That gap is also why new, deliberately held-out benchmarks keep appearing: the old ones stop discriminating.
The evaluation setup does the work
The same model produces very different numbers depending on how it was run. pass@1 — one attempt,
scored — is what you get in production. pass@64 — sixty-four attempts with an oracle picking the
correct one — is a number you will never see again outside the chart it appeared in.
Reasoning effort, temperature, prompt format, and whether tools were
available all move scores by more than the margins being claimed. Two labs can report on "the same
benchmark" and not be measuring the same thing.
Aggregate scores hide the task you care about
A model that leads on an aggregate index can be behind on your workload, because the index is an average over tasks that are not your task — and averaging shrinks whatever it averages. An index built from ten benchmarks turns a decisive five-point win on one of them into half a point on the headline, so the differences you are shopping for are exactly the ones the aggregate smooths away. That is why the top of an index is usually a cluster separated by noise while the same models on a single skill are separated by a mile: the leader at code generation is frequently not the leader at long-context retrieval, at instruction-following, or at cost per token.
Real-World Applications
Choosing a model. This is where the concept actually costs money. A team picks the model at the top of a leaderboard, ships it, and finds it worse at their own task and several times the price of the runner-up. The useful question is never "what is SOTA" but "what is best at my task at a price I will pay" — and our model catalog exists for that comparison, while a live leaderboard answers the ranking question better than any written page can.
Five things decide whether a chart in front of you means anything:
- How many questions? Divide 100 by that number. That is what a single question is worth, and on a benchmark of a few hundred items it is usually a large fraction of the lead being claimed.
pass@1orpass@k, and under what settings? Reasoning effort, temperature and tool access move a score further than the margin does, and onlypass@1resembles what you will run.- Who ran it? A lab reporting its own model on its own harness against baselines it chose is an interested party, not an auditor.
- Which version of the index, and dated when? An aggregate score quoted without its version is not comparable to the other score you are holding it against.
- What does it cost per token, and how slow is it? Neither is on the chart, and between two models you cannot statistically distinguish, they are the entire decision.
Or skip the chart and measure it yourself. Pull thirty to fifty real examples off your own traffic, run the two or three candidates through them, and score the outputs. The arithmetic that deflates the launch chart applies to your test too: thirty examples resolve a gap of about twenty points and nothing finer, and it takes around 120 to see ten. That sounds like a weak instrument until you notice it is pointed at the right resolution — a model twenty points better on your work is worth a migration, and one you cannot separate from the incumbent on your own data was never going to repay the switching cost, whatever it scored on GPQA.
The publication bar in research. "Beats SOTA" is, in practice, a condition of acceptance at major ML venues. That single convention shapes what gets researched: it rewards small gains on established benchmarks over work that is harder to score, and it is a large part of why benchmark chasing and contamination are so persistent. The label is not just a description — it is an incentive.
Regulation and procurement. Rules aimed at "frontier" models need a definition of frontier, and in practice they reach for either compute thresholds or capability benchmarks. The EU AI Act takes the first: a general-purpose model trained with more than 10²⁵ FLOP is presumed to carry systemic risk, with the obligations that follow. Compute is the cleaner threshold of the two, because it cannot be optimised for without actually being spent. When a benchmark score becomes the legal or purchasing threshold instead, the pressure to optimise for the benchmark rather than the capability gets much stronger.
Key Concepts
- Benchmark saturation: a test stops working well before anyone gets 100% on it, because a benchmark's own labelling errors set a ceiling below the real one — the 2024 audit Are We Done with MMLU? found errors in about 6.5% of that benchmark's questions, so a score in the mid-90s there is partly a measure of how well a model reproduces the errata. Once the leaders cluster in that band, what separates them is mistakes and noise, and a harder benchmark has to be built.
- Index versions are not comparable: an aggregate score such as the Artificial Analysis Intelligence Index is revised, and the same model can move by ten points or more between index versions with no change to the model at all. A score quoted without its version number is not a fact. (This site treats a bare score as an error for exactly that reason.)
- Elo and human preference are a different measurement: arena-style rankings ask humans which answer they prefer, which captures style and helpfulness that benchmarks miss — and rewards agreeable, well-formatted answers that are not necessarily more correct. They are honest about their own noise, publishing a confidence interval next to every rating, and those intervals routinely overlap for a dozen models at a time. The rank ordering is far more precise than the measurement underneath it.
Challenges
The label rots faster than the page describing it. Any content that names the current SOTA model is wrong within months, which is why this page does not name one. Treat a written ranking as a historical note and go to a live source.
Being SOTA and being useful have come apart. A model can hold the top score on a reasoning benchmark and still be the wrong choice for a production system that needs low latency, a large context window, predictable cost, or a permissive licence. None of those appear in the score that earns the label.
Self-reported evaluations are the norm. Most frontier scores are published by the lab that trained the model, run on its own harness, against baselines it selected. Independent replication is the exception, not the rule — so the comparison you are reading was constructed by an interested party.
A benchmark measures what is easy to measure. Hallucination rates, long-horizon reliability, and whether a model knows it is wrong are the properties that decide whether a system survives contact with users, and they are the hardest to reduce to a number. The leaderboard is bright, and the important questions are outside the light.
Future Trends
Evaluation is moving away from single-turn question answering, because that format has largely been exhausted: the frontier now sits on long-horizon agentic tasks — multi-step work with tools, where success is a completed task rather than a matched string. Those evaluations are far more expensive to run and far harder to game, but they are also noisier and less reproducible, which means the tidy single-number ranking is likely to get less reliable, not more.
The second shift is toward contamination-resistant design: answer keys that are never published, questions drawn from sources newer than any current model's training cutoff, and test sets that are rotated on a schedule so that optimising against last month's copy buys nothing. Each of these buys integrity at the cost of transparency — you can trust the number more, and verify it less. The end point of that trade is an evaluation you cannot reproduce, run by people you have to take on trust, which is a different problem from the one it solves rather than an absence of one.