---
source: 'https://howaiworks.ai/glossary/state-of-the-art-model'
section: glossary
title: State of the Art Model (SOTA)
description: >-
  There is no single SOTA model. How to read a benchmark claim — and why a
  3-point lead on GPQA Diamond is six questions, well inside the noise.
tags:
  - Benchmarks
  - Performance
  - AI Models
  - AI Research
category: Artificial Intelligence
datePublished: '2025-09-19'
lastUpdated: '2026-07-21'
---

# State of the Art Model (SOTA)

> There is no single SOTA model. How to read a benchmark claim — and why a 3-point lead on GPQA Diamond is six questions, well inside the noise.

## Definition

**There is no single state-of-the-art model, and the number in the launch chart is smaller than it
looks.** "SOTA" means one thing only: on one [benchmark](https://howaiworks.ai/glossary/benchmark), on one day, under
one evaluation setup, a model posted the best published score. It is a claim about a test set, not
a verdict about a model — and the margin that earns the label is often a handful of questions.

GPQA Diamond, the graduate-level science benchmark that appears in most frontier-model launch
charts, contains **198 questions**. One question is therefore worth **0.51 percentage points**. A
model that beats another 88% to 85% — a three-point lead, the kind that gets a headline — has
answered **six more questions correctly**. Whether it wins at all depends on which six questions
happened to make it into the set.

## How It Works

A SOTA claim is produced by running a model against a fixed set of questions and comparing the
score with the best previously published one. Nothing about that process is dishonest. It is simply
much weaker evidence than the presentation implies, and it bends in four specific ways.

### The margin is usually inside the noise

A benchmark score is a proportion measured on a sample, so it carries sampling error like any other
poll. Read the two runs as independent samples — the conservative reading, and the only one the
published data lets you compute — and for two models scoring around 85–88% on GPQA Diamond's 198
questions, the standard error of the *difference* between them is about 3.4 points, which puts the
95% confidence interval at roughly **±6.7 percentage points**. That is more than twice the
three-point gap.

The real comparison is tighter than that, because both models answered the *same* 198 questions.
That makes it paired data, and the appropriate test is McNemar's, which throws away every question
the two models got the same and looks only at the ones they disagree on. Pairing does help — but
not enough to rescue the headline. With a net lead of six questions, the gap clears significance
only if the two models disagree on **roughly ten questions or fewer in total**, meaning they agree
on 95% of the set. And you cannot check whether they do: labs publish the percentage, never the
per-question results, so the data the sharper test needs is not in the chart.

Sampling error is not the only noise, either. At any temperature above zero the same model does not
answer the same question the same way twice, so a published score is one draw from a distribution
rather than a property of the model. Launch charts almost never report the spread across reruns,
and a benchmark item the model gets right half the time contributes a coin flip to the total.

The honest reading of "88% vs 85%" is *we cannot distinguish these two models*, and the same
arithmetic applies to every small benchmark: noise scales with 1/√n, so a 200-question test is
inherently about three times noisier than a 2,000-question one. To resolve a three-point difference
at all — an interval that merely excludes zero — takes roughly **1,000 questions**, and about twice
that for a comparison anyone would call adequately powered. A leaderboard sorted to two decimal
places is sorting mostly on luck.

### The test set may be in the training data

Benchmarks are published on the web, and training corpora are scraped from the web. When the
questions and answers leak into pretraining, the model recalls rather than reasons, and the score
stops estimating the only thing a held-out test set was ever there to estimate —
[generalization](https://howaiworks.ai/glossary/generalization) to questions the model has not seen. This is called
**contamination**, and it is [overfitting](https://howaiworks.ai/glossary/overfitting) at the scale of a whole field: a
test everyone has been optimising against for two years is no longer a test.

Catching it is harder than it sounds, and there are three usual attempts. Benchmark authors embed a
**canary string**, a unique identifier that a careful lab filters out of its corpus — which
demonstrates good faith rather than absence, since nothing forces anyone to look for it. Analysts
measure **n-gram overlap** between test items and training data, which fails precisely where it
matters, because frontier training corpora are not published. The strongest evidence is a
**rebuild**: write a fresh test at matched difficulty and see whether the scores hold. Scale AI did
this to GSM8K in 2024, regenerating the same grade-school arithmetic task as GSM1k; several model
families dropped by close to 13 points on the new set while others were unchanged — which is what
contamination looks like on the rare occasion you can see it at all. That gap is also why new,
deliberately held-out benchmarks keep appearing: the old ones stop discriminating.

### The evaluation setup does the work

The same model produces very different numbers depending on how it was run. `pass@1` — one attempt,
scored — is what you get in production. `pass@64` — sixty-four attempts with an oracle picking the
correct one — is a number you will never see again outside the chart it appeared in.
[Reasoning effort](https://howaiworks.ai/glossary/test-time-compute), temperature, prompt format, and whether tools were
available all move scores by more than the margins being claimed. Two labs can report on "the same
benchmark" and not be measuring the same thing.

### Aggregate scores hide the task you care about

A model that leads on an aggregate index can be behind on your workload, because the index is an
average over tasks that are not your task — and averaging shrinks whatever it averages. An index
built from ten benchmarks turns a decisive five-point win on one of them into **half a point** on
the headline, so the differences you are shopping for are exactly the ones the aggregate smooths
away. That is why the top of an index is usually a cluster separated by noise while the same models
on a single skill are separated by a mile: the leader at code generation is frequently not the
leader at long-context retrieval, at instruction-following, or at cost per token.

## Real-World Applications

**Choosing a model.** This is where the concept actually costs money. A team picks the model at the
top of a leaderboard, ships it, and finds it worse at their own task and several times the price of
the runner-up. The useful question is never "what is SOTA" but "what is best at *my* task at a
price I will pay" — and our [model catalog](https://howaiworks.ai/models) exists for that comparison, while a live
leaderboard answers the ranking question better than any written page can.

Five things decide whether a chart in front of you means anything:

- **How many questions?** Divide 100 by that number. That is what a single question is worth, and
  on a benchmark of a few hundred items it is usually a large fraction of the lead being claimed.
- **`pass@1` or `pass@k`, and under what settings?** Reasoning effort, temperature and tool access
  move a score further than the margin does, and only `pass@1` resembles what you will run.
- **Who ran it?** A lab reporting its own model on its own harness against baselines it chose is an
  interested party, not an auditor.
- **Which version of the index, and dated when?** An aggregate score quoted without its version is
  not comparable to the other score you are holding it against.
- **What does it cost per token, and how slow is it?** Neither is on the chart, and between two
  models you cannot statistically distinguish, they are the entire decision.

Or skip the chart and measure it yourself. Pull thirty to fifty real examples off your own traffic,
run the two or three candidates through them, and score the outputs. The arithmetic that deflates
the launch chart applies to your test too: thirty examples resolve a gap of about **twenty points**
and nothing finer, and it takes around **120** to see ten. That sounds like a weak instrument until
you notice it is pointed at the right resolution — a model twenty points better on *your* work is
worth a migration, and one you cannot separate from the incumbent on your own data was never going
to repay the switching cost, whatever it scored on GPQA.

**The publication bar in research.** "Beats SOTA" is, in practice, a condition of acceptance at
major ML venues. That single convention shapes what gets researched: it rewards small gains on
established benchmarks over work that is harder to score, and it is a large part of why benchmark
chasing and contamination are so persistent. The label is not just a description — it is an
incentive.

**Regulation and procurement.** Rules aimed at "frontier" models need a definition of frontier, and
in practice they reach for either compute thresholds or capability benchmarks. The EU AI Act takes
the first: a general-purpose model trained with more than **10²⁵ FLOP** is presumed to carry
systemic risk, with the obligations that follow. Compute is the cleaner threshold of the two,
because it cannot be optimised for without actually being spent. When a *benchmark* score becomes
the legal or purchasing threshold instead, the pressure to optimise for the benchmark rather than
the capability gets much stronger.

## Key Concepts

- **Benchmark saturation**: a test stops working well before anyone gets 100% on it, because a
  benchmark's own labelling errors set a ceiling below the real one — the 2024 audit *Are We Done
  with MMLU?* found errors in about 6.5% of that benchmark's questions, so a score in the mid-90s
  there is partly a measure of how well a model reproduces the errata. Once the leaders cluster in
  that band, what separates them is mistakes and noise, and a harder benchmark has to be built.
- **Index versions are not comparable**: an aggregate score such as the Artificial Analysis
  Intelligence Index is revised, and the same model can move by ten points or more between index
  versions with no change to the model at all. A score quoted without its version number is not a
  fact. (This site treats a bare score as an error for exactly that reason.)
- **Elo and human preference are a different measurement**: arena-style rankings ask humans which
  answer they prefer, which captures style and helpfulness that benchmarks miss — and rewards
  agreeable, well-formatted answers that are not necessarily more correct. They are honest about
  their own noise, publishing a confidence interval next to every rating, and those intervals
  routinely overlap for a dozen models at a time. The rank ordering is far more precise than the
  measurement underneath it.

## Challenges

**The label rots faster than the page describing it.** Any content that names the current SOTA
model is wrong within months, which is why this page does not name one. Treat a written ranking as
a historical note and go to a live source.

**Being SOTA and being useful have come apart.** A model can hold the top score on a reasoning
benchmark and still be the wrong choice for a production system that needs low latency, a large
[context window](https://howaiworks.ai/glossary/context-window), predictable cost, or a permissive licence. None of
those appear in the score that earns the label.

**Self-reported evaluations are the norm.** Most frontier scores are published by the lab that
trained the model, run on its own harness, against baselines it selected. Independent replication
is the exception, not the rule — so the comparison you are reading was constructed by an
interested party.

**A benchmark measures what is easy to measure.** [Hallucination](https://howaiworks.ai/glossary/hallucinations) rates,
long-horizon reliability, and whether a model knows it is wrong are the properties that decide
whether a system survives contact with users, and they are the hardest to reduce to a number. The
leaderboard is bright, and the important questions are outside the light.

## Future Trends

Evaluation is moving away from single-turn question answering, because that format has largely been
exhausted: the frontier now sits on long-horizon [agentic tasks](https://howaiworks.ai/glossary/agentic-workflow) —
multi-step work with tools, where success is a completed task rather than a matched string. Those
evaluations are far more expensive to run and far harder to game, but they are also noisier and
less reproducible, which means the
tidy single-number ranking is likely to get *less* reliable, not more.

The second shift is toward contamination-resistant design: answer keys that are never published,
questions drawn from sources newer than any current model's training cutoff, and test sets that are
rotated on a schedule so that optimising against last month's copy buys nothing. Each of these buys
integrity at the cost of transparency — you can trust the number more, and verify it less. The
end point of that trade is an evaluation you cannot reproduce, run by people you have to take on
trust, which is a different problem from the one it solves rather than an absence of one.

## Frequently Asked Questions

### Which model is state of the art right now?

There is no single answer, and any page that gives you one is out of date. SOTA is per-task: the leader on coding is rarely the leader on long-context retrieval or on cost per token. Check a live leaderboard for the task you actually care about, and read the model catalog for what each model is strong at.

### What makes a model 'state of the art'?

A model is called SOTA when it posts the best published score on a benchmark. That is a claim about one test set on one day under one evaluation setup — not a general statement that the model is the best available.

### Is a 3-point benchmark lead meaningful?

Usually not. GPQA Diamond has 198 questions, so a single question is worth 0.51 percentage points and a 3-point lead is six questions. Read as independent samples, the 95% confidence interval on that difference is roughly ±6.7 points, so the lead sits inside the noise.

### How many questions does a benchmark need before a 3-point lead means something?

Around 1,000, and roughly twice that for a comparison anyone would call adequately powered. GPQA Diamond has 198, MMLU has about 14,000 — which is one reason a three-point gap means very different things on the two.

### What is benchmark contamination?

It is when benchmark questions and answers appear in a model's training data, usually because both were scraped from the web. The model then scores high by recall rather than reasoning, and the benchmark stops measuring what it was built to measure.

### Why do two leaderboards disagree about the best model?

Because they measure different things: different task mixes, different prompting, different sampling settings, and sometimes different versions of an index whose scores are not comparable across versions.

## Related

### Related terms

- [Benchmark](https://howaiworks.ai/glossary/benchmark)
- [Generalization](https://howaiworks.ai/glossary/generalization)
- [Overfitting](https://howaiworks.ai/glossary/overfitting)
- [Test-Time Compute](https://howaiworks.ai/glossary/test-time-compute)
- [Performance](https://howaiworks.ai/glossary/performance)
- [Large Language Model (LLM)](https://howaiworks.ai/glossary/large-language-model)

---

Source: https://howaiworks.ai/glossary/state-of-the-art-model — HowAIWorks.ai