---
source: 'https://howaiworks.ai/glossary/llm-as-a-judge'
section: glossary
title: LLM-as-a-Judge
description: >-
  Using one model to grade another's output against a rubric: how it works, the
  biases that break it, and the human agreement rate you can actually reach.
tags:
  - LLM
  - evaluation
  - AI Testing
  - benchmark
  - MLOps
  - Prompt Engineering
category: Machine Learning
datePublished: '2026-07-24'
lastUpdated: '2026-07-24'
---

# LLM-as-a-Judge

> Using one model to grade another's output against a rubric: how it works, the biases that break it, and the human agreement rate you can actually reach.

## Definition

**LLM-as-a-judge** is the practice of having one language model grade another model's output
against a written rubric, in place of a human rater. The judge is shown the original question, one
answer or a pair of competing answers, and instructions describing what counts as good; it returns
a verdict — a score, a winner, or a pass/fail label — usually with a sentence of reasoning attached.

It exists for a cost reason. A careful human comparison of two long answers takes minutes of expert
attention; a judge call takes seconds and a fraction of a cent, which is the difference between
evaluating a two-hundred-case regression suite on every pull request and evaluating it twice a year.
The surprising part is that it works at all. In the paper that named the technique, Zheng et al.
(2023) found GPT-4's pairwise judgments agreed with human experts on **85%** of non-tied MT-Bench
comparisons, while the human experts agreed with *each other* on only **81%**. A good judge is about
as close to a panel of people as the people are to one another.

That result is also the trap, because 85% agreement is not 85% accuracy, and the missing 15% is not
random noise. It is systematic: the judge prefers the answer it read first, prefers the longer
answer, mildly prefers answers from its own model family, and — the failure that matters most —
makes mistakes that are *correlated with the generator's*, because it is often the same model, or a
close relative, with the same blind spots. A judge cannot see an error it would have made itself.
Everything useful about this technique lives in what you do about that.

## How It Works

A judge is a prompt, not a model. There is no separate architecture and usually no training: you
take a capable general model and give it a template with four parts — the user's original question,
the response or responses under evaluation, a rubric spelling out the criteria, and a strict output
format so the verdict can be parsed. Zheng et al.'s pairwise template is a good specimen and is
worth reading in full, because most production judges are variations on it: it lists the criteria
(helpfulness, relevance, accuracy, depth, creativity, level of detail), explicitly instructs the
model to ignore position and length, and demands the verdict as `[[A]]`, `[[B]]` or `[[C]]` so a
script can read it. Notably, those anti-bias instructions are in the prompt *and the biases below
were measured with them present*. Telling a judge not to be biased does not make it unbiased.

Three mechanical details separate a judge that produces a usable number from one that produces
noise.

**Parse the verdict, do not read it.** A judgment that arrives as free text has to be reduced to a
symbol before it can be counted, and the reduction is where silent failures live. Zheng et al.'s
Table 2 reports an "Error" column for judgments in the wrong output format — 1.2% to 3.8% for the
weaker judges, 0.0% for GPT-4. A pipeline that treats an unparseable verdict as a tie, or worse as a
loss, is putting the judge's formatting reliability into your quality metric. Count parse failures
separately and look at them.

**Pin everything and log it.** A judge is a non-deterministic component: the model identifier, the
[temperature](https://howaiworks.ai/glossary/temperature), the rubric wording and the template are all inputs to the
number you are about to plot on a dashboard. Change any of them and the score moves for reasons that
have nothing to do with the system under test. This is why [monitoring](https://howaiworks.ai/glossary/monitoring) a
generative system treats the judge as a monitored dependency in its own right: a quality "regression"
that coincides with a judge upgrade is a measurement artefact, and you cannot tell the two apart
after the fact unless the judge's version sat in the change log next to the model's.

**Run every comparison in both orders.** This is the single highest-value protocol change and it is
cheap to reason about. Zheng et al. call it the conservative approach: judge the pair as (A, B), then
again as (B, A), and declare a winner only if the same answer wins both times. Disagreement becomes a
tie. The cost is exact — two calls per comparison instead of one — and the yield depends on how
consistent your judge is. At GPT-4's measured 65% swap consistency on near-identical answers, a
200-pair suite costs 400 judge calls and returns about 130 decided verdicts and 70 forced ties. If
you need 200 decided verdicts you must run 308 pairs, or 615 calls. That is a 3× budget multiplier
over the naive single-call design, and it buys the only thing that makes the output trustworthy.

## Types

The literature recognises three grading modes, and they are not interchangeable.

**Pairwise comparison** shows the judge two answers to the same question and asks which is better,
with ties allowed. **Single-answer grading** (also called direct scoring or pointwise) shows one
answer and asks for a score, typically 1–10 against a rubric. **Reference-guided grading** adds a
known-good answer to the prompt and asks the judge to compare against it before deciding.

Pairwise is the more reliable default, for a reason that has a close parallel in
[reward models](https://howaiworks.ai/glossary/reward-model): an absolute score has no fixed zero and no fixed unit. It is
whatever the judge's internal rubric happens to be that day. Zheng et al. put it directly — single
grading "may be unable to discern subtle differences between specific pairs, and its results may
become unstable, as absolute scores are likely to fluctuate more than relative pairwise results if
the judge model changes." Direct scores also compress: judges cluster their ratings into a narrow
band near the top of whatever scale you hand them, so a rubric with ten levels resolves far fewer
than ten. Pairwise asks only for an ordering, which is the one judgment a model makes dependably, and
orderings are directly convertible into win rates and Elo-style ratings.

The measured gap is real but modest. On Chatbot Arena data, GPT-4 in pairwise mode agreed with humans
on 87% of non-tied votes against 85% for the same model doing single-answer grading; on MT-Bench both
reached 85%. Pairwise's decisive advantage is stability across judge versions, not raw accuracy on
any one day.

What pairwise costs is scale. Comparisons grow quadratically: ranking 10 candidate prompts pairwise
is 45 comparisons, and with order-swapping, 90 calls; 20 candidates is 190 comparisons and 380 calls.
Direct scoring is linear and is the right tool when you have many candidates and only need a coarse
sort, or when there is no natural opponent to compare against — which is the usual situation in
production, where a live response has no counterfactual twin.

Reference-guided grading is the mode people reach for last and should reach for first whenever a
reference exists. Zheng et al. tested GPT-4 on ten maths questions where one answer was wrong,
scoring each pair in both orders: with the default prompt the judge called an incorrect answer
correct **14 times out of 20**; with chain-of-thought prompting, 6 out of 20; with a reference answer
in the prompt, **3 out of 20**. Handing the judge the answer key cut its failure rate by more than
four-fifths on exactly the task where an unaided judge is weakest.

## Real-World Applications

**Public leaderboards run on judges.** AlpacaEval scores instruction-following models by having a
judge compare each model's answers against a fixed baseline's, and its length-controlled variant
reports a Spearman correlation of **0.98** with human votes on LMSYS Chatbot Arena, up from 0.94
before the correction (Dubois et al., 2024). That is a leaderboard built entirely from model
judgments tracking a leaderboard built entirely from human ones, which is the strongest single piece
of evidence that the technique carries real signal.

**Internal regression evaluation is the highest-value use and the least discussed.** If you ship a
product on an [LLM](https://howaiworks.ai/glossary/large-language-model), every prompt edit and model upgrade is a silent
risk, and a suite of your own cases with known-good outputs is the only way to catch a break. For
anything open-ended — a summary, a support reply, a rewritten paragraph — there is no string match
that can score it, so the judge *is* the assertion. This is the mechanism that turns a
[benchmark](https://howaiworks.ai/glossary/benchmark) from a public leaderboard into a test suite you run on every pull
request, and it is where most of the judge calls in the industry are actually spent.

**Production quality monitoring.** A deployed generative system has no accuracy metric to degrade: no
labels arrive, and a fluent, confident, wrong answer returns a perfectly normal 200 response of
perfectly normal length. Sampling live traffic and scoring it with a judge nightly is the closest
available substitute for the accuracy curve a classifier gives you for free. It only works if you
hold the judge fixed and treat its variance as part of the measurement — see
[monitoring](https://howaiworks.ai/glossary/monitoring).

**Guardrails and routing.** Judges run inline as well as offline: a cheap model checks whether a
response is grounded in the retrieved documents, whether it leaked a system prompt, or whether a
draft is good enough to send or should be escalated. Here the judge's latency and cost are on the
critical path, which usually forces a smaller judge than an offline evaluation would use — and
smaller judges are markedly more biased, as the numbers below show.

**Training signal.** Constitutional AI and RLAIF replace some or all human preference labels with
model-generated ones, which is a judge wired into the training loop rather than the test harness.
This is where the distinction from a [reward model](https://howaiworks.ai/glossary/reward-model) matters: a reward model
is a scalar head trained on preference pairs, fast enough to call millions of times during
[RLHF](https://howaiworks.ai/glossary/reinforcement-learning-from-human-feedback); a judge is a prompted general model
that writes its comparison out in text. Judges dominate evaluation because they are flexible and need
no training; reward models dominate training because they are cheap per call.

## Key Concepts

**Agreement is not accuracy.** Every headline figure in this area is an agreement rate against human
labels, and humans disagree with each other. On non-tied MT-Bench votes, human experts agreed 81% of
the time. That is the ceiling: a judge scoring 85% is not 15% wrong, it is 15% *different from a
particular panel*, and some of that difference is the panel's.

**Raw agreement flatters, so convert it.** Cohen's kappa subtracts the agreement two random judges
would reach anyway. On a forced binary choice, chance is 50%, so 85% raw agreement is a kappa of
0.70, and the human panel's 81% is 0.62. Those are respectable numbers, not the near-perfect ones the
percentages suggest. Report kappa when you validate a judge on your own data.

**Agreement depends on the size of the gap you are measuring.** Zheng et al. broke agreement down by
how far apart the two models were and found it rose from about 70% for closely matched pairs to
nearly 100% for badly mismatched ones. This is the most practically important caveat on the whole
technique: a judge is most reliable exactly when you least need it, and least reliable in the regime
you actually work in — comparing yesterday's prompt with today's. A validation set built from obvious
wins will certify a judge that cannot do your job.

**Judge self-consistency is not judge quality.** GPT-4 in pairwise mode agreed with GPT-4 in
single-answer mode on 95–97% of non-tied MT-Bench votes, while agreeing with humans on 85%. A judge
is far more reproducible than it is right, and reproducibility is what a dashboard shows you. Stable
numbers are evidence of a stable judge, not a correct one.

**The only real validation is human labels on your own data.** Public agreement figures were measured
on chat assistant answers to open-ended questions; nothing carries them over to your medical
summaries or your SQL rewrites. Hand-label a few hundred of your own cases — weighted towards close
calls — and measure the judge against them before you trust a single number it produces. Re-measure
when the judge version changes.

## Challenges

**Position bias.** The judge favours whichever answer it read first, and the effect is large. With two
deliberately similar answers, Zheng et al. found GPT-4 gave the same verdict after swapping only
**65.0%** of the time, choosing the first answer in 30.0% of cases; GPT-3.5 was consistent 46.2% of
the time and Claude-v1 just **23.8%**, picking the first answer in 75.0% of cases. Wang et al. (2023)
measured the same thing as a "conflict rate" and showed how it can be exploited: with GPT-3.5 as the
evaluator, Vicuna-13B won 2.5% of comparisons against it in one position and 82.5% in the other, and
the abstract's headline is that Vicuna could be made to beat ChatGPT on **66 of 80** queries purely by
choosing the presentation order. The severity tracks how close the answers are — the same paper
reports a 46.3% conflict rate for GPT-4 on a close pair and 5.0% on a mismatched one. Swapping is the
fix; few-shot examples help too, lifting GPT-4's consistency from 65.0% to 77.5%.

**Verbosity bias.** Judges prefer longer answers, independent of whether they are better. Zheng et al.
built a "repetitive list" attack: take an answer containing a numbered list, ask a model to rephrase
the list without adding information, and prepend the rephrasing, doubling the length while adding
nothing. Claude-v1 and GPT-3.5 preferred the padded version **91.3%** of the time; GPT-4 fell for it
8.7% of the time. The effect is not confined to adversarial constructions. Dubois et al. (2024) took
one model, compared it against *itself* on AlpacaEval, and varied only the verbosity instruction in
its prompt: its win rate moved from **22.9% to 64.3%**. Same model, same capability, same judge — a
41-point swing bought with an instruction to be wordy. Their length-controlled correction, a
regression that predicts what the preference would have been at zero length difference, narrows that
range to 41.9%–51.6% and cuts the normalised standard deviation across verbosity prompts from 25% to
10%.

**Self-preference.** A judge tends to favour its own family's output. Zheng et al. measured GPT-4
giving its own answers a **10%** higher win rate than humans did, and Claude-v1 giving its own a 25%
higher one — while cautioning that with their sample size they could not establish this as a
controlled effect, since you cannot restyle a response into another model's voice without changing
its quality. Panickssery et al. (2024) found the mechanism: LLMs can recognise their own outputs at
above-chance accuracy out of the box, and fine-tuning a model to be better at self-recognition
increases its self-preference in step — a linear relationship that survives their controls for
obvious confounders. The practical rule is simple: never let a model be the sole judge of its own
generations, and be suspicious of any evaluation where the judge and the top-scoring model come from
the same lab.

**Correlated errors — the failure the other three are symptoms of.** A judge is useful only to the
extent that its mistakes are independent of the generator's. They are not. Zheng et al.'s Chatbot
Arena figures make this measurable: GPT-4 as judge agreed with humans on 87% of non-tied votes and
GPT-3.5 as judge on 83%, but the two judges agreed *with each other* on 94%. If their errors were
independent, two judges that good would agree with each other 87% × 83% + 13% × 17% = **74.4%** of the
time. The observed 94% is nearly twenty points of shared error — mistakes both models make, in the
same direction, on the same items, invisible to any amount of cross-checking between them.

The mechanism is plain once stated. A judge drawn from the same pretraining distribution as the
generator has the same gaps in its knowledge, the same misconceptions and the same stylistic tastes.
Ask it to check a claim the generator hallucinated and it will often find the claim plausible for the
identical reason the generator produced it. A closely related failure is documented directly: Zheng
et al. show GPT-4 grading an elementary maths problem it can solve correctly when asked in isolation,
and being talked out of the right answer by the wrong answer it was grading. Whether the judge
inherits the mistake or is argued into it, the result is the same — the output under evaluation
shapes the evaluation. This is the same structure that
makes a model a poor checker of its own work in any [error-handling](https://howaiworks.ai/glossary/error-handling)
pipeline, and it puts a hard floor under what judge-based evaluation can catch. Verifiable checks — a
unit test, an executed query, a retrieved citation — are not the same kind of thing as a judge and
should be used instead of one wherever the task admits them.

**Goodharting the judge.** The moment a judge score becomes a target, the system starts optimising
for the judge rather than the quality. This is exactly reward-model overoptimisation with a prompt
where the scalar head used to be, and it arrives faster than people expect, because the judge's
biases are the cheapest gradient available: adding length, adding structure, adding confident phrasing
and adding markdown headers all raise scores without touching substance. LMArena's own answer was
style-controlled ratings: a regression that carries answer length and the counts of markdown headers,
bold spans and lists as covariates, so credit for formatting is attributed to the formatting rather
than to the model. If your judge is also your optimisation target, you need a held-out evaluation the
optimiser never sees.

**Cost and latency stop being negligible at scale.** The per-call price is small, but the protocol
multiplies it: two calls for order-swapping, several judges if you use a panel, several samples if
you average out judge variance. A single-call design that becomes a swapped three-judge panel is 6×
the calls and 6× the bill for the same suite, and a judge that reads two long answers pays input
tokens for both.

## Future Trends

**Juries of small models instead of one large judge.** Panels of several cheaper models, voting,
consistently trade well against a single frontier judge — partly on cost and partly because a panel
drawn from different labs has less correlated error than one model consulted repeatedly. The design
question is which axis of diversity actually buys independence: different prompts of the same model
buy almost none, different model families buy some, and a rule-based or execution-based checker
alongside the models buys the most.

**Statistical corrections applied on top of the judge.** Length-controlled AlpacaEval is the template:
rather than trying to prompt the bias away, fit a model of the bias and condition it out afterwards.
The approach generalises to any confound you can measure — formatting, hedging, the presence of code
blocks — and it is more honest than an instruction the judge demonstrably ignores.

**Verifiable rewards displacing judges where a checker exists.** The clear trend in training is to
replace subjective scoring with programmatic verification wherever the task allows: unit tests for
code, exact-match for maths, executable checks for structured output. Judges retreat to the genuinely
open-ended qualities — tone, helpfulness, whether a summary is faithful — which is where they were
always least replaceable and, awkwardly, also where their agreement with humans is lowest.

**Judge versions becoming first-class artefacts.** As evaluation results start feeding release
decisions, the judge model, prompt hash, temperature and rubric are moving into the same version
control and change log as the system under test, with historical scores recomputed when the judge
changes. An evaluation number without a judge version attached is becoming as meaningless as a
benchmark score without a harness.

## Code Example

Three calculations worth doing before you trust a judge-based result: convert raw agreement into
kappa, check how much of two judges' agreement is shared error rather than shared correctness, and
size the sample the swap protocol leaves you with.

```python
from math import sqrt

# Cohen's kappa: what survives of a raw agreement rate once you subtract the
# agreement two coin-flipping judges would reach anyway. Non-tie votes only,
# so chance is 50%. Figures from Zheng et al. (2023), Tables 5 and 6.
def kappa(agreement, chance=0.5):
    return (agreement - chance) / (1 - chance)

rows = [("GPT-4 judge vs human   (Arena)", 0.87),
        ("GPT-3.5 judge vs human (Arena)", 0.83),
        ("GPT-4 vs GPT-3.5 judge (Arena)", 0.94),
        ("human vs human      (MT-bench)", 0.81)]
for name, a in rows:
    print(f"{name}  agreement {a:.0%}   kappa {kappa(a):.2f}")

# If two judges erred independently, their agreement with each other would be
# pinned by their agreement with humans: both right, or both wrong the same
# way. Everything above that line is error the two judges share.
p4, p35, observed = 0.87, 0.83, 0.94
expected = p4 * p35 + (1 - p4) * (1 - p35)
print(f"\nindependent-error prediction {expected:.1%} vs observed {observed:.0%}"
      f"  ->  {(observed - expected) * 100:.1f} pts of shared error")

# The conservative swap rule: judge every pair twice, in both orders, and keep
# only the verdicts that survive. 65% is GPT-4's measured swap consistency.
consistency, suite = 0.65, 200
decided = suite * consistency
print(f"\n{suite} pairs -> {suite * 2} judge calls -> {decided:.0f} decided,"
      f" {suite - decided:.0f} forced to a tie")
print(f"to keep {suite} decided verdicts: {suite / consistency:.0f} pairs,"
      f" {2 * suite / consistency:.0f} calls")

# And what a win rate off that many verdicts can actually resolve.
for n in (130, 200, 1000):
    margin = 1.96 * sqrt(0.25 / n) * 100
    print(f"n={n:5d}   95% CI on a 50% win rate: +/- {margin:.1f} pts")
```

Running it prints:

```
GPT-4 judge vs human   (Arena)  agreement 87%   kappa 0.74
GPT-3.5 judge vs human (Arena)  agreement 83%   kappa 0.66
GPT-4 vs GPT-3.5 judge (Arena)  agreement 94%   kappa 0.88
human vs human      (MT-bench)  agreement 81%   kappa 0.62

independent-error prediction 74.4% vs observed 94%  ->  19.6 pts of shared error

200 pairs -> 400 judge calls -> 130 decided, 70 forced to a tie
to keep 200 decided verdicts: 308 pairs, 615 calls
n=  130   95% CI on a 50% win rate: +/- 8.6 pts
n=  200   95% CI on a 50% win rate: +/- 6.9 pts
n= 1000   95% CI on a 50% win rate: +/- 3.1 pts
```

Read the last block before designing an evaluation. A 200-case suite run through the swap protocol
leaves about 130 decided comparisons, and 130 comparisons cannot resolve a win rate to better than
roughly ±8.6 points. If the prompt change you are testing moves quality by three points, this suite
will report noise and you will ship on it. The judge is rarely the binding constraint on a small
evaluation — the sample size is, and the swap protocol you need for the judge to be trustworthy makes
the sample smaller.

## Frequently Asked Questions

### What is LLM-as-a-judge in simple terms?

It is asking one language model to grade another model's answer against a written rubric, instead of paying a human to read it. The judge sees the question, the answer or a pair of competing answers, and instructions on what counts as good, and returns a verdict with a short explanation.

### How accurate is an LLM judge compared to a human?

On MT-Bench, GPT-4's pairwise judgments matched human experts on 85% of non-tied comparisons, while the human experts matched each other on 81% (Zheng et al., 2023). That is the realistic ceiling: a good judge is about as close to a panel of humans as the humans are to each other, and no closer.

### Why is pairwise comparison better than scoring answers 1 to 10?

An absolute score has no fixed scale. It drifts between judge versions, bunches up around 7 and 8, and cannot resolve small differences. Pairwise comparison only asks which of two answers is better, which is the one thing a judge does reliably — at the cost of growing quadratically with the number of candidates.

### What is position bias in an LLM judge?

The judge favours whichever answer it reads first. With two near-identical answers, GPT-4 gave the same verdict after swapping their order in only 65% of cases and picked the first answer 30% of the time; Claude-v1 was consistent 23.8% of the time (Zheng et al., 2023). The fix is to run every comparison in both orders and count a flip as a tie.

### Can a model judge its own output?

It can, but its mistakes will be correlated with the ones it made while generating. A judge cannot see a blind spot it shares with the generator, and Zheng et al. found GPT-4 favoured its own answers with a 10% higher win rate than humans gave them. Use a different model family where you can, and never let self-evaluation be the only gate.

### How do I know if my judge is any good?

Hand-label a sample of your own cases and measure agreement against the judge. Raw agreement flatters — convert it to Cohen's kappa so chance agreement is subtracted, and check that agreement holds on the close calls, not just the obvious ones. Any judge scores near 100% on comparisons where one answer is plainly terrible.

## Related

### Related terms

- [Benchmark](https://howaiworks.ai/glossary/benchmark)
- [Reward Model (RM)](https://howaiworks.ai/glossary/reward-model)
- [Monitoring](https://howaiworks.ai/glossary/monitoring)
- [Error Handling in AI Systems](https://howaiworks.ai/glossary/error-handling)
- [Reinforcement Learning from Human Feedback (RLHF)](https://howaiworks.ai/glossary/reinforcement-learning-from-human-feedback)
- [Prompt Engineering](https://howaiworks.ai/glossary/prompt-engineering)

---

Source: https://howaiworks.ai/glossary/llm-as-a-judge — HowAIWorks.ai