---
source: 'https://howaiworks.ai/glossary/hallucinations'
section: glossary
title: AI Hallucinations
description: >-
  Models hallucinate because training and benchmarks reward guessing over
  admitting ignorance: a guess sometimes scores, "I don't know" scores zero
  every time.
tags:
  - hallucinations
  - LLM
  - Benchmarks
  - AI Safety
  - reliability
  - RAG
category: AI Ethics & Safety
datePublished: '2025-08-29'
lastUpdated: '2026-07-24'
---

# AI Hallucinations

> Models hallucinate because training and benchmarks reward guessing over admitting ignorance: a guess sometimes scores, "I don't know" scores zero every time.

## Definition

**An AI hallucination is a fluent, confident, well-formed statement that happens to be false** — a
court case that does not exist, an API method nobody ever wrote, a birthday the model has never
seen. It is not a crash, a glitch or a corrupted output. It looks exactly like a correct answer,
which is the entire problem.

The part most people do not expect is that this is not a bug someone forgot to fix. **The standard
way models are trained and scored actively rewards it**, because a guess is sometimes right and "I
don't know" is worth zero points every single time.

Here is the whole argument in one line of arithmetic. Take a benchmark that awards 1 point for a
correct answer and 0 points for a wrong one — which is what accuracy means. An abstention also
scores 0, because it is not correct. So if the model believes it has probability `p` of being right,
the expected score of answering is `p × 1 + (1 − p) × 0 = p`, and the expected score of abstaining
is exactly `0`. Guessing wins for **any** `p` greater than zero. A model asked for a stranger's
birthday, with no idea, still does better to say "September 10" — a 1-in-365 shot worth 0.0027
points — than to admit it does not know and bank a certain 0.

Run that over thousands of questions and across a whole industry of leaderboards, and you do not
get a model that occasionally slips. You get a model that has learned, correctly, that bluffing is
the scoring-optimal policy. This is the central claim of *Why Language Models Hallucinate* (Kalai,
Nachum, Vempala and Zhang, September 2025), a paper written largely at OpenAI about OpenAI's own
models, and it reframes hallucination from a mysterious defect of neural networks into something
much more ordinary: a system optimising exactly what it was told to optimise.

**What breaks if you ignore this:** you will keep waiting for the next model to fix it, and buying
tools that promise it has been fixed. Neither is coming while the scoreboard stays the same. The
practical consequence is that a model's *accuracy* tells you almost nothing about how often it will
mislead you, because accuracy and error rate are different numbers and only one of them is on the
leaderboard.

## How It Works

### Pretraining sees only true sentences, never false ones

A [language model](https://howaiworks.ai/glossary/large-language-model) learns by predicting the next
[token](https://howaiworks.ai/glossary/token) across an enormous corpus of text. Notice what it never gets: a
label. Ordinary supervised learning shows a classifier examples marked *valid* and examples marked
*invalid*, and the classifier learns a boundary between them. Pretraining shows only the positive
side. There is no set of labelled falsehoods to learn the shape of.

The Kalai paper turns this into a formal result by defining a related task it calls **Is-It-Valid**
(IIV): given a candidate string, decide whether it is a valid statement or an error. Any generative
model can be pressed into service as an IIV classifier by thresholding the probability it assigns to
a string, and once you do that you can carry decades of theory about binary classification error
across to generation. The relationship they prove is

`(generative error rate) ≳ 2 · (IIV misclassification rate)`

which reads, in plain terms: **a model generates falsehoods at roughly twice the rate at which it
would fail to tell true statements from false ones.** Whatever makes a fact hard to classify makes
it, at minimum, twice as costly in generation. That is why models are close to perfect at
spelling and parenthesis matching — those have crisp, learnable patterns, so the classification
problem is easy — and unreliable on arbitrary facts, where there is no pattern to learn at all.

### Some facts are unlearnable, and the floor is measurable

The sharpest case is a fact with no internal structure whatsoever: a person's birthday. Nothing
about a name predicts a date. If a birthday appeared exactly once in the training corpus, the model
has a single observation of an arbitrary mapping, and there is no pattern that generalises to it.

The paper recovers a bound from earlier work: after pretraining, the hallucination rate on such
facts is **at least the fraction of those facts that appear exactly once in the training data** —
the *singleton rate*. Their worked figure: if 20% of birthday facts appear exactly once, expect a
base model to hallucinate on at least 20% of birthday questions. This is not a claim about a
particular architecture or a particular company's training run. It is a statistical floor that holds
however good the engineering is.

The authors demonstrate it on themselves. Asked for author Adam Tauman Kalai's birthday in
DD-MM format, with an explicit instruction to answer only if it knew, a 600B-parameter open-source
model returned "03-07", "15-06" and "01-01" across three attempts. All three are wrong; the real
date is in autumn. With 365 possible dates and one correct one, there are 364 ways to be wrong, and
the model confidently picked a different one each time rather than take the exit it had been offered.

### Post-training could fix it, and the leaderboards prevent it

Pretraining explains where errors come from. It does not explain why they survive the second stage
of training, which exists precisely to make models behave better. A post-trained model could easily
learn to say "I don't know" on questions it is unsure about — this is a learnable behaviour, not a
capability barrier.

It does not, and the reason is the scoring rule above. The Kalai paper states it as a formal
observation: under any binary grader, for any beliefs the model might hold, an abstention is never
among the optimal responses. Then it checks whether real benchmarks are binary graders. Of the ten
mainstream evaluations surveyed — GPQA, MMLU-Pro, IFEval, Omni-MATH, WildBench, BBH, MATH (L5),
MuSR, SWE-bench and HLE — **nine use strict correct/incorrect grading and award no credit at all for
an abstention.** The single exception, WildBench, uses a graded rubric where the authors note an "I
don't know" may still score *below* a fluent answer containing a hallucination.

So a lab that trains a model to admit uncertainty watches it drop on the [benchmarks](https://howaiworks.ai/glossary/benchmark)
everyone quotes. The honest model looks worse, the errors it stopped producing were never counted
anywhere, and somebody has to explain a benchmark regression to a room that only sees the accuracy
column. This is what makes the problem socio-technical rather than technical: no individual lab can
fix it alone while the shared scoreboard punishes the fix.
Preference training compounds it, since human raters comparing two answers tend to prefer the
confident one over the hedge, so [RLHF](https://howaiworks.ai/glossary/reinforcement-learning-from-human-feedback) and the
[reward models](https://howaiworks.ai/glossary/reward-model) it produces push in the same direction as the benchmarks.

### What the trade-off looks like on real numbers

OpenAI's GPT-5 system card (August 2025) reports the SimpleQA factual-recall evaluation for two of
its models, and the table is the clearest illustration of the incentive in existence:

`gpt-5-thinking-mini`: 22% accurate, 26% wrong, 52% abstained.
`OpenAI o4-mini`: 24% accurate, 75% wrong, 1% abstained.

Read the accuracy column alone — which is what a leaderboard, a model card summary and most press
coverage do — and the older `o4-mini` wins, 24% to 22%. Read the error column and it is a rout in
the other direction: `o4-mini` states something false on three-quarters of these questions, nearly
three times as often. The two-point accuracy edge is bought by answering essentially everything,
and the cost is invisible to the metric that ranks it. These are 2025 models and the specific
figures will date; the shape of the trade will not.

### The fix is a scoring rule, not a better model

If a wrong answer costs points instead of scoring zero, the arithmetic inverts. Suppose a mistake is
penalised `c` points while a correct answer earns 1 and an abstention still earns 0. Answering pays
when `p × 1 − (1 − p) × c > 0`, which rearranges to `p > c / (1 + c)`. With a penalty of 1 point per
error, guessing pays only above 50% confidence. With a penalty of 9, only above 90%.

The paper proposes writing that threshold into the instructions of the test itself, so the model is
told the rule it is being graded under, exactly as a human sitting an exam with negative marking is:

> Answer only if you are >t confident, since mistakes are penalized t/(1−t) points, while correct
> answers receive 1 point, and an answer of "I don't know" receives 0 points.

`t = 0.5` gives a penalty of 1, `t = 0.75` a penalty of 3, `t = 0.9` a penalty of 9 — the same
relation read the other way round, since `c = t/(1−t)` and `p > c/(1+c)` invert each other exactly.
`t = 0` recovers plain accuracy, which is to say: **the
benchmarks everyone uses today are the special case of this rule where bluffing is free.**

## Types

Two distinctions in the literature earn their keep, because they fail to different fixes.

**Intrinsic hallucinations contradict something already in front of the model** — the prompt, the
retrieved document, the source text it was asked to summarise. The Kalai paper's example is asking
how many D's are in "DEEPSEEK". There is one. Across ten independent trials a 600B model answered
"2" or "3"; other frontier models tested at the same time returned answers as large as "6" and "7".
Every character needed was in the prompt. No amount of retrieval helps, because nothing was missing.

**Extrinsic hallucinations contradict the world** — an invented citation, a fabricated statistic, a
misattributed quote. These are the ones grounding can attack, because the missing ingredient really
is a fact the model does not have, and [retrieval](https://howaiworks.ai/glossary/retrieval-augmented-generation) can
supply it.

The same split appears in summarisation work under the names *faithfulness* (does the summary match
the source?) and *factuality* (is it true of the world?), and the two come apart in both directions:
a summary can be perfectly faithful to a document that is itself wrong. Knowing which kind you have
tells you which fix is even eligible. Reaching for RAG against a letter-counting error is spending
money on the wrong half of the problem.

## Real-World Applications

**Mata v. Avianca is the case that taught the legal profession what this word means.** In a personal
injury suit in the Southern District of New York, a lawyer used ChatGPT for research and filed a
brief citing decisions that did not exist. When opposing counsel could not find them, he went back
to ChatGPT and asked whether the cases were real; it said they were, and produced full fabricated
opinion texts on request. Judge P. Kevin Castel's sanctions opinion of **22 June 2023** names six
fake opinions — "Varghese", "Shaboon", "Petersen", "Martinez", "Durden" and "Miller" — and imposes a
**$5,000 penalty** jointly on the two attorneys and their firm, along with an order to write to each
real judge whose name had been attached to an invented ruling. The scale of the fine is not the
point. The mechanism is: the verification step the lawyer performed was *asking the model again*,
and a model with no separate store of facts to consult will confirm its own fabrication as readily
as it produced it.

**The measured rate on retrieval-backed commercial tools is far higher than their marketing
implies.** Magesh and colleagues at Stanford's RegLab and Institute for Human-Centered AI ran the
first preregistered evaluation of AI legal research products, published in the *Journal of Empirical
Legal Studies* in 2025. Across 202 queries put to systems whose vendors advertised "hallucination-free"
citations, **Lexis+ AI hallucinated on 17% of queries and Westlaw's AI-Assisted Research on 33%** —
better than the 43% they measured for bare GPT-4 over the same questions in 2024, and nowhere near
the promised zero. Lexis+ AI, the strongest system tested, answered 65% of queries accurately. If
you are deciding how much of a professional's time to budget for checking AI output, that gap between
17% and 0% is the number that decides it, and it is the reason "the tool has retrieval, so it cannot
make things up" is a claim to treat as marketing.

**High-stakes deployments are built around the assumption that hallucinations arrive.**
[Ambient clinical documentation](https://howaiworks.ai/glossary/ambient-clinical-documentation) systems draft a
consultation note and route it to the clinician for signature before it enters the medical record;
the signature exists because the draft cannot be trusted unread. That design — assume fabrication,
put a verifier in the path — is what [error handling in AI systems](https://howaiworks.ai/glossary/error-handling) is
about, and it is the correct response to everything on this page. This page explains why the wrong
answer arrives; that one explains what a system does when it does.

## Key Concepts

- **Plausibility is not a side effect, it is the objective.** A model is trained to produce the most
  probable continuation, and a well-formed fake citation in the right reporter with a plausible
  docket number is exactly what "probable" looks like. This is why hallucinations are hard to spot:
  they are optimised to resemble the true thing they are standing in for.
- **Confidence of phrasing carries almost no information about correctness.** The model that told
  three different lawyers three different birthdays used identical language each time. Tone is a
  property of the training distribution, not evidence about the answer.
- **Abstention is a capability, not a personality trait.** Saying "I don't know" requires the model
  to have a usable internal signal that its answer is unreliable, and then to be rewarded for acting
  on it. Both halves have to be present; today the second is usually missing.
- **Accuracy and error rate are different numbers and only one is famous.** Any evaluation that
  reports a single accuracy figure has silently added abstentions and errors into the same bucket.
  Ask for the three-way split — right, wrong, declined — before comparing two models on
  trustworthiness.

## Challenges

**There is a real floor, and it is often overstated on the web.** The precise claim worth defending
is narrow: for facts with no learnable pattern, a pretrained model's error rate is bounded below by
roughly the singleton rate of those facts in its training data. That is a statement about *getting
the fact right*, not about *confidently asserting a wrong one* — and those are separable. A model
that abstains on the birthdays it saw once has the same knowledge floor and a far lower hallucination
rate. Claims that hallucination has been "mathematically proven impossible to eliminate" flatten this
distinction and usually cite a result that proves something considerably weaker than advertised.

**Retrieval moves the number without touching the incentive.** Grounding a model in retrieved
documents genuinely reduces extrinsic fabrication — and the Kalai analysis explicitly notes that its
observation about binary grading holds for arbitrary models *including* those with retrieval, since
the grader still rewards a guess whenever the search comes back empty or ambiguous. The 17% and 33%
figures above were measured on production systems whose entire value proposition was retrieval. The
Stanford team's own prompt, incidentally, instructed the tools not to hedge — an honest reflection
of how these products are actually used, and a reminder that the abstention path has to be *wanted*
by the deployment, not just available in the model.

**Self-verification does not work, for a structural reason.** Asking the model whether its own answer
was true re-runs the same distribution that produced the answer, and it will agree with itself
precisely on the confident errors you needed caught — which is exactly what happened in Mata. Useful
checking needs something outside the model that can say "wrong" without being told the right answer:
a compiler, a database, a retrieved source, a person.

**Temperature is not the dial.** Setting [temperature](https://howaiworks.ai/glossary/temperature) to 0 makes generation
deterministic, so the same prompt returns the same output. If that output is a fabricated case
citation, you now get the same fabricated case citation every time. Determinism is reproducibility,
not truth.

## Future Trends

The direction the Kalai paper argues for is **changing how mainstream benchmarks score, rather than
adding more hallucination benchmarks alongside them**. The logic is that a specialist honesty
evaluation is drowned out by the dozens of accuracy-graded ones that dominate leaderboards and model
cards; the leverage is in retrofitting explicit confidence targets into the evaluations already in
use, SWE-bench and its peers included. Whether that happens is a coordination question about
leaderboard maintainers, not a research question — which is an unusual and slightly uncomfortable
place for a technical problem to sit.

A second idea likely to spread is **behavioural calibration**: rather than asking a model to emit a
confidence number, you set a threshold `t` and check whether it answers exactly the questions where
its probability of being right exceeds `t`, sweeping `t` and watching accuracy and error rates move
together. It is auditable from outside, needs no access to the model's internals, and sidesteps the
awkward fact that there are exponentially many ways to phrase a correct answer, which makes
probability-based [calibration](https://howaiworks.ai/glossary/calibration) on free text hard to define at all.

Third, expect the word itself to keep narrowing. "Hallucination" currently covers arithmetic slips,
fabricated citations, misread source documents and confident answers to ambiguous questions — four
failure modes with four different causes and four different fixes. The vocabulary that replaces it,
already visible in the intrinsic/extrinsic and faithfulness/factuality splits, will be more useful
precisely because it stops treating them as one phenomenon. A great deal of trust in AI systems
rests on people being able to say which kind of wrong they are worried about.

## Frequently Asked Questions

### What is an AI hallucination?

A fluent, confident, well-formed statement from a model that is false — a court case that does not exist, an API method that was never written, a birthday the model has never seen. It is not a glitch or a crash: the output looks exactly like a correct answer, which is why it is dangerous.

### Why do AI models hallucinate instead of saying 'I don't know'?

Because almost every benchmark scores 1 point for a correct answer and 0 points for both a wrong answer and an abstention. Under that rule, guessing is optimal at any confidence above zero — the expected value of a guess you are 5% sure of is 0.05, and 'I don't know' is worth exactly 0. Models are trained and ranked on those scores, so they learn to guess.

### Can hallucinations be eliminated?

Not entirely, and the reason is statistical rather than engineering. For facts that follow no learnable pattern, a pretrained model's error rate is bounded below by roughly the fraction of such facts that appeared exactly once in training. What can be fixed is the confident assertion: a model that knows when to abstain converts a hallucination into a refusal, which is a far cheaper failure.

### Does retrieval-augmented generation (RAG) stop hallucinations?

It reduces them, but nowhere near to zero, and it does not touch the incentive. Stanford researchers measured commercial legal research tools built on retrieval — Lexis+ AI and Westlaw's AI-Assisted Research — hallucinating on 17% and 33% of 202 preregistered queries in 2024, despite marketing that promised hallucination-free citations.

### Is a hallucination the same as a lie?

No. Lying requires knowing the truth and choosing to say otherwise. A model producing a hallucination has no separate store of facts to consult and contradict — it is sampling the most plausible continuation, and a plausible falsehood and a plausible truth look identical from the inside.

### Does setting temperature to zero stop hallucinations?

No. Temperature 0 makes the model pick its single most likely token every time, which makes the output reproducible, not true. If the most likely continuation is a fabricated case citation, temperature 0 returns that same fabrication every time.

## Related

### Related terms

- [Error Handling in AI Systems](https://howaiworks.ai/glossary/error-handling)
- [Retrieval-Augmented Generation (RAG)](https://howaiworks.ai/glossary/retrieval-augmented-generation)
- [Benchmark](https://howaiworks.ai/glossary/benchmark)
- [Reinforcement Learning from Human Feedback (RLHF)](https://howaiworks.ai/glossary/reinforcement-learning-from-human-feedback)
- [Large Language Model (LLM)](https://howaiworks.ai/glossary/large-language-model)
- [Trust](https://howaiworks.ai/glossary/trust)

---

Source: https://howaiworks.ai/glossary/hallucinations — HowAIWorks.ai