Definition
An AI hallucination is a fluent, confident, well-formed statement that happens to be false — a court case that does not exist, an API method nobody ever wrote, a birthday the model has never seen. It is not a crash, a glitch or a corrupted output. It looks exactly like a correct answer, which is the entire problem.
The part most people do not expect is that this is not a bug someone forgot to fix. The standard way models are trained and scored actively rewards it, because a guess is sometimes right and "I don't know" is worth zero points every single time.
Here is the whole argument in one line of arithmetic. Take a benchmark that awards 1 point for a
correct answer and 0 points for a wrong one — which is what accuracy means. An abstention also
scores 0, because it is not correct. So if the model believes it has probability p of being right,
the expected score of answering is p × 1 + (1 − p) × 0 = p, and the expected score of abstaining
is exactly 0. Guessing wins for any p greater than zero. A model asked for a stranger's
birthday, with no idea, still does better to say "September 10" — a 1-in-365 shot worth 0.0027
points — than to admit it does not know and bank a certain 0.
Run that over thousands of questions and across a whole industry of leaderboards, and you do not get a model that occasionally slips. You get a model that has learned, correctly, that bluffing is the scoring-optimal policy. This is the central claim of Why Language Models Hallucinate (Kalai, Nachum, Vempala and Zhang, September 2025), a paper written largely at OpenAI about OpenAI's own models, and it reframes hallucination from a mysterious defect of neural networks into something much more ordinary: a system optimising exactly what it was told to optimise.
What breaks if you ignore this: you will keep waiting for the next model to fix it, and buying tools that promise it has been fixed. Neither is coming while the scoreboard stays the same. The practical consequence is that a model's accuracy tells you almost nothing about how often it will mislead you, because accuracy and error rate are different numbers and only one of them is on the leaderboard.
How It Works
Pretraining sees only true sentences, never false ones
A language model learns by predicting the next token across an enormous corpus of text. Notice what it never gets: a label. Ordinary supervised learning shows a classifier examples marked valid and examples marked invalid, and the classifier learns a boundary between them. Pretraining shows only the positive side. There is no set of labelled falsehoods to learn the shape of.
The Kalai paper turns this into a formal result by defining a related task it calls Is-It-Valid (IIV): given a candidate string, decide whether it is a valid statement or an error. Any generative model can be pressed into service as an IIV classifier by thresholding the probability it assigns to a string, and once you do that you can carry decades of theory about binary classification error across to generation. The relationship they prove is
(generative error rate) ≳ 2 · (IIV misclassification rate)
which reads, in plain terms: a model generates falsehoods at roughly twice the rate at which it would fail to tell true statements from false ones. Whatever makes a fact hard to classify makes it, at minimum, twice as costly in generation. That is why models are close to perfect at spelling and parenthesis matching — those have crisp, learnable patterns, so the classification problem is easy — and unreliable on arbitrary facts, where there is no pattern to learn at all.
Some facts are unlearnable, and the floor is measurable
The sharpest case is a fact with no internal structure whatsoever: a person's birthday. Nothing about a name predicts a date. If a birthday appeared exactly once in the training corpus, the model has a single observation of an arbitrary mapping, and there is no pattern that generalises to it.
The paper recovers a bound from earlier work: after pretraining, the hallucination rate on such facts is at least the fraction of those facts that appear exactly once in the training data — the singleton rate. Their worked figure: if 20% of birthday facts appear exactly once, expect a base model to hallucinate on at least 20% of birthday questions. This is not a claim about a particular architecture or a particular company's training run. It is a statistical floor that holds however good the engineering is.
The authors demonstrate it on themselves. Asked for author Adam Tauman Kalai's birthday in DD-MM format, with an explicit instruction to answer only if it knew, a 600B-parameter open-source model returned "03-07", "15-06" and "01-01" across three attempts. All three are wrong; the real date is in autumn. With 365 possible dates and one correct one, there are 364 ways to be wrong, and the model confidently picked a different one each time rather than take the exit it had been offered.
Post-training could fix it, and the leaderboards prevent it
Pretraining explains where errors come from. It does not explain why they survive the second stage of training, which exists precisely to make models behave better. A post-trained model could easily learn to say "I don't know" on questions it is unsure about — this is a learnable behaviour, not a capability barrier.
It does not, and the reason is the scoring rule above. The Kalai paper states it as a formal observation: under any binary grader, for any beliefs the model might hold, an abstention is never among the optimal responses. Then it checks whether real benchmarks are binary graders. Of the ten mainstream evaluations surveyed — GPQA, MMLU-Pro, IFEval, Omni-MATH, WildBench, BBH, MATH (L5), MuSR, SWE-bench and HLE — nine use strict correct/incorrect grading and award no credit at all for an abstention. The single exception, WildBench, uses a graded rubric where the authors note an "I don't know" may still score below a fluent answer containing a hallucination.
So a lab that trains a model to admit uncertainty watches it drop on the benchmarks everyone quotes. The honest model looks worse, the errors it stopped producing were never counted anywhere, and somebody has to explain a benchmark regression to a room that only sees the accuracy column. This is what makes the problem socio-technical rather than technical: no individual lab can fix it alone while the shared scoreboard punishes the fix. Preference training compounds it, since human raters comparing two answers tend to prefer the confident one over the hedge, so RLHF and the reward models it produces push in the same direction as the benchmarks.
What the trade-off looks like on real numbers
OpenAI's GPT-5 system card (August 2025) reports the SimpleQA factual-recall evaluation for two of its models, and the table is the clearest illustration of the incentive in existence:
gpt-5-thinking-mini: 22% accurate, 26% wrong, 52% abstained.
OpenAI o4-mini: 24% accurate, 75% wrong, 1% abstained.
Read the accuracy column alone — which is what a leaderboard, a model card summary and most press
coverage do — and the older o4-mini wins, 24% to 22%. Read the error column and it is a rout in
the other direction: o4-mini states something false on three-quarters of these questions, nearly
three times as often. The two-point accuracy edge is bought by answering essentially everything,
and the cost is invisible to the metric that ranks it. These are 2025 models and the specific
figures will date; the shape of the trade will not.
The fix is a scoring rule, not a better model
If a wrong answer costs points instead of scoring zero, the arithmetic inverts. Suppose a mistake is
penalised c points while a correct answer earns 1 and an abstention still earns 0. Answering pays
when p × 1 − (1 − p) × c > 0, which rearranges to p > c / (1 + c). With a penalty of 1 point per
error, guessing pays only above 50% confidence. With a penalty of 9, only above 90%.
The paper proposes writing that threshold into the instructions of the test itself, so the model is told the rule it is being graded under, exactly as a human sitting an exam with negative marking is:
Answer only if you are >t confident, since mistakes are penalized t/(1−t) points, while correct answers receive 1 point, and an answer of "I don't know" receives 0 points.
t = 0.5 gives a penalty of 1, t = 0.75 a penalty of 3, t = 0.9 a penalty of 9 — the same
relation read the other way round, since c = t/(1−t) and p > c/(1+c) invert each other exactly.
t = 0 recovers plain accuracy, which is to say: the
benchmarks everyone uses today are the special case of this rule where bluffing is free.
Types
Two distinctions in the literature earn their keep, because they fail to different fixes.
Intrinsic hallucinations contradict something already in front of the model — the prompt, the retrieved document, the source text it was asked to summarise. The Kalai paper's example is asking how many D's are in "DEEPSEEK". There is one. Across ten independent trials a 600B model answered "2" or "3"; other frontier models tested at the same time returned answers as large as "6" and "7". Every character needed was in the prompt. No amount of retrieval helps, because nothing was missing.
Extrinsic hallucinations contradict the world — an invented citation, a fabricated statistic, a misattributed quote. These are the ones grounding can attack, because the missing ingredient really is a fact the model does not have, and retrieval can supply it.
The same split appears in summarisation work under the names faithfulness (does the summary match the source?) and factuality (is it true of the world?), and the two come apart in both directions: a summary can be perfectly faithful to a document that is itself wrong. Knowing which kind you have tells you which fix is even eligible. Reaching for RAG against a letter-counting error is spending money on the wrong half of the problem.
Real-World Applications
Mata v. Avianca is the case that taught the legal profession what this word means. In a personal injury suit in the Southern District of New York, a lawyer used ChatGPT for research and filed a brief citing decisions that did not exist. When opposing counsel could not find them, he went back to ChatGPT and asked whether the cases were real; it said they were, and produced full fabricated opinion texts on request. Judge P. Kevin Castel's sanctions opinion of 22 June 2023 names six fake opinions — "Varghese", "Shaboon", "Petersen", "Martinez", "Durden" and "Miller" — and imposes a $5,000 penalty jointly on the two attorneys and their firm, along with an order to write to each real judge whose name had been attached to an invented ruling. The scale of the fine is not the point. The mechanism is: the verification step the lawyer performed was asking the model again, and a model with no separate store of facts to consult will confirm its own fabrication as readily as it produced it.
The measured rate on retrieval-backed commercial tools is far higher than their marketing implies. Magesh and colleagues at Stanford's RegLab and Institute for Human-Centered AI ran the first preregistered evaluation of AI legal research products, published in the Journal of Empirical Legal Studies in 2025. Across 202 queries put to systems whose vendors advertised "hallucination-free" citations, Lexis+ AI hallucinated on 17% of queries and Westlaw's AI-Assisted Research on 33% — better than the 43% they measured for bare GPT-4 over the same questions in 2024, and nowhere near the promised zero. Lexis+ AI, the strongest system tested, answered 65% of queries accurately. If you are deciding how much of a professional's time to budget for checking AI output, that gap between 17% and 0% is the number that decides it, and it is the reason "the tool has retrieval, so it cannot make things up" is a claim to treat as marketing.
High-stakes deployments are built around the assumption that hallucinations arrive. Ambient clinical documentation systems draft a consultation note and route it to the clinician for signature before it enters the medical record; the signature exists because the draft cannot be trusted unread. That design — assume fabrication, put a verifier in the path — is what error handling in AI systems is about, and it is the correct response to everything on this page. This page explains why the wrong answer arrives; that one explains what a system does when it does.
Key Concepts
- Plausibility is not a side effect, it is the objective. A model is trained to produce the most probable continuation, and a well-formed fake citation in the right reporter with a plausible docket number is exactly what "probable" looks like. This is why hallucinations are hard to spot: they are optimised to resemble the true thing they are standing in for.
- Confidence of phrasing carries almost no information about correctness. The model that told three different lawyers three different birthdays used identical language each time. Tone is a property of the training distribution, not evidence about the answer.
- Abstention is a capability, not a personality trait. Saying "I don't know" requires the model to have a usable internal signal that its answer is unreliable, and then to be rewarded for acting on it. Both halves have to be present; today the second is usually missing.
- Accuracy and error rate are different numbers and only one is famous. Any evaluation that reports a single accuracy figure has silently added abstentions and errors into the same bucket. Ask for the three-way split — right, wrong, declined — before comparing two models on trustworthiness.
Challenges
There is a real floor, and it is often overstated on the web. The precise claim worth defending is narrow: for facts with no learnable pattern, a pretrained model's error rate is bounded below by roughly the singleton rate of those facts in its training data. That is a statement about getting the fact right, not about confidently asserting a wrong one — and those are separable. A model that abstains on the birthdays it saw once has the same knowledge floor and a far lower hallucination rate. Claims that hallucination has been "mathematically proven impossible to eliminate" flatten this distinction and usually cite a result that proves something considerably weaker than advertised.
Retrieval moves the number without touching the incentive. Grounding a model in retrieved documents genuinely reduces extrinsic fabrication — and the Kalai analysis explicitly notes that its observation about binary grading holds for arbitrary models including those with retrieval, since the grader still rewards a guess whenever the search comes back empty or ambiguous. The 17% and 33% figures above were measured on production systems whose entire value proposition was retrieval. The Stanford team's own prompt, incidentally, instructed the tools not to hedge — an honest reflection of how these products are actually used, and a reminder that the abstention path has to be wanted by the deployment, not just available in the model.
Self-verification does not work, for a structural reason. Asking the model whether its own answer was true re-runs the same distribution that produced the answer, and it will agree with itself precisely on the confident errors you needed caught — which is exactly what happened in Mata. Useful checking needs something outside the model that can say "wrong" without being told the right answer: a compiler, a database, a retrieved source, a person.
Temperature is not the dial. Setting temperature to 0 makes generation deterministic, so the same prompt returns the same output. If that output is a fabricated case citation, you now get the same fabricated case citation every time. Determinism is reproducibility, not truth.
Future Trends
The direction the Kalai paper argues for is changing how mainstream benchmarks score, rather than adding more hallucination benchmarks alongside them. The logic is that a specialist honesty evaluation is drowned out by the dozens of accuracy-graded ones that dominate leaderboards and model cards; the leverage is in retrofitting explicit confidence targets into the evaluations already in use, SWE-bench and its peers included. Whether that happens is a coordination question about leaderboard maintainers, not a research question — which is an unusual and slightly uncomfortable place for a technical problem to sit.
A second idea likely to spread is behavioural calibration: rather than asking a model to emit a
confidence number, you set a threshold t and check whether it answers exactly the questions where
its probability of being right exceeds t, sweeping t and watching accuracy and error rates move
together. It is auditable from outside, needs no access to the model's internals, and sidesteps the
awkward fact that there are exponentially many ways to phrase a correct answer, which makes
probability-based calibration on free text hard to define at all.
Third, expect the word itself to keep narrowing. "Hallucination" currently covers arithmetic slips, fabricated citations, misread source documents and confident answers to ambiguous questions — four failure modes with four different causes and four different fixes. The vocabulary that replaces it, already visible in the intrinsic/extrinsic and faithfulness/factuality splits, will be more useful precisely because it stops treating them as one phenomenon. A great deal of trust in AI systems rests on people being able to say which kind of wrong they are worried about.