Definition
Error handling in an AI system is mostly the work of catching failures that never raise an exception. Ordinary code gives you two outcomes to program against: a function returns, or it throws. A language model has a third, and it is the common one — it returns a fluent, well-formed, confidently worded answer that is wrong. Nothing in the stack notices. There is no stack trace for a fabricated citation, no non-zero exit status for a tool call whose arguments are perfectly valid JSON and name the wrong customer record.
This is why the AI version of the subject is not the try/except article. The exception-handling
part still applies unchanged — rate limits, timeouts, a service returning 500 — and it is well
covered by forty years of distributed systems practice. The part
that is new is everything the runtime will never tell you about.
Here is the shape of the problem in arithmetic. Suppose one step of your pipeline is right 95% of
the time, which for anything involving judgement is a good step. Chain ten of them, so the final
answer depends on all ten being right, and you get 0.95¹⁰ = 59.9%. Twenty steps gives 0.95²⁰ = 35.8%. Two runs in five are wrong at ten steps, and not one of them logged an error — every
call returned 200 with a plausible payload, every parse succeeded, every dashboard is green.
That last clause is what separates this from the availability arithmetic on the distributed computing page, which multiplies the same way. There, each component fails visibly — it times out, it refuses, it returns an error — and the product tells you how often a user sees a failure. Here the product tells you how often the user gets a confident answer that happens to be false, and there is no signal anywhere in the system distinguishing that run from a good one.
How It Works
Four failure classes, ordered by how hard they are to see
Raised failures are the easy ones: a 429, a socket timeout, an out-of-memory kill. The runtime tells you, and the standard toolkit — retries with jitter, budgets, circuit breakers, idempotent operations — applies without modification.
Malformed output is next: you asked for JSON and got prose, a trailing comma, or a missing required field. Nothing threw, but a parser can decide the question mechanically, which makes this a raised error you have to raise yourself.
Well-formed but invalid output is where it gets AI-specific. A function call can satisfy its schema perfectly — right field names, right types, right enum values — and still reference an order ID that does not exist, a date outside the allowed range, or a unit the API does not accept. Detecting it needs something to check against: a database, a constraint, a second lookup.
Silently wrong output is the floor. Well-formed, schema-valid, referentially consistent, and false. A summary that inverts a clause. A number that is plausible and invented. This is the class that shares a mechanism with hallucinations — that page explains why models produce them; this one is about what a system does when they arrive.
Almost all useful engineering here consists of moving errors up that list: converting an undetectable failure into a detectable one, because only detectable failures can be retried, routed, logged or refused.
Validation at the boundary, and constrained decoding underneath it
The cheap move is to validate everything crossing a boundary: parse the JSON, check it against the schema, reject and retry on failure. The stronger move is to make the invalid output unproducible. Constrained decoding applies the schema as a mask over the token distribution at sampling time, so a token that would break the grammar has its probability zeroed before selection — the model cannot emit a trailing comma because "comma" is not among the legal next tokens.
When OpenAI shipped Structured Outputs in August 2024 it reported that gpt-4o-2024-08-06 scored
100% on its internal complex-schema-following eval, against under 40% for gpt-4-0613 relying on
prompting alone. The same machinery exists outside hosted APIs — GBNF grammars in llama.cpp,
Outlines and similar libraries — and it retires the malformed-output class entirely.
It also has a cost that is easy to miss, because it constrains the shape of an answer while the answer is still being computed. The EMNLP 2024 Industry Track paper Let Me Speak Freely? measured this on GSM8K across the mid-2024 frontier: GPT-3.5-Turbo scored 75.99% answering in free text and 49.25% under a JSON schema; Claude-3-Haiku fell from 86.51% to 23.44%; Gemini-1.5-Flash barely moved (89.33% to 89.21%). Those are all mid-2024 models — a full model generation behind the current catalog — and the gaps have narrowed on later generations, but the mechanism has not gone anywhere: reasoning needs room, and a grammar that forces the answer field early takes the room away. The measured fix is a second call — reason in prose, then convert to the schema — which buys back the accuracy for the price of one cheap extra request.
A verifier is a step whose whole job is to raise
If nothing else can see the error, add something that can. Suppose each step is correct with
probability p, and you add a checker that catches a fraction r of the incorrect outputs and
triggers one retry that succeeds at the same base rate. Effective per-step correctness becomes:
p' = p + (1 − p) · r · p
With p = 0.95 and a verifier that catches 70% of errors, p' = 0.983, and a ten-step run climbs
from 59.9% to 84.5%. A perfect verifier would give 97.5%. And with r = 0 you get exactly p
back — nothing. That identity is the most important thing on this page: retries do not improve
correctness, detection does. Retrying a hallucination buys a second, differently worded
hallucination.
The cost side is favourable more often than teams assume, because checking an answer means reading it while producing it meant writing it, and output tokens are the expensive ones everywhere. A ten-step run generating 500 tokens per step emits 5,000 output tokens; adding a 100-token verdict per step adds 1,000, about 20% more output, to move the run from 59.9% to 84.5% correct. Whether that is a good trade depends entirely on what a wrong answer costs you — the number most teams have never written down, and the one that should decide the whole design.
Failing closed
When no oracle exists, the last lever is what you do with an answer you could not verify. Shipping it is failing open, and it is the default in most LLM products — not because anyone chose it, but because nobody chose. Failing closed means refusing, escalating, or returning "I could not confirm this", and it needs two things the model cannot supply on its own: an abstention signal, and somewhere for the escalated case to go. The second is almost always the harder engineering problem, and it is a product decision rather than a machine-learning one.
Real-World Applications
Coding agents work because the domain ships with an oracle. A compiler, a type checker and a test suite will all say "wrong" without being told the right answer, so a coding agent can generate, run, read the failure and revise — converting silent errors into raised ones on every iteration. This is the single best explanation for why software engineering is the most reliable agent domain today and why a research or planning agent with the same model underneath is far less dependable: nothing in its environment ever objects.
Constrained decoding deleted a class of bug rather than mitigating it. Before August 2024, every production LLM integration carried retry-on-parse-failure code and a regex to strip markdown fences from JSON. Schema enforcement at the provider, and GBNF grammars in local runtimes, made that code dead — a rare outright elimination in a field that mostly mitigates.
τ-bench was designed around exactly this failure. Sierra's 2024 benchmark puts an agent in
airline and retail customer-service scenarios and grades the final database state against an
annotated goal, not the conversation transcript — so a confidently wrong action scores as wrong.
Its pass^k metric asks whether all k independent attempts at a task succeed. An agent above 60%
on a single attempt dropped below 25% at pass^8 in the retail domain. Averaged success hid an
inconsistency that only a reliability-shaped metric could see.
Retrieval systems verify against the passage they retrieved.
RAG pipelines increasingly run a grounding check —
does each claim in the answer appear in the retrieved context? — which is a verifier with a real,
domain-specific oracle attached. Its r is well below 1, since a claim can be grounded and still
misleading, but it is decisively better than zero, and it is why citation-linked answers are worth
the extra pass.
Clinical documentation puts a human in the abstention path by design. Ambient clinical documentation systems draft a note from a consultation and route it to the clinician for signature before anything enters the record. The signature is the verifier, the refusal path is "edit it", and the regulatory posture makes failing closed non-optional. It is the shape most high-stakes deployments converge on.
Key Concepts
- An oracle is anything that can say "wrong" without knowing the right answer. A parser, a compiler, a unit test, a schema, a database lookup, a retrieved source, a second model, a human. The reliability ceiling of an AI system is set largely by which oracles exist in its domain — not by the model, which is why the same model is trustworthy at code and unreliable at research.
- Detectable and correctable are not the same property. A checker that flags 90% of bad answers but cannot produce a good one still earns its keep: it converts a silent error into a refusal, which is a far cheaper failure than a confident one.
- Budget error per step, not per system. Invert the compounding to size a step: hitting 90% over
a twenty-step run needs
0.9^(1/20) = 99.47%per step. If your steps are at 95%, twenty steps is not a target you can reach by prompting — you have to shorten the chain or add verifiers. - Confidence is not calibration. A model's certainty of phrasing carries almost no information about correctness, and its self-reported confidence scores are only as good as their measured calibration. Treat both as inputs to a check, never as the check.
Challenges
The verifier is usually the same model, and its errors are correlated. Asking a model to grade
its own output shares the priors and the training data that produced the mistake, so it agrees with
itself on precisely the cases you needed caught. Real r is far lower than a demo suggests,
because the demo tested on errors someone injected deliberately — which are, by construction, the
kind a checker finds.
Fixing the visible errors can hide a drop in the invisible ones. Turn on constrained decoding and your parse-failure rate goes to zero while task accuracy may quietly fall, as the GSM8K figures above show. If the dashboard tracks malformed responses and not correctness, that reads as an unambiguous improvement. Every reliability intervention needs an outcome metric, not just an error metric — which is where this page hands off to monitoring.
There is no stack trace, so localisation is manual. In a ten-step agentic workflow, step 7's output is wrong because step 3 invented a customer ID and steps 4 through 6 faithfully carried it forward. Nothing errored, nothing branched, and no tool refused. Finding the origin means a human reading the whole transcript, which is why step-level traces and intermediate-output logging are not optional instrumentation here.
Non-determinism defeats the usual definition of "fixed". At any temperature
above zero the same prompt can produce a different answer, so a bug may not reproduce. Worse, the
absence of a bug proves very little: a 5% failure rate survives twenty clean test runs 0.95²⁰ = 35.8% of the time. More than a third of "we fixed it, it passed twenty times" conclusions about a
one-in-twenty bug are simply wrong.
Failing closed has a cost nobody budgets. Every abstention becomes a task a human has to do, and an agent that refuses 15% of the time needs a queue, a rota and a service level for that 15%. Teams routinely ship the refusal path and discover afterwards that it has nowhere to land — at which point the operational pressure is to loosen the threshold, and the check quietly stops firing.
Future Trends
The clearest direction is reshaping tasks so that an oracle exists, rather than waiting for models that need none. Asking for SQL instead of an answer, code instead of a calculation, an extraction with span offsets into a source document instead of a free summary — each converts a judgement you cannot check into an artefact you can execute, diff or trace back. The design question is shifting from "can the model do this?" to "how will I know if it did?"
Second, verification is moving from the end of the run to every step. Process reward models and
step-level critics score intermediate reasoning rather than the final answer, which is what the
p' = p + (1 − p)·r·p arithmetic rewards: catching an error at step 3 costs one retry, catching it
at step 10 costs the whole run. Expect frameworks to make per-step checks the default rather than a
pattern each team reinvents.
Third, abstention is becoming a first-class output. Structured refusals, "insufficient evidence" as a legitimate schema branch, and calibrated confidence exposed by the API are all early, and all point the same way: a model that reliably says "I don't know" is worth more in a pipeline than one a few points better on average, because the first can be routed and the second cannot.
Code Example
The three pieces of arithmetic above, computed rather than asserted. None of it is about a particular model — it is about the shape of a pipeline, which is why it stays true as models change.
def chain_correct(p: float, steps: int) -> float:
"""Probability every step in the chain is right."""
return p ** steps
def with_verifier(p: float, recall: float) -> float:
"""Per-step correctness when a checker catches `recall` of errors and retries once."""
return p + (1 - p) * recall * p
def step_budget(target: float, steps: int) -> float:
"""Per-step correctness needed to hit `target` over `steps` steps."""
return target ** (1 / steps)
for n in (5, 10, 20):
print(f"{n:>2} steps at 95%: {chain_correct(0.95, n):.1%}")
for r in (0.0, 0.5, 0.7, 1.0):
p = with_verifier(0.95, r)
print(f"verifier recall {r:.0%}: step {p:.3%}, 10-step run {chain_correct(p, 10):.1%}")
print(f"to reach 90% over 20 steps, each step must be {step_budget(0.90, 20):.2%}")
# 5 steps at 95%: 77.4%
# 10 steps at 95%: 59.9%
# 20 steps at 95%: 35.8%
# verifier recall 0%: step 95.000%, 10-step run 59.9%
# verifier recall 50%: step 97.375%, 10-step run 76.6%
# verifier recall 70%: step 98.325%, 10-step run 84.5%
# verifier recall 100%: step 99.750%, 10-step run 97.5%
# to reach 90% over 20 steps, each step must be 99.47%
Read the middle block as the design rule. The first line and the last differ only in whether something in the loop can tell a wrong answer from a right one; the model, the prompt and the number of retries are identical. That is where the effort belongs.