---
source: 'https://howaiworks.ai/glossary/error-handling'
section: glossary
title: Error Handling in AI Systems
description: >-
  Ordinary code fails by raising an exception. A model fails by returning a
  confident, well-formed, wrong answer that nothing catches. What to do about
  it.
tags:
  - reliability
  - AI Agents
  - Tool Calling
  - fault tolerance
  - hallucinations
  - AI Safety
  - LLM
category: Software Development
datePublished: '2025-08-17'
lastUpdated: '2026-07-24'
---

# Error Handling in AI Systems

> Ordinary code fails by raising an exception. A model fails by returning a confident, well-formed, wrong answer that nothing catches. What to do about it.

## Definition

**Error handling in an AI system is mostly the work of catching failures that never raise an
exception.** Ordinary code gives you two outcomes to program against: a function returns, or it
throws. A [language model](https://howaiworks.ai/glossary/large-language-model) has a third, and it is the common one —
it returns a fluent, well-formed, confidently worded answer that is wrong. Nothing in the stack
notices. There is no stack trace for a fabricated citation, no non-zero exit status for a tool call
whose arguments are perfectly valid JSON and name the wrong customer record.

This is why the AI version of the subject is not the `try`/`except` article. The exception-handling
part still applies unchanged — rate limits, timeouts, a service returning 500 — and it is well
covered by forty years of [distributed systems](https://howaiworks.ai/glossary/distributed-computing) practice. The part
that is new is everything the runtime will never tell you about.

Here is the shape of the problem in arithmetic. Suppose one step of your pipeline is right 95% of
the time, which for anything involving judgement is a good step. Chain ten of them, so the final
answer depends on all ten being right, and you get `0.95¹⁰ = 59.9%`. Twenty steps gives `0.95²⁰ =
35.8%`. Two runs in five are wrong at ten steps, and **not one of them logged an error** — every
call returned 200 with a plausible payload, every parse succeeded, every dashboard is green.

That last clause is what separates this from the availability arithmetic on the
[distributed computing](https://howaiworks.ai/glossary/distributed-computing) page, which multiplies the same way. There,
each component fails *visibly* — it times out, it refuses, it returns an error — and the product
tells you how often a user sees a failure. Here the product tells you how often the user gets a
confident answer that happens to be false, and there is no signal anywhere in the system
distinguishing that run from a good one.

## How It Works

### Four failure classes, ordered by how hard they are to see

**Raised failures** are the easy ones: a 429, a socket timeout, an out-of-memory kill. The runtime
tells you, and the standard toolkit — retries with jitter, budgets, circuit breakers, idempotent
operations — applies without modification.

**Malformed output** is next: you asked for JSON and got prose, a trailing comma, or a missing
required field. Nothing threw, but a parser can decide the question mechanically, which makes this
a raised error you have to raise yourself.

**Well-formed but invalid** output is where it gets AI-specific. A [function
call](https://howaiworks.ai/glossary/function-calling) can satisfy its schema perfectly — right field names, right types,
right enum values — and still reference an order ID that does not exist, a date outside the allowed
range, or a unit the API does not accept. Detecting it needs something to check *against*: a
database, a constraint, a second lookup.

**Silently wrong** output is the floor. Well-formed, schema-valid, referentially consistent, and
false. A summary that inverts a clause. A number that is plausible and invented. This is the class
that shares a mechanism with [hallucinations](https://howaiworks.ai/glossary/hallucinations) — that page explains why
models produce them; this one is about what a system does when they arrive.

Almost all useful engineering here consists of **moving errors up that list**: converting an
undetectable failure into a detectable one, because only detectable failures can be retried,
routed, logged or refused.

### Validation at the boundary, and constrained decoding underneath it

The cheap move is to validate everything crossing a boundary: parse the JSON, check it against the
schema, reject and retry on failure. The stronger move is to make the invalid output unproducible.
**Constrained decoding** applies the schema as a mask over the token distribution at sampling time,
so a token that would break the grammar has its probability zeroed before selection — the model
cannot emit a trailing comma because "comma" is not among the legal next
[tokens](https://howaiworks.ai/glossary/token).

When OpenAI shipped Structured Outputs in August 2024 it reported that `gpt-4o-2024-08-06` scored
100% on its internal complex-schema-following eval, against under 40% for `gpt-4-0613` relying on
prompting alone. The same machinery exists outside hosted APIs — GBNF grammars in llama.cpp,
Outlines and similar libraries — and it retires the malformed-output class entirely.

It also has a cost that is easy to miss, because **it constrains the shape of an answer while the
answer is still being computed**. The EMNLP 2024 Industry Track paper *Let Me Speak Freely?*
measured this on GSM8K across the mid-2024 frontier: GPT-3.5-Turbo scored 75.99% answering in free
text and 49.25% under a JSON schema; Claude-3-Haiku fell from 86.51% to 23.44%; Gemini-1.5-Flash
barely moved (89.33% to 89.21%). Those are all mid-2024 models — a full model generation behind the
current catalog — and the gaps have narrowed on later generations, but the mechanism has not gone
anywhere: reasoning needs room, and a grammar that forces the answer field early takes the room
away. The measured fix is a second call — reason in prose, then convert to the schema — which
buys back the accuracy for the price of one cheap extra request.

### A verifier is a step whose whole job is to raise

If nothing else can see the error, add something that can. Suppose each step is correct with
probability `p`, and you add a checker that catches a fraction `r` of the incorrect outputs and
triggers one retry that succeeds at the same base rate. Effective per-step correctness becomes:

`p' = p + (1 − p) · r · p`

With `p = 0.95` and a verifier that catches 70% of errors, `p' = 0.983`, and a ten-step run climbs
from **59.9% to 84.5%**. A perfect verifier would give 97.5%. And with `r = 0` you get exactly `p`
back — nothing. That identity is the most important thing on this page: **retries do not improve
correctness, detection does.** Retrying a hallucination buys a second, differently worded
hallucination.

The cost side is favourable more often than teams assume, because checking an answer means reading
it while producing it meant writing it, and output tokens are the expensive ones everywhere. A
ten-step run generating 500 tokens per step emits 5,000 output tokens; adding a 100-token verdict
per step adds 1,000, about **20% more output**, to move the run from 59.9% to 84.5% correct.
Whether that is a good trade depends entirely on what a wrong answer costs you — the number most
teams have never written down, and the one that should decide the whole design.

### Failing closed

When no oracle exists, the last lever is what you do with an answer you could not verify. Shipping
it is *failing open*, and it is the default in most LLM products — not because anyone chose it, but
because nobody chose. *Failing closed* means refusing, escalating, or returning "I could not
confirm this", and it needs two things the model cannot supply on its own: an abstention signal,
and somewhere for the escalated case to go. The second is almost always the harder engineering
problem, and it is a product decision rather than a machine-learning one.

## Real-World Applications

**Coding agents work because the domain ships with an oracle.** A compiler, a type checker and a
test suite will all say "wrong" without being told the right answer, so a coding agent can
generate, run, read the failure and revise — converting silent errors into raised ones on every
iteration. This is the single best explanation for why software engineering is the most reliable
agent domain today and why a research or planning [agent](https://howaiworks.ai/glossary/ai-agent) with the same model
underneath is far less dependable: nothing in its environment ever objects.

**Constrained decoding deleted a class of bug rather than mitigating it.** Before August 2024,
every production LLM integration carried retry-on-parse-failure code and a regex to strip markdown
fences from JSON. Schema enforcement at the provider, and GBNF grammars in local runtimes, made
that code dead — a rare outright elimination in a field that mostly mitigates.

**τ-bench was designed around exactly this failure.** Sierra's 2024 benchmark puts an agent in
airline and retail customer-service scenarios and grades the **final database state** against an
annotated goal, not the conversation transcript — so a confidently wrong action scores as wrong.
Its `pass^k` metric asks whether *all* k independent attempts at a task succeed. An agent above 60%
on a single attempt dropped below 25% at `pass^8` in the retail domain. Averaged success hid an
inconsistency that only a reliability-shaped metric could see.

**Retrieval systems verify against the passage they retrieved.**
[RAG](https://howaiworks.ai/glossary/retrieval-augmented-generation) pipelines increasingly run a grounding check —
does each claim in the answer appear in the retrieved context? — which is a verifier with a real,
domain-specific oracle attached. Its `r` is well below 1, since a claim can be grounded and still
misleading, but it is decisively better than zero, and it is why citation-linked answers are worth
the extra pass.

**Clinical documentation puts a human in the abstention path by design.** [Ambient clinical
documentation](https://howaiworks.ai/glossary/ambient-clinical-documentation) systems draft a note from a consultation
and route it to the clinician for signature before anything enters the record. The signature is the
verifier, the refusal path is "edit it", and the regulatory posture makes failing closed
non-optional. It is the shape most high-stakes deployments converge on.

## Key Concepts

- **An oracle is anything that can say "wrong" without knowing the right answer.** A parser, a
  compiler, a unit test, a schema, a database lookup, a retrieved source, a second model, a human.
  The reliability ceiling of an AI system is set largely by which oracles exist in its domain — not
  by the model, which is why the same model is trustworthy at code and unreliable at research.
- **Detectable and correctable are not the same property.** A checker that flags 90% of bad answers
  but cannot produce a good one still earns its keep: it converts a silent error into a refusal,
  which is a far cheaper failure than a confident one.
- **Budget error per step, not per system.** Invert the compounding to size a step: hitting 90% over
  a twenty-step run needs `0.9^(1/20) = 99.47%` per step. If your steps are at 95%, twenty steps is
  not a target you can reach by prompting — you have to shorten the chain or add verifiers.
- **Confidence is not calibration.** A model's certainty of phrasing carries almost no information
  about correctness, and its self-reported confidence scores are only as good as their measured
  calibration. Treat both as inputs to a check, never as the check.

## Challenges

**The verifier is usually the same model, and its errors are correlated.** Asking a model to grade
its own output shares the priors and the training data that produced the mistake, so it agrees with
itself on precisely the cases you needed caught. Real `r` is far lower than a demo suggests,
because the demo tested on errors someone injected deliberately — which are, by construction, the
kind a checker finds.

**Fixing the visible errors can hide a drop in the invisible ones.** Turn on [constrained decoding](https://howaiworks.ai/glossary/structured-outputs)
and your parse-failure rate goes to zero while task accuracy may quietly fall, as the GSM8K figures
above show. If the dashboard tracks malformed responses and not correctness, that reads as an
unambiguous improvement. Every reliability intervention needs an outcome metric, not just an error
metric — which is where this page hands off to [monitoring](https://howaiworks.ai/glossary/monitoring).

**There is no stack trace, so localisation is manual.** In a ten-step [agentic
workflow](https://howaiworks.ai/glossary/agentic-workflow), step 7's output is wrong because step 3 invented a customer
ID and steps 4 through 6 faithfully carried it forward. Nothing errored, nothing branched, and no
tool refused. Finding the origin means a human reading the whole transcript, which is why
step-level traces and intermediate-output logging are not optional instrumentation here.

**Non-determinism defeats the usual definition of "fixed".** At any [temperature](https://howaiworks.ai/glossary/temperature)
above zero the same prompt can produce a different answer, so a bug may not reproduce. Worse, the
absence of a bug proves very little: a 5% failure rate survives twenty clean test runs `0.95²⁰ =
35.8%` of the time. More than a third of "we fixed it, it passed twenty times" conclusions about a
one-in-twenty bug are simply wrong.

**Failing closed has a cost nobody budgets.** Every abstention becomes a task a human has to do,
and an agent that refuses 15% of the time needs a queue, a rota and a service level for that
15%. Teams routinely ship the refusal path and discover afterwards that it has nowhere to
land — at which point the operational pressure is to loosen the threshold, and the check quietly
stops firing.

## Future Trends

The clearest direction is **reshaping tasks so that an oracle exists**, rather than waiting for
models that need none. Asking for SQL instead of an answer, code instead of a calculation, an
extraction with span offsets into a source document instead of a free summary — each converts a
judgement you cannot check into an artefact you can execute, diff or trace back. The design
question is shifting from "can the model do this?" to "how will I know if it did?"

Second, **verification is moving from the end of the run to every step**. Process reward models and
step-level critics score intermediate reasoning rather than the final answer, which is what the
`p' = p + (1 − p)·r·p` arithmetic rewards: catching an error at step 3 costs one retry, catching it
at step 10 costs the whole run. Expect frameworks to make per-step checks the default rather than a
pattern each team reinvents.

Third, **abstention is becoming a first-class output**. Structured refusals, "insufficient
evidence" as a legitimate schema branch, and calibrated confidence exposed by the API are all
early, and all point the same way: a model that reliably says "I don't know" is worth more in a
pipeline than one a few points better on average, because the first can be routed and the second
cannot.

## Code Example

The three pieces of arithmetic above, computed rather than asserted. None of it is about a
particular model — it is about the shape of a pipeline, which is why it stays true as models
change.

```python
def chain_correct(p: float, steps: int) -> float:
    """Probability every step in the chain is right."""
    return p ** steps

def with_verifier(p: float, recall: float) -> float:
    """Per-step correctness when a checker catches `recall` of errors and retries once."""
    return p + (1 - p) * recall * p

def step_budget(target: float, steps: int) -> float:
    """Per-step correctness needed to hit `target` over `steps` steps."""
    return target ** (1 / steps)

for n in (5, 10, 20):
    print(f"{n:>2} steps at 95%: {chain_correct(0.95, n):.1%}")

for r in (0.0, 0.5, 0.7, 1.0):
    p = with_verifier(0.95, r)
    print(f"verifier recall {r:.0%}: step {p:.3%}, 10-step run {chain_correct(p, 10):.1%}")

print(f"to reach 90% over 20 steps, each step must be {step_budget(0.90, 20):.2%}")

#  5 steps at 95%: 77.4%
# 10 steps at 95%: 59.9%
# 20 steps at 95%: 35.8%
# verifier recall 0%: step 95.000%, 10-step run 59.9%
# verifier recall 50%: step 97.375%, 10-step run 76.6%
# verifier recall 70%: step 98.325%, 10-step run 84.5%
# verifier recall 100%: step 99.750%, 10-step run 97.5%
# to reach 90% over 20 steps, each step must be 99.47%
```

Read the middle block as the design rule. The first line and the last differ only in whether
something in the loop can tell a wrong answer from a right one; the model, the prompt and the
number of retries are identical. That is where the effort belongs.

## Frequently Asked Questions

### How is error handling in an AI system different from ordinary error handling?

Ordinary code has two outcomes you can program against — it returns or it throws. A model has a third and it is the common one: it returns a fluent, well-formed answer that is wrong, with no exception, no error code and no stack trace. Most of the work is inventing a way to detect that outcome, because the runtime will never tell you about it.

### Why do small error rates matter so much in an agent?

Because they multiply. A step that is right 95% of the time, chained ten times, gives a run that is right 0.95^10 = 59.9% of the time; twenty steps gives 35.8%. Nothing failed, no step raised anything, and four runs in ten are wrong.

### Does retrying fix an AI error?

Only if something detected it. Retrying a rate limit or a malformed JSON payload works because a parser or an HTTP status said 'wrong'. Retrying a hallucination just buys a second, differently worded hallucination, because nothing in the loop knows the first one was false.

### Does constrained decoding solve the problem?

It solves one class of it. Masking invalid tokens at sampling time makes schema violations impossible, and OpenAI reported moving from under 40% to 100% on its complex-schema eval when it shipped Structured Outputs in August 2024. It guarantees the shape of the answer, never its truth, and forcing the shape too early can cost accuracy.

### What is an oracle in this context?

Anything that can say 'wrong' without being told the right answer: a JSON parser, a compiler, a type checker, a unit test, a database lookup, a retrieved source document, a human reviewer. The reliability ceiling of an AI system is set mostly by which oracles exist in its domain.

### What does it mean to fail closed?

To refuse or escalate rather than ship an answer you could not verify. It costs coverage and needs somewhere to escalate to, which is usually a product decision rather than a model one — but shipping the unverified answer is also a decision, just an unexamined one.

## Related

### Related terms

- [AI Hallucinations](https://howaiworks.ai/glossary/hallucinations)
- [Robustness](https://howaiworks.ai/glossary/robustness)
- [Function Calling (Tool Calling)](https://howaiworks.ai/glossary/function-calling)
- [Agentic Workflow](https://howaiworks.ai/glossary/agentic-workflow)
- [Monitoring](https://howaiworks.ai/glossary/monitoring)
- [Distributed Computing](https://howaiworks.ai/glossary/distributed-computing)

---

Source: https://howaiworks.ai/glossary/error-handling — HowAIWorks.ai