---
source: 'https://howaiworks.ai/glossary/structured-outputs'
section: glossary
title: Structured Outputs (Constrained Decoding)
description: >-
  Constrained decoding masks a language model's token distribution against a
  JSON schema or grammar at each step, so the output is parseable by
  construction.
tags:
  - LLM
  - Tool Calling
  - reliability
  - Tokens
  - Inference
  - Developer Tools
category: Software Development
datePublished: '2026-07-24'
lastUpdated: '2026-07-24'
---

# Structured Outputs (Constrained Decoding)

> Constrained decoding masks a language model's token distribution against a JSON schema or grammar at each step, so the output is parseable by construction.

## Definition

**Structured outputs** are the guarantee that a language model's response is not just
probably-well-formed but *provably* parseable — valid JSON, or text matching a supplied grammar,
every time. The mechanism that delivers the guarantee is **constrained decoding** (also called
grammar-constrained or guided decoding): at each step of generation, the sampler is only allowed to
pick a token that keeps the output valid against the target schema. Illegal tokens are removed
before anything is sampled, so the model *cannot* produce a trailing comma, an unquoted key, or a
field of the wrong type.

That is a different thing from asking nicely. "Return JSON and nothing else" in the prompt is a
request the model can decline — under load, on a long output, or when the schema is deep, it
sometimes wraps the JSON in prose, truncates a brace, or invents a field. Constrained decoding
makes that class of failure structurally impossible rather than merely unlikely, which is why it is
the difference between a parser you have to wrap in a retry loop and one that never throws.

The catch, and the reason this is a real engineering decision rather than a free upgrade, is that a
constraint operates on the answer *while the answer is still being computed*. It can guarantee the
shape of a response without improving — and sometimes while degrading — what the response actually
says.

## How It Works

A language model generates one token at a time. At each step it produces a vector of logits, one
score per entry in its vocabulary (tens of thousands of entries), which a softmax turns into a
probability distribution; the sampler then draws the next [token](https://howaiworks.ai/glossary/token) from that
distribution. Ordinary "prompt-and-hope" generation samples from the full distribution and trusts
the model to have learned to stay inside JSON. Nothing enforces it.

Constrained decoding inserts one step in between. A constraint engine tracks the output produced so
far, works out the set of tokens that would keep it valid against the schema or grammar, and sets
the logits of every other token to negative infinity — a **mask**. After the softmax, a masked
token has probability exactly zero, so it can never be drawn no matter how the sampler is
configured. This is worth stressing because it is the source of the guarantee: masking happens
*before* sampling, so [temperature](https://howaiworks.ai/glossary/temperature), top-p and every other sampling knob
choose only among tokens the grammar already permits. If the partial output is `{"age":` and the
schema says `age` is an integer, the only legal next tokens are digits (and perhaps a minus sign) —
the token for `"` is masked, so the model physically cannot open a string where a number belongs.

The schema is not checked with an `if` statement per token, which would be far too slow. It is
compiled ahead of time into a formal grammar and usually a **finite-state machine**, so that "which
tokens are legal in this state" is a lookup rather than a computation. The widely-cited *Efficient
Guided Generation for Large Language Models* paper (the Outlines method) makes this its central
result: by indexing the state machine against the vocabulary once, up front, the per-token cost of
enforcing the constraint becomes *O(1) on average* — a constant, independent of vocabulary size —
where the naive approach of scanning the whole vocabulary at every step is O(N). The paper's own
summary is that the technique "adds little overhead to the token sequence generation process." That
efficiency is what makes the guarantee practical to switch on for every request instead of only the
ones that failed.

### Why prompt-and-hope pays a retry tax and constraints do not

The value of turning "usually valid" into "always valid" is easiest to see on the retry budget it
eliminates. Suppose unconstrained JSON generation on some moderately nested schema is malformed at
a rate of `f = 0.05` — five parses in a hundred fail — and suppose failures are independent, so a
retry fails at the same rate. The expected number of attempts to get one clean parse is a geometric
series, `1 / (1 − f) = 1 / 0.95 = 1.0526`. That is a **5.26% tax** on every generation on the JSON
path: 5.26% more calls, more output tokens, and more wall-clock latency, just to reach the output
you asked for the first time.

The average understates the problem, because what actually hurts is the tail. The chance an item is
*still* malformed after you have retried it is `f` raised to the number of attempts:

- after 1 retry (2 attempts): `0.05² = 0.0025` → **2,500 malformed per million** items
- after 2 retries: `0.05³ = 0.000125` → **125 per million**
- after 3 retries: `0.05⁴ ≈ 6.25 × 10⁻⁶` → **about 6 per million**

So even a fairly aggressive cap of three retries leaves a handful of items per million that never
parse and must fall through to a human, a default, or a dropped record — and it triples-plus the
latency of every unlucky item along the way. Push the failure rate to `f = 0.20`, which a deep or
unusual schema can easily reach, and the arithmetic gets ugly fast: `1 / 0.8 = 1.25` means a **25%**
tax on the JSON path, and `0.20⁴ = 0.0016` leaves **1,600 malformed per million** even after three
retries.

Constrained decoding drives `f` for that failure class to **zero**. Every sample is parseable by
construction, so the retry loop, the tax, and the tail all disappear together — not shrink, vanish.
That is the whole case for it: not that it makes malformed output rare, but that it makes it
impossible, and impossible is the only failure rate you never have to write fallback code for.

## Real-World Applications

**Hosted API strict modes.** When OpenAI shipped Structured Outputs in August 2024 it reported that
`gpt-4o-2024-08-06` with the feature enabled scored 100% on its internal complex-schema-following
evaluation, against under 40% for an earlier model (`gpt-4-0613`) relying on prompting alone — a
provider figure, but the gap is the point: prompting was a coin-flip on hard schemas and the
constraint was not. The same idea is exposed through the API in two places at once — a strict
`response_format` for the final answer, and a strict flag on tool/function definitions so a
[function call's](https://howaiworks.ai/glossary/function-calling) arguments are guaranteed to match the declared
parameter schema. (Which providers offer strict guarantees, and for which schema features, changes
release to release; treat any specific behavior as dated to when you read it.)

**Local and open-source inference.** The mechanism is not a hosted-API secret. `llama.cpp` ships
GBNF grammars, a compact grammar format you hand to the runtime to constrain output token by token;
Outlines, Guidance and XGrammar provide the same for the Python ecosystem; and serving stacks such
as vLLM expose "guided decoding" that accepts a JSON schema or regular expression directly. An
engineer running an open-weights model on their own hardware makes exactly the decision this page
is about: pass a grammar and get a guarantee, or prompt and wrap the result in a validator.

**The boundary of an agent or pipeline.** Anywhere one program has to read a model's output
mechanically — a step in an agent loop populating the arguments for the next tool call, a data
extraction job writing rows into a database, a router picking one of a fixed set of labels — the
receiving code needs the output to parse and to match a shape. Constrained decoding is how that
boundary stops being a source of exceptions. This is the same territory covered by
[error handling in AI systems](https://howaiworks.ai/glossary/error-handling): the strongest form of handling a malformed
output is to make it unproducible in the first place.

## Key Concepts

**Strict mode versus JSON mode.** These are often confused and are not the same guarantee. A plain
"JSON mode" promises only that the output is *syntactically* valid JSON — it will parse — but says
nothing about *which* JSON: fields can be missing, extra, or the wrong type. A strict *schema* mode
constrains the output to a specific structure, so the promise is both "it parses" and "it has the
fields and types you declared." The first removes syntax errors; the second removes schema
violations on top of them.

**Shape is not truth.** Constrained decoding removes two failure classes and leaves a third
untouched. *Malformed* output (won't parse) and *schema-invalid* output (parses, wrong shape) both
become impossible. *Well-formed but wrong* output does not: a model can emit `{"order_id": "A-40021",
"refund": 250.00}` that validates perfectly and refers to an order that does not exist, a refund
outside policy, or a currency the API rejects. Detecting that still requires checking the values
against something real — a database, a constraint, a second lookup — which no grammar can do,
because the grammar only knows the schema, not the world.

**Where the constraint binds, the model stops choosing.** Every token the mask forbids is a token
the model's own distribution no longer influences. Usually that is exactly what you want — you did
not want the trailing comma either. But it means a badly-designed schema can force the model down a
path it would not have taken, which is the mechanism behind the quality effects below.

## Challenges

**Schema rigidity forces early commitment.** A schema that puts the answer field first makes the
model emit its conclusion before it has generated any of the reasoning that would have supported it.
For a [reasoning model](https://howaiworks.ai/glossary/reasoning-model) or any chain-of-thought prompt, that is
removing the room where the actual work happens. The design tension is real: the schema you want for
clean downstream parsing (answer first, tightly typed) is often the opposite of the schema that lets
the model think (a free-text `reasoning` field before a `final_answer` field).

**Measured quality regressions under tight constraints.** This is not hypothetical. The EMNLP 2024
Industry Track paper *Let Me Speak Freely?* measured it on GSM8K across the mid-2024 frontier:
GPT-3.5-Turbo scored 75.99% answering in free text and 49.25% under a JSON schema; Claude-3-Haiku
fell from 86.51% to 23.44%; Gemini-1.5-Flash barely moved (89.33% to 89.21%). Two things are load-
bearing about those numbers. First, the effect is model- and schema-dependent — one model lost
almost nothing while another lost two-thirds of its accuracy — so you cannot assume constraints are
free *or* assume they are ruinous without measuring your own case. Second, these are mid-2024
models, a full generation behind the current catalog, and the gaps have narrowed on later
generations; but the mechanism that caused them has not gone anywhere, because reasoning still needs
room and a grammar that forces the answer early still takes it away.

**The two-pass fix, and what it costs.** The standard mitigation is to separate thinking from
formatting: let the model reason in unconstrained prose, then make a second, cheap call that
converts that reasoning into the strict schema. It buys back most of the lost accuracy for the price
of one extra request — but it is a request, so it doubles the round trips and reintroduces some of
the latency the guarantee was supposed to save. A related, cheaper option is to build the reasoning
*into* the schema as a leading free-text field, so a single constrained call still lets the model
narrate before it commits.

**Grammar coverage and portability.** Real JSON Schema is large — nested `oneOf`, recursive
definitions, regex-constrained strings, unbounded arrays — and constraint engines support different
subsets of it. A schema that a hosted API accepts may not compile under a local GBNF grammar, and
vice versa, so a system that treats "structured output" as one portable feature can break when it
changes backends. Check what your specific engine supports rather than assuming the full spec.

## Future Trends

The clearest direction is standardization: JSON Schema is becoming the common interface for
constraints across hosted APIs and open-source runtimes, which is what will eventually let a single
schema drive OpenAI, a local `llama.cpp` model, and a vLLM deployment without rewriting it three
ways — the portability gap in the previous section is the thing being closed. The second is the
maturing of "reason-then-format" as a default rather than a workaround: as reasoning models make
extended thinking the norm, the tooling is moving toward separating an unconstrained thinking phase
from a constrained output phase automatically, so a developer gets both the guarantee and the
reasoning room without hand-building a two-pass pipeline. Both trends point the same way — toward
constrained decoding being the ordinary case for machine-consumed output, with the quality cost
engineered down rather than accepted.

## Code Example

The heart of constrained decoding is a single masking step applied to the logits before sampling.
This toy version shows the mechanism directly: the grammar engine supplies the set of token ids that
are legal in the current state, and every other token's score is driven to negative infinity so its
post-softmax probability is exactly zero.

```python
import math

def mask_logits(logits, legal_token_ids):
    """logits: dict of {token_id: score} for the whole vocabulary.
    legal_token_ids: the set of ids the compiled grammar allows as the
    NEXT token, given everything generated so far."""
    return {
        tid: (score if tid in legal_token_ids else -math.inf)
        for tid, score in logits.items()
    }

def softmax(logits):
    # -inf logits contribute exp(-inf) = 0, i.e. zero probability.
    m = max(v for v in logits.values() if v != -math.inf)
    exps = {t: math.exp(v - m) if v != -math.inf else 0.0
            for t, v in logits.items()}
    total = sum(exps.values())
    return {t: e / total for t, e in exps.items()}

# Vocabulary: after `{"age":` the grammar permits only digits and a minus
# sign; the double-quote token that would (wrongly) open a string is illegal.
raw_logits = {"0": 2.1, "1": 3.4, "2": 1.0, "-": 0.5, '"': 4.9}
legal = {"0", "1", "2", "-"}          # supplied by the FSM for this state

probs = softmax(mask_logits(raw_logits, legal))
for token, p in sorted(probs.items(), key=lambda kv: -kv[1]):
    print(f"{token!r:>4}  {p:.4f}")
```

Running it prints:

```text
 '1'  0.7051
 '0'  0.1922
 '2'  0.0640
 '-'  0.0388
 '"'  0.0000
```

The `"` token had the highest raw score of all — 4.9, the value the unconstrained model most wanted
to emit — and constrained decoding still gives it probability `0.0000`. That is the guarantee made
concrete: the illegal token is not made *unlikely*, it is made *impossible*, and no temperature or
top-p setting can bring a zero-probability token back into play. A real engine differs only in
scale: the vocabulary is tens of thousands of tokens, `legal` is produced by a compiled finite-state
machine in O(1) per step rather than written by hand, and the masking runs on every one of the
hundreds of tokens in the response — but the operation the reader just watched is the whole idea.

## Frequently Asked Questions

### What is the difference between structured outputs and function calling?

Function calling is about the model deciding which tool to invoke and with what arguments. Structured outputs (a provider's strict mode) is the separate guarantee that whatever JSON comes back validates against a schema. Modern APIs apply the same constrained-decoding machinery to both, so a strict function call and a strict response share the same mechanism.

### Does constrained decoding guarantee a correct answer?

No. It guarantees parseable, schema-valid output — it removes the malformed and schema-invalid failure classes, not the well-formed-but-wrong class. A field can pass every type and enum check and still be factually false or reference an ID that does not exist.

### Can forcing a schema hurt answer quality?

It can. Constraining the shape of an answer while the answer is still being computed can cut a model's reasoning room; on the GSM8K math benchmark some 2024 models dropped sharply under a JSON schema. The usual fix is to reason in free text first, then convert to the schema in a second pass.

### How is a JSON schema turned into a decoding constraint?

The schema is compiled to a formal grammar and typically a finite-state machine. At each decoding step the engine computes which next tokens keep the partial output valid and masks the rest — sets their probability to zero — before a token is sampled.

### Do I still need output validation if I use structured outputs?

Yes, for semantics. Constraints guarantee shape, not truth or referential consistency. You still check that an order ID exists, a date is in range, or a unit is one the downstream API accepts.

## Related

### Related terms

- [Function Calling (Tool Calling)](https://howaiworks.ai/glossary/function-calling)
- [Error Handling in AI Systems](https://howaiworks.ai/glossary/error-handling)
- [Token](https://howaiworks.ai/glossary/token)
- [Reasoning Model](https://howaiworks.ai/glossary/reasoning-model)
- [Temperature](https://howaiworks.ai/glossary/temperature)
- [Prompt Engineering](https://howaiworks.ai/glossary/prompt-engineering)

---

Source: https://howaiworks.ai/glossary/structured-outputs — HowAIWorks.ai