Structured Outputs (Constrained Decoding)

Definition

Structured outputs are the guarantee that a language model's response is not just probably-well-formed but provably parseable — valid JSON, or text matching a supplied grammar, every time. The mechanism that delivers the guarantee is constrained decoding (also called grammar-constrained or guided decoding): at each step of generation, the sampler is only allowed to pick a token that keeps the output valid against the target schema. Illegal tokens are removed before anything is sampled, so the model cannot produce a trailing comma, an unquoted key, or a field of the wrong type.

That is a different thing from asking nicely. "Return JSON and nothing else" in the prompt is a request the model can decline — under load, on a long output, or when the schema is deep, it sometimes wraps the JSON in prose, truncates a brace, or invents a field. Constrained decoding makes that class of failure structurally impossible rather than merely unlikely, which is why it is the difference between a parser you have to wrap in a retry loop and one that never throws.

The catch, and the reason this is a real engineering decision rather than a free upgrade, is that a constraint operates on the answer while the answer is still being computed. It can guarantee the shape of a response without improving — and sometimes while degrading — what the response actually says.

How It Works

A language model generates one token at a time. At each step it produces a vector of logits, one score per entry in its vocabulary (tens of thousands of entries), which a softmax turns into a probability distribution; the sampler then draws the next token from that distribution. Ordinary "prompt-and-hope" generation samples from the full distribution and trusts the model to have learned to stay inside JSON. Nothing enforces it.

Constrained decoding inserts one step in between. A constraint engine tracks the output produced so far, works out the set of tokens that would keep it valid against the schema or grammar, and sets the logits of every other token to negative infinity — a mask. After the softmax, a masked token has probability exactly zero, so it can never be drawn no matter how the sampler is configured. This is worth stressing because it is the source of the guarantee: masking happens before sampling, so temperature, top-p and every other sampling knob choose only among tokens the grammar already permits. If the partial output is {"age": and the schema says age is an integer, the only legal next tokens are digits (and perhaps a minus sign) — the token for " is masked, so the model physically cannot open a string where a number belongs.

The schema is not checked with an if statement per token, which would be far too slow. It is compiled ahead of time into a formal grammar and usually a finite-state machine, so that "which tokens are legal in this state" is a lookup rather than a computation. The widely-cited Efficient Guided Generation for Large Language Models paper (the Outlines method) makes this its central result: by indexing the state machine against the vocabulary once, up front, the per-token cost of enforcing the constraint becomes O(1) on average — a constant, independent of vocabulary size — where the naive approach of scanning the whole vocabulary at every step is O(N). The paper's own summary is that the technique "adds little overhead to the token sequence generation process." That efficiency is what makes the guarantee practical to switch on for every request instead of only the ones that failed.

Why prompt-and-hope pays a retry tax and constraints do not

The value of turning "usually valid" into "always valid" is easiest to see on the retry budget it eliminates. Suppose unconstrained JSON generation on some moderately nested schema is malformed at a rate of f = 0.05 — five parses in a hundred fail — and suppose failures are independent, so a retry fails at the same rate. The expected number of attempts to get one clean parse is a geometric series, 1 / (1 − f) = 1 / 0.95 = 1.0526. That is a 5.26% tax on every generation on the JSON path: 5.26% more calls, more output tokens, and more wall-clock latency, just to reach the output you asked for the first time.

The average understates the problem, because what actually hurts is the tail. The chance an item is still malformed after you have retried it is f raised to the number of attempts:

• after 1 retry (2 attempts): 0.05² = 0.0025 → 2,500 malformed per million items
• after 2 retries: 0.05³ = 0.000125 → 125 per million
• after 3 retries: 0.05⁴ ≈ 6.25 × 10⁻⁶ → about 6 per million

So even a fairly aggressive cap of three retries leaves a handful of items per million that never parse and must fall through to a human, a default, or a dropped record — and it triples-plus the latency of every unlucky item along the way. Push the failure rate to f = 0.20, which a deep or unusual schema can easily reach, and the arithmetic gets ugly fast: 1 / 0.8 = 1.25 means a 25% tax on the JSON path, and 0.20⁴ = 0.0016 leaves 1,600 malformed per million even after three retries.

Constrained decoding drives f for that failure class to zero. Every sample is parseable by construction, so the retry loop, the tax, and the tail all disappear together — not shrink, vanish. That is the whole case for it: not that it makes malformed output rare, but that it makes it impossible, and impossible is the only failure rate you never have to write fallback code for.

Real-World Applications

Hosted API strict modes. When OpenAI shipped Structured Outputs in August 2024 it reported that gpt-4o-2024-08-06 with the feature enabled scored 100% on its internal complex-schema-following evaluation, against under 40% for an earlier model (gpt-4-0613) relying on prompting alone — a provider figure, but the gap is the point: prompting was a coin-flip on hard schemas and the constraint was not. The same idea is exposed through the API in two places at once — a strict response_format for the final answer, and a strict flag on tool/function definitions so a function call's arguments are guaranteed to match the declared parameter schema. (Which providers offer strict guarantees, and for which schema features, changes release to release; treat any specific behavior as dated to when you read it.)

Local and open-source inference. The mechanism is not a hosted-API secret. llama.cpp ships GBNF grammars, a compact grammar format you hand to the runtime to constrain output token by token; Outlines, Guidance and XGrammar provide the same for the Python ecosystem; and serving stacks such as vLLM expose "guided decoding" that accepts a JSON schema or regular expression directly. An engineer running an open-weights model on their own hardware makes exactly the decision this page is about: pass a grammar and get a guarantee, or prompt and wrap the result in a validator.

The boundary of an agent or pipeline. Anywhere one program has to read a model's output mechanically — a step in an agent loop populating the arguments for the next tool call, a data extraction job writing rows into a database, a router picking one of a fixed set of labels — the receiving code needs the output to parse and to match a shape. Constrained decoding is how that boundary stops being a source of exceptions. This is the same territory covered by error handling in AI systems: the strongest form of handling a malformed output is to make it unproducible in the first place.

Key Concepts

Strict mode versus JSON mode. These are often confused and are not the same guarantee. A plain "JSON mode" promises only that the output is syntactically valid JSON — it will parse — but says nothing about which JSON: fields can be missing, extra, or the wrong type. A strict schema mode constrains the output to a specific structure, so the promise is both "it parses" and "it has the fields and types you declared." The first removes syntax errors; the second removes schema violations on top of them.

Shape is not truth. Constrained decoding removes two failure classes and leaves a third untouched. Malformed output (won't parse) and schema-invalid output (parses, wrong shape) both become impossible. Well-formed but wrong output does not: a model can emit {"order_id": "A-40021", "refund": 250.00} that validates perfectly and refers to an order that does not exist, a refund outside policy, or a currency the API rejects. Detecting that still requires checking the values against something real — a database, a constraint, a second lookup — which no grammar can do, because the grammar only knows the schema, not the world.

Where the constraint binds, the model stops choosing. Every token the mask forbids is a token the model's own distribution no longer influences. Usually that is exactly what you want — you did not want the trailing comma either. But it means a badly-designed schema can force the model down a path it would not have taken, which is the mechanism behind the quality effects below.

Challenges

Schema rigidity forces early commitment. A schema that puts the answer field first makes the model emit its conclusion before it has generated any of the reasoning that would have supported it. For a reasoning model or any chain-of-thought prompt, that is removing the room where the actual work happens. The design tension is real: the schema you want for clean downstream parsing (answer first, tightly typed) is often the opposite of the schema that lets the model think (a free-text reasoning field before a final_answer field).

Measured quality regressions under tight constraints. This is not hypothetical. The EMNLP 2024 Industry Track paper Let Me Speak Freely? measured it on GSM8K across the mid-2024 frontier: GPT-3.5-Turbo scored 75.99% answering in free text and 49.25% under a JSON schema; Claude-3-Haiku fell from 86.51% to 23.44%; Gemini-1.5-Flash barely moved (89.33% to 89.21%). Two things are load- bearing about those numbers. First, the effect is model- and schema-dependent — one model lost almost nothing while another lost two-thirds of its accuracy — so you cannot assume constraints are free or assume they are ruinous without measuring your own case. Second, these are mid-2024 models, a full generation behind the current catalog, and the gaps have narrowed on later generations; but the mechanism that caused them has not gone anywhere, because reasoning still needs room and a grammar that forces the answer early still takes it away.

The two-pass fix, and what it costs. The standard mitigation is to separate thinking from formatting: let the model reason in unconstrained prose, then make a second, cheap call that converts that reasoning into the strict schema. It buys back most of the lost accuracy for the price of one extra request — but it is a request, so it doubles the round trips and reintroduces some of the latency the guarantee was supposed to save. A related, cheaper option is to build the reasoning into the schema as a leading free-text field, so a single constrained call still lets the model narrate before it commits.

Grammar coverage and portability. Real JSON Schema is large — nested oneOf, recursive definitions, regex-constrained strings, unbounded arrays — and constraint engines support different subsets of it. A schema that a hosted API accepts may not compile under a local GBNF grammar, and vice versa, so a system that treats "structured output" as one portable feature can break when it changes backends. Check what your specific engine supports rather than assuming the full spec.

Future Trends

The clearest direction is standardization: JSON Schema is becoming the common interface for constraints across hosted APIs and open-source runtimes, which is what will eventually let a single schema drive OpenAI, a local llama.cpp model, and a vLLM deployment without rewriting it three ways — the portability gap in the previous section is the thing being closed. The second is the maturing of "reason-then-format" as a default rather than a workaround: as reasoning models make extended thinking the norm, the tooling is moving toward separating an unconstrained thinking phase from a constrained output phase automatically, so a developer gets both the guarantee and the reasoning room without hand-building a two-pass pipeline. Both trends point the same way — toward constrained decoding being the ordinary case for machine-consumed output, with the quality cost engineered down rather than accepted.

Code Example

The heart of constrained decoding is a single masking step applied to the logits before sampling. This toy version shows the mechanism directly: the grammar engine supplies the set of token ids that are legal in the current state, and every other token's score is driven to negative infinity so its post-softmax probability is exactly zero.

import math

def mask_logits(logits, legal_token_ids):
    """logits: dict of {token_id: score} for the whole vocabulary.
    legal_token_ids: the set of ids the compiled grammar allows as the
    NEXT token, given everything generated so far."""
    return {
        tid: (score if tid in legal_token_ids else -math.inf)
        for tid, score in logits.items()
    }

def softmax(logits):
    # -inf logits contribute exp(-inf) = 0, i.e. zero probability.
    m = max(v for v in logits.values() if v != -math.inf)
    exps = {t: math.exp(v - m) if v != -math.inf else 0.0
            for t, v in logits.items()}
    total = sum(exps.values())
    return {t: e / total for t, e in exps.items()}

# Vocabulary: after `{"age":` the grammar permits only digits and a minus
# sign; the double-quote token that would (wrongly) open a string is illegal.
raw_logits = {"0": 2.1, "1": 3.4, "2": 1.0, "-": 0.5, '"': 4.9}
legal = {"0", "1", "2", "-"}          # supplied by the FSM for this state

probs = softmax(mask_logits(raw_logits, legal))
for token, p in sorted(probs.items(), key=lambda kv: -kv[1]):
    print(f"{token!r:>4}  {p:.4f}")

Running it prints:

 '1'  0.7051
 '0'  0.1922
 '2'  0.0640
 '-'  0.0388
 '"'  0.0000

The " token had the highest raw score of all — 4.9, the value the unconstrained model most wanted to emit — and constrained decoding still gives it probability 0.0000. That is the guarantee made concrete: the illegal token is not made unlikely, it is made impossible, and no temperature or top-p setting can bring a zero-probability token back into play. A real engine differs only in scale: the vocabulary is tens of thousands of tokens, legal is produced by a compiled finite-state machine in O(1) per step rather than written by hand, and the masking runs on every one of the hundreds of tokens in the response — but the operation the reader just watched is the whole idea.