Conformal Prediction

Definition

Conformal prediction is a wrapper you put around an already-trained model that converts its single best guess into a prediction set — a small collection of candidate answers — that is mathematically guaranteed to contain the true answer at a rate you choose in advance, such as 90%. You pick the error rate; the procedure hands back sets that hit it. It does this without retraining the model, without knowing what kind of model it is, and without assuming anything about the shape of the data distribution.

That last part is what makes it unusual. Most ways of attaching a confidence to a prediction — a softmax probability, a Bayesian credible interval — are only trustworthy if some modelling assumption happens to be true, and softmax scores in particular are notoriously overconfident. Conformal prediction instead gives a distribution-free, finite-sample guarantee that rests on a single, checkable assumption: that your calibration data and your test data are drawn the same way (exchangeability). Formally, for a target error rate α, the set C(X) satisfies

1 − α  ≤  P(Y_test ∈ C(X_test))  ≤  1 − α + 1/(n + 1)

where n is the size of the calibration set. Set α = 0.1 and the true label lands in the set at least 90% of the time — and by no more than a hair over 90% for reasonably large n. The price you pay is that the set can be large: a guarantee to cover the answer says nothing about the set being small, and on a hard input the honest set may list many candidates. That size is the feature, not a defect — it is the model telling you, in a unit you can act on, that it does not know.

How It Works

The whole method is three steps: score, calibrate, and predict.

Step 1 — Define a nonconformity score. Pick any function s(x, y) that is large when the label y looks wrong for input x and small when it looks right. For a classifier that outputs a softmax probability for each class, the standard choice is s(x, y) = 1 − f(x)_y, the probability mass the model did not put on the true class. A confident, correct prediction scores near 0; a confident mistake scores near 1. The score can be anything you like — the guarantee does not depend on it being good, only on it being fixed before you look at the calibration data.

Step 2 — Calibrate on held-out data. Take a calibration set of n labelled examples that the model never trained on, and compute the score s(X_i, Y_i) on each. Now sort those n scores and find the conformal quantile q̂. This is the one piece of arithmetic that matters, and it carries a finite-sample correction that is easy to get wrong. Per Angelopoulos and Bates, q̂ is the

ceil((n + 1)(1 − α)) / n

empirical quantile of the calibration scores — "essentially the 1 − α quantile, but with a small correction," in their words. That +1 and the ceiling are what turn an asymptotic heuristic into an exact finite-sample guarantee.

It is worth seeing why the correction bites. Suppose you want 90% coverage (α = 0.1) from only n = 10 calibration points. The level is ceil((10 + 1)(0.9)) / 10 = ceil(9.9) / 10 = 10/10 = 1.0, so q̂ is the largest of your 10 scores — not the 9th, which a naive "90th percentile" would give you. With ten points, guaranteeing 90% coverage means using the single worst case you observed. Now scale up: at n = 1000, the level is ceil(1001 × 0.9) / 1000 = ceil(900.9) / 1000 = 901/1000 = 0.901, so q̂ is the 901st-smallest score — the ordinary 90th percentile nudged up by one-tenth of one percentile. The correction is enormous when data is scarce and vanishes as n grows, which is exactly the 1/(n+1) slack in the coverage bound above.

Step 3 — Form the prediction set. For a new input X_test, include every label whose score falls at or below the threshold:

C(X_test) = { y : s(X_test, y) ≤ q̂ }

For the softmax score, this is { classes k : f(X_test)_k ≥ 1 − q̂ } — keep every class the model rated at least 1 − q̂. Because q̂ was chosen so that at least a 1 − α fraction of calibration scores fall below it, and the test point is exchangeable with them, the true label's score falls below q̂ at least 1 − α of the time. That single exchangeability step is the entire proof, and it is why the guarantee needs no assumption about the model or the data-generating distribution.

Exchangeability, and what it is not. The one assumption is that the calibration examples and the test example are exchangeable — loosely, that their joint distribution does not care about ordering, which the usual i.i.d. sampling satisfies. It is much weaker than "the model is correct" and much weaker than "the data is Gaussian." But it is not free: it fails the moment your test data comes from a different distribution than your calibration data, which is the caveat the Challenges section returns to.

Types

The three-step recipe above is split (inductive) conformal prediction, and it is what nearly everyone means in practice, because it costs one extra held-out set and no extra model fits. Two other named variants are worth knowing:

• Full (transductive) conformal prediction is the original Vovk–Gammerman–Saunders formulation. Instead of a single held-out calibration set, it refits the scoring procedure for every candidate label of every test point. It wastes no data on calibration and can give tighter sets, but the repeated refitting is usually far too expensive for modern models.
• Conformalized quantile regression (CQR) targets regression rather than classification. It wraps a quantile-regression model — one already trained to predict, say, the 5th and 95th percentiles — and conformalizes its interval widths so the resulting intervals carry the exact coverage guarantee even when the underlying quantile estimates are off. It combines the adaptivity of quantile regression with the validity of conformal prediction.

These are genuine, named methods people select between; they are not a decorative taxonomy. If a "type" of conformal prediction is not one of these families or a documented score function, be suspicious of it.

Real-World Applications

Conformal prediction earns its place wherever a wrong answer is expensive and "I am not sure" is a usable output.

In medical image classification, a model that returns a single diagnosis forces a false choice; a conformal set that is guaranteed to contain the true condition 90% of the time turns the model into a screening aid. The Angelopoulos and Bates tutorial builds its running example on exactly this shape — ImageNet classification where an easy image yields a one-element set and a genuinely ambiguous one yields a set of several plausible classes — and the same pattern maps directly onto clinical triage, where a large set is a signal to route the case to a human.

The method is mainstream enough that it ships as tooling. MAPIE (Model-Agnostic Prediction Interval Estimator), part of the scikit-learn-contrib ecosystem, wraps any scikit-learn estimator to produce conformal sets and intervals; TorchCP does the same for PyTorch models. Because the wrapper is model-agnostic, teams adopt it without touching their existing training pipeline — the decision that changes is downstream: when the set is large, abstain, escalate, or gather more data.

In drug discovery and cheminformatics, conformal prediction is used to attach calibrated confidence to QSAR property predictions, so that a screening pipeline can rank compounds not just by predicted activity but by whether the model's confidence is high enough to act on. In demand and load forecasting, conformalized intervals give planners a stated coverage on a forecast band — a stock buffer or a capacity margin sized to a real probability rather than a hopeful point estimate. And because a conformal p-value is itself a test of exchangeability with the training data, the same machinery underpins out-of-distribution and anomaly detection: an input whose score exceeds every calibration score is, quantifiably, an outlier — the connection this page's related term on anomaly detection makes precise.

Key Concepts

Marginal versus conditional coverage. The guarantee is marginal: the 90% is averaged over the randomness in both the calibration draw and the test point. It does not promise 90% coverage for every subgroup or every individual input. A model can hit 90% overall while covering easy inputs 99% of the time and a hard minority only 70% — the averages balance. True conditional coverage (90% for every X) is provably impossible to guarantee distribution-free in finite samples, which is why the honest claim is always the marginal one. When per-group validity matters, you calibrate per group (Mondrian conformal prediction) and accept a coverage statement per group rather than one global number.

How this differs from Bayesian uncertainty. A Bayesian model places a prior over its parameters and produces a posterior predictive distribution, from which a 90% credible interval falls out. That interval is calibrated only if the model is correctly specified — wrong prior or wrong likelihood, and the "90%" is aspirational. Bayesian intervals also aim, at least in principle, at input-conditional uncertainty. Conformal prediction makes the opposite trade: it abandons the ambition of per-input calibration in exchange for a marginal guarantee that holds under only exchangeability, treating the model — Bayesian, frequentist, a random forest, a neural net — as an opaque score generator. The two compose well: conformalize a Bayesian model's posterior, and you keep the posterior's adaptive interval shapes while gaining a guarantee that survives the model being misspecified.

Validity is free; efficiency is not. Coverage is guaranteed for any nonconformity score. What a better score buys you is smaller sets — the difference between a set that names two candidate diagnoses and one that names nine. So the score function is where domain knowledge and effort go, while the coverage stays pinned by construction.

Challenges

Conformal prediction is unusually honest about what it does and does not deliver, and its failure modes are specific to the method.

• Distribution shift voids the guarantee. Exchangeability between calibration and test data is load-bearing, and it is the first thing to break in production. A model calibrated on last quarter's traffic and deployed against this quarter's — the phenomenon the sibling concept of concept drift describes — no longer has an exchangeable test point, and the 90% coverage is simply no longer promised. This is the single most important caveat: conformal prediction does not detect or fix drift, it merely assumes it away.
• Marginal coverage can hide undercovered subgroups. As above, the global 90% can mask a subgroup sitting well below it. If that subgroup is a protected class or a safety-critical regime, marginal coverage is the wrong metric and you need group-conditional calibration.
• Coverage without usefulness. The guarantee constrains the set to contain the truth, not to be small. A poor base model produces valid but enormous sets — in the limit, a set containing every class covers the truth 100% of the time and tells you nothing. Set size, not coverage, is the number to watch for whether the system is actually helping.
• Calibration data has a cost. Split conformal spends a held-out set that could have trained the model. On small datasets this is a real trade-off, and it is why cross-conformal and jackknife+ variants exist — to recycle data — at the cost of extra computation.
• Small calibration sets make coverage jittery. The 1/(n+1) slack is not the only finite-sample effect: for small n, the coverage you actually achieve on a given calibration draw varies around the target. The guarantee holds in expectation over calibration draws; a single unlucky draw can land a couple of points off, which is the practical reason behind the ~1,000-point rule of thumb.

Future Trends

The active research on conformal prediction is aimed squarely at its one weak assumption. Weighted conformal prediction re-weights calibration points to correct for known covariate shift, recovering a guarantee when the shift is understood. Adaptive conformal inference (ACI) and related online methods drop exchangeability entirely for streaming and time-series settings, adjusting the target level on the fly so long-run coverage holds even as the distribution moves. And as language models spread, conformal methods for LLMs are being used to build calibrated abstention and factuality filters — sets of generated claims sized so that the true or supported content is retained at a guaranteed rate, turning an unconstrained generator into one with a stated error budget. Each of these keeps the same three-step skeleton; what changes is how q̂ is computed when the data can no longer be assumed exchangeable.

Code Example

The block below runs split conformal prediction end to end on a synthetic 10-class problem: it computes the conformal quantile with the finite-sample correction, forms prediction sets on fresh test data, and measures the empirical coverage and average set size. It uses the same adjusted-quantile line as the Angelopoulos–Bates tutorial (np.ceil((n+1)*(1-alpha))/n with method='higher'). The output shown is the actual output of this exact code.

import numpy as np
rng = np.random.default_rng(0)
alpha = 0.1

# Small hand-check: n=10 calibration scores, alpha=0.1
scores10 = np.array([0.05,0.11,0.18,0.22,0.29,0.34,0.41,0.55,0.63,0.88])
n = 10
q_level = np.ceil((n+1)*(1-alpha))/n
qhat10 = np.quantile(scores10, q_level, method='higher')
print(f"[n=10] q_level = ceil(11*0.9)/10 = {q_level}  -> qhat = {qhat10} (the max)")

# Synthetic 10-class classifier, split-conformal, alpha=0.1
K, N = 10, 6000
y = rng.integers(0, K, size=N)
logits = rng.normal(0, 1, size=(N, K))
logits[np.arange(N), y] += 2.0          # imperfect but informative model
ex = np.exp(logits - logits.max(1, keepdims=True))
smx = ex / ex.sum(1, keepdims=True)

idx = rng.permutation(N)
cal, test = idx[:1000], idx[1000:]
n = len(cal)
cal_scores = 1 - smx[cal, y[cal]]                 # nonconformity score
q_level = np.ceil((n+1)*(1-alpha))/n
qhat = np.quantile(cal_scores, q_level, method='higher')
print(f"[n=1000] q_level = {q_level}  -> qhat = {qhat:.4f}")

sets = smx[test] >= (1 - qhat)                    # prediction sets
covered = sets[np.arange(len(test)), y[test]]
print(f"empirical coverage = {covered.mean():.4f}  (target {1-alpha})")
print(f"average set size    = {sets.sum(1).mean():.4f}")
print(f"top-1 accuracy      = {(smx[test].argmax(1)==y[test]).mean():.4f}")

Output:

[n=10] q_level = ceil(11*0.9)/10 = 1.0  -> qhat = 0.88 (the max)
[n=1000] q_level = 0.901  -> qhat = 0.8878
empirical coverage = 0.9076  (target 0.9)
average set size    = 2.5396
top-1 accuracy      = 0.6676

Read the numbers together. The model is only 67% accurate top-1, so a single guess is wrong a third of the time — but the conformal sets cover the true class 90.8% of the time, right on the 90% target, at an average cost of about 2.5 candidate labels per input. Nothing about the model changed; the guarantee came entirely from the calibration quantile. Swap in a stronger model and the coverage stays near 90% while the average set size shrinks toward 1 — validity fixed, efficiency improving — which is exactly the split the Key Concepts section describes.