Explainable AI (XAI)

Definition

Explainable AI (XAI) is the set of methods for producing a human-readable account of why a model produced a particular output — and the phrase hides a split that decides everything else about it. An interpretable model is one whose mechanism you can read off the model itself: the coefficients of a logistic regression, the path through a shallow decision tree. A post-hoc explanation is something different in kind — a second procedure that inspects a model it did not build and reports a guess about what it did. The first is true by construction. The second is an estimate, and it can be badly wrong while looking entirely convincing.

Most people arrive at this page because someone — an auditor, a regulator, a model risk committee — has said their system must be explainable, and they want to know what that obliges them to produce. The honest answer is that the obligations are almost always for a local explanation: not "how does the model work in general" but "name the specific reasons this application was declined." US credit law has required that for decades, and the EU AI Act's Article 86 now says something similar. That framing matters, because the tools most teams reach for — LIME, SHAP, saliency maps — were designed to produce exactly that kind of per-decision output, and are exactly the tools whose faithfulness is least guaranteed. (The acronym is not new marketing: DARPA ran a research program literally called Explainable AI from 2017 to 2021, which is where XAI comes from.)

How It Works

What a post-hoc explainer actually computes

LIME (Local Interpretable Model-agnostic Explanations) takes one input, generates a cloud of perturbed variants around it, asks the black box for a prediction on each, weights those samples by how close they are to the original, and fits a small sparse linear model to the result. The coefficients of that little linear model are the explanation. Note what is being explained: not the black box, but a linear approximation of the black box in a neighbourhood that LIME itself invented.

SHAP (SHapley Additive exPlanations) borrows the Shapley value from cooperative game theory. Treat each feature as a player and the prediction as a payout. A feature's attribution is its average marginal contribution — how much the prediction moves when you add that feature — averaged over every possible subset of the other features. This has properties LIME does not: the attributions provably sum to the gap between this prediction and the average prediction, and a feature the model ignores gets exactly zero.

The cost of doing SHAP exactly, worked

Those properties are bought with an exponential. For n features, the average runs over all subsets of the remaining features, and the full computation touches all 2ⁿ subsets of the feature set:

n = 10 features  ->  2^10 =         1,024 subsets
n = 20 features  ->  2^20 =     1,048,576 subsets
n = 30 features  ->  2^30 = 1,073,741,824 subsets

Each subset needs a model evaluation, and in practice several — the "absent" features have to be marginalised out by averaging over a background dataset, which multiplies the count again by the background set's size. Put a cheap gradient-boosted model at 1 ms per evaluation behind that: 20 features costs about 17.5 minutes for a single prediction's explanation, and 30 features costs about 12.4 days. The 137-field COMPAS instrument discussed below is not in the same universe as a feasible exact computation.

So no production implementation does it exactly. KernelSHAP samples a few thousand coalitions and recovers the attributions by weighted linear regression. TreeSHAP escapes the exponential altogether for tree ensembles by exploiting the tree structure, which is a large part of why SHAP became the default for gradient boosting and random forests specifically. The consequence for everything else is simple and under-stated: the SHAP bar you are looking at is a Monte Carlo estimate, and the plot does not draw an error bar on it.

Fidelity: does the explanation match the model at all?

Both methods fit a simple surrogate, and a surrogate has a goodness of fit. That number — the local R², how much of the black box's behaviour in the neighbourhood the linear stand-in actually reproduces — is the single most useful diagnostic in explainability, and it is almost never shown. When Tan et al. measured it for image classifiers (GLIME, NeurIPS 2023), LIME's local R² at its own default kernel width (σ = 0.25) came out near zero. An explanation with a local R² near zero is not a simplified description of the model. It is a description of a surface the model does not have.

Types

Three axes are genuinely used in the literature, and the useful observation is that they are not independent — one of them decides truth, and the other two only decide convenience.

Ante-hoc versus post-hoc is the axis that matters. An ante-hoc (or intrinsic) method makes the model itself readable: a depth-3 tree, a rule list, a generalised additive model, a scorecard. There is nothing to verify, because the explanation is the model. A post-hoc method leaves the model alone and estimates an explanation afterwards. Everything that goes wrong later in this page goes wrong on the post-hoc side of this line.

Global versus local decides scope. A global explanation describes the model's behaviour across all inputs — permutation importance, partial dependence — and answers questions about design and feature selection. A local explanation describes one decision. Legal obligations are essentially always local, and a global feature-importance chart is not a substitute for one, however impressive it looks in a model card.

Model-agnostic versus model-specific decides reach. Agnostic methods treat the model as a query oracle: feed inputs, read outputs, infer. That generality is why LIME and KernelSHAP work on anything. It is also precisely the assumption that the attack in the next section breaks, because a model that can tell it is being queried by an explainer can behave differently while it is being watched. Model-specific methods — TreeSHAP, gradient-based attribution, reading an attention mechanism — give up generality and get either exactness or speed back.

Real-World Applications

Credit denials in the United States are the oldest hard requirement and the clearest picture of what "explainable" means operationally. The Equal Credit Opportunity Act and Regulation B require a declined applicant to be told the specific principal reasons, and Regulation B's official commentary to 12 CFR 1002.9(b)(2) adds that "disclosure of more than four reasons is not likely to be helpful to the applicant." CFPB Circular 2022-03 closed the obvious escape route: a creditor cannot excuse non-compliance on the grounds that its model is too complicated to interpret. So the deliverable is roughly four ranked, plain-language causes for one specific denial — and it has to be stable, because two runs of the same pipeline producing two different letters for the same applicant is a compliance incident, not a rounding error. AI in finance covers the wider regulatory picture.

Criminal risk assessment produced the most-cited demonstration that explaining a black box was the wrong goal in the first place. COMPAS scores recidivism risk from a 137-question instrument and is proprietary. CORELS, an algorithm for learning provably optimal rule lists, was run on the ProPublica two-year recidivism data and produced a list of three rules over age and prior offences: mean test accuracy 0.665 (sd 0.018) across ten folds, against COMPAS's 0.660 (sd 0.019). This is the empirical backbone of Cynthia Rudin's 2019 argument in Nature Machine Intelligence that high-stakes decisions should use inherently interpretable models rather than black boxes with explanations bolted on. When the accuracy gap is half a percentage point, the black box was never buying anything.

Clinical risk models supply the case where readability caught something lethal. Caruana et al. (KDD 2015) fitted an intelligible additive model to pneumonia mortality data and could see, in the model itself, a rule to the effect that having asthma lowers your risk of dying of pneumonia. It was a real pattern in the data — asthmatic patients were routed straight to intensive care and so survived more often — and acting on it would have sent exactly the wrong patients home. A neural network trained on the same data learns the same pattern and shows you nothing. The explanation did not merely reassure anyone; it changed the model. That is the standard to judge an explanation by. See AI in healthcare for how these models are deployed.

Debugging is the everyday application and the one with the fewest failure modes, because the audience can go and check. If a saliency map says the classifier is reading the hospital's watermark rather than the lung, you test that by removing the watermark. As a hypothesis generator a post-hoc explanation is cheap and useful; as evidence to a third party it is carrying weight it was never validated for.

Key Concepts

• Faithfulness is not plausibility: an explanation that a human finds satisfying and an explanation that matches what the model did are different properties, and human evaluation measures only the first. A forged explanation scores perfectly on plausibility.
• Attribution is not causation: SHAP tells you how much a feature moved this model's output, not what would happen in the world if the feature changed. Two models with identical accuracy on correlated features can distribute attributions completely differently. See causal reasoning for the distinction.
• Counterfactuals give recourse, attributions do not: "your loan is declined; had your outstanding balance been below $4,000 it would have been approved" is actionable in a way that "balance contributed −0.31 to the logit" never is, which is why counterfactual explanations map more naturally onto what the law actually asks for.
• The interpretability tax is smaller than assumed on tabular data: the accuracy gap between a well-tuned interpretable model and a black box on structured data is often within noise — the CORELS result above is 0.005 — and much larger on perception tasks, where no readable model competes at all.

Challenges

An explanation can be forged, and the attack is cheap. This is the strongest single result in the field and it deserves to be known outside it. Slack, Hilgard, Jia, Singh and Lakkaraju (Fooling LIME and SHAP, AIES 2020) built a "scaffolded" classifier: an out-of-distribution detector spots the synthetic points LIME and SHAP generate, routes those to an innocuous model, and routes real inputs to a classifier that predicts purely on race. On the COMPAS data — 6,172 records, 51.4% African-American defendants — the attack moved the top-ranked feature from race to a meaningless decoy on 100% of held-out points under LIME, and on 84% of points under SHAP (100% on the Communities and Crime data). The model's real predictions were unchanged throughout. Nothing about the resulting explanation looks wrong; the explainer simply never saw the model that made the decision. The attack works whenever the detector's F1 exceeds about 0.8, which is not a demanding bar for structured data.

The same prediction explained twice gives two answers. LIME's neighbourhood is sampled randomly, so the explanation is a random variable. The GLIME authors measured the top-20 Jaccard index between LIME explanations of the same image under two random seeds at below 0.4 — with 16,384 perturbation samples already drawn. Fewer than half the features on the list survive a change of seed. And because the surrogate is fitted by weighted least squares, the noise on a coefficient falls only as 1/√N: going from 1,000 samples to 5,000 costs five times the compute and buys a factor of √5 ≈ 2.2. You cannot cheaply buy your way to a stable explanation.

Instability is not unique to post-hoc methods, and that is the deeper problem. The decision trees page shows a single training row flipping which feature wins the root split. Put the two facts side by side: an interpretable model's mechanism can be unstable under the data, and a post-hoc explanation's output is unstable under its own random seed. Interpretability guarantees that the explanation describes the model. It guarantees nothing about whether the model — or the explanation — would survive being refit tomorrow. This is why explanation stability belongs in monitoring alongside accuracy drift.

Explanations shift liability without reducing harm. Shipping a SHAP plot with every decision produces an artefact that looks like accountability and, in the adversarial case above, certifies a discriminatory model as clean. No regulator can tell a faithful explanation from a forged one by reading it. So the useful compliance question is not "can you explain it" but "can you show the explanation is faithful" — which has a measurable answer (local R², stability across seeds, fidelity to held-out behaviour) that almost no explainability dashboard reports.

Future Trends

A deadline turns this from best practice into a deliverable. The EU AI Act's high-risk regime carries both Article 13 (transparency and instructions sufficient for deployers to interpret outputs) and Article 86, a right for an affected person to obtain "clear and meaningful explanations of the role of the AI system in the decision-making procedure." As of July 2026 those apply from 2 December 2027 for stand-alone Annex III systems and 2 August 2028 for AI embedded in already-regulated products, having been pushed back from 2026 and 2027 by the Digital Omnibus (Parliament 16 June 2026, Council 29 June 2026); the separate Article 50 transparency rules still apply from 2 August 2026. Treat every date here as perishable — this timeline has already moved once. AI governance covers which tier a given system falls into.

Faithfulness metrics displace user studies. For a decade, explanation quality was largely evaluated by asking people whether they liked the explanation — a metric the forgery attack scores full marks on. The measurable alternatives are becoming the reported numbers: deletion and insertion curves, local R², seed stability, and the sanity check that randomising a model's weights ought to destroy its explanation and frequently does not.

Mechanistic interpretability is the ante-hoc route for models you cannot replace with a rule list. Nobody will swap a frontier large language model for a scorecard, so the intrinsic path for these systems runs through the computation itself — features, circuits, sparse autoencoders over activations — rather than a surrogate fitted to the outputs. Its claims are meant to be checkable by intervention on the model, which is exactly what LIME and SHAP cannot offer.

Code Example

The Shapley definition is short enough to implement directly, and doing so makes both its guarantees and its cost concrete. This brute-force version enumerates every subset, which is exactly what production libraries refuse to do.

from itertools import combinations
from math import factorial

FEATURES = ["income", "priors", "age", "zip"]
n = len(FEATURES)

# A toy model. v(S) is its output when only the features in S are "present".
def v(S):
    w = {"income": 2.0, "priors": -3.0, "age": 1.0, "zip": 0.0}
    interaction = 1.5 if {"income", "priors"}.issubset(S) else 0.0
    return sum(w[f] for f in S) + interaction

def shapley(i):
    others = [f for f in FEATURES if f != i]
    total = 0.0
    for k in range(len(others) + 1):
        for S in combinations(others, k):
            weight = factorial(k) * factorial(n - k - 1) / factorial(n)
            total += weight * (v(list(S) + [i]) - v(list(S)))
    return total

for f in FEATURES:
    print(f"{f:>7}: {shapley(f):+.3f}")
# income: +2.750   priors: -2.250   age: +1.000   zip: +0.000

# Efficiency axiom: attributions sum to v(all) - v(none).
print(round(sum(shapley(f) for f in FEATURES), 6), v(FEATURES) - v([]))   # 1.5 1.5

# The cost of the loop above, by feature count.
for k in (4, 10, 20, 30):
    print(k, 2 ** k)     # 4 16 / 10 1024 / 20 1048576 / 30 1073741824

Three things are visible in the output. zip has weight zero in the model and receives an attribution of exactly 0.000 — the dummy axiom, and the property permutation importance does not guarantee. The 1.5 interaction term, which belongs to income and priors jointly and to neither alone, is split evenly between them: income scores 2.0 + 0.75 and priors scores −3.0 + 0.75. That even split is a convention Shapley values impose, not a fact discovered in the model, and it is the reason two correlated features can trade attribution between them with no change in behaviour. And the final loop is the wall: the same code on 30 features runs a billion times, which is why the SHAP value you see in a notebook is nearly always sampled rather than computed.