---
source: 'https://howaiworks.ai/glossary/data-leakage'
section: glossary
title: Data Leakage
description: >-
  Data leakage is training a model on information it will not have at prediction
  time. It pushes scores up, not down, which is why review never catches it.
tags:
  - Machine Learning
  - overfitting
  - cross-validation
  - model evaluation
  - data quality
category: Machine Learning
datePublished: '2026-07-24'
lastUpdated: '2026-07-24'
---

# Data Leakage

> Data leakage is training a model on information it will not have at prediction time. It pushes scores up, not down, which is why review never catches it.

## Definition

Data leakage is what happens when a model is trained on information it will not have at the moment
it is asked to make a real prediction. The information can arrive as a column that quietly encodes
the answer, as a test row that is also sitting in the training set, or as a statistic computed over
the whole dataset before it was split. However it arrives, the effect is the same: the model learns
a shortcut that exists only in your files, and the score you measure is a measurement of that
shortcut rather than of the model.

The reason leakage deserves its own page — rather than a paragraph inside
[overfitting](https://howaiworks.ai/glossary/overfitting) — is that **it does not behave like a bug.** A bug makes the
number worse, so someone investigates. Leakage makes the number *better*. Accuracy goes up, the
validation curve looks clean, the ROC curve hugs the corner, and every person who reviews the work
sees a result that exceeds expectations. Nothing in the training run raises a flag, nothing in
[cross-validation](https://howaiworks.ai/glossary/cross-validation) raises a flag, and the pull request looks fine line
by line. Leakage is therefore usually discovered in production, weeks later, when the live numbers
refuse to match the offline ones and nobody can explain the difference.

The scale of the gap is not subtle. In KDD Cup 2008, a breast-cancer detection contest, patient IDs
alone split the entrants' data into ranges whose malignancy rate ran from 36% down to 1% — a
meaningless identifier that was a 36-fold swing in the base rate of the thing being predicted.
In a set of political-science papers reproduced by Sayash Kapoor and Arvind Narayanan, an AdaBoost
model that reported an AUC of 0.94 against logistic regression's 0.76 fell to 0.77 once the
leakage was repaired — while the logistic regression, which had nothing to leak, went *up* to 0.78.
Both cases are covered in detail below.

One clarification before going further, because the phrase is overloaded. In security, a "data
leak" means confidential records escaping to people who should not see them. That is a different
problem with a different fix. This page is about the machine-learning sense: information flowing
from the target into the model's inputs, corrupting the measurement rather than exposing anyone.

## How It Works

Every case of leakage fails the same one-sentence test. **For each value the model is given, ask:
would this value have been knowable, for this exact row, at the moment the prediction is needed?**
If the answer is no — because the value was recorded afterwards, or computed from rows that are
supposed to be in the future, or derived from the test set — it is leakage, regardless of how
reasonable the code looks.

Kapoor and Narayanan's survey of ML-based science, which found 17 research fields with documented
leakage errors affecting 329 papers between them, organises the failure into a taxonomy of eight
types. The groupings below cover the same ground in the order you are likely to meet them.

### Target leakage: a column that is the answer in disguise

The clean case is a feature that only ever gets filled in *after* the outcome it is supposed to
predict. A `cancellation_reason` field is populated when a customer cancels; a
`days_in_intensive_care` value exists because the patient was admitted; a `discount_applied` flag
is set by the retention team that was called in because the account was already lost. Train on
these and the model reaches near-perfect accuracy by reading the answer off the input. Deploy it
and the field is empty, because the event has not happened yet. This is the mechanism
[supervised learning](https://howaiworks.ai/glossary/supervised-learning) is most often wrecked by, and it is why the
first pass over a new dataset should be a column-by-column interrogation of *when* each field is
written, not what it contains.

The harder case is a feature that encodes the answer indirectly. An identifier, a row number, a
file path, a timestamp of when the record was inserted, the name of the source system — none of
these look like predictors, and all of them can be. If positive and negative examples were
collected from different places, anything that betrays the source is a proxy for the label.

### Train/test contamination: the same rows on both sides

The test set only measures generalization if the model has genuinely never seen its contents.
Duplicated rows break that quietly: scraped datasets, merged exports and datasets assembled from
other datasets routinely contain the same record more than once, and a duplicate landing on both
sides of the split converts memorisation into apparent generalization. Near-duplicates are worse,
because deduplicating by exact hash misses them — two crops of the same photograph, two revisions
of the same document, the same transaction recorded by two systems.

The same failure at internet scale is why [benchmark](https://howaiworks.ai/glossary/benchmark) scores for large
language models are contested: if the evaluation set was published on the web before the training
corpus was crawled, the benchmark is partly a memory test. See
[generalization](https://howaiworks.ai/glossary/generalization) for what happens when a matched, freshly written
benchmark is used instead.

### Temporal leakage: learning from the future

On time-ordered data, a random split scatters future observations into the training set. The model
is then scored on a day whose immediate neighbours it has already seen, which is a much easier
problem than the one it will face in production, where every prediction is about a genuinely
unseen future. The same error arrives through feature engineering: normalising a series by its
overall mean, or attaching a value that is only known in hindsight, imports future information
into a past row without touching the split at all. [Time series](https://howaiworks.ai/glossary/time-series) covers the
forward-chaining alternative in full.

### Group leakage: the same entity on both sides

Rows are frequently not independent. Four X-rays belong to one patient, twenty sessions to one
user, a year of hourly readings to one machine. A random split puts some of that entity's rows in
training and the rest in testing, so the model can memorise the entity and recognise it again
rather than learn anything transferable. The measured score then answers the question "have I seen
this patient before?" instead of "can I diagnose a new patient?". The fix is to split by group, not
by row — and the prerequisite is knowing that row and subject are different things, which is a
property of the data nobody will tell you about.

### Preprocessing leakage: fitting before splitting

This is the most common instance in real code and the least visible. Any transformation that
*learns* something from data — a scaler learning a mean and standard deviation, an imputer learning
a median, an oversampler generating synthetic minority rows, a feature selector learning which
columns matter — must be fitted on training rows only. Fit it on the full table before splitting
and the test rows have influenced the training pipeline.

The severity varies enormously by transformation, and this is worth internalising rather than
memorising a rule. A `StandardScaler` fitted on everything usually shifts the result by a hair,
because a mean estimated from 1,000 rows barely differs from one estimated from 800. An
oversampler fitted on everything is catastrophic, because copies of the same minority rows end up
on both sides of the split. A feature selector fitted on everything is worse still: with more
candidate columns than rows, selecting the columns that correlate best with the labels and then
"validating" on a subset of the same data measures how well luck reproduces itself.
[Feature selection](https://howaiworks.ai/glossary/feature-selection) works that arithmetic out in full, on pure noise,
and the answer is an 8% error rate on data with no signal in it whatsoever.

The mechanical remedy is to make every learned transformation part of the model object — a
scikit-learn `Pipeline` handed to `cross_val_score`, so that fitting happens inside each fold
automatically — rather than a step you run once on a dataframe before the modelling starts.

### Why it survives review

Read those five mechanisms as diffs and none of them looks wrong. `scaler.fit(X)` before
`train_test_split` is one line and reads as tidy. `KFold(shuffle=True)` is the default everyone
uses. Keeping `patient_id` in the frame is laziness, not malice. Each individual decision is
defensible in isolation; only the combination of *this* transformation with *this* split is the
error, and a reviewer looking at code cannot see the split. This is why leakage detection is an
audit of the data and the pipeline shape, not a code-style question.

## Real-World Applications

### KDD Cup 2008: the identifier that predicted cancer

The 2008 KDD Cup asked competitors to detect breast cancer from mammography data. The winning IBM
and Tel Aviv University team reported that the "Patient ID" field, ignored by most entrants,
carried substantial information about the target. The training data covered 1,712 patients, 118 of
whom had cancer — an overall rate of 6.9%. Discretised, the IDs fell into natural bins with wildly
different outcomes: 254 patients with IDs below 20,000, of whom 36% were malignant; 414 patients
between 100,000 and 500,000, of whom 1% were malignant; and 1,044 patients above 4,000,000, of whom
1.7% were malignant. Within that last bin, all 18 malignant patients had IDs between 4,000,000 and
4,870,000 — a range containing only 3 healthy patients.

The team's explanation was that the competition data had been compiled from several medical
institutions, and IDs were assigned consecutively per source. A preventive-screening centre and a
treatment-oriented hospital have very different cancer prevalence, so the source — leaked through
the ID — predicted the outcome without any medical content whatsoever.

The follow-up experiment is the part worth remembering. Suspecting that removing the ID would not
be enough, the team tried to predict the *source bin* from the 117 image features alone, using only
benign candidates so that the label could not help. Two of the four bins were identifiable with
AUCs of 0.86 and 0.75. The leak was not in the identifier column; the identifier merely made it
visible. Differences in equipment and calibration between institutions had been baked into the
imagery itself, so every model built on that dataset — including the winning one — was probably
inflated.

### Civil war prediction: the fanciest model had the most to lose

Kapoor and Narayanan reproduced four political-science papers claiming that machine learning
substantially outperforms decades-old logistic regression at predicting civil war onset. All four
failed to reproduce, each for a leakage reason: one imputed training and test data together, two
reused that already-imputed dataset, and one used proxies for the target variable.

The numbers from their reproduction of Wang (2019) are the clearest illustration on this page,
because they show *who* the leakage benefits. The underlying dataset has 17.4% of its values
missing, and one variable — agricultural exports as a share of GDP — is missing for 49.8% of rows,
so imputation is doing a great deal of work. Imputing before splitting lets the test rows inform
the values filled into the training rows. Reported AUC for AdaBoost was 0.94. Repairing the
imputation so it happens inside each fold dropped it to 0.82, and evaluating on a genuinely
out-of-sample period dropped it again to 0.77. The random forest in the same table fell from a
reported 0.92 to 0.78 and then to 0.73 on that out-of-sample test. Meanwhile
the logistic-regression baseline moved from 0.76 to 0.77 to 0.78 — *up*, because a model with
almost no capacity to memorise had almost nothing to gain from the leak.

That asymmetry is the general lesson. Leakage does not inflate all models equally; it rewards
whichever model is most capable of memorising the leaked structure, which is usually the flexible
one you were hoping to justify. In the summary of that paper, the AUC gap between AdaBoost and
logistic regression collapsed from 0.14 to 0.01 once the error was fixed — that is, the entire
reported advantage of modern machine learning over a 2003 regression was the leak.

### COVID-19 imaging: 2,212 papers, none usable

The AIX-COVNET collaboration's systematic review of machine-learning models for diagnosing COVID-19
from chest X-rays and CT scans (Roberts et al., arXiv:2008.06388, later published in *Nature
Machine Intelligence*) identified 2,212 studies, kept 415 after initial screening, and carried 61
through quality screening into the review. Not one of the 61 was judged to be of potential clinical
use.

Two of the causes are textbook leakage. The first is what the authors call **Frankenstein
datasets**: public datasets assembled from other public datasets and redistributed under a new
name. They document a case where one such compilation merged several sources without noticing that
one component already contained another, so models were trained and tested on overlapping data
while believing the sources to be distinct. The second is a sampling failure with the same effect:
16 of the 61 papers used a well-known paediatric pneumonia dataset as their control group, without
noting that those patients were aged between one and five. A model separating adult COVID-19
patients from toddlers with pneumonia can score superbly by detecting children versus adults. A
related analysis cited in the review found that a model could still "diagnose" COVID-19 with an AUC
of 0.68 after the lung region had been excluded from the images entirely.

## Key Concepts

**Leakage and overfitting are different diseases with opposite symptoms.** Overfitting produces a
*gap*: excellent training accuracy, mediocre validation accuracy. That gap is visible, which is why
[regularization](https://howaiworks.ai/glossary/regularization) and held-out evaluation work as remedies. Leakage
*closes* the gap, because the leaked information is present in the validation set too. Any
diagnostic built on comparing training and validation scores is blind to it by construction.

**The most reliable detector is disbelief.** Before auditing the pipeline, ask whether the score is
possible. A model that beats radiologists on a first attempt, forecasts next quarter's revenue to
within a percent, or reaches 0.99 AUC on a problem where the domain literature reports 0.75 has
almost certainly found a shortcut. Every documented case above was first noticed because someone
found the result too good rather than because a test failed.

**Three cheap checks catch most of it.** Re-split the data by time and see how far the score falls;
re-split by entity — patient, user, device — and see how far it falls again; and inspect feature
importances for a single column carrying implausible weight, then drop it and re-run. A score that
survives all three is not proof of cleanliness, but a score that collapses under any of them has
told you exactly where to look.

**Offline-versus-online divergence is the last line of defence, and it is expensive.** By the time
[monitoring](https://howaiworks.ai/glossary/monitoring) shows that live precision is half the offline figure, the model
is already deployed and decisions have already been made on it. Treating a large offline/online gap
as a leakage hypothesis, rather than assuming the world changed between
[deployment](https://howaiworks.ai/glossary/model-deployment) and now, shortens that investigation considerably.

## Challenges

**Removing the leaked column often does not remove the leak.** The KDD Cup case is the canonical
demonstration: the patient ID was the visible symptom, and the source signature persisted in the
image features after the ID was deleted. Whenever positives and negatives come from different
places, the collection process itself is the leak, and no amount of column-dropping fixes it. The
repair is at the data-collection level — sampling controls from the same institutions, scanners and
time windows as the cases.

**Some legitimate features are genuinely hard to compute honestly.** "Average order value over the
last 90 days" is a perfectly sensible predictor, and it is correct only if the window is recomputed
relative to each row's prediction timestamp rather than to the date the export was taken.
Getting this right across dozens of aggregates is the point-in-time-correctness problem that
feature stores exist to solve, and it is real engineering, not a matter of remembering a rule.

**Repeated evaluation leaks the test set slowly, through the analyst.** Every time you look at the
held-out score and change something in response, a little information moves from the test set into
the model — not through the code, but through your decisions. After a hundred such rounds the
"held-out" number is optimistic, which is why a genuinely untouched final test set, opened once, is
worth the data it costs.

**Target encoding leaks inside a single training fold.** Replacing a high-cardinality categorical
with the mean target for that category means each row's encoded value is computed partly from its
own label. For a category with one member, the encoding *is* the label. This leak happens entirely
within the training data, so no split protects against it; it is why
[gradient boosting](https://howaiworks.ai/glossary/gradient-boosting) libraries implement ordered or out-of-fold variants
of the encoding rather than the naive version.

**You cannot group by a structure you do not know exists.** Grouped splits require knowing the
grouping variable, and datasets frequently arrive without one — no patient key, no device ID, no
indication that 40% of the rows came from a single hospital. When the group is unknown and
unrecoverable, the honest response is a wider uncertainty band on the reported score, not a
confident number from a random split.

## Code Example

The script below builds one synthetic churn dataset and scores it three ways with the same
[random forest](https://howaiworks.ai/glossary/random-forest) and the same five folds. Two hundred customers each
contribute five monthly rows. One feature is a noisy measurement of a genuine driver of churn;
eight more are stable account traits that carry no information about churn but do identify the
customer. A separate `exit_survey` column is filled in only after a customer has already left.

```python
import numpy as np
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import GroupKFold, KFold, cross_val_score

rng = np.random.default_rng(0)
n_cust, rows_per = 200, 5
n = n_cust * rows_per

# Per-customer truth: one genuine driver of churn, plus eight stable account
# traits that say nothing about churn but do identify the customer.
driver = rng.standard_normal(n_cust)
churn = (driver + 0.7 * rng.standard_normal(n_cust) > 0).astype(int)
traits = rng.standard_normal((n_cust, 8))

cust = np.repeat(np.arange(n_cust), rows_per)          # 5 monthly rows per customer
X = np.column_stack([
    driver[cust] + 0.3 * rng.standard_normal(n),        # noisy monthly measurement
    traits[cust] + 0.05 * rng.standard_normal((n, 8)),  # near-constant per customer
])
# A field only ever filled in after the customer has already churned.
exit_survey = churn[cust] * rng.normal(3, 1, n) + rng.normal(0, 1, n)
y = churn[cust]

clf = RandomForestClassifier(n_estimators=200, random_state=0)
group_cv, random_cv = GroupKFold(5), KFold(5, shuffle=True, random_state=0)

honest = cross_val_score(clf, X, y, cv=group_cv, groups=cust).mean()
target_leak = cross_val_score(clf, np.column_stack([X, exit_survey]), y,
                              cv=group_cv, groups=cust).mean()
group_leak = cross_val_score(clf, X, y, cv=random_cv).mean()

print(f"honest      GroupKFold, no exit survey : {honest:.1%}")
print(f"target leak GroupKFold + exit survey   : {target_leak:.1%}")
print(f"group leak  random KFold, no survey    : {group_leak:.1%}")
```

Output:

```
honest      GroupKFold, no exit survey : 73.6%
target leak GroupKFold + exit survey   : 91.9%
group leak  random KFold, no survey    : 96.7%
```

Three numbers, one dataset, and only the honest one is real. **73.6%** is what this problem
actually supports: the driver is noisy, the label is a coin flip biased by it, and no model can do
much better. Adding the exit survey lifts the score to **91.9%** — an 18.3-point gain, from a
column that in production is blank for every customer you would want to predict. Switching from
`GroupKFold` to a plain shuffled `KFold` produces **96.7%** without adding any feature at all: with
five rows per customer and five folds, roughly four-fifths of each customer's rows sit in training
whenever the remaining ones are being scored, so the forest can identify the customer from their
eight near-constant traits and recall the answer. That is a 23.1-point gain bought entirely by
changing one argument in the cross-validation call.

Notice what a reviewer would see. The leaky runs use the same estimator, the same fold count and
the same metric as the honest one. There is no exception, no warning, no divergent training curve —
just a better number, produced by two lines that look completely ordinary. A person shown only
96.7% has no way to tell it from a real result, which is the whole difficulty of the subject and
the reason a leakage audit has to be a deliberate step in [training](https://howaiworks.ai/glossary/training) rather than
something you expect the metrics to surface on their own.

## Frequently Asked Questions

### What is data leakage in machine learning?

It is when information that would not be available at prediction time reaches the model during training — a column that encodes the answer, a test row that also appears in training, a statistic computed over the whole dataset before splitting it. The model learns from it, the validation score rises, and the gain disappears the moment the model runs on genuinely new data.

### How is data leakage different from overfitting?

Overfitting shows up as a gap: training accuracy far above validation accuracy. Leakage closes that gap, because the leaked information is present in the validation set too. Overfitting is caught by holding data out; leakage survives holding data out, which is exactly what makes it dangerous.

### How do I know if my model has data leakage?

The strongest signal is a score that is better than the problem allows — a model beating clinicians, forecasters or the domain's known ceiling on a first attempt. After that: one feature carrying implausible importance, a big drop when you re-split by time or by entity instead of at random, and a live deployment that scores far below the offline number.

### Why does fitting a scaler before splitting count as leakage?

Because the scaler's mean and standard deviation are computed from rows that later become the test set, so the test rows influenced the transformation applied to the training rows. The effect of a scaler alone is usually small; the same mistake with an imputer, an oversampler or a feature selector can be large enough to invent a result out of pure noise.

### Is data leakage the same as a data leak or data breach?

No. In security, a data leak means confidential information escaping to outsiders. In machine learning, data leakage is an evaluation error: information moving from the answer into the model's inputs. The words overlap, the problems do not.

## Related

### Related terms

- [Cross-Validation (CV)](https://howaiworks.ai/glossary/cross-validation)
- [Overfitting](https://howaiworks.ai/glossary/overfitting)
- [Feature Selection (FS)](https://howaiworks.ai/glossary/feature-selection)
- [Generalization](https://howaiworks.ai/glossary/generalization)
- [Time Series](https://howaiworks.ai/glossary/time-series)
- [Training](https://howaiworks.ai/glossary/training)

---

Source: https://howaiworks.ai/glossary/data-leakage — HowAIWorks.ai