Definition
Data leakage is what happens when a model is trained on information it will not have at the moment it is asked to make a real prediction. The information can arrive as a column that quietly encodes the answer, as a test row that is also sitting in the training set, or as a statistic computed over the whole dataset before it was split. However it arrives, the effect is the same: the model learns a shortcut that exists only in your files, and the score you measure is a measurement of that shortcut rather than of the model.
The reason leakage deserves its own page — rather than a paragraph inside overfitting — is that it does not behave like a bug. A bug makes the number worse, so someone investigates. Leakage makes the number better. Accuracy goes up, the validation curve looks clean, the ROC curve hugs the corner, and every person who reviews the work sees a result that exceeds expectations. Nothing in the training run raises a flag, nothing in cross-validation raises a flag, and the pull request looks fine line by line. Leakage is therefore usually discovered in production, weeks later, when the live numbers refuse to match the offline ones and nobody can explain the difference.
The scale of the gap is not subtle. In KDD Cup 2008, a breast-cancer detection contest, patient IDs alone split the entrants' data into ranges whose malignancy rate ran from 36% down to 1% — a meaningless identifier that was a 36-fold swing in the base rate of the thing being predicted. In a set of political-science papers reproduced by Sayash Kapoor and Arvind Narayanan, an AdaBoost model that reported an AUC of 0.94 against logistic regression's 0.76 fell to 0.77 once the leakage was repaired — while the logistic regression, which had nothing to leak, went up to 0.78. Both cases are covered in detail below.
One clarification before going further, because the phrase is overloaded. In security, a "data leak" means confidential records escaping to people who should not see them. That is a different problem with a different fix. This page is about the machine-learning sense: information flowing from the target into the model's inputs, corrupting the measurement rather than exposing anyone.
How It Works
Every case of leakage fails the same one-sentence test. For each value the model is given, ask: would this value have been knowable, for this exact row, at the moment the prediction is needed? If the answer is no — because the value was recorded afterwards, or computed from rows that are supposed to be in the future, or derived from the test set — it is leakage, regardless of how reasonable the code looks.
Kapoor and Narayanan's survey of ML-based science, which found 17 research fields with documented leakage errors affecting 329 papers between them, organises the failure into a taxonomy of eight types. The groupings below cover the same ground in the order you are likely to meet them.
Target leakage: a column that is the answer in disguise
The clean case is a feature that only ever gets filled in after the outcome it is supposed to
predict. A cancellation_reason field is populated when a customer cancels; a
days_in_intensive_care value exists because the patient was admitted; a discount_applied flag
is set by the retention team that was called in because the account was already lost. Train on
these and the model reaches near-perfect accuracy by reading the answer off the input. Deploy it
and the field is empty, because the event has not happened yet. This is the mechanism
supervised learning is most often wrecked by, and it is why the
first pass over a new dataset should be a column-by-column interrogation of when each field is
written, not what it contains.
The harder case is a feature that encodes the answer indirectly. An identifier, a row number, a file path, a timestamp of when the record was inserted, the name of the source system — none of these look like predictors, and all of them can be. If positive and negative examples were collected from different places, anything that betrays the source is a proxy for the label.
Train/test contamination: the same rows on both sides
The test set only measures generalization if the model has genuinely never seen its contents. Duplicated rows break that quietly: scraped datasets, merged exports and datasets assembled from other datasets routinely contain the same record more than once, and a duplicate landing on both sides of the split converts memorisation into apparent generalization. Near-duplicates are worse, because deduplicating by exact hash misses them — two crops of the same photograph, two revisions of the same document, the same transaction recorded by two systems.
The same failure at internet scale is why benchmark scores for large language models are contested: if the evaluation set was published on the web before the training corpus was crawled, the benchmark is partly a memory test. See generalization for what happens when a matched, freshly written benchmark is used instead.
Temporal leakage: learning from the future
On time-ordered data, a random split scatters future observations into the training set. The model is then scored on a day whose immediate neighbours it has already seen, which is a much easier problem than the one it will face in production, where every prediction is about a genuinely unseen future. The same error arrives through feature engineering: normalising a series by its overall mean, or attaching a value that is only known in hindsight, imports future information into a past row without touching the split at all. Time series covers the forward-chaining alternative in full.
Group leakage: the same entity on both sides
Rows are frequently not independent. Four X-rays belong to one patient, twenty sessions to one user, a year of hourly readings to one machine. A random split puts some of that entity's rows in training and the rest in testing, so the model can memorise the entity and recognise it again rather than learn anything transferable. The measured score then answers the question "have I seen this patient before?" instead of "can I diagnose a new patient?". The fix is to split by group, not by row — and the prerequisite is knowing that row and subject are different things, which is a property of the data nobody will tell you about.
Preprocessing leakage: fitting before splitting
This is the most common instance in real code and the least visible. Any transformation that learns something from data — a scaler learning a mean and standard deviation, an imputer learning a median, an oversampler generating synthetic minority rows, a feature selector learning which columns matter — must be fitted on training rows only. Fit it on the full table before splitting and the test rows have influenced the training pipeline.
The severity varies enormously by transformation, and this is worth internalising rather than
memorising a rule. A StandardScaler fitted on everything usually shifts the result by a hair,
because a mean estimated from 1,000 rows barely differs from one estimated from 800. An
oversampler fitted on everything is catastrophic, because copies of the same minority rows end up
on both sides of the split. A feature selector fitted on everything is worse still: with more
candidate columns than rows, selecting the columns that correlate best with the labels and then
"validating" on a subset of the same data measures how well luck reproduces itself.
Feature selection works that arithmetic out in full, on pure noise,
and the answer is an 8% error rate on data with no signal in it whatsoever.
The mechanical remedy is to make every learned transformation part of the model object — a
scikit-learn Pipeline handed to cross_val_score, so that fitting happens inside each fold
automatically — rather than a step you run once on a dataframe before the modelling starts.
Why it survives review
Read those five mechanisms as diffs and none of them looks wrong. scaler.fit(X) before
train_test_split is one line and reads as tidy. KFold(shuffle=True) is the default everyone
uses. Keeping patient_id in the frame is laziness, not malice. Each individual decision is
defensible in isolation; only the combination of this transformation with this split is the
error, and a reviewer looking at code cannot see the split. This is why leakage detection is an
audit of the data and the pipeline shape, not a code-style question.
Real-World Applications
KDD Cup 2008: the identifier that predicted cancer
The 2008 KDD Cup asked competitors to detect breast cancer from mammography data. The winning IBM and Tel Aviv University team reported that the "Patient ID" field, ignored by most entrants, carried substantial information about the target. The training data covered 1,712 patients, 118 of whom had cancer — an overall rate of 6.9%. Discretised, the IDs fell into natural bins with wildly different outcomes: 254 patients with IDs below 20,000, of whom 36% were malignant; 414 patients between 100,000 and 500,000, of whom 1% were malignant; and 1,044 patients above 4,000,000, of whom 1.7% were malignant. Within that last bin, all 18 malignant patients had IDs between 4,000,000 and 4,870,000 — a range containing only 3 healthy patients.
The team's explanation was that the competition data had been compiled from several medical institutions, and IDs were assigned consecutively per source. A preventive-screening centre and a treatment-oriented hospital have very different cancer prevalence, so the source — leaked through the ID — predicted the outcome without any medical content whatsoever.
The follow-up experiment is the part worth remembering. Suspecting that removing the ID would not be enough, the team tried to predict the source bin from the 117 image features alone, using only benign candidates so that the label could not help. Two of the four bins were identifiable with AUCs of 0.86 and 0.75. The leak was not in the identifier column; the identifier merely made it visible. Differences in equipment and calibration between institutions had been baked into the imagery itself, so every model built on that dataset — including the winning one — was probably inflated.
Civil war prediction: the fanciest model had the most to lose
Kapoor and Narayanan reproduced four political-science papers claiming that machine learning substantially outperforms decades-old logistic regression at predicting civil war onset. All four failed to reproduce, each for a leakage reason: one imputed training and test data together, two reused that already-imputed dataset, and one used proxies for the target variable.
The numbers from their reproduction of Wang (2019) are the clearest illustration on this page, because they show who the leakage benefits. The underlying dataset has 17.4% of its values missing, and one variable — agricultural exports as a share of GDP — is missing for 49.8% of rows, so imputation is doing a great deal of work. Imputing before splitting lets the test rows inform the values filled into the training rows. Reported AUC for AdaBoost was 0.94. Repairing the imputation so it happens inside each fold dropped it to 0.82, and evaluating on a genuinely out-of-sample period dropped it again to 0.77. The random forest in the same table fell from a reported 0.92 to 0.78 and then to 0.73 on that out-of-sample test. Meanwhile the logistic-regression baseline moved from 0.76 to 0.77 to 0.78 — up, because a model with almost no capacity to memorise had almost nothing to gain from the leak.
That asymmetry is the general lesson. Leakage does not inflate all models equally; it rewards whichever model is most capable of memorising the leaked structure, which is usually the flexible one you were hoping to justify. In the summary of that paper, the AUC gap between AdaBoost and logistic regression collapsed from 0.14 to 0.01 once the error was fixed — that is, the entire reported advantage of modern machine learning over a 2003 regression was the leak.
COVID-19 imaging: 2,212 papers, none usable
The AIX-COVNET collaboration's systematic review of machine-learning models for diagnosing COVID-19 from chest X-rays and CT scans (Roberts et al., arXiv:2008.06388, later published in Nature Machine Intelligence) identified 2,212 studies, kept 415 after initial screening, and carried 61 through quality screening into the review. Not one of the 61 was judged to be of potential clinical use.
Two of the causes are textbook leakage. The first is what the authors call Frankenstein datasets: public datasets assembled from other public datasets and redistributed under a new name. They document a case where one such compilation merged several sources without noticing that one component already contained another, so models were trained and tested on overlapping data while believing the sources to be distinct. The second is a sampling failure with the same effect: 16 of the 61 papers used a well-known paediatric pneumonia dataset as their control group, without noting that those patients were aged between one and five. A model separating adult COVID-19 patients from toddlers with pneumonia can score superbly by detecting children versus adults. A related analysis cited in the review found that a model could still "diagnose" COVID-19 with an AUC of 0.68 after the lung region had been excluded from the images entirely.
Key Concepts
Leakage and overfitting are different diseases with opposite symptoms. Overfitting produces a gap: excellent training accuracy, mediocre validation accuracy. That gap is visible, which is why regularization and held-out evaluation work as remedies. Leakage closes the gap, because the leaked information is present in the validation set too. Any diagnostic built on comparing training and validation scores is blind to it by construction.
The most reliable detector is disbelief. Before auditing the pipeline, ask whether the score is possible. A model that beats radiologists on a first attempt, forecasts next quarter's revenue to within a percent, or reaches 0.99 AUC on a problem where the domain literature reports 0.75 has almost certainly found a shortcut. Every documented case above was first noticed because someone found the result too good rather than because a test failed.
Three cheap checks catch most of it. Re-split the data by time and see how far the score falls; re-split by entity — patient, user, device — and see how far it falls again; and inspect feature importances for a single column carrying implausible weight, then drop it and re-run. A score that survives all three is not proof of cleanliness, but a score that collapses under any of them has told you exactly where to look.
Offline-versus-online divergence is the last line of defence, and it is expensive. By the time monitoring shows that live precision is half the offline figure, the model is already deployed and decisions have already been made on it. Treating a large offline/online gap as a leakage hypothesis, rather than assuming the world changed between deployment and now, shortens that investigation considerably.
Challenges
Removing the leaked column often does not remove the leak. The KDD Cup case is the canonical demonstration: the patient ID was the visible symptom, and the source signature persisted in the image features after the ID was deleted. Whenever positives and negatives come from different places, the collection process itself is the leak, and no amount of column-dropping fixes it. The repair is at the data-collection level — sampling controls from the same institutions, scanners and time windows as the cases.
Some legitimate features are genuinely hard to compute honestly. "Average order value over the last 90 days" is a perfectly sensible predictor, and it is correct only if the window is recomputed relative to each row's prediction timestamp rather than to the date the export was taken. Getting this right across dozens of aggregates is the point-in-time-correctness problem that feature stores exist to solve, and it is real engineering, not a matter of remembering a rule.
Repeated evaluation leaks the test set slowly, through the analyst. Every time you look at the held-out score and change something in response, a little information moves from the test set into the model — not through the code, but through your decisions. After a hundred such rounds the "held-out" number is optimistic, which is why a genuinely untouched final test set, opened once, is worth the data it costs.
Target encoding leaks inside a single training fold. Replacing a high-cardinality categorical with the mean target for that category means each row's encoded value is computed partly from its own label. For a category with one member, the encoding is the label. This leak happens entirely within the training data, so no split protects against it; it is why gradient boosting libraries implement ordered or out-of-fold variants of the encoding rather than the naive version.
You cannot group by a structure you do not know exists. Grouped splits require knowing the grouping variable, and datasets frequently arrive without one — no patient key, no device ID, no indication that 40% of the rows came from a single hospital. When the group is unknown and unrecoverable, the honest response is a wider uncertainty band on the reported score, not a confident number from a random split.
Code Example
The script below builds one synthetic churn dataset and scores it three ways with the same
random forest and the same five folds. Two hundred customers each
contribute five monthly rows. One feature is a noisy measurement of a genuine driver of churn;
eight more are stable account traits that carry no information about churn but do identify the
customer. A separate exit_survey column is filled in only after a customer has already left.
import numpy as np
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import GroupKFold, KFold, cross_val_score
rng = np.random.default_rng(0)
n_cust, rows_per = 200, 5
n = n_cust * rows_per
# Per-customer truth: one genuine driver of churn, plus eight stable account
# traits that say nothing about churn but do identify the customer.
driver = rng.standard_normal(n_cust)
churn = (driver + 0.7 * rng.standard_normal(n_cust) > 0).astype(int)
traits = rng.standard_normal((n_cust, 8))
cust = np.repeat(np.arange(n_cust), rows_per) # 5 monthly rows per customer
X = np.column_stack([
driver[cust] + 0.3 * rng.standard_normal(n), # noisy monthly measurement
traits[cust] + 0.05 * rng.standard_normal((n, 8)), # near-constant per customer
])
# A field only ever filled in after the customer has already churned.
exit_survey = churn[cust] * rng.normal(3, 1, n) + rng.normal(0, 1, n)
y = churn[cust]
clf = RandomForestClassifier(n_estimators=200, random_state=0)
group_cv, random_cv = GroupKFold(5), KFold(5, shuffle=True, random_state=0)
honest = cross_val_score(clf, X, y, cv=group_cv, groups=cust).mean()
target_leak = cross_val_score(clf, np.column_stack([X, exit_survey]), y,
cv=group_cv, groups=cust).mean()
group_leak = cross_val_score(clf, X, y, cv=random_cv).mean()
print(f"honest GroupKFold, no exit survey : {honest:.1%}")
print(f"target leak GroupKFold + exit survey : {target_leak:.1%}")
print(f"group leak random KFold, no survey : {group_leak:.1%}")
Output:
honest GroupKFold, no exit survey : 73.6%
target leak GroupKFold + exit survey : 91.9%
group leak random KFold, no survey : 96.7%
Three numbers, one dataset, and only the honest one is real. 73.6% is what this problem
actually supports: the driver is noisy, the label is a coin flip biased by it, and no model can do
much better. Adding the exit survey lifts the score to 91.9% — an 18.3-point gain, from a
column that in production is blank for every customer you would want to predict. Switching from
GroupKFold to a plain shuffled KFold produces 96.7% without adding any feature at all: with
five rows per customer and five folds, roughly four-fifths of each customer's rows sit in training
whenever the remaining ones are being scored, so the forest can identify the customer from their
eight near-constant traits and recall the answer. That is a 23.1-point gain bought entirely by
changing one argument in the cross-validation call.
Notice what a reviewer would see. The leaky runs use the same estimator, the same fold count and the same metric as the honest one. There is no exception, no warning, no divergent training curve — just a better number, produced by two lines that look completely ordinary. A person shown only 96.7% has no way to tell it from a real result, which is the whole difficulty of the subject and the reason a leakage audit has to be a deliberate step in training rather than something you expect the metrics to surface on their own.