Cross-Validation (CV)

Definition

Cross-validation is a way to estimate how well a model will perform on data it has never seen: you split the training data into k equal parts, train k times with a different part held out each round, and average the k scores — so every row is tested exactly once, instead of only the rows that happened to land in a single test set. Use k = 5 unless you have a reason not to, and k = 10 when the dataset is small enough that you can afford twice the training runs.

The single most important thing to understand about it is what it does not do. Cross-validation does not make your model better. It makes your estimate better. The model that comes out the other side is exactly as overfit as the model that went in — what has changed is that you can now see it. Treating a good cross-validation score as evidence that you have fixed something is the most common misreading of the technique, and it is how a model with a reported 94% ends up doing 71% in production.

How It Works

Start with the problem it solves. Suppose you have 100 labelled rows and you do the usual thing: hold out 20 for testing, train on 80, and score. Now assume, generously, that your model's true accuracy is 80%. The estimate you just computed is the average of 20 coin flips weighted 80/20, so its standard error is √(0.8 × 0.2 / 20) ≈ 8.9 percentage points. A 95% interval around your measurement spans roughly 62% to 98%. Change the random seed and the same model, on the same data, can honestly report 70% one day and 90% the next. Nothing about the model changed. You measured a different 20 rows.

Cross-validation attacks that directly. With 5-fold, the 100 rows are split into five blocks of 20. Round one trains on blocks 2–5 and tests on block 1; round two trains on blocks 1 and 3–5 and tests on block 2; and so on. Five training runs later, every one of the 100 rows has been predicted exactly once by a model that never saw it during training. The reported score is the mean of the five fold scores, and because it rests on 100 test predictions rather than 20, its standard error falls toward √(0.16 / 100) = 4 percentage points. The real reduction is somewhat less than that, because the five training sets overlap in 60 of their 80 rows and the fold scores are therefore correlated rather than independent — but the direction is right and the effect is large.

The second output is the one people throw away. Alongside the mean, cross-validation hands you the spread across folds, and that spread is information. Fold scores of 0.79, 0.81, 0.80, 0.82, 0.78 and fold scores of 0.91, 0.65, 0.88, 0.72, 0.89 both average to about 0.80, and they mean completely different things: the first is a model you can plan around, the second is a model whose behaviour depends heavily on which rows it was shown, and averaging it into a single headline number hides exactly the fact you needed.

Choosing k, with the arithmetic

Larger k pulls in two directions at once.

Each fold's model trains on (k−1)/k of the data, so with k = 5 it sees 80% of your rows and with k = 10 it sees 90%. Since less training data usually means a slightly worse model, small k gives an estimate that is pessimistically biased — it tells you how good a model trained on 80% of your data is, when what you will actually ship is trained on 100%. Raising k shrinks that bias. It also multiplies your compute bill by exactly k: 10-fold is precisely twice the training runs of 5-fold, and the folds become more similar to each other (their training sets now overlap in 8 blocks out of 9), so their scores are more correlated and averaging them helps less than the count suggests.

Five and ten are conventional because they sit where those curves cross for most datasets, not because of any theorem. The honest rule is that k matters much less than people think, and the things below — leakage, grouping, temporal order — matter far more.

Leave-one-out (LOOCV) is the extreme: k = n, one row held out per round. It is nearly unbiased, since each model trains on n−1 of n rows, and it is almost always the wrong choice. On 1,000 rows it is 1,000 training runs — at 30 seconds each that is 8.3 hours against 2.5 minutes for 5-fold, for an estimate that is usually no more trustworthy. The reason it is not more trustworthy is subtle and worth holding onto: the 1,000 models differ from each other by a single row, so they are nearly identical, and their errors are nearly perfectly correlated. Averaging 1,000 correlated numbers reduces variance far less than averaging 1,000 independent ones. In classification it is worse still, because each fold's score can only be 0 or 1 — you are averaging a thousand coin flips, not a thousand accuracies. LOOCV earns its cost on genuinely tiny datasets (tens of rows) where losing 20% of the data to a test fold is unaffordable, and rarely anywhere else.

Types

The variants are not interchangeable styles. Each one exists because plain random k-fold breaks in a specific, identifiable way, and picking the wrong one does not degrade your estimate gently — it invalidates it.

K-fold

The default. Shuffle, cut into k blocks, rotate the held-out block. Appropriate when rows are independent of each other and order does not matter — most tabular problems where each row is a separate customer, transaction or measurement.

Stratified k-fold

Fixes: rare classes distributed unevenly by chance. With a 2% positive rate and five folds of 100 rows, random splitting can easily give one fold 4 positives and another zero — and a fold with zero positives makes recall undefined and the average meaningless. Stratified splitting forces each fold to carry the same class proportions as the full dataset. It costs nothing and is the right default for every classification problem, which is why StratifiedKFold is what scikit-learn quietly uses when you pass an integer cv to a classifier.

Leave-one-out

Fixes: not having enough rows to spare 20% of them. Covered above; k = n, nearly unbiased, high variance, n training runs.

Group k-fold

Fixes: multiple rows describing the same underlying entity. Say you have 200 chest X-rays from 50 patients — four images each. Under random 5-fold, for any given test image the chance that all three of its siblings also land in that same test fold is about (1/5)³ ≈ 0.8%, which means roughly 99% of your test images have another image of the same patient sitting in the training set. The model does not have to learn pathology; it can learn the patient. GroupKFold takes a group label and guarantees that every row of a group lands on the same side of every split. The same problem appears with repeated measurements per user, per device, per store and per experiment run, and it is the single most common reason a medical or industrial model that validated beautifully collapses on a new site.

Time-series split (forward chaining)

Fixes: training on the future to predict the past. If your rows are ordered in time, random folds put January in the test set and December in the training set, and the model gets to see the answer before the question — a market regime, a viral product, a policy change. The score you get is not optimistic, it is invalid: it measures a capability the deployed model will never have. Forward chaining trains on rows 1…t and tests on rows t+1…t+m, then extends the window and repeats, so training data always precedes test data. Any time series problem — demand, prices, sensor readings, churn — needs this, and a random k-fold reported on temporal data should be treated as a result that has not been produced.

Nested cross-validation

Fixes: the score being contaminated by the tuning it was used for. Covered in Key Concepts below, because it is less a splitting scheme than an argument about what a reported number means.

Real-World Applications

scikit-learn's defaults are the de facto standard. cross_val_score, GridSearchCV and RandomizedSearchCV all default to 5-fold, and for classifiers that 5-fold is stratified. That default is the reason "5-fold CV" appears in so many papers and notebooks, and it is why Pipeline matters so much: putting your preprocessing inside a Pipeline is what makes each transformation refit inside each fold rather than once on everything.

Kaggle competitions turned "trust your CV, not the leaderboard" into folklore. The public leaderboard is a single split scored on a fraction of the test set, so it is exactly the high-variance one-shot estimate described above. Teams that tune against it are fitting a few thousand rows of noise and routinely drop dozens of places when the private leaderboard is revealed; teams that build a reliable local cross-validation scheme and trust it over the public score are the ones who stay put. This is the clearest real-world demonstration that a single split is a noisy instrument.

Medical imaging enforces patient-level splits. Reviewers at radiology and pathology venues routinely reject work that splits at the image or slice level rather than the patient level, precisely because of the group-leakage arithmetic above. Grouped splitting is now an expected part of the methods section, not an optional refinement.

Credit risk and fraud use out-of-time validation. Financial model-risk review expects a model to be tested on a period it was not trained on, because the thing that breaks a credit scorecard is not a novel borrower but a novel year. Forward-chaining validation is how that requirement is operationalised, and a random k-fold on transaction history would not satisfy it.

Genomics learned the feature-selection lesson the expensive way. In early microarray studies it was common to screen tens of thousands of genes down to a handful of "informative" ones using the whole dataset, then cross-validate a classifier on those selected genes. The resulting accuracies were spectacular and largely fictitious — the selection step had already looked at the labels of the test rows. The correction, re-running selection inside every fold, is now standard practice in bioinformatics pipelines, and the code example at the end of this page reproduces the effect in about twenty lines.

Key Concepts

Nested cross-validation, and why the flat version flatters you

Here is the mechanism, because it is easy to state and easy to miss. You run 5-fold cross-validation across a grid of 12 hyperparameter settings, take the best average score, and report it. That reported number is the maximum of 12 noisy estimates, and the maximum of a set of noisy numbers is biased upward — some of that winning setting's advantage is real skill and some of it is that setting having got lucky on these particular folds. You used the same folds to choose and to report, so the choice contaminated the report.

Nested cross-validation separates the two jobs. An outer loop splits the data and, for each outer fold, an entire inner cross-validation runs on the outer training portion to pick hyperparameters; the chosen model is then scored once on the outer test fold, which no part of the selection ever touched. The outer scores are what you report.

The cost is the reason people skip it. With 12 hyperparameter combinations, 5 inner folds and 5 outer folds, you pay 12 × 5 = 60 fits per outer fold plus one refit, times 5 outer folds = 305 training runs, against 61 for a flat grid search — a 5× multiplier that grows linearly with your outer k. Skipping it is often a defensible engineering decision. Skipping it and then quoting the tuned score as an unbiased estimate of production performance is not, and it is a large part of why published numbers are so hard to reproduce.

When cross-validation is the wrong tool

Cross-validation is a technique for when data is scarce relative to what you would like to spend on testing. Two situations remove that premise.

Large datasets. Take a million labelled rows and hold out 20%. That test set has 200,000 rows, so at 80% accuracy your estimate carries a standard error of √(0.16 / 200,000) ≈ 0.09 percentage points. The number is already precise to a tenth of a point; 5-fold would multiply your training cost by five to sharpen something that is not blunt. Use a single hold-out — and spend the saved compute on a second, permanently untouched test set instead.

Deep learning. If a single training run costs 12 GPU-hours, 5-fold costs 60, and any hyperparameter search multiplies from there. This is not a failure of rigour in the literature; it is arithmetic. The convention of a fixed train/validation/test split, sometimes repeated over a handful of random seeds to show variance, exists because full cross-validation on large models is unaffordable — and knowing that is part of reading those results correctly.

Challenges

Leakage: the failure that actually kills projects

Every other problem on this page costs you accuracy at the margin. Leakage costs you the whole result, silently, while every metric on your screen improves. It happens whenever information that would not be available at prediction time reaches the model during training, and cross-validation does not protect you from it — a leaky pipeline produces five beautifully consistent folds.

• Preprocessing fitted before the split. This is the most common instance and the least visible. StandardScaler().fit(X) on the full dataset computes a mean and standard deviation that include your test rows; an imputer fitted on everything fills training gaps using test-set statistics; a feature selector run once on the whole table has already read the test labels. The fix is mechanical: every transformation that learns anything from data must be fitted inside each fold, on that fold's training rows only, which is precisely what a scikit-learn Pipeline passed to cross_val_score does for you.
• Grouped rows split across folds. Four X-rays per patient, twenty sessions per user, hourly readings from one machine — random folds test the model on an entity it has already memorised. Use GroupKFold with the entity ID, and be suspicious of any dataset where "row" and "subject" are not the same thing.
• Temporal order ignored. Random folds on time-ordered data let the model learn from the future. Use forward chaining, and treat a random k-fold score on a forecasting problem as a number that was never measured.
• Duplicate and near-duplicate rows. Scraped datasets, augmented datasets and merged exports routinely contain the same record twice. A duplicate landing in both training and test turns memorisation into apparent generalisation, and deduplicating before splitting is a five-minute check that is worth running before you trust any surprisingly good score.

The estimate is not a guarantee

Cross-validation tells you how a model performs on data drawn from the same distribution as your training set. It says nothing about data from a different hospital, a different season, or after a product redesign. Every fold in your evaluation came from the same pool, so a shift in that pool is outside what the method can see, and monitoring after deployment remains a separate obligation rather than something a good CV score discharges.

Code Example

Feature selection performed before the split, on data that contains no signal whatsoever. Both runs use identical data, an identical model and identical 5-fold splits; the only difference is where the selection step happens.

import numpy as np
from sklearn.feature_selection import SelectKBest, f_classif
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score
from sklearn.pipeline import Pipeline

rng = np.random.default_rng(0)
X = rng.normal(size=(200, 5000))      # pure noise
y = rng.integers(0, 2, size=200)      # labels independent of X

model = LogisticRegression(max_iter=1000)

# WRONG: pick the 20 "best" features using every row, then cross-validate
X_leaked = SelectKBest(f_classif, k=20).fit_transform(X, y)
leaky = cross_val_score(model, X_leaked, y, cv=5)

# RIGHT: selection happens inside each fold, on training rows only
honest = cross_val_score(
    Pipeline([("select", SelectKBest(f_classif, k=20)), ("clf", model)]),
    X, y, cv=5,
)

print(f"selected before splitting: {leaky.mean():.3f}  folds {np.round(leaky, 2)}")
print(f"selected inside each fold: {honest.mean():.3f}  folds {np.round(honest, 2)}")

Output (scikit-learn 1.9.0):

selected before splitting: 0.795  folds [0.8  0.72 0.82 0.78 0.85]
selected inside each fold: 0.510  folds [0.55 0.48 0.52 0.57 0.42]

There is nothing to learn here — y was generated independently of X. The correct pipeline says so: 51%, which is chance. The leaky version reports 79.5% with five agreeable-looking folds and a reassuringly small spread, because searching 5,000 random columns for the 20 that best match the labels will always find 20 that match well, and once those columns are chosen, the test rows they were chosen to fit are no longer unseen.

A 29-point lift out of nothing at all is what leakage looks like from the inside: not an error message, but a result that finally seems to be working.