---
source: 'https://howaiworks.ai/glossary/cross-validation'
section: glossary
title: Cross-Validation (CV)
description: >-
  How cross-validation turns one lucky train/test split into a stable estimate,
  why 5 or 10 folds, and the leakage that quietly inflates every score.
tags:
  - cross-validation
  - Machine Learning
  - generalization
  - hyperparameters
  - model evaluation
category: Machine Learning
datePublished: '2025-08-20'
lastUpdated: '2026-07-24'
---

# Cross-Validation (CV)

> How cross-validation turns one lucky train/test split into a stable estimate, why 5 or 10 folds, and the leakage that quietly inflates every score.

## Definition

Cross-validation is a way to estimate how well a model will perform on data it has never seen: you
split the training data into k equal parts, train k times with a different part held out each
round, and average the k scores — so every row is tested exactly once, instead of only the rows
that happened to land in a single test set. **Use k = 5 unless you have a reason not to**, and
k = 10 when the dataset is small enough that you can afford twice the training runs.

The single most important thing to understand about it is what it does *not* do. Cross-validation
does not make your model better. It makes your *estimate* better. The model that comes out the
other side is exactly as [overfit](https://howaiworks.ai/glossary/overfitting) as the model that went in — what has
changed is that you can now see it. Treating a good cross-validation score as evidence that you
have fixed something is the most common misreading of the technique, and it is how a model with a
reported 94% ends up doing 71% in production.

## How It Works

Start with the problem it solves. Suppose you have 100 labelled rows and you do the usual thing:
hold out 20 for testing, train on 80, and score. Now assume, generously, that your model's true
accuracy is 80%. The estimate you just computed is the average of 20 coin flips weighted 80/20, so
its standard error is √(0.8 × 0.2 / 20) ≈ **8.9 percentage points**. A 95% interval around your
measurement spans roughly **62% to 98%**. Change the random seed and the same model, on the same
data, can honestly report 70% one day and 90% the next. Nothing about the model changed. You
measured a different 20 rows.

Cross-validation attacks that directly. With 5-fold, the 100 rows are split into five blocks of
20. Round one trains on blocks 2–5 and tests on block 1; round two trains on blocks 1 and 3–5 and
tests on block 2; and so on. Five training runs later, every one of the 100 rows has been predicted
exactly once by a model that never saw it during training. The reported score is the mean of the
five fold scores, and because it rests on 100 test predictions rather than 20, its standard error
falls toward √(0.16 / 100) = **4 percentage points**. The real reduction is somewhat less than
that, because the five training sets overlap in 60 of their 80 rows and the fold scores are
therefore correlated rather than independent — but the direction is right and the effect is large.

The second output is the one people throw away. Alongside the mean, cross-validation hands you the
spread across folds, and that spread is information. Fold scores of 0.79, 0.81, 0.80, 0.82, 0.78
and fold scores of 0.91, 0.65, 0.88, 0.72, 0.89 both average to about 0.80, and they mean
completely different things: the first is a model you can plan around, the second is a model whose
behaviour depends heavily on which rows it was shown, and averaging it into a single headline
number hides exactly the fact you needed.

### Choosing k, with the arithmetic

Larger k pulls in two directions at once.

Each fold's model trains on (k−1)/k of the data, so with k = 5 it sees 80% of your rows and with
k = 10 it sees 90%. Since less training data usually means a slightly worse model, small k gives an
estimate that is **pessimistically biased** — it tells you how good a model trained on 80% of your
data is, when what you will actually ship is trained on 100%. Raising k shrinks that bias. It also
multiplies your compute bill by exactly k: 10-fold is precisely twice the training runs of 5-fold,
and the folds become more similar to each other (their training sets now overlap in 8 blocks out of
9), so their scores are more correlated and averaging them helps less than the count suggests.

Five and ten are conventional because they sit where those curves cross for most datasets, not
because of any theorem. The honest rule is that k matters much less than people think, and the
things below — leakage, grouping, temporal order — matter far more.

**Leave-one-out** (LOOCV) is the extreme: k = n, one row held out per round. It is nearly unbiased,
since each model trains on n−1 of n rows, and it is almost always the wrong choice. On 1,000 rows
it is 1,000 training runs — at 30 seconds each that is **8.3 hours** against **2.5 minutes** for
5-fold, for an estimate that is usually no more trustworthy. The reason it is not more trustworthy
is subtle and worth holding onto: the 1,000 models differ from each other by a single row, so they
are nearly identical, and their errors are nearly perfectly correlated. Averaging 1,000 correlated
numbers reduces variance far less than averaging 1,000 independent ones. In classification it is
worse still, because each fold's score can only be 0 or 1 — you are averaging a thousand coin
flips, not a thousand accuracies. LOOCV earns its cost on genuinely tiny datasets (tens of rows)
where losing 20% of the data to a test fold is unaffordable, and rarely anywhere else.

## Types

The variants are not interchangeable styles. Each one exists because plain random k-fold breaks in
a specific, identifiable way, and picking the wrong one does not degrade your estimate gently — it
invalidates it.

### K-fold

The default. Shuffle, cut into k blocks, rotate the held-out block. Appropriate when rows are
independent of each other and order does not matter — most tabular problems where each row is a
separate customer, transaction or measurement.

### Stratified k-fold

Fixes: **rare classes distributed unevenly by chance.** With a 2% positive rate and five folds of
100 rows, random splitting can easily give one fold 4 positives and another zero — and a fold with
zero positives makes recall undefined and the average meaningless. Stratified splitting forces each
fold to carry the same class proportions as the full dataset. It costs nothing and is the right
default for every [classification](https://howaiworks.ai/glossary/classification) problem, which is why
`StratifiedKFold` is what scikit-learn quietly uses when you pass an integer `cv` to a classifier.

### Leave-one-out

Fixes: **not having enough rows to spare 20% of them.** Covered above; k = n, nearly unbiased, high
variance, n training runs.

### Group k-fold

Fixes: **multiple rows describing the same underlying entity.** Say you have 200 chest X-rays from
50 patients — four images each. Under random 5-fold, for any given test image the chance that all
three of its siblings also land in that same test fold is about (1/5)³ ≈ 0.8%, which means roughly
**99% of your test images have another image of the same patient sitting in the training set**. The
model does not have to learn pathology; it can learn the patient. `GroupKFold` takes a group label
and guarantees that every row of a group lands on the same side of every split. The same problem
appears with repeated measurements per user, per device, per store and per experiment run, and it
is the single most common reason a medical or industrial model that validated beautifully collapses
on a new site.

### Time-series split (forward chaining)

Fixes: **training on the future to predict the past.** If your rows are ordered in time, random
folds put January in the test set and December in the training set, and the model gets to see the
answer before the question — a market regime, a viral product, a policy change. The score you get
is not optimistic, it is *invalid*: it measures a capability the deployed model will never have.
Forward chaining trains on rows 1…t and tests on rows t+1…t+m, then extends the window and repeats,
so training data always precedes test data. Any [time series](https://howaiworks.ai/glossary/time-series) problem —
demand, prices, sensor readings, churn — needs this, and a random k-fold reported on temporal data
should be treated as a result that has not been produced.

### Nested cross-validation

Fixes: **the score being contaminated by the tuning it was used for.** Covered in Key Concepts
below, because it is less a splitting scheme than an argument about what a reported number means.

## Real-World Applications

**scikit-learn's defaults are the de facto standard.** `cross_val_score`, `GridSearchCV` and
`RandomizedSearchCV` all default to 5-fold, and for classifiers that 5-fold is stratified. That
default is the reason "5-fold CV" appears in so many papers and notebooks, and it is why
`Pipeline` matters so much: putting your preprocessing inside a `Pipeline` is what makes each
transformation refit inside each fold rather than once on everything.

**Kaggle competitions turned "trust your CV, not the leaderboard" into folklore.** The public
leaderboard is a single split scored on a fraction of the test set, so it is exactly the
high-variance one-shot estimate described above. Teams that tune against it are fitting a few
thousand rows of noise and routinely drop dozens of places when the private leaderboard is
revealed; teams that build a reliable local cross-validation scheme and trust it over the public
score are the ones who stay put. This is the clearest real-world demonstration that a single split
is a noisy instrument.

**Medical imaging enforces patient-level splits.** Reviewers at radiology and pathology venues
routinely reject work that splits at the image or slice level rather than the patient level,
precisely because of the group-leakage arithmetic above. Grouped splitting is now an expected part
of the methods section, not an optional refinement.

**Credit risk and fraud use out-of-time validation.** Financial model-risk review expects a model
to be tested on a period it was not trained on, because the thing that breaks a credit scorecard is
not a novel borrower but a novel year. Forward-chaining validation is how that requirement is
operationalised, and a random k-fold on transaction history would not satisfy it.

**Genomics learned the feature-selection lesson the expensive way.** In early microarray studies it
was common to screen tens of thousands of genes down to a handful of "informative" ones using the
whole dataset, then cross-validate a classifier on those selected genes. The resulting accuracies
were spectacular and largely fictitious — the selection step had already looked at the labels of
the test rows. The correction, re-running selection inside every fold, is now standard practice in
bioinformatics pipelines, and the code example at the end of this page reproduces the effect in
about twenty lines.

## Key Concepts

### Nested cross-validation, and why the flat version flatters you

Here is the mechanism, because it is easy to state and easy to miss. You run 5-fold
cross-validation across a grid of 12 hyperparameter settings, take the best average score, and
report it. That reported number is the **maximum** of 12 noisy estimates, and the maximum of a set
of noisy numbers is biased upward — some of that winning setting's advantage is real skill and some
of it is that setting having got lucky on these particular folds. You used the same folds to choose
and to report, so the choice contaminated the report.

Nested cross-validation separates the two jobs. An outer loop splits the data and, for each outer
fold, an entire inner cross-validation runs on the outer training portion to pick hyperparameters;
the chosen model is then scored once on the outer test fold, which no part of the selection ever
touched. The outer scores are what you report.

The cost is the reason people skip it. With 12 hyperparameter combinations, 5 inner folds and 5
outer folds, you pay 12 × 5 = 60 fits per outer fold plus one refit, times 5 outer folds =
**305 training runs**, against **61** for a flat grid search — a 5× multiplier that grows linearly
with your outer k. Skipping it is often a defensible engineering decision. Skipping it and then
quoting the tuned score as an unbiased estimate of production performance is not, and it is a large
part of why published numbers are so hard to reproduce.

### When cross-validation is the wrong tool

Cross-validation is a technique for when data is scarce relative to what you would like to spend on
testing. Two situations remove that premise.

**Large datasets.** Take a million labelled rows and hold out 20%. That test set has 200,000 rows,
so at 80% accuracy your estimate carries a standard error of √(0.16 / 200,000) ≈ **0.09 percentage
points**. The number is already precise to a tenth of a point; 5-fold would multiply your training
cost by five to sharpen something that is not blunt. Use a single hold-out — and spend the saved
compute on a second, permanently untouched test set instead.

**Deep learning.** If a single [training](https://howaiworks.ai/glossary/training) run costs 12 GPU-hours, 5-fold costs
60, and any hyperparameter search multiplies from there. This is not a failure of rigour in the
literature; it is arithmetic. The convention of a fixed train/validation/test split, sometimes
repeated over a handful of random seeds to show variance, exists because full cross-validation on
large models is unaffordable — and knowing that is part of reading those results correctly.

## Challenges

### [Leakage](https://howaiworks.ai/glossary/data-leakage): the failure that actually kills projects

Every other problem on this page costs you accuracy at the margin. Leakage costs you the whole
result, silently, while every metric on your screen improves. It happens whenever information that
would not be available at prediction time reaches the model during training, and cross-validation
does not protect you from it — a leaky pipeline produces five beautifully consistent folds.

- **Preprocessing fitted before the split.** This is the most common instance and the least
  visible. `StandardScaler().fit(X)` on the full dataset computes a mean and standard deviation
  that include your test rows; an imputer fitted on everything fills training gaps using test-set
  statistics; a feature selector run once on the whole table has already read the test labels. The
  fix is mechanical: every transformation that learns anything from data must be fitted **inside**
  each fold, on that fold's training rows only, which is precisely what a scikit-learn `Pipeline`
  passed to `cross_val_score` does for you.
- **Grouped rows split across folds.** Four X-rays per patient, twenty sessions per user, hourly
  readings from one machine — random folds test the model on an entity it has already memorised.
  Use `GroupKFold` with the entity ID, and be suspicious of any dataset where "row" and "subject"
  are not the same thing.
- **Temporal order ignored.** Random folds on time-ordered data let the model learn from the future.
  Use forward chaining, and treat a random k-fold score on a forecasting problem as a number that
  was never measured.
- **Duplicate and near-duplicate rows.** Scraped datasets, augmented datasets and merged exports
  routinely contain the same record twice. A duplicate landing in both training and test turns
  memorisation into apparent generalisation, and deduplicating before splitting is a five-minute
  check that is worth running before you trust any surprisingly good score.

### The estimate is not a guarantee

Cross-validation tells you how a model performs on data drawn from the same distribution as your
training set. It says nothing about data from a different hospital, a different season, or after a
product redesign. Every fold in your evaluation came from the same pool, so a shift in that pool is
outside what the method can see, and monitoring after [deployment](https://howaiworks.ai/glossary/model-deployment)
remains a separate obligation rather than something a good CV score discharges.

## Code Example

Feature selection performed before the split, on data that contains no signal whatsoever. Both runs
use identical data, an identical model and identical 5-fold splits; the only difference is *where*
the selection step happens.

```python
import numpy as np
from sklearn.feature_selection import SelectKBest, f_classif
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score
from sklearn.pipeline import Pipeline

rng = np.random.default_rng(0)
X = rng.normal(size=(200, 5000))      # pure noise
y = rng.integers(0, 2, size=200)      # labels independent of X

model = LogisticRegression(max_iter=1000)

# WRONG: pick the 20 "best" features using every row, then cross-validate
X_leaked = SelectKBest(f_classif, k=20).fit_transform(X, y)
leaky = cross_val_score(model, X_leaked, y, cv=5)

# RIGHT: selection happens inside each fold, on training rows only
honest = cross_val_score(
    Pipeline([("select", SelectKBest(f_classif, k=20)), ("clf", model)]),
    X, y, cv=5,
)

print(f"selected before splitting: {leaky.mean():.3f}  folds {np.round(leaky, 2)}")
print(f"selected inside each fold: {honest.mean():.3f}  folds {np.round(honest, 2)}")
```

Output (scikit-learn 1.9.0):

```
selected before splitting: 0.795  folds [0.8  0.72 0.82 0.78 0.85]
selected inside each fold: 0.510  folds [0.55 0.48 0.52 0.57 0.42]
```

There is nothing to learn here — `y` was generated independently of `X`. The correct pipeline says
so: **51%**, which is chance. The leaky version reports **79.5%** with five agreeable-looking folds
and a reassuringly small spread, because searching 5,000 random columns for the 20 that best match
the labels will always find 20 that match well, and once those columns are chosen, the test rows
they were chosen to fit are no longer unseen.

A 29-point lift out of nothing at all is what leakage looks like from the inside: not an error
message, but a result that finally seems to be working.

## Frequently Asked Questions

### How many folds should I use for cross-validation?

Use 5 unless you have a reason not to. Ten folds gives each model slightly more training data and a slightly less pessimistic estimate, at exactly double the training cost. Below a few hundred rows, 10 is often worth it; above a few hundred thousand, a single hold-out set is already precise enough that k-fold buys nothing.

### Does cross-validation prevent overfitting?

No. Cross-validation measures generalization, it does not improve it — the model you get out is exactly as overfit as the model you put in. It makes the problem *visible* by giving you an honest estimate, which you then fix with [regularization](https://howaiworks.ai/glossary/regularization), more data or a simpler model. See [overfitting](https://howaiworks.ai/glossary/overfitting) for the fixes themselves.

### Why is my cross-validation score much better than production?

Almost always leakage. Check three things in order: was any scaler, imputer or feature-selection step fitted before the split rather than inside each fold; do multiple rows share a patient, user or device that is now on both sides of a fold; and is the data temporal, in which case random folds trained on the future to predict the past.

### What is the difference between k-fold and leave-one-out cross-validation?

K-fold splits the data into k parts and trains k times. Leave-one-out is the extreme case k = n: one row held out per round, n training runs. It is nearly unbiased but its fold scores are almost perfectly correlated — each model saw all but one of the same rows — so averaging them reduces variance far less than the n-fold cost suggests.

### When should I use stratified cross-validation?

Whenever the target is a class label, and especially when classes are imbalanced. With a 2% positive rate and 5 folds of 100 rows, plain random folds can hand one fold 4 positives and another zero, which makes recall on that fold undefined. StratifiedKFold keeps each fold's class proportions equal to the whole dataset's, at no extra cost.

### Do I need cross-validation for deep learning?

Usually you cannot afford it. If one training run takes 12 GPU-hours, 5-fold turns that into 60, and a hyperparameter sweep multiplies it again — which is why almost every deep learning paper reports a single train/validation/test split, sometimes averaged over a few random seeds. Cross-validation is a small-and-medium-data technique.

### What is nested cross-validation and do I need it?

It wraps an inner cross-validation that picks hyperparameters inside an outer one that scores the result, so the number you report was never used to make a choice. You need it whenever you are publishing or comparing a tuned model's performance; you can skip it when you only need to rank candidates and have a separate untouched test set.

## Related

### Related terms

- [Generalization](https://howaiworks.ai/glossary/generalization)
- [Overfitting](https://howaiworks.ai/glossary/overfitting)
- [Model Deployment](https://howaiworks.ai/glossary/model-deployment)
- [Optimization](https://howaiworks.ai/glossary/optimization)
- [Machine Learning (ML)](https://howaiworks.ai/glossary/machine-learning)
- [Performance](https://howaiworks.ai/glossary/performance)

---

Source: https://howaiworks.ai/glossary/cross-validation — HowAIWorks.ai