Definition
Early stopping ends model training at the point where performance on a held-out validation set stops improving — usually several passes before the training loss has flattened — because past that point the model is fitting quirks of the training data rather than the structure that carries over to new data. It is the cheapest and one of the most widely used defences against overfitting: it costs nothing extra to compute, adds no term to the loss, and often improves the final model simply by declining to keep training it.
The core observation is that training loss and validation loss part ways. As training proceeds, the loss measured on the data the model is fitting falls almost monotonically — a big enough network can drive it close to zero by memorising. The loss on data the model has never updated on falls too, but only for a while; once the model starts encoding noise specific to the training set, the validation loss bottoms out and begins to climb. The gap between the two curves is the overfitting. Early stopping watches the validation curve and freezes the model at its lowest point.
How It Works
The mechanism has three moving parts, and the second and third are the ones first drafts get wrong.
First, a monitored metric on held-out data. After each epoch (or every few hundred steps) you evaluate the model on a validation split that is never used to compute gradients. Usually this is the validation loss; it can be any metric you care about, such as accuracy or F1, as long as you are consistent about whether lower or higher is better.
Second, patience. Validation loss is noisy — a single epoch can tick upward and then resume falling to a new low. If you stopped the instant it rose, you would stop on noise. Patience is the number of consecutive epochs you allow with no new best before you give up. A patience of 0 stops far too eagerly; values from roughly 5 to 20 epochs are typical, and patience is itself a hyperparameter you tune.
Third, restoring the best weights. This is the step people forget. When you stop after patience runs out, the current weights are not the best ones — they are several epochs of overfitting past the best ones. You must have checkpointed the weights from the epoch with the lowest validation loss and load them back. Stopping without restoring throws away the whole point: you would ship the memorised model you were trying to avoid.
Put together, the loop is: after each epoch, compute the validation metric; if it beats the
best seen so far, save the weights and reset a counter; otherwise increment the counter; when
the counter reaches patience, stop and reload the saved weights.
A worked training curve
Take a run where the best model is reached well before training would naturally end. The training loss decays geometrically toward a floor while the validation loss is U-shaped — falling, bottoming out, then rising as overfitting sets in:
- The best validation loss is 0.3531, reached at epoch 9.
- With
patience = 3, the run stops at epoch 12, having seen three straight epochs (10, 11, 12) with no improvement, and restores the epoch-9 weights. - Had you instead trained a fixed 20 epochs and kept the final model, its validation loss would be about 0.43 — roughly 22% worse than the model early stopping keeps.
So the naive fixed-length run does more work and produces a worse model. The best epoch, 9,
sits well before both the patience-triggered stop at 12 and the fixed budget of 20 — early
stopping recovers that best model automatically. The ## Code Example below runs exactly this
loop and prints the table.
Real-World Applications
Early stopping is the default in every major training framework, which is the clearest sign
of how routine it is. Keras exposes it as the EarlyStopping callback with patience and
restore_best_weights arguments; PyTorch Lightning ships an EarlyStopping callback keyed to
any logged metric; Hugging Face's Trainer provides EarlyStoppingCallback alongside
load_best_model_at_end, and XGBoost and LightGBM both take an early_stopping_rounds
argument that halts boosting when a validation metric stalls.
Where it does the most work is fine-tuning on small datasets. When you adapt a large pretrained model to a few thousand labelled examples, the model has more than enough capacity to memorise them in a handful of epochs, and the validation curve turns up fast. Here early stopping, often with patience as low as 2 or 3 epochs, is frequently what separates a model that generalises from one that has simply learned the training set by heart.
Gradient-boosted trees rely on it structurally: boosting keeps adding trees, each
correcting the last, and without a stopping rule it will keep adding them until it overfits.
early_stopping_rounds on a validation fold is standard practice in the Kaggle-style tabular
work where XGBoost and LightGBM dominate, and it doubles as a way to discover the right
number of trees rather than guessing it.
It is worth being honest about where the term does not cleanly apply. In frontier large-language-model pretraining, the corpus is so large that each token is typically seen only about once — the model never re-fits the same data, so the classic picture of validation loss turning back up does not occur. Validation loss keeps falling, and training is halted by the compute or token budget set by scaling laws, not by an early-stopping trigger. The concept returns in force during fine-tuning, where epochs repeat over a small dataset.
Key Concepts
Why it acts as a regularizer
Early stopping is not just a convenience that saves compute — it is a genuine form of regularization, and you can see why on the mechanics of gradient descent. Weights start small (near their random initialisation) and move outward with each gradient-descent update toward the values that minimise training loss. Stopping early leaves them partway along that journey, closer to their small starting point than the fully-fit solution — and keeping weights small is exactly what an L2 penalty does.
Make it quantitative on a single parameter with a quadratic loss. Let the learning rate be
η = 0.1 and the local curvature be λ = 2, so ηλ = 0.2. After t gradient steps from a zero
start, the parameter reaches a fraction 1 − (1 − ηλ)^t of its unregularised least-squares
value:
- after 3 updates: 1 − 0.8³ = 1 − 0.512 = 48.8% of the way there;
- after 10 updates: 1 − 0.8¹⁰ = 1 − 0.107 = 89.3%;
- after 30 updates: 1 − 0.8³⁰ ≈ 99.9%.
Stopping at 10 updates leaves the weight shrunk to about 89% of its overfit value — the same pull toward the origin an explicit L2 penalty would apply. This is why the quantity that matters is the number of effective updates, roughly η·t: more updates let the weights travel farther from initialisation and use more of the model's capacity. Goodfellow, Bengio and Courville (Deep Learning, 2016, §7.8) make the correspondence precise: for a quadratic loss, early stopping is equivalent to L2 regularization, with η·t playing the inverse role of the penalty coefficient — fewer updates behave like a stronger penalty.
The validation set is spent, not free
Early stopping chooses an epoch, and any choice made by looking at data uses up that data. The stopped model has been selected to look good on the validation set, so its validation loss is an optimistic estimate of true performance. This is why you need three splits, not two: a training set to compute gradients, a validation set to decide when to stop, and a genuinely untouched test set to report the final number. Reusing the validation set as the headline metric quietly inflates it. With little data, this is where cross-validation helps — though combining it with early stopping means each fold stops at a different epoch, which is a subtlety worth handling deliberately.
Challenges
A noisy validation curve fools a small patience. If your validation set is tiny or your batches are small, the per-epoch metric jitters, and a short patience will stop on a random dip before the model has converged. The fixes are a larger or more representative validation split, a longer patience, or smoothing the monitored metric — but each trades away some of the compute saving that made early stopping attractive.
Local plateaus look like the end. Loss curves are not always a clean U. A model can sit on a plateau for many epochs and then drop sharply to a new, lower basin — behaviour seen with warmup-and-decay learning-rate schedules and with some architectures. A patience shorter than the plateau stops the run just before the improvement it was waiting for. If you use a learning- rate schedule, patience has to outlast the schedule's flat regions.
It couples with the learning-rate schedule. The epoch at which validation loss bottoms out depends on how fast the weights are moving, which the learning rate controls. Change the schedule and the best epoch moves, so a patience tuned for one schedule can be wrong for another. Early stopping and the learning-rate schedule are two knobs on the same thing — the number of effective updates — and they must be tuned together, not in isolation.
Restoring best weights costs memory or I/O. Keeping the best checkpoint means holding a second copy of the model in memory or writing it to disk each time the metric improves. For a large model that is a real cost, and skipping the restore to save it silently defeats the whole technique — you stop, but keep the overfit weights.
Code Example
The loop below implements early stopping over a deterministic synthetic curve: training loss decays geometrically while validation loss is U-shaped. It tracks the best epoch, applies a patience of 3, and reports which weights it would restore. This is the run described in How It Works; the output is the program's real output.
# A deterministic training run: train loss keeps falling, validation loss
# turns back up once the model starts memorising. Early stopping watches the
# validation loss, waits `patience` epochs for a new best, then stops and
# restores the weights from the best epoch.
def train_loss(e):
return round(1.00 * 0.75 ** e + 0.05, 4) # monotonically decreasing
def val_loss(e):
return round(0.60 * 0.75 ** e + 0.012 * e + 0.20, 4) # U-shaped
patience = 3
best_val = float("inf")
best_epoch = -1
wait = 0
stopped_at = None
for e in range(20):
tl, vl = train_loss(e), val_loss(e)
if vl < best_val:
best_val, best_epoch, wait = vl, e, 0
marker = " <- new best (weights saved)"
else:
wait += 1
marker = f" (no improvement for {wait})"
print(f"epoch {e:2d} | train {tl:.4f} | val {vl:.4f}{marker}")
if wait >= patience:
stopped_at = e
break
print()
print(f"stopped at epoch {stopped_at}; restored weights from epoch {best_epoch} "
f"(val {best_val:.4f})")
print(f"epochs trained past the best model: {stopped_at - best_epoch}")
Output:
epoch 0 | train 1.0500 | val 0.8000 <- new best (weights saved)
epoch 1 | train 0.8000 | val 0.6620 <- new best (weights saved)
epoch 2 | train 0.6125 | val 0.5615 <- new best (weights saved)
epoch 3 | train 0.4719 | val 0.4891 <- new best (weights saved)
epoch 4 | train 0.3664 | val 0.4378 <- new best (weights saved)
epoch 5 | train 0.2873 | val 0.4024 <- new best (weights saved)
epoch 6 | train 0.2280 | val 0.3788 <- new best (weights saved)
epoch 7 | train 0.1835 | val 0.3641 <- new best (weights saved)
epoch 8 | train 0.1501 | val 0.3561 <- new best (weights saved)
epoch 9 | train 0.1251 | val 0.3531 <- new best (weights saved)
epoch 10 | train 0.1063 | val 0.3538 (no improvement for 1)
epoch 11 | train 0.0922 | val 0.3573 (no improvement for 2)
epoch 12 | train 0.0817 | val 0.3630 (no improvement for 3)
stopped at epoch 12; restored weights from epoch 9 (val 0.3531)
epochs trained past the best model: 3
Notice that at the stop, training loss (0.0817) is still falling fast and less than a quarter of the validation loss (0.3630) — the model is still "improving" on the data it can see. Watching training loss would have told you to keep going. The validation curve is what caught the turn.