--- source: 'https://howaiworks.ai/glossary/early-stopping' section: glossary title: Early Stopping description: >- Early stopping halts training once validation loss stops improving, so the model generalises instead of memorising. How patience and best-weight restore work. tags: - Machine Learning - overfitting - regularization - training - generalization - gradient descent - hyperparameters - model evaluation category: Machine Learning datePublished: '2026-07-25' lastUpdated: '2026-07-25' --- # Early Stopping > Early stopping halts training once validation loss stops improving, so the model generalises instead of memorising. How patience and best-weight restore work. ## Definition Early stopping ends model training at the point where performance on a held-out validation set stops improving — usually several passes before the training loss has flattened — because past that point the model is fitting quirks of the training data rather than the structure that carries over to new data. It is the cheapest and one of the most widely used defences against [overfitting](https://howaiworks.ai/glossary/overfitting): it costs nothing extra to compute, adds no term to the loss, and often improves the final model simply by declining to keep training it. The core observation is that training loss and validation loss part ways. As [training](https://howaiworks.ai/glossary/training) proceeds, the loss measured on the data the model is fitting falls almost monotonically — a big enough network can drive it close to zero by memorising. The loss on data the model has never updated on falls too, but only for a while; once the model starts encoding noise specific to the training set, the validation loss bottoms out and begins to climb. The gap between the two curves is the overfitting. Early stopping watches the validation curve and freezes the model at its lowest point. ## How It Works The mechanism has three moving parts, and the second and third are the ones first drafts get wrong. First, a **monitored metric on held-out data.** After each epoch (or every few hundred steps) you evaluate the model on a validation split that is never used to compute gradients. Usually this is the validation loss; it can be any metric you care about, such as accuracy or F1, as long as you are consistent about whether lower or higher is better. Second, **patience.** Validation loss is noisy — a single epoch can tick upward and then resume falling to a new low. If you stopped the instant it rose, you would stop on noise. Patience is the number of consecutive epochs you allow with no new best before you give up. A patience of 0 stops far too eagerly; values from roughly 5 to 20 epochs are typical, and patience is itself a [hyperparameter](https://howaiworks.ai/glossary/hyperparameter) you tune. Third, **restoring the best weights.** This is the step people forget. When you stop after patience runs out, the current weights are *not* the best ones — they are several epochs of overfitting past the best ones. You must have checkpointed the weights from the epoch with the lowest validation loss and load them back. Stopping without restoring throws away the whole point: you would ship the memorised model you were trying to avoid. Put together, the loop is: after each epoch, compute the validation metric; if it beats the best seen so far, save the weights and reset a counter; otherwise increment the counter; when the counter reaches `patience`, stop and reload the saved weights. ### A worked training curve Take a run where the best model is reached well before training would naturally end. The training loss decays geometrically toward a floor while the validation loss is U-shaped — falling, bottoming out, then rising as overfitting sets in: - The **best validation loss is 0.3531, reached at epoch 9.** - With `patience = 3`, the run stops at **epoch 12**, having seen three straight epochs (10, 11, 12) with no improvement, and restores the epoch-9 weights. - Had you instead trained a fixed 20 epochs and kept the final model, its validation loss would be about **0.43** — roughly 22% worse than the model early stopping keeps. So the naive fixed-length run does more work and produces a worse model. The best epoch, 9, sits well before both the patience-triggered stop at 12 and the fixed budget of 20 — early stopping recovers that best model automatically. The `## Code Example` below runs exactly this loop and prints the table. ## Real-World Applications Early stopping is the default in every major training framework, which is the clearest sign of how routine it is. Keras exposes it as the `EarlyStopping` callback with `patience` and `restore_best_weights` arguments; PyTorch Lightning ships an `EarlyStopping` callback keyed to any logged metric; Hugging Face's `Trainer` provides `EarlyStoppingCallback` alongside `load_best_model_at_end`, and XGBoost and LightGBM both take an `early_stopping_rounds` argument that halts boosting when a validation metric stalls. Where it does the most work is **fine-tuning on small datasets.** When you adapt a large pretrained model to a few thousand labelled examples, the model has more than enough capacity to memorise them in a handful of epochs, and the validation curve turns up fast. Here early stopping, often with patience as low as 2 or 3 epochs, is frequently what separates a model that generalises from one that has simply learned the training set by heart. **Gradient-boosted trees** rely on it structurally: boosting keeps adding trees, each correcting the last, and without a stopping rule it will keep adding them until it overfits. `early_stopping_rounds` on a validation fold is standard practice in the Kaggle-style tabular work where XGBoost and LightGBM dominate, and it doubles as a way to *discover* the right number of trees rather than guessing it. It is worth being honest about where the term does **not** cleanly apply. In frontier [large-language-model](https://howaiworks.ai/glossary/large-language-model) pretraining, the corpus is so large that each token is typically seen only about once — the model never re-fits the same data, so the classic picture of validation loss turning back up does not occur. Validation loss keeps falling, and training is halted by the compute or token budget set by [scaling laws](https://howaiworks.ai/glossary/scaling-laws), not by an early-stopping trigger. The concept returns in force during fine-tuning, where epochs repeat over a small dataset. ## Key Concepts ### Why it acts as a regularizer Early stopping is not just a convenience that saves compute — it is a genuine form of [regularization](https://howaiworks.ai/glossary/regularization), and you can see why on the mechanics of gradient descent. Weights start small (near their random initialisation) and move outward with each [gradient-descent](https://howaiworks.ai/glossary/gradient-descent) update toward the values that minimise training loss. Stopping early leaves them partway along that journey, closer to their small starting point than the fully-fit solution — and keeping weights small is exactly what an L2 penalty does. Make it quantitative on a single parameter with a quadratic loss. Let the learning rate be η = 0.1 and the local curvature be λ = 2, so ηλ = 0.2. After `t` gradient steps from a zero start, the parameter reaches a fraction 1 − (1 − ηλ)^t of its unregularised least-squares value: - after **3 updates**: 1 − 0.8³ = 1 − 0.512 = **48.8%** of the way there; - after **10 updates**: 1 − 0.8¹⁰ = 1 − 0.107 = **89.3%**; - after **30 updates**: 1 − 0.8³⁰ ≈ **99.9%**. Stopping at 10 updates leaves the weight shrunk to about 89% of its overfit value — the same pull toward the origin an explicit L2 penalty would apply. This is why the quantity that matters is the number of *effective updates*, roughly η·t: more updates let the weights travel farther from initialisation and use more of the model's capacity. Goodfellow, Bengio and Courville (Deep Learning, 2016, §7.8) make the correspondence precise: for a quadratic loss, early stopping is equivalent to L2 regularization, with η·t playing the inverse role of the penalty coefficient — fewer updates behave like a stronger penalty. ### The validation set is spent, not free Early stopping *chooses* an epoch, and any choice made by looking at data uses up that data. The stopped model has been selected to look good on the validation set, so its validation loss is an optimistic estimate of true performance. This is why you need three splits, not two: a training set to compute gradients, a validation set to decide when to stop, and a genuinely untouched test set to report the final number. Reusing the validation set as the headline metric quietly inflates it. With little data, this is where [cross-validation](https://howaiworks.ai/glossary/cross-validation) helps — though combining it with early stopping means each fold stops at a different epoch, which is a subtlety worth handling deliberately. ## Challenges **A noisy validation curve fools a small patience.** If your validation set is tiny or your batches are small, the per-epoch metric jitters, and a short patience will stop on a random dip before the model has converged. The fixes are a larger or more representative validation split, a longer patience, or smoothing the monitored metric — but each trades away some of the compute saving that made early stopping attractive. **Local plateaus look like the end.** Loss curves are not always a clean U. A model can sit on a plateau for many epochs and then drop sharply to a new, lower basin — behaviour seen with warmup-and-decay learning-rate schedules and with some architectures. A patience shorter than the plateau stops the run just before the improvement it was waiting for. If you use a learning- rate schedule, patience has to outlast the schedule's flat regions. **It couples with the learning-rate schedule.** The epoch at which validation loss bottoms out depends on how fast the weights are moving, which the learning rate controls. Change the schedule and the best epoch moves, so a patience tuned for one schedule can be wrong for another. Early stopping and the learning-rate schedule are two knobs on the same thing — the number of effective updates — and they must be tuned together, not in isolation. **Restoring best weights costs memory or I/O.** Keeping the best checkpoint means holding a second copy of the model in memory or writing it to disk each time the metric improves. For a large model that is a real cost, and skipping the restore to save it silently defeats the whole technique — you stop, but keep the overfit weights. ## Code Example The loop below implements early stopping over a deterministic synthetic curve: training loss decays geometrically while validation loss is U-shaped. It tracks the best epoch, applies a patience of 3, and reports which weights it would restore. This is the run described in *How It Works*; the output is the program's real output. ```python # A deterministic training run: train loss keeps falling, validation loss # turns back up once the model starts memorising. Early stopping watches the # validation loss, waits `patience` epochs for a new best, then stops and # restores the weights from the best epoch. def train_loss(e): return round(1.00 * 0.75 ** e + 0.05, 4) # monotonically decreasing def val_loss(e): return round(0.60 * 0.75 ** e + 0.012 * e + 0.20, 4) # U-shaped patience = 3 best_val = float("inf") best_epoch = -1 wait = 0 stopped_at = None for e in range(20): tl, vl = train_loss(e), val_loss(e) if vl < best_val: best_val, best_epoch, wait = vl, e, 0 marker = " <- new best (weights saved)" else: wait += 1 marker = f" (no improvement for {wait})" print(f"epoch {e:2d} | train {tl:.4f} | val {vl:.4f}{marker}") if wait >= patience: stopped_at = e break print() print(f"stopped at epoch {stopped_at}; restored weights from epoch {best_epoch} " f"(val {best_val:.4f})") print(f"epochs trained past the best model: {stopped_at - best_epoch}") ``` Output: ``` epoch 0 | train 1.0500 | val 0.8000 <- new best (weights saved) epoch 1 | train 0.8000 | val 0.6620 <- new best (weights saved) epoch 2 | train 0.6125 | val 0.5615 <- new best (weights saved) epoch 3 | train 0.4719 | val 0.4891 <- new best (weights saved) epoch 4 | train 0.3664 | val 0.4378 <- new best (weights saved) epoch 5 | train 0.2873 | val 0.4024 <- new best (weights saved) epoch 6 | train 0.2280 | val 0.3788 <- new best (weights saved) epoch 7 | train 0.1835 | val 0.3641 <- new best (weights saved) epoch 8 | train 0.1501 | val 0.3561 <- new best (weights saved) epoch 9 | train 0.1251 | val 0.3531 <- new best (weights saved) epoch 10 | train 0.1063 | val 0.3538 (no improvement for 1) epoch 11 | train 0.0922 | val 0.3573 (no improvement for 2) epoch 12 | train 0.0817 | val 0.3630 (no improvement for 3) stopped at epoch 12; restored weights from epoch 9 (val 0.3531) epochs trained past the best model: 3 ``` Notice that at the stop, training loss (0.0817) is still falling fast and less than a quarter of the validation loss (0.3630) — the model is still "improving" on the data it can see. Watching training loss would have told you to keep going. The validation curve is what caught the turn. ## Frequently Asked Questions ### When should I stop training a neural network? Stop when the loss on a held-out validation set stops improving, not when the training loss flattens. Training loss keeps falling as the model memorises, so it is the wrong thing to watch. Early stopping automates this: it tracks validation loss and halts once it has failed to improve for a set number of epochs. ### What is patience in early stopping? Patience is how many epochs of no improvement you tolerate before stopping. Validation loss is noisy and often ticks up for an epoch or two before finding a new best, so patience of 0 stops far too early. A patience of 5 to 20 epochs is common; you then restore the weights from the best epoch, not the last one. ### Is early stopping a form of regularization? Yes. By limiting how many gradient updates the weights receive, early stopping keeps them closer to their small initial values, which shrinks the effective model capacity. For a quadratic loss it is provably equivalent to L2 regularization, with the number of updates times the learning rate playing the inverse role of the penalty strength. ### Does early stopping apply to large language model pretraining? Not in its classic form. Frontier LLM pretraining usually passes over each token roughly once, so the model never re-fits the same data and validation loss keeps falling; training stops when the compute or token budget runs out, not when a validation metric turns up. Classic early stopping still matters for fine-tuning on smaller datasets, where epochs repeat. ### Do I need a separate validation set for early stopping? Yes. Early stopping selects the epoch, which is a decision made using the data you measure it on, so that data cannot also be your test set — the stopped model would look better than it really is. Use a training set to fit, a validation set to decide when to stop, and a untouched test set to report the final number. ## Related ### Related terms - [Overfitting](https://howaiworks.ai/glossary/overfitting) - [Regularization](https://howaiworks.ai/glossary/regularization) - [Generalization](https://howaiworks.ai/glossary/generalization) - [Cross-Validation (CV)](https://howaiworks.ai/glossary/cross-validation) - [Loss Function](https://howaiworks.ai/glossary/loss-function) - [Hyperparameter](https://howaiworks.ai/glossary/hyperparameter) --- Source: https://howaiworks.ai/glossary/early-stopping — HowAIWorks.ai