Overfitting

Overfitting is when a model memorizes its training data, including the noise, so it scores high on training data but fails on new, unseen data.

Published Updated

On this page

Definition

Overfitting is when a machine learning model learns its training data too well — including the random noise and one-off quirks that will not appear again — so it scores high on the data it trained on and poorly on new, unseen data. That split is the whole phenomenon: a model that answers the exam it has already seen but not the one it is about to sit. The distance between "how it did on training data" and "how it does on fresh data" is called the generalization gap, and a large gap is the signature of overfitting.

It matters because the number most people look at — accuracy on the data used to build the model — is exactly the number overfitting inflates. If you evaluate a model only on its training data and ship it, you are trusting the one measurement designed to look good while the model is broken. It passes every check you ran and then fails in production the first time it sees an input it did not memorize.

How It Works

A model has some amount of capacity — how many distinct patterns it can represent. A straight line has very little; a deep neural network with millions of weights has an enormous amount. When capacity is large relative to the number of training examples, the model has enough freedom to do something lazy but effective on paper: instead of learning the general rule behind the data, it fits the exact points it was given, noise and all. On the training set this looks like success — the error keeps falling — but the model has partly memorized rather than generalized.

The cleanest way to see this is polynomial fitting. A polynomial of degree d has d + 1 free coefficients, and d + 1 coefficients are exactly enough to pass a curve through d + 1 points with zero error. So fit a degree-11 polynomial to 12 noisy points and it will thread every single one — training error 0 — by bending violently between them into a shape that has nothing to do with the underlying signal. A degree-3 polynomial cannot chase every point, so it is forced to find the smooth trend, and that trend is what actually transfers to new data. The ## Code Example below runs exactly this and prints the numbers: at degree 11 training error is 0.000 while error on unseen points is 3.225 — roughly eight times worse than the noise in the data.

This is why you cannot detect overfitting from training performance alone. You need a second, held-out slice of data the model never trains on — a validation set. Train while watching both curves: at first training and validation error fall together, because the model is learning real signal that helps everywhere. Then they part. Validation error bottoms out and starts to climb while training error keeps dropping toward zero. That fork — the point where the model stops learning the world and starts memorizing this particular sample of it — is overfitting happening in front of you, and its location tells you when to stop.

In the language of the bias-variance tradeoff, overfitting is high variance: retrain the same flexible model on a slightly different sample and it produces a very different fit, because it is chasing the noise, which differs every time. The opposite failure, underfitting, is high bias — a model so rigid it misses the real pattern and does badly even on training data. A model that underfits has high bias and low variance; overfitting is the mirror image, low bias and high variance. Neither extreme generalizes well, and the job of every prevention technique below is to buy a little bias back in exchange for a lot less variance.

Real-World Applications

The costs of overfitting are most visible where a hidden test set later grades a model that looked excellent on the data its builders could see.

Kaggle machine-learning competitions make this a spectator sport. Entrants tune against a public leaderboard scored on one slice of the test data, but final rankings use a private slice they never saw. Teams that adjusted their models to squeeze out fractions of a percent on the public slice are, in effect, overfitting to it, and the "leaderboard shakeup" when private scores post regularly drops such teams dozens or hundreds of places while steadier models rise. It is overfitting to a hold-out set, played out publicly.

Research benchmarks show a subtler version. When Recht and colleagues built a fresh test set for ImageNet in 2019, following the original collection procedure as closely as they could, the accuracy of well-known image classifiers dropped by 11 to 14 points on the new images (and 3 to 15 points on a rebuilt CIFAR-10). The authors argue the cause is mainly that the new images are slightly harder rather than the field having overfit the old test set outright — but either way it is the same warning: a score measured on one fixed dataset overstates how a model does on genuinely fresh data drawn the same way.

In applied settings the failure is quieter and more expensive. A medical model that learned to read a scanner's watermark or a hospital-specific artifact instead of the disease will report excellent accuracy in validation and then collapse at a new hospital with a different scanner. A fraud detector tuned on last year's fraud patterns memorizes attacks that have already stopped and misses the ones now happening. In each case the model was evaluated on data that shared the very quirks it overfit to, so the evaluation lied — which is why "how does it do on data from a genuinely different source?" is the question that separates a deployable model from a demo.

Key Concepts

Preventing overfitting means one of two things: give the model less room to memorize, or give it more constraints on what it is allowed to learn. The practical toolkit, roughly in the order you should reach for it:

  • A held-out split, always. Before anything else, set aside data the model never trains on and judge it only there. Cross-validation does this several times over, rotating which slice is held out, so your estimate of true performance does not hinge on one lucky split. This does not prevent overfitting; it makes it visible, which is the prerequisite for everything else.
  • Regularization. Regularization adds a penalty on model complexity to the training objective. L2 (weight decay) adds a term proportional to the sum of the squared weights, pushing every weight toward zero unless the data strongly argues otherwise; L1 adds the sum of absolute weights, which drives some weights exactly to zero and effectively removes features. Either way the model must now justify complexity with real signal, because complexity itself costs it something.
  • Dropout, for neural networks. On each training step it randomly deletes a fraction of the units. Srivastava et al. (2014) showed that keeping each hidden unit with probability p = 0.5 is close to optimal across a wide range of tasks; because a net with n units then samples from 2^n possible "thinned" sub-networks, dropout approximates training an ensemble of exponentially many networks that share weights. At test time no units are dropped and the outgoing weights are multiplied by p, so each unit's expected output matches training. On MNIST, adding dropout took a standard network from 1.60% to 1.25% test error.
  • Early stopping. Watch validation error while training and stop at its lowest point, before it turns upward. It is nearly free and directly targets the fork described above.
  • More data, or augmentation. More examples leave less slack for memorizing noise. When real data is scarce, data augmentation fabricates plausible variants — crops, flips, rotations, added noise — so the model sees each example many ways and cannot latch onto any single presentation of it.
  • A simpler model. The degree-3 polynomial above beat the degree-11 one on unseen data not despite being simpler but because of it. If a smaller model generalizes as well, it is the better model.

Challenges

The hard part of overfitting is that its worst form is invisible to the check most people run. A model can be badly overfit and still post a flawless training score and a strong score on a validation set — if that validation set was drawn from the same batch, the same source, the same time window as the training data, it shares the exact quirks the model memorized and cannot expose the problem. This is why a leak of information from the test set into training, or a validation split that is not truly independent, is so dangerous: it produces confident numbers that are simply wrong, and nothing in the training run flags it.

Overfitting also compounds with distribution shift. The generalization gap you measure at build time assumes tomorrow's inputs resemble today's; when the world moves — new users, new fraud tactics, a new sensor — a model that overfit even slightly to the old distribution degrades faster than a robustly-fit one, and the drop can be gradual enough to miss until it is expensive. And every time you consult a held-out set to make a decision — which model, which hyperparameters — you spend a little of its independence, so a test set reused across hundreds of experiments slowly becomes a training set you are overfitting to indirectly. Guarding against overfitting is not a step you complete once; it is a discipline about which data is allowed to influence which decision, maintained for as long as the model is in use.

Code Example

This fits polynomials of three complexities to the same 12 noisy points and reports error on the training points versus 200 unseen points from the true curve. Run it as-is to reproduce the numbers cited above.

import numpy as np

rng = np.random.default_rng(3)

# True signal: a gentle curve. The model never sees this — only noisy samples.
f = lambda x: np.sin(1.5 * x)

# 12 training points with measurement noise; 200 clean test points, same range.
x_train = np.linspace(-3, 3, 12)
y_train = f(x_train) + rng.normal(0, 0.4, size=x_train.shape)
x_test = np.linspace(-3, 3, 200)
y_test = f(x_test)

def rmse(a, b):
    return float(np.sqrt(np.mean((a - b) ** 2)))

print(f"{'degree':>6} {'train RMSE':>11} {'test RMSE':>10}")
for degree in (1, 3, 11):
    coeffs = np.polyfit(x_train, y_train, degree)   # least-squares fit
    tr = rmse(np.polyval(coeffs, x_train), y_train)
    te = rmse(np.polyval(coeffs, x_test), y_test)
    print(f"{degree:>6} {tr:>11.3f} {te:>10.3f}")

Output:

degree  train RMSE  test RMSE
     1       0.973      0.695
     3       0.437      0.380
    11       0.000      3.225

Read the three rows as underfit, fit, overfit. Degree 1 (a straight line) is too rigid to follow the curve, so it does mediocre everywhere — high bias, underfitting. Degree 3 finds the real shape: low error on both training and unseen data, with a small gap. Degree 11 drives training error to exactly 0.000 by threading all 12 points, yet its error on unseen data explodes to 3.225. The training score is the best of the three and the model is the worst — which is the entire lesson of overfitting in one table.

Frequently Asked Questions

Track training and validation error on the same chart while you train. As long as both fall together the model is still learning signal; the moment validation error flattens and starts rising while training error keeps dropping, the widening gap is overfitting. A model that scores 99% on training data and 78% on held-out data is overfitting by 21 points.
Overfitting is a model too flexible for the data: it drives training error to near zero by memorizing noise and does poorly on new data (low bias, high variance). Underfitting is the opposite: a model too simple to capture the real pattern, so it does poorly on both training and new data (high bias, low variance).
Often, yes. Overfitting happens when the model has more capacity than the data can constrain, so adding data leaves less room to memorize noise. When collecting real data is expensive, data augmentation manufactures more of it — random crops, flips and rotations of images give the model many views of each example.
A held-out split, before any technique. It does not prevent overfitting; it makes it visible, which everything else depends on. Then regularization, which charges the model for complexity — L2 pushes every weight toward zero unless the data argues otherwise, L1 drives some to exactly zero and removes the feature outright. Reach for architectural fixes after you can measure the gap, not before.
No, and chasing zero is the wrong goal. Some gap between training and validation performance is normal; the aim is the model that generalizes best, not the one with the smallest gap. A model that underfits has no gap at all and is still worse than a slightly-overfit model that learned the real pattern.

Related Terms

Underfitting

Underfitting is when a model is too simple to capture the real pattern, so it does poorly on both training and new data — the mirror of overfitting.

Regularization

Regularization deliberately constrains a model so it fits training data less perfectly, trading a little training accuracy for much better generalization.

Generalization

Why a model that scores 99% in training fails in the world: the generalization gap, the test-set size needed to measure it, and why classical theory is wrong.

Cross-Validation (CV)

How cross-validation turns one lucky train/test split into a stable estimate, why 5 or 10 folds, and the leakage that quietly inflates every score.

High Bias

Statistical bias, not AI fairness: the error a model class cannot escape even with infinite data — worked in closed form on a line fitted to a parabola.

Low Variance

Low variance means values sit close to their average — and for a model, that retraining it on fresh data barely moves its predictions. Worked on numbers.

Continue Learning

Explore our use-case guides and prompts to deepen your AI knowledge.

Browse AI Tools← Back to Glossary