Bias-Variance Tradeoff

Definition

The bias-variance tradeoff is the fact that a model's expected error splits into exactly three pieces — bias², variance and irreducible noise — and that the first two move in opposite directions as you make the model more flexible. Bias here is statistical: the error that remains because the model is too rigid to represent the truth, an error a rainfall model with no people in it can have. It has nothing to do with the discrimination sense of the word, which is a separate subject at Bias in AI (Algorithmic Bias). Nor is it the learnable b in wx + b that a neuron carries. Three unrelated things, one word.

Here is the whole idea as one table. Fit polynomials of increasing degree to 30 noisy points drawn from y = sin(3x), where the noise variance is exactly 0.09, and refit each one 400 times to measure how much its predictions move:

The two bad models are bad for opposite reasons and by wildly different amounts. The constant is 83% bias — it is wrong in the same direction on every refit, and no amount of data changes that. The degree-8 polynomial is 98% variance — averaged over 400 refits it sits close to the truth, and almost every individual fit is a disaster. It carries 62 times the error of the model in the middle.

Now read the middle row, because it contains the thing most explanations of this term get wrong. At the best model, bias² is 0.0031 and variance is 0.0163: the variance is five times larger. "Tradeoff" gets read as "keep the two in balance", and the optimum is nowhere near where they are equal. What is true instead is that 82% of the best model's remaining error is the noise floor — the 0.09 that nothing can remove. The tradeoff was won not by balancing anything but by pushing the sum down until almost all of what was left was not the model's fault.

What breaks if you ignore this is specific: the two failures have opposite fixes, and the reflex response to a disappointing score makes one of them worse. "Collect more data" is the standard move and it does nothing at all for bias. "Use a bigger model" is the standard move for underfitting and it buys variance you then have to pay for in the data you did not have. Choosing between them requires knowing which term you are looking at, and the diagnostic most courses teach — the gap between training and test error — does not tell you.

How It Works

The decomposition, derived

Suppose the world generates labels as y = f(x) + ε, where f is some true function and ε is noise with mean 0 and variance σ². You draw a training set D, fit a model f̂_D, and ask what squared error to expect at a fixed input x — averaged over both the noise and over which training set you happened to draw.

Write f̄(x) = E_D[f̂_D(x)] for the average prediction across all the training sets you might have drawn. Then add and subtract it inside the squared error:

E[(y − f̂(x))²]  =  E[(f(x) + ε − f̂(x))²]
                 =  E[ε²]  +  E[(f(x) − f̂(x))²]
                 =  σ²  +  E[((f(x) − f̄(x)) + (f̄(x) − f̂(x)))²]
                 =  σ²  +  (f(x) − f̄(x))²  +  E[(f̄(x) − f̂(x))²]

Every cross term vanishes. The first one goes because ε is independent of the training set and has mean zero. The second goes because f(x) − f̄(x) is a constant once you have fixed x, and the other factor f̄(x) − f̂(x) has mean zero by the definition of f̄. What is left has names:

expected error = bias² + variance + irreducible noise

• bias² = (f(x) − f̄(x))², how far the average model sits from the truth. Nothing about your sample appears in it, which is why more data cannot move it.
• variance = E[(f̄(x) − f̂(x))²], how far an individual fit scatters around that average.
• σ², what the label does that no input explains.

Two things follow from the shape of that expression alone. The first is that this is an identity, not a theory — it is three lines of algebra and it cannot be wrong. Anything you later read about the tradeoff "breaking down" is a claim about how the terms move, never about whether they add up. The second is that bias is squared and variance is not, so they are not symmetric quantities: halving a bias divides its contribution by four, while halving a variance divides its contribution by two.

Both terms, measured, as the model gets more flexible

The ## Code Example runs the sweep behind the three-row table above. Full output:

The last two columns are the identity checked rather than asserted. bias² + variance + σ² is computed from 400 refits; measured is the mean squared error of those same fits against fresh noisy labels they never saw. They agree to within 0.002 in all nine rows.

Read the two middle columns in opposite directions. Bias² falls from 0.5220 to 0.0003, a factor of roughly 1,740. Variance rises from 0.0209 to 6.7183, a factor of 321. Their sum bottoms out at degree 3, and the two arms around it are not symmetric. Near the bottom, being one step too flexible is cheap — degree 4 costs 13% (0.1236 against 0.1094) — while being one step too simple costs 186% (0.3132). Further out the asymmetry reverses hard: the simplest model on the table is 5.8× the optimum and the most flexible is 62.5×. Erring simple is expensive immediately and then stops getting worse; erring flexible is nearly free at first and then falls off a cliff, which is the worse shape to be walking blindfolded.

Two rows are worth more than the trend. Degree 1 to degree 2 buys almost nothing: bias² goes from 0.1667 to 0.1683, a fraction worse, while variance more than doubles. The reason is that sin(3x) is an odd function and a quadratic term is even, so the extra parameter can represent nothing the truth actually contains. Capacity only reduces bias when it is capacity the target can use — which is why "the model is underfitting, make it bigger" so often fails and "add the missing feature" so often works.

And degree 5, not degree 3, is the lowest-bias model on the table (0.0003 against 0.0031). It is also 76% worse overall. Selecting on bias alone picks it; selecting on variance alone picks the constant, which is 5.8× worse than the best. Only the sum is the thing to minimise, and neither half of the tradeoff can be optimised on its own without producing a model nobody should ship.

Why the two terms move against each other

The mechanism is that a fixed number of training rows has to pay for every parameter. For ordinary least squares with p parameters fitted on n rows of Gaussian inputs, the variance term is

variance ≈ σ² · p / (n − p − 1)

and this is exact enough to check. In the second experiment below — a linear model with 50 training rows and noise variance 0.25 — the formula predicts 0.3947 at p = 30, 1.1111 at p = 40 and 12.0000 at p = 48. The simulation measured 0.3990, 1.1209 and 12.1171: about 1% high at all three, the residual being Monte Carlo error over 200 refits.

That denominator is the entire story. At p = 5 out of 50 rows the marginal parameter costs σ²/44 of variance and is nearly free. At p = 48 out of 50 the same parameter costs σ²/1 — 44 times as much — and the term as a whole has grown from 0.028 to 12.0. As p approaches n the denominator approaches zero and the variance diverges, which is not a metaphor: it is the arithmetic reason the classical curve has a right arm at all, and, as it turns out, the reason for something stranger.

Where the textbook curve stops describing modern practice

Everything above assumed you stop before p = n. Modern models do not. A frontier network has more parameters than it has training tokens to constrain them, is trained until its training loss is essentially zero, and generalizes anyway. The naive U-curve says this is impossible; it happens every day, so the naive U-curve is not the whole story.

Push the same linear experiment straight through p = n and out the other side, using the minimum-norm solution that least squares returns once the system is underdetermined. There are 50 training rows, 30 columns carrying real signal, and every column past the 30th is pure noise:

The peak at p = n is 3,752× worse than the classical optimum, and it is almost entirely the variance term — 99.5% of it. Then the direction reverses. Going from 50 columns to 800 multiplies the parameter count by 16 and divides the variance by 16,088. Adding parameters that carry no signal whatsoever made the model more stable, not less.

That shape has a name and a literature. Belkin, Hsu, Ma and Mandal (PNAS 2019) fitted random Fourier features to a 10,000-example subset of MNIST, hit the interpolation threshold at exactly 10⁴ features, and wrote that "the model class that achieves interpolation with fewest parameters (N = n random features) yields the least accurate predictor. (In fact, it has no predictive ability for classification.) But as the number of features increases beyond n, the accuracy improves dramatically, exceeding that of the predictor corresponding to the bottom of the U-shaped curve." They named the curve double descent, and described it as one that "subsumes the textbook U-shaped bias-variance trade-off curve".

The stronger result is that on real networks the variance term does not merely recover — it never rose in the first place. Neal et al., A Modern Take on the Bias-Variance Tradeoff in Neural Networks (arXiv:1810.08591), took 50 bootstrap resamples of each training set, trained 50 single-hidden-layer networks per width, and measured the two terms directly on MNIST, CIFAR-10, SVHN and a regression task. Their finding: "for all tasks that we study, bias and variance both decrease as we scale network width." They also went back to the paper that established the folklore — Geman, Bienenstock and Doursat's Neural Networks and the Bias/Variance Dilemma (Neural Computation 4(1):1–58, 1992), which asserted that "the basic trend is what we expect: bias falls and variance increases with the number of hidden units" — and observed that Geman et al.'s own figures do not support their claim.

Be careful about what this does and does not overturn, because the folklore reading of double descent is as wrong as the folklore it replaced.

The decomposition is untouched: it is algebra, and the tables above are it, measured, on both sides of the threshold. What is false is the empirical generalisation that variance increases with parameter count. It rises up to p = n and falls after, so parameter count is not a monotone proxy for capacity and was never the right axis to plot.

And "bigger is always better" does not follow. The peak is real, it is catastrophic, and it sits exactly where a classically-sized model sits — at p = 50 the model here was three thousand times worse than at p = 30. Nor did the second descent win: in this experiment the best over-parameterized model scored 0.9577 at 80 columns, still 2.4× worse than the classical optimum of 0.4012. Belkin's MNIST setting did exceed its classical optimum; this one does not. Which of the two minima is lower is a property of the problem, not a law. Generalization goes further into what replaced capacity as the explanation.

Real-World Applications

Self-consistency in language models, and why it works on some errors and not others. Wang et al., Self-Consistency Improves Chain of Thought Reasoning in Language Models (ICLR 2023, arXiv:2203.11171), sample many chain-of-thought traces at non-zero temperature instead of greedily decoding one, then take the majority answer. On GSM8K it added 17.9 percentage points over chain-of-thought prompting. What makes it a bias-variance result rather than a prompting trick is their Table 7, which puts it beside two other things you could average over. On LaMDA-137B: plain chain-of-thought scores 17.1, ensembling over 40 prompt permutations scores 19.2, and self-consistency over 40 sampled reasoning paths scores 27.7. Same model, same budget of 40 forward passes, five times the gain (+10.6 against +2.1) — because decoding stochasticity is variance and averages away, while the model's systematic errors are bias and survive any vote. The operational consequence is sharp: on a task where a model is confidently and consistently wrong, forty samples buy nothing and cost forty times the tokens.

Shrinkage covariance estimation, where the sample estimate is not merely noisy but singular. An S&P 500-sized portfolio has 500 assets, so its covariance matrix has 500 × 501 / 2 = 125,250 free parameters. Ledoit and Wolf's evaluation in Honey, I Shrunk the Sample Covariance Matrix (Journal of Portfolio Management 30(4):110–119, 2004) estimates it from the last T = 60 monthly returns, across benchmark sizes of N = 30, 50, 100, 225 and 500. At N = 500 that is 30,000 observed numbers for 125,250 parameters — 4.2 parameters per number — and the sample covariance matrix has rank at most 59, so 441 of its 500 eigenvalues are exactly zero. A mean-variance optimiser reads a zero eigenvalue as a riskless direction and levers into it. Their conclusion is the bluntest sentence in the bias-variance literature: "nobody should be using the sample covariance matrix for the purpose of portfolio optimization." The fix is to pull the estimate toward a rigid, structured target — deliberately biased, dramatically less variable, and invertible. It is standard enough to ship in scikit-learn as sklearn.covariance.LedoitWolf.

The ten-events-per-variable rule, a variance budget written as a headcount. Clinical prediction models are built under a rule of thumb that comes from one simulation: Peduzzi, Concato, Kemper, Holford and Feinstein, A simulation study of the number of events per variable in logistic regression analysis (Journal of Clinical Epidemiology 49(12):1373–9, 1996). They took a cardiac trial of 673 patients with 252 deaths and 7 predictors — 252 / 7 = 36 events per variable — and resampled it 500 times at each of EPV = 2, 5, 10, 15, 20 and 25. Their result: "for EPV values of 10 or greater, no major problems occurred", while below 10 the coefficients were "biased in both positive and negative directions", the 90% confidence limits lost their coverage, and significance appeared in the wrong direction. That is variance in the estimated coefficients, converted into a rule anyone can apply without simulating anything: an outcome that occurs 80 times in your data supports about 8 predictors. Note also what the rule cannot do — it says nothing about whether those 8 are the right predictors, which is the other term entirely.

Key Concepts

• Bias is squared and variance is not. Halving a bias divides its contribution by four; halving a variance divides it by two. So at the top of the U-curve's left arm, a small reduction in rigidity is worth far more than the same-sized reduction in scatter, and the reverse is true on the right arm.
• The optimum is not where the two are equal. At the best degree in the table, variance is 5.3× bias² and 82% of the total error is irreducible noise. Reading "tradeoff" as "balance" produces the wrong target; the target is the sum, and the sum's minimum has no interest in the ratio.
• σ² is what you are aiming at, and you do not know it. In a simulation it was set to 0.09; on real data nothing tells you what it is. The workable substitute is comparative — fit something far more flexible than you intend to ship and treat its plateau as a stand-in floor. That gives a lower bound on your bias, never a measurement of it.
• Both terms are properties of a procedure at a sample size, not of a saved model. The same architecture is a high-variance choice on 30 rows and a low-variance one on 30,000. A model file has no variance; a training pipeline does, which is why the question is always "at what n".
• Parameter count is not the capacity axis. Variance rose with p up to p = n and fell by four orders of magnitude after it. Anything that assumes "more parameters, more variance" is describing only the left half of a curve whose right half is where every model this site covers lives.

Challenges

You cannot measure the split on your own data, which is why almost nobody does. Both terms are defined as expectations over training sets you might have drawn, and you drew one; bias additionally needs the true function f, which is the thing you were trying to estimate. Every number in the tables on this page exists because the truth was known and the sampling could be repeated 400 times. On a real problem you get neither. What is actually available is indirect: repeated cross-validation or several random seeds to see the scatter, and a deliberately over-powered reference model to bound the bias. Both cost a multiple of the training budget, which is why published results usually report one run and one number that contains both terms fused together.

The diagnostic everyone learns does not distinguish the two failures. The train/test gap is taught as the signature of variance, and it is not: for any fixed model class the gap shrinks toward zero as n grows, so a large gap means "not enough data yet" and a small one means "enough", in both regimes. The high bias and low variance pages each show a version of this on numbers. What separates the two is the level the curves converge to relative to the noise floor, and that requires the reference model above — which is exactly the measurement people skip.

The dial has half a dozen handles and they compound silently. λ in a ridge penalty, weight decay, max_depth, min_samples_leaf, dropout rate, early stopping, and an aggressive feature-selected subset are all the same purchase: bias bought to shed variance. Each was set separately, often by a default, and their effects multiply. A model that plateaus above the noise floor is at least as likely to be carrying four accumulated hyperparameter decisions as to be the wrong model class — and the first experiment is to turn them off, not to make the model bigger. Everything a regularizer does is on this axis, and every ensemble is the same trade run in the other direction, which ensemble methods works out on numbers.

Nothing here tells you what to do when the tradeoff is between models, not within one. The decomposition assumes one model class and one flexibility dial. Real decisions are not shaped that way: a gradient-boosted ensemble and a fine-tuned transformer do not sit on a common capacity axis, and neither is a more-flexible version of the other. The decomposition explains why any given dial has an interior optimum. It does not rank two dials against each other, and treating it as if it does is how "we need a bigger model" gets said in rooms where the real problem was a missing feature.

Code Example

Two things worth computing rather than asserting: that the three terms actually add up to the error you measure, and that the U-shaped curve they produce reverses if you keep going.

import numpy as np

rng = np.random.default_rng(0)
NOISE = 0.3                      # sd of the label noise, so the irreducible term is 0.09
N_TRAIN, TRIALS = 30, 400

def f(x):                        # the truth every model is trying to recover
    return np.sin(3.0 * x)

x_test = np.linspace(-1.0, 1.0, 400)
truth = f(x_test)

def fit_predict(degree):
    """Refit a degree-d polynomial TRIALS times; return one row of predictions each."""
    preds = np.empty((TRIALS, len(x_test)))
    noisy_mse = np.empty(TRIALS)
    for t in range(TRIALS):
        x = rng.uniform(-1.0, 1.0, N_TRAIN)
        y = f(x) + NOISE * rng.standard_normal(N_TRAIN)
        c = np.polyfit(x, y, degree)
        preds[t] = np.polyval(c, x_test)
        y_fresh = truth + NOISE * rng.standard_normal(len(x_test))   # unseen labels
        noisy_mse[t] = np.mean((preds[t] - y_fresh) ** 2)
    return preds, noisy_mse.mean()

print("polynomial degree sweep, 30 training rows, truth = sin(3x)")
print(f"{'degree':>6} {'bias^2':>8} {'variance':>9} {'noise':>7} {'sum':>8} {'measured':>9}")
for d in range(0, 9):
    preds, measured = fit_predict(d)
    bias2 = np.mean((preds.mean(axis=0) - truth) ** 2)
    var = np.mean(preds.var(axis=0))
    print(f"{d:>6} {bias2:>8.4f} {var:>9.4f} {NOISE**2:>7.4f}"
          f" {bias2 + var + NOISE**2:>8.4f} {measured:>9.4f}")

# --- past the interpolation threshold ---------------------------------------
D_SIG, N, SD, TRIALS2, PMAX = 30, 50, 0.5, 200, 800
beta = np.zeros(PMAX)
beta[:D_SIG] = rng.standard_normal(D_SIG) / np.sqrt(D_SIG)   # only 30 columns carry signal
Xte = rng.standard_normal((3_000, PMAX))
yte = Xte @ beta

print("\nlinear model, 50 training rows, 30 useful columns, the rest pure noise")
print(f"{'columns':>8} {'bias^2':>9} {'variance':>12} {'test MSE':>12}")
for p in (1, 5, 10, 20, 30, 40, 48, 50, 52, 60, 80, 150, 300, 800):
    preds = np.empty((TRIALS2, len(Xte)))
    for t in range(TRIALS2):
        X = rng.standard_normal((N, PMAX))
        y = X @ beta + SD * rng.standard_normal(N)
        w, *_ = np.linalg.lstsq(X[:, :p], y, rcond=None)   # min-norm when p > N
        preds[t] = Xte[:, :p] @ w
    bias2 = np.mean((preds.mean(axis=0) - yte) ** 2)
    var = np.mean(preds.var(axis=0))
    print(f"{p:>8} {bias2:>9.4f} {var:>12.4f} {np.mean((preds - yte) ** 2):>12.4f}")

Output:

polynomial degree sweep, 30 training rows, truth = sin(3x)
degree   bias^2  variance   noise      sum  measured
     0   0.5220    0.0209  0.0900   0.6329    0.6349
     1   0.1667    0.0215  0.0900   0.2782    0.2781
     2   0.1683    0.0548  0.0900   0.3132    0.3131
     3   0.0031    0.0163  0.0900   0.1094    0.1094
     4   0.0034    0.0302  0.0900   0.1236    0.1236
     5   0.0003    0.1027  0.0900   0.1930    0.1935
     6   0.0006    0.1435  0.0900   0.2341    0.2332
     7   0.0059    2.4072  0.0900   2.5031    2.5053
     8   0.0250    6.7183  0.0900   6.8333    6.8322

linear model, 50 training rows, 30 useful columns, the rest pure noise
 columns    bias^2     variance     test MSE
       1    1.3670       0.0418       1.4089
       5    1.1205       0.1592       1.2798
      10    0.7429       0.2474       0.9903
      20    0.2981       0.3914       0.6895
      30    0.0022       0.3990       0.4012
      40    0.0040       1.1209       1.1249
      48    0.0824      12.1171      12.1995
      50    7.3746    1497.7829    1505.1575
      52    0.0566      12.5329      12.5895
      60    0.0445       1.5119       1.5564
      80    0.2005       0.7573       0.9577
     150    0.5996       0.4353       1.0348
     300    0.9657       0.2417       1.2074
     800    1.2326       0.0931       1.3258

The first block is the identity, checked: the sum column is built from bias and variance measured across 400 refits, the measured column is error against labels those fits never saw, and they agree everywhere. The second block is what the classical picture leaves out. Test error falls to 0.4012 at 30 columns, is destroyed at 50 — one column per training row, 1505.1575, of which 99.5% is variance — and then falls back to 0.9577 by 80 columns, while the variance term keeps dropping all the way to 0.0931 at 800.

Change SD = 0.5 to SD = 0.0 and the peak does not shrink — it disappears. Every row from 30 columns through 50 then prints bias², variance and test MSE of exactly 0.0000, because with no noise in the labels there is nothing left for a model with one parameter per row to chase, and it recovers the true function outright. That is the shortest statement of what the variance term is. It was never about how many parameters a model has; it is about how much of the sample those parameters are free to memorise, and at exactly one parameter per row they are free to memorise all of it.