High Bias

Definition

High bias is the part of a model's error that comes from the shape of the model rather than from its training — the error that would still be there if you had infinite data and a perfect optimiser. This is the statistical sense of the word, the one in the bias-variance decomposition, and it has nothing to do with the discrimination sense: an AI system that treats groups unfairly is a separate subject, covered at Bias in AI (Algorithmic Bias). A weather model with no people in it can have high bias.

Here is the whole idea as one number. Take data that genuinely curves — y = x² over an interval two units wide — and fit the best possible straight line to it. The best line is y = 2x − ⅔, and its mean squared error is exactly 4/45 ≈ 0.0889. Not 0.0889 on a bad day, or 0.0889 until you tune it: 0.0889 is what remains when the line is chosen optimally, with no noise in the data, with every point in the universe available to fit on. It is 0.0889 at twenty training rows and 0.0889 at twenty million.

That is what makes bias different from every other way a model fails. A model that has memorised its training set can be fixed with more data. A model that has not converged can be fixed with more training. A model with high bias cannot be fixed by either, because nothing in the training procedure is what went wrong — the answer was never inside the set of functions the model can express. You change the model or you keep the error.

How It Works

The error is fixed before you see any data

Expected squared error splits into three pieces: bias², variance, and irreducible noise. Variance is how much the fitted model moves when you resample the training set, and it shrinks as the training set grows. Noise is whatever the target does that no input explains. Bias is what is left — the distance between the true function and the best member of your model class, chosen with perfect knowledge. (The three-way decomposition is worked on numbers, with an ensemble in the middle of it, on ensemble methods; this page is about the first term alone.)

Because bias is defined against the best member of the class, it can be computed before any data exists. For x drawn uniformly from [0, 2] and y = x², the best line has slope Cov(x, x²) / Var(x) = (2 − 4/3) / (1/3) = 2 and intercept 4/3 − 2 = −2/3. The error it leaves is Var(x²) − 2²·Var(x) = 64/45 − 4/3 = 4/45.

Two things about that residual are worth more than the number itself.

The first is that the fit looks good. Its R² is (4/3) / (64/45) = 15/16 = 93.75%. A model explaining 94% of the variance is not one anybody flags in review, and it is nonetheless sitting on a permanent ceiling — it will explain 93.75% of the variance forever, on any amount of data, and the remaining 6.25% is not noise but a shape the model is structurally unable to see.

The second is that the leftover error has structure. Random error is scattered; bias is a pattern. The degree-1 fit is too high in the middle of the range and too low at both ends, in every single sample, because that is what a straight line does to a parabola. This is the practical test and it costs nothing: plot the residuals against each input feature. A cloud means you are at the noise floor. A curve, a smile, a drift means the missing structure is right there in the picture.

Why more data cannot touch it, on the numbers

The ## Code Example below fits polynomials of degree 1, 2 and 8 to y = x² plus Gaussian noise with standard deviation 0.3 — so the irreducible noise variance is exactly 0.09 — and reports mean train and test error over 200 fits at each training-set size. Three rows of that output are the argument:

Read the bottom row. Given 10,000 rows, the degree-2 and degree-8 models have both landed on 0.090 — the noise floor, the best any model can do on this problem. The degree-1 model has landed on 0.178, and 0.178 − 0.090 = 0.088, which is 4/45 measured rather than derived. Half of that model's total error is bias, and the five-hundred-fold increase in data from 20 rows to 10,000 removed none of it.

The degree-8 model is the instructive contrast. At 20 rows it is a catastrophe — a test MSE of 276 against the biased model's 0.20, a factor of 1,377 — because sixteen coefficients fitted to twenty noisy points swing wildly from sample to sample. That is pure variance, and data cures it completely: by 10,000 rows it is indistinguishable from the correctly-specified model. Everything that was wrong with the flexible model was temporary. Nothing that is wrong with the simple one is.

The learning-curve diagnostic that the textbook version gets wrong

The folklore is that high variance shows up as a persistent train/test gap and high bias as no gap. The gap column says otherwise:

Both gaps close, and by 10,000 rows both are effectively zero. A gap is a statement about how much data you have, not about which failure you have — it is the generalization gap, and it shrinks for any fixed model class as n grows.

The signal is the level the curves converge to, not the distance between them. Degree 8 converges onto 0.090; degree 1 converges onto 0.178. So the question to ask of a learning curve is not "is there a gap" but "is the plateau above the noise floor" — and that reframing changes what you do next. A large gap means collect more data. A high plateau means stop collecting; the data you already have is not the constraint.

The catch is that you have to know the noise floor to use this, and on real data you usually do not. The workable substitute is comparative: fit something far more flexible than you intend to ship — a deep gradient-boosted ensemble, a large network — on all the data you have. If it plateaus meaningfully lower than your model, the difference is your bias budget and it is available. If it plateaus in the same place, you are at the noise floor and the remaining error is not yours to remove.

What actually lowers bias, and by how much

For a line fitted to a parabola over an interval of width w, the approximation error is w⁴ / 180 — verified against a direct fit in the code below to seven decimal places. The fourth power is the mechanism behind most of the ways bias gets reduced in practice, because it says that a model asked to cover a narrower range is a far better model.

Split the domain in half and fit a separate line to each piece and the bias falls by 16×; split it into four and it falls by 256×. That is what a decision tree with linear leaf models is doing, and what a piecewise-linear network with ReLU activations does automatically. A plain regression tree, which fits a constant rather than a line in each leaf, only gets about 4× per split — the same computation with constant leaves gives 1.422, 0.422, 0.110, 0.0277 for one, two, four and eight pieces — which is why such trees need to be deeper than intuition suggests.

Cheaper still is adding the feature. Give the degree-1 model an x² column and it becomes the degree-2 model: bias goes to exactly zero, at a cost of one column, no extra data and no extra training. Most real high bias is a missing feature rather than missing capacity. Cheaper again is relaxing a constraint you imposed yourself, since every regularizer is bias bought deliberately in exchange for variance.

Raising capacity — more parameters, more layers, more boosting rounds — is the option that always works and always costs the most, because the capacity comes back as variance to be paid for in data. Gradient boosting is the method built around that trade, adding shallow high-bias trees one at a time so bias falls in controlled increments. Averaging models, by contrast, does nothing for bias at all: the mean of ten models each wrong by b is wrong by b.

Real-World Applications

Vision Transformers, and what removing an inductive bias costs. A convolutional network builds in two assumptions about images: that nearby pixels matter together, and that a cat is a cat wherever it appears. Both are restrictions on the function class — high bias, chosen on purpose, and correct often enough to have defined computer vision for a decade. The Vision Transformer removed them. Dosovitskiy et al. (ICLR 2021) state the consequence plainly: Transformers "lack some of the inductive biases inherent to CNNs, such as translation equivariance and locality", and trained on ImageNet alone "these models yield modest accuracies of a few percentage points below ResNets of comparable size". Pre-trained instead on ImageNet-21k (14M images) or JFT-300M (303M images), they win — the paper's own summary is that "large scale training trumps inductive bias". That is the bias-variance trade at the frontier: lower bias is available, and its price is measured in hundreds of millions of labelled images.

Naive Bayes, whose central assumption is known to be false. A naive Bayes classifier assumes every feature is conditionally independent given the class. In text this is obviously wrong — "New" and "York" are not independent — so the probabilities it produces are badly biased, routinely saturating at 0.999 or 0.001. It nonetheless classifies well, and Domingos and Pazzani (Machine Learning 29, 1997) explained why: under zero-one loss only the argmax matters, so a decision survives large errors in the probabilities as long as the ranking is preserved. This is the sharpest statement of when high bias is affordable — it depends on the loss. The same model used to produce calibrated probabilities, for a downstream expected-cost calculation, is unusable.

Linear probes, where high bias is the measuring instrument. In interpretability work, the standard way to ask what a network's intermediate layer has encoded is to train a linear classifier on the frozen activations (Alain and Bengio, 2016). The probe is kept linear precisely because it is high bias: a probe flexible enough to fit anything would score well on any representation and measure nothing. Its inability to construct features is what makes its accuracy a statement about the layer rather than about the probe. Bias here is not a defect to minimise; it is the experimental control.

The Black-Scholes volatility smile. The option-pricing model assumes a single constant volatility for an underlying asset, which implies that implied volatility backed out of market prices should be flat across strike prices. It is not — it curves, and since the 1987 crash equity index options have shown a persistent skew. The smile is the model's bias made visible in exactly the form described above: not scattered pricing error, but a shape in the residuals, present in the same direction every day, which no amount of market data removes because the assumption rather than the estimation is what is wrong. Traders did not fix it by fitting harder. They changed the model.

Key Concepts

• Approximation error versus estimation error: the two halves of a model's excess error. Approximation error is the gap between the truth and the best function your model class contains — bias, set by the choice of class. Estimation error is the gap between that best function and the one you actually fitted — variance, set by how much data you have. Only the second responds to n, which is the entire content of this page.
• Inductive bias is the cause, not a synonym: locality in a CNN, additivity in a linear model, axis-aligned splits in a tree. Each is a named assumption, and each produces statistical bias only when it happens to be false of your data — the same assumption that costs a ViT several points on ImageNet is what lets a CNN learn from a thousand examples. Generalization covers why some such assumption is mandatory.
• The plateau, not the gap: a converged learning curve sitting above the noise floor is the only reliable signature of bias. A gap tells you about sample size. Both failures show a gap at n = 20 and neither shows one at n = 10,000.
• Structured residuals: bias leaves a pattern correlated with the inputs, variance leaves a cloud. Plotting residuals against each feature is the cheapest bias diagnostic in existence and almost nobody runs it.
• Bias is a price, not only a fault: at 20 training rows the deliberately-biased model beat the flexible one by 1,377×. Whether high bias is a bug depends on how much data you have, and the crossover in the table above sits between 50 and 200 rows.

Challenges

Bias is not directly measurable, because it is defined against an answer you do not have. Bias is the distance between the true function f and the best member of your model class, and if you knew f you would not be fitting anything. Everything you can measure is bias plus noise, and separating the two requires either repeated observations at identical inputs — which most datasets do not contain — or a more flexible reference model whose plateau you treat as a stand-in for the noise floor. The reference-model trick is what practitioners actually use, and it is a lower bound on your bias, never a measurement of it.

The reflex fix is the most expensive one. "The model is underfitting, make it bigger" is the default response, and the degree-8 column above is what it costs at n = 20: capacity added to reduce bias comes back as variance and has to be paid for in data you may not have. Two cheaper experiments come first. Check whether the missing structure is a feature — an interaction, a ratio, a log, a lag — because no amount of depth recovers a variable that is not in the input. And check what you have already biased on purpose: a ridge penalty, weight decay, a shallow max-depth, early stopping and an aggressive feature-selected subset are all deliberate bias, so a high plateau is as likely to be a hyperparameter you set as a model class you chose.

Bias gets worse when the input range widens, and it does so fast. The w⁴/180 result cuts both ways: a linear model that is adequate over its training range has sixteen times the approximation error over a range twice as wide. This is why extrapolation from linear models fails so much harder than intuition suggests, and why a model that has been fine for two years can degrade sharply when the distribution of an input drifts outward rather than shifting sideways. The training loss will not have moved.

Code Example

Two things worth computing instead of asserting: that the learning curves of a high-bias and a high-variance model converge to different levels, and that the closed-form approximation error is the number they converge to.

import numpy as np

rng = np.random.default_rng(0)
NOISE = 0.3                      # standard deviation, so the noise floor is 0.09

def sample(n, width=2.0):
    x = rng.uniform(0.0, width, n)
    return x, x**2 + NOISE * rng.standard_normal(n)

def curve(degree, sizes, trials=200):
    """Mean train and test MSE for a degree-d polynomial, per training-set size."""
    xte, yte = sample(50_000)
    for n in sizes:
        tr, te = [], []
        for _ in range(trials):
            x, y = sample(n)
            c = np.polyfit(x, y, degree)
            tr.append(np.mean((np.polyval(c, x) - y) ** 2))
            te.append(np.mean((np.polyval(c, xte) - yte) ** 2))
        print(f"  n={n:>6}  train={np.mean(tr):7.4f}  test={np.mean(te):9.4f}"
              f"  gap={np.mean(te) - np.mean(tr):+9.4f}")

SIZES = (20, 50, 200, 1_000, 10_000)
print("degree 1 - too simple: high bias")
curve(1, SIZES)
print("degree 2 - exactly right: no bias")
curve(2, SIZES)
print("degree 8 - too flexible: high variance")
curve(8, SIZES)

# The bias of a straight line on a parabola, in closed form: width^4 / 180.
print("\nwidth   fitted MSE    w^4/180")
for w in (2.0, 1.0, 0.5, 0.25):
    x = np.linspace(0.0, w, 2_000_000)
    c = np.polyfit(x, x**2, 1)
    print(f"  {w:<5}  {np.mean((np.polyval(c, x) - x**2) ** 2):.8f}  {w**4 / 180:.8f}")

Output:

degree 1 - too simple: high bias
  n=    20  train= 0.1571  test=   0.2010  gap=  +0.0439
  n=    50  train= 0.1704  test=   0.1858  gap=  +0.0154
  n=   200  train= 0.1769  test=   0.1805  gap=  +0.0036
  n=  1000  train= 0.1784  test=   0.1788  gap=  +0.0004
  n= 10000  train= 0.1788  test=   0.1784  gap=  -0.0005
degree 2 - exactly right: no bias
  n=    20  train= 0.0773  test=   0.1063  gap=  +0.0291
  n=    50  train= 0.0846  test=   0.0959  gap=  +0.0113
  n=   200  train= 0.0889  test=   0.0916  gap=  +0.0027
  n=  1000  train= 0.0900  test=   0.0904  gap=  +0.0004
  n= 10000  train= 0.0898  test=   0.0901  gap=  +0.0004
degree 8 - too flexible: high variance
  n=    20  train= 0.0498  test= 276.8416  gap=+276.7918
  n=    50  train= 0.0754  test=   0.3670  gap=  +0.2916
  n=   200  train= 0.0857  test=   0.0946  gap=  +0.0089
  n=  1000  train= 0.0888  test=   0.0909  gap=  +0.0021
  n= 10000  train= 0.0900  test=   0.0901  gap=  +0.0001

width   fitted MSE    w^4/180
  2.0    0.08888907  0.08888889
  1.0    0.00555557  0.00555556
  0.5    0.00034722  0.00034722
  0.25   0.00002170  0.00002170

The degree-1 block converges to 0.1788 and stays there; 0.1788 − 0.0900 = 0.0888, which is 4/45 recovered from a simulation that was never told about it. The degree-8 block converges to 0.0901, the noise floor, having started 3,000× worse. And the last block confirms w⁴/180 to seven decimal places, so the fourth-power scaling is arithmetic rather than a rule of thumb.

Change one line to see the only fix that matters. Set the degree to 2 and the 0.0888 disappears entirely — not reduced, gone — because the truth is now inside the set of functions the model can express. That is the whole distinction between this failure and every other one: it is not solved by more of anything.