---
source: 'https://howaiworks.ai/glossary/high-bias'
section: glossary
title: High Bias
description: >-
  Statistical bias, not AI fairness: the error a model class cannot escape even
  with infinite data — worked in closed form on a line fitted to a parabola.
tags:
  - bias-variance tradeoff
  - Machine Learning
  - underfitting
  - overfitting
  - Model Training
category: Machine Learning
datePublished: '2025-08-20'
lastUpdated: '2026-07-22'
---

# High Bias

> Statistical bias, not AI fairness: the error a model class cannot escape even with infinite data — worked in closed form on a line fitted to a parabola.

## Definition

High bias is the part of a model's error that comes from the **shape of the model** rather than
from its training — the error that would still be there if you had infinite data and a perfect
optimiser. This is the statistical sense of the word, the one in the bias-variance decomposition,
and it has nothing to do with the discrimination sense: an AI system that treats groups unfairly is
a separate subject, covered at [Bias in AI (Algorithmic Bias)](https://howaiworks.ai/glossary/bias). A weather model
with no people in it can have high bias.

Here is the whole idea as one number. Take data that genuinely curves — y = x² over an interval two
units wide — and fit the best possible straight line to it. The best line is y = 2x − ⅔, and its
mean squared error is exactly **4/45 ≈ 0.0889**. Not 0.0889 on a bad day, or 0.0889 until you tune
it: 0.0889 is what remains when the line is chosen optimally, with no noise in the data, with every
point in the universe available to fit on. It is 0.0889 at twenty training rows and 0.0889 at
twenty million.

That is what makes bias different from every other way a model fails. A model that has memorised
its training set can be fixed with more data. A model that has not converged can be fixed with more
training. A model with high bias cannot be fixed by either, because nothing in the training
procedure is what went wrong — the answer was never inside the set of functions the model can
express. You change the model or you keep the error.

## How It Works

### The error is fixed before you see any data

Expected squared error splits into three pieces: **bias²**, **variance**, and **irreducible noise**.
Variance is how much the fitted model moves when you resample the training set, and it shrinks as
the training set grows. Noise is whatever the target does that no input explains. Bias is what is
left — the distance between the true function and the *best* member of your model class, chosen
with perfect knowledge. (The three-way decomposition is worked on numbers, with an ensemble in the
middle of it, on [ensemble methods](https://howaiworks.ai/glossary/ensemble-methods); this page is about the first term
alone.)

Because bias is defined against the best member of the class, it can be computed before any data
exists. For x drawn uniformly from [0, 2] and y = x², the best line has slope
Cov(x, x²) / Var(x) = (2 − 4/3) / (1/3) = **2** and intercept 4/3 − 2 = **−2/3**. The error it
leaves is Var(x²) − 2²·Var(x) = 64/45 − 4/3 = **4/45**.

Two things about that residual are worth more than the number itself.

The first is that the fit looks *good*. Its R² is (4/3) / (64/45) = **15/16 = 93.75%**. A model
explaining 94% of the variance is not one anybody flags in review, and it is nonetheless sitting on
a permanent ceiling — it will explain 93.75% of the variance forever, on any amount of data, and
the remaining 6.25% is not noise but a shape the model is structurally unable to see.

The second is that the leftover error has **structure**. Random error is scattered; bias is a
pattern. The degree-1 fit is too high in the middle of the range and too low at both ends, in every
single sample, because that is what a straight line does to a parabola. This is the practical test
and it costs nothing: plot the residuals against each input feature. A cloud means you are at the
noise floor. A curve, a smile, a drift means the missing structure is right there in the picture.

### Why more data cannot touch it, on the numbers

The `## Code Example` below fits polynomials of degree 1, 2 and 8 to y = x² plus Gaussian noise
with standard deviation 0.3 — so the irreducible noise variance is exactly 0.09 — and reports mean
train and test error over 200 fits at each training-set size. Three rows of that output are the
argument:

| Training rows | Degree 1 (too simple) | Degree 2 (right) | Degree 8 (too flexible) |
|---|---|---|---|
| 20 | 0.2010 | 0.1063 | 276.8416 |
| 200 | 0.1805 | 0.0916 | 0.0946 |
| 10,000 | **0.1784** | **0.0901** | **0.0901** |

Read the bottom row. Given 10,000 rows, the degree-2 and degree-8 models have both landed on
**0.090** — the noise floor, the best any model can do on this problem. The degree-1 model has
landed on **0.178**, and 0.178 − 0.090 = **0.088**, which is 4/45 measured rather than derived.
**Half of that model's total error is bias**, and the five-hundred-fold increase in data from 20
rows to 10,000 removed none of it.

The degree-8 model is the instructive contrast. At 20 rows it is a catastrophe — a test MSE of 276
against the biased model's 0.20, a factor of 1,377 — because sixteen coefficients fitted to twenty
noisy points swing wildly from sample to sample. That is pure variance, and data cures it
completely: by 10,000 rows it is indistinguishable from the correctly-specified model. Everything
that was wrong with the flexible model was temporary. Nothing that is wrong with the simple one is.

### The learning-curve diagnostic that the textbook version gets wrong

The folklore is that high variance shows up as a persistent train/test gap and high bias as no gap.
The gap column says otherwise:

| Training rows | Degree 1 gap | Degree 8 gap |
|---|---|---|
| 20 | +0.0439 | +276.79 |
| 50 | +0.0154 | +0.2916 |
| 200 | +0.0036 | +0.0089 |
| 1,000 | +0.0004 | +0.0021 |
| 10,000 | −0.0005 | +0.0001 |

Both gaps close, and by 10,000 rows both are effectively zero. A gap is a statement about how much
data you have, not about which failure you have — it is the [generalization](https://howaiworks.ai/glossary/generalization)
gap, and it shrinks for any fixed model class as n grows.

**The signal is the level the curves converge to, not the distance between them.** Degree 8
converges onto 0.090; degree 1 converges onto 0.178. So the question to ask of a learning curve is
not "is there a gap" but "is the plateau above the noise floor" — and that reframing changes what
you do next. A large gap means collect more data. A high plateau means stop collecting; the data
you already have is not the constraint.

The catch is that you have to know the noise floor to use this, and on real data you usually do
not. The workable substitute is comparative: fit something far more flexible than you intend to
ship — a deep gradient-boosted ensemble, a large network — on all the data you have. If it plateaus
meaningfully lower than your model, the difference is your bias budget and it is available. If it
plateaus in the same place, you are at the noise floor and the remaining error is not yours to
remove.

### What actually lowers bias, and by how much

For a line fitted to a parabola over an interval of width w, the approximation error is
**w⁴ / 180** — verified against a direct fit in the code below to seven decimal places. The fourth
power is the mechanism behind most of the ways bias gets reduced in practice, because it says that
**a model asked to cover a narrower range is a far better model.**

| Interval width | Approximation error | Improvement |
|---|---|---|
| 2.0 | 0.08888889 | — |
| 1.0 | 0.00555556 | 16× |
| 0.5 | 0.00034722 | 256× |
| 0.25 | 0.00002170 | 4,096× |

Split the domain in half and fit a separate line to each piece and the bias falls by 16×; split it
into four and it falls by 256×. That is what a [decision tree](https://howaiworks.ai/glossary/decision-trees) with linear
leaf models is doing, and what a piecewise-linear network with ReLU activations does automatically.
A plain regression tree, which fits a constant rather than a line in each leaf, only gets about 4×
per split — the same computation with constant leaves gives 1.422, 0.422, 0.110, 0.0277 for one,
two, four and eight pieces — which is why such trees need to be deeper than intuition suggests.

Cheaper still is **adding the feature**. Give the degree-1 model an x² column and it becomes the
degree-2 model: bias goes to exactly zero, at a cost of one column, no extra data and no extra
training. Most real high bias is a missing feature rather than missing capacity. Cheaper again is
**relaxing a constraint you imposed yourself**, since every [regularizer](https://howaiworks.ai/glossary/regularization)
is bias bought deliberately in exchange for variance.

**Raising capacity** — more parameters, more layers, more boosting rounds — is the option that
always works and always costs the most, because the capacity comes back as variance to be paid for
in data. [Gradient boosting](https://howaiworks.ai/glossary/gradient-boosting) is the method built around that trade,
adding shallow high-bias trees one at a time so bias falls in controlled increments. Averaging
models, by contrast, does nothing for bias at all: the mean of ten models each wrong by b is wrong
by b.

## Real-World Applications

**Vision Transformers, and what removing an inductive bias costs.** A convolutional network builds
in two assumptions about images: that nearby pixels matter together, and that a cat is a cat
wherever it appears. Both are restrictions on the function class — high bias, chosen on purpose,
and correct often enough to have defined [computer vision](https://howaiworks.ai/glossary/computer-vision) for a decade.
The Vision Transformer removed them. Dosovitskiy et al. ([ICLR
2021](https://arxiv.org/abs/2010.11929)) state the consequence plainly: Transformers "lack some of
the inductive biases inherent to CNNs, such as translation equivariance and locality", and trained
on ImageNet alone "these models yield modest accuracies of a few percentage points below ResNets of
comparable size". Pre-trained instead on ImageNet-21k (14M images) or JFT-300M (303M images), they
win — the paper's own summary is that "large scale training trumps inductive bias". That is the
bias-variance trade at the frontier: lower bias is available, and its price is measured in hundreds
of millions of labelled images.

**[Naive Bayes](https://howaiworks.ai/glossary/naive-bayes), whose central assumption is known to be false.** A naive Bayes classifier assumes
every feature is conditionally independent given the class. In text this is obviously wrong — "New"
and "York" are not independent — so the probabilities it produces are badly biased, routinely
saturating at 0.999 or 0.001. It nonetheless classifies well, and Domingos and Pazzani (*Machine
Learning* 29, 1997) explained why: under zero-one loss only the *argmax* matters, so a decision
survives large errors in the probabilities as long as the ranking is preserved. This is the sharpest
statement of when high bias is affordable — it depends on the loss. The same model used to
produce calibrated probabilities, for a downstream expected-cost calculation, is unusable.

**Linear probes, where high bias is the measuring instrument.** In interpretability work, the
standard way to ask what a network's intermediate layer has encoded is to train a linear classifier
on the frozen activations ([Alain and Bengio, 2016](https://arxiv.org/abs/1610.01644)). The probe is
kept linear precisely *because* it is high bias: a probe flexible enough to fit anything would score
well on any representation and measure nothing. Its inability to construct features is what makes
its accuracy a statement about the layer rather than about the probe. Bias here is not a defect to
minimise; it is the experimental control.

**The Black-Scholes volatility smile.** The option-pricing model assumes a single constant
volatility for an underlying asset, which implies that implied volatility backed out of market
prices should be flat across strike prices. It is not — it curves, and since the 1987 crash equity
index options have shown a persistent skew. The smile is the model's bias made visible in exactly
the form described above: not scattered pricing error, but a *shape* in the residuals, present in
the same direction every day, which no amount of market data removes because the assumption rather
than the estimation is what is wrong. Traders did not fix it by fitting harder. They changed the
model.

## Key Concepts

- **Approximation error versus estimation error**: the two halves of a model's excess error.
  Approximation error is the gap between the truth and the best function your model class contains —
  bias, set by the choice of class. Estimation error is the gap between that best function and the
  one you actually fitted — variance, set by how much data you have. Only the second responds to n,
  which is the entire content of this page.
- **Inductive bias is the cause, not a synonym**: locality in a CNN, additivity in a linear model,
  axis-aligned splits in a tree. Each is a named assumption, and each produces statistical bias only
  when it happens to be false of your data — the same assumption that costs a ViT several points on
  ImageNet is what lets a CNN learn from a thousand examples.
  [Generalization](https://howaiworks.ai/glossary/generalization) covers why some such assumption is mandatory.
- **The plateau, not the gap**: a converged learning curve sitting above the noise floor is the only
  reliable signature of bias. A gap tells you about sample size. Both failures show a gap at n = 20
  and neither shows one at n = 10,000.
- **Structured residuals**: bias leaves a pattern correlated with the inputs, variance leaves a
  cloud. Plotting residuals against each feature is the cheapest bias diagnostic in existence and
  almost nobody runs it.
- **Bias is a price, not only a fault**: at 20 training rows the deliberately-biased model beat the
  flexible one by 1,377×. Whether high bias is a bug depends on how much data you have, and the
  crossover in the table above sits between 50 and 200 rows.

## Challenges

**Bias is not directly measurable, because it is defined against an answer you do not have.**
Bias is the distance between the true function f and the best member of your model class, and if you
knew f you would not be fitting anything. Everything you can measure is bias plus noise, and
separating the two requires either repeated observations at identical inputs — which most datasets
do not contain — or a more flexible reference model whose plateau you treat as a stand-in for the
noise floor. The reference-model trick is what practitioners actually use, and it is a lower bound
on your bias, never a measurement of it.

**The reflex fix is the most expensive one.** "The model is underfitting, make it bigger" is the
default response, and the degree-8 column above is what it costs at n = 20: capacity added to
reduce bias comes back as variance and has to be paid for in data you may not have. Two cheaper
experiments come first. Check whether the missing structure is a *feature* — an interaction, a
ratio, a log, a lag — because no amount of depth recovers a variable that is not in the input. And
check what you have already biased on purpose: a ridge penalty, weight decay, a shallow max-depth,
early stopping and an aggressive [feature-selected](https://howaiworks.ai/glossary/feature-selection) subset are all
deliberate bias, so a high plateau is as likely to be a hyperparameter you set as a model class you
chose.

**Bias gets worse when the input range widens, and it does so fast.** The w⁴/180 result cuts both
ways: a linear model that is adequate over its training range has sixteen times the approximation
error over a range twice as wide. This is why extrapolation from linear models fails so much harder
than intuition suggests, and why a model that has been fine for two years can degrade sharply when
the distribution of an input drifts outward rather than shifting sideways. The training loss will
not have moved.

## Code Example

Two things worth computing instead of asserting: that the learning curves of a high-bias and a
high-variance model converge to *different levels*, and that the closed-form approximation error is
the number they converge to.

```python
import numpy as np

rng = np.random.default_rng(0)
NOISE = 0.3                      # standard deviation, so the noise floor is 0.09

def sample(n, width=2.0):
    x = rng.uniform(0.0, width, n)
    return x, x**2 + NOISE * rng.standard_normal(n)

def curve(degree, sizes, trials=200):
    """Mean train and test MSE for a degree-d polynomial, per training-set size."""
    xte, yte = sample(50_000)
    for n in sizes:
        tr, te = [], []
        for _ in range(trials):
            x, y = sample(n)
            c = np.polyfit(x, y, degree)
            tr.append(np.mean((np.polyval(c, x) - y) ** 2))
            te.append(np.mean((np.polyval(c, xte) - yte) ** 2))
        print(f"  n={n:>6}  train={np.mean(tr):7.4f}  test={np.mean(te):9.4f}"
              f"  gap={np.mean(te) - np.mean(tr):+9.4f}")

SIZES = (20, 50, 200, 1_000, 10_000)
print("degree 1 - too simple: high bias")
curve(1, SIZES)
print("degree 2 - exactly right: no bias")
curve(2, SIZES)
print("degree 8 - too flexible: high variance")
curve(8, SIZES)

# The bias of a straight line on a parabola, in closed form: width^4 / 180.
print("\nwidth   fitted MSE    w^4/180")
for w in (2.0, 1.0, 0.5, 0.25):
    x = np.linspace(0.0, w, 2_000_000)
    c = np.polyfit(x, x**2, 1)
    print(f"  {w:<5}  {np.mean((np.polyval(c, x) - x**2) ** 2):.8f}  {w**4 / 180:.8f}")
```

Output:

```text
degree 1 - too simple: high bias
  n=    20  train= 0.1571  test=   0.2010  gap=  +0.0439
  n=    50  train= 0.1704  test=   0.1858  gap=  +0.0154
  n=   200  train= 0.1769  test=   0.1805  gap=  +0.0036
  n=  1000  train= 0.1784  test=   0.1788  gap=  +0.0004
  n= 10000  train= 0.1788  test=   0.1784  gap=  -0.0005
degree 2 - exactly right: no bias
  n=    20  train= 0.0773  test=   0.1063  gap=  +0.0291
  n=    50  train= 0.0846  test=   0.0959  gap=  +0.0113
  n=   200  train= 0.0889  test=   0.0916  gap=  +0.0027
  n=  1000  train= 0.0900  test=   0.0904  gap=  +0.0004
  n= 10000  train= 0.0898  test=   0.0901  gap=  +0.0004
degree 8 - too flexible: high variance
  n=    20  train= 0.0498  test= 276.8416  gap=+276.7918
  n=    50  train= 0.0754  test=   0.3670  gap=  +0.2916
  n=   200  train= 0.0857  test=   0.0946  gap=  +0.0089
  n=  1000  train= 0.0888  test=   0.0909  gap=  +0.0021
  n= 10000  train= 0.0900  test=   0.0901  gap=  +0.0001

width   fitted MSE    w^4/180
  2.0    0.08888907  0.08888889
  1.0    0.00555557  0.00555556
  0.5    0.00034722  0.00034722
  0.25   0.00002170  0.00002170
```

The degree-1 block converges to 0.1788 and stays there; 0.1788 − 0.0900 = 0.0888, which is 4/45
recovered from a simulation that was never told about it. The degree-8 block converges to 0.0901,
the noise floor, having started 3,000× worse. And the last block confirms w⁴/180 to seven decimal
places, so the fourth-power scaling is arithmetic rather than a rule of thumb.

Change one line to see the only fix that matters. Set the degree to 2 and the 0.0888 disappears
entirely — not reduced, gone — because the truth is now inside the set of functions the model can
express. That is the whole distinction between this failure and every other one: it is not solved by
more of anything.

## Frequently Asked Questions

### Is high bias the same as biased or unfair AI?

No, and the two are unrelated despite sharing the word. High bias is statistical bias — systematic error caused by a model being too simple for its data — and it applies equally to a model with no people in it at all. Unfair treatment of groups is a different subject, covered at [Bias in AI](https://howaiworks.ai/glossary/bias). A model can have zero statistical bias and be grossly unfair, or high statistical bias and be perfectly fair.

### What is high bias in machine learning?

It is the error that comes from the shape of the model rather than from its training — the error that would remain if you had infinite data and a perfect optimiser. A straight line fitted to genuinely curved data has a floor it cannot go below, and that floor is the bias.

### Will more training data fix high bias?

No. That is the defining property. In the simulation on this page, a straight line fitted to y = x² plateaus at a test MSE of about 0.179 whether it sees 1,000 rows or 10,000, while the noise floor is 0.090 — half the error is bias and data never touches it. Only changing the model, or adding a feature it can use, moves that number.

### How do I tell high bias from high variance on a learning curve?

Read the level the curves settle at, not the gap between them. Both failures show a gap at small n and both gaps close as data arrives. The difference is where they close: a high-variance model converges down onto the noise floor, and a high-bias model converges onto a plateau above it. A big gap means you have not collected enough data yet; a high plateau means more data will not help.

### How do you reduce high bias?

Give the model something it can use. Add the missing feature — one x² column removes 100% of the bias in the worked example here, at no cost in data. Otherwise increase capacity (a higher-degree polynomial, a deeper network, more boosting rounds), reduce regularization, or narrow the input range the model has to cover, since the approximation error of a line on a curve scales as the fourth power of that range.

### Is high bias always bad?

No — at small sample sizes it is frequently the better trade. In the table on this page, at 20 training rows the deliberately too-simple degree-1 model scored a test MSE of 0.201 while the flexible degree-8 model scored 276.8. The flexible model only overtakes it somewhere between 50 and 200 rows. Bias is the price of stability, and when data is scarce that price is worth paying.

## Related

### Related terms

- [Underfitting](https://howaiworks.ai/glossary/underfitting)
- [Low Variance](https://howaiworks.ai/glossary/low-variance)
- [Overfitting](https://howaiworks.ai/glossary/overfitting)
- [Generalization](https://howaiworks.ai/glossary/generalization)
- [Regularization](https://howaiworks.ai/glossary/regularization)
- [Ensemble Methods](https://howaiworks.ai/glossary/ensemble-methods)

---

Source: https://howaiworks.ai/glossary/high-bias — HowAIWorks.ai