Low Variance

Definition

Low variance means a set of numbers sits close to its own average. Take 48, 49, 50, 51, 52. The mean is 50, the deviations from it are −2, −1, 0, 1, 2, and squaring those gives 4, 1, 0, 1, 4. The variance is the average of those squares: 10 / 5 = 2. Now take 10, 30, 50, 70, 90. The mean is still exactly 50, but the squared deviations are 1600, 400, 0, 400, 1600, so the variance is 4000 / 5 = 800 — four hundred times larger, from a set that any summary quoting only the average would describe as identical to the first one.

That is the plain statistical sense, and it is what most people typing "low variance" are after. In machine learning the phrase carries a second, related meaning that is easy to confuse with it: a model has low variance when refitting it on a different sample of the same data barely changes what it predicts. The spread being measured is not the spread of the data — it is the spread of the model's own answers across the training sets it might have been given. In the simulation further down this page, a straight line refitted on two separate 40-row samples changed its prediction at one input by 0.32, while a fully grown decision tree trained on those same two samples changed by 2.90. Nine times the movement, from the same data-generating process.

Both senses share one trap, and it is the reason this page has arithmetic in it. Low variance is not a synonym for good. A model that ignores its input entirely and always predicts 7 has a variance of exactly zero — it is perfectly stable, perfectly reproducible, and in the decomposition below it carries 24 times the total error of the best model tested. Stability is a property you can have for free by refusing to learn anything.

How It Works

The calculation, and the two things people get wrong about it

Variance is the mean squared deviation from the mean. Written out for the two sets above:

The first thing people get wrong is the divisor. Dividing by n gives the variance of the five numbers as the entire population. If they are a sample drawn from something larger, the conventional divisor is n − 1, which gives 10/4 = 2.5 and 4000/4 = 1000 instead. The correction exists because deviations are measured from the sample's own mean, and the sample mean is pulled toward whatever values happened to be drawn, so squared deviations around it are systematically too small. This is why a calculator's σ and s keys disagree on the same data. It is also why the disagreement rarely matters for the question being asked: the ratio between the two sets is 400× under either divisor.

The second thing people get wrong is that a variance is in squared units and is therefore meaningless on its own. If those numbers are delivery times in minutes, set A's variance is "2 minutes squared" — a quantity no one can picture. Measure the identical deliveries in seconds and the variance becomes 7,200, because the units square along with everything else (60² = 3,600). The square root fixes this: a standard deviation of 1.41 minutes is a typical distance from the average and can be said out loud.

"Low" compared to what

Variance has no absolute scale, so "low variance" is only a claim relative to the mean. Divide the standard deviation by the mean and you get the coefficient of variation: 1.41 / 50 = 2.8% for set A against 28.28 / 50 = 56.6% for set B. Those percentages are comparable across quantities in a way the raw variances are not.

This is the first thing that breaks in practice. A variance of 2 sitting around a mean of 50 is a tightly controlled process. The identical variance of 2 sitting around a mean of 0.1 is chaos — a standard deviation fourteen times the average value, a coefficient of variation of 1,414%. A variance quoted without its mean is not a fact, it is a number. Dashboards that track "prediction variance" or "response-time variance" as a standalone metric, with no denominator, routinely report an improvement that is really a shift in scale.

The model sense: what moves when the data changes

The machine-learning meaning takes the same formula and applies it somewhere unexpected. Fix a single input — say x = 5. Now imagine drawing a fresh training set from the same population, fitting your model, and recording what it predicts at x = 5. Do that many times and you get a distribution of predictions at that one input. The variance of that distribution is the model's variance. It says nothing about how spread out the data is, and everything about how much of the model is a reaction to the particular rows it happened to see.

The ## Code Example at the end fits models to data generated by y = 2x + 3 sin(x) plus Gaussian noise with a standard deviation of 1.0. At x = 5 the truth is 7.12. Fitting on two different 40-row samples gives:

The line is the low-variance model: its two answers differ by 0.32, about 3% of their size. It is also wrong both times, missing the true 7.12 by roughly 3.2 in the same direction — because a straight line cannot follow a wave, and no amount of extra data will teach it to. That is high bias, and the pairing of low variance with high bias is not a coincidence: both come from the model being unable to react to what it is shown.

The tree moves nine times as much. On sample A it lands on 7.09, almost exactly right; on sample B it lands on 9.99, worse than the line. Its accuracy on any single run is largely luck.

The decomposition, with the numbers filled in

The reason neither model is obviously better is that expected squared error splits into three parts that trade against each other:

expected error = bias² + variance + irreducible noise

Bias² is how far the average prediction across refits sits from the truth. Variance is how much individual refits scatter around that average. Noise is what no model can remove — here it is exactly 1.00, because that is the variance of the noise added to the data. Averaging over 500 refits at 40 training rows each:

Read the variance column downward: 0.00, 0.22, 0.92, 1.23, rising monotonically as the models get more flexible. Now read the total: 49.53, 5.18, 2.02, 2.25. The best model on the list has the second-highest variance. Selecting on low variance alone picks the constant, which is 24.5× worse than the depth-3 tree; selecting on low bias alone picks the full-depth tree, which is 11% worse than the depth-3 tree. Only the sum is the thing to minimise.

The straight line is the case worth staring at, because it is the one that survives review. Its variance of 0.22 is the second-lowest on the table, so it passes every stability check you could write. But 3.96 of its 4.18 reducible error — 95% — is bias. Driving its variance all the way to zero would improve its total error by 4.2%, and getting the shape of the model right improves it by 61% (5.18 → 2.02). The stable model is not slightly worse than the right one; it is worse by a factor of two and a half, and its stability is what hides that.

Note also that the depth-3 tree beats the full-depth tree by trading in the other direction: it accepts five times the bias² (0.10 against 0.02) to cut variance by 25% (1.23 → 0.92). Limiting tree depth, regularization, early stopping and feature subsetting are all purchases of low variance paid for in bias, and they are worth making exactly when the variance you remove exceeds the bias you buy — which the table decides and intuition does not.

Why zero variance is a warning sign

The constant model is not a straw man; it is what an underfit model degrades toward, and it looks excellent on every monitor that watches for instability. Retrain it nightly and the predictions do not drift. Ship it to two regions and they agree. Run it on last year's data and it reproduces. Everything a stability dashboard is designed to catch, it passes, because it is not responding to anything. If a production model's predictions have suspiciously little spread — narrower than the spread of the target it is predicting — that is not reliability, it is a model that has learned the mean and stopped. Underfitting and low variance are the same event described from two directions.

Real-World Applications

Random forests exist to buy low variance without paying in bias. A fully grown decision tree is the high-variance model par excellence — the 1.23 in the table above, and worse on real data. Leo Breiman's answer in Bagging Predictors (Machine Learning 24(2):123–140, 1996) and Random Forests (Machine Learning 45(1):5–32, 2001) was to fit many such trees on bootstrap resamples and average them, then decorrelate them further by restricting each split to a random subset of features. Averaging leaves bias exactly where it was and attacks only the variance term, which is why the base learner has to be a deep, low-bias, wildly unstable tree rather than a stable one. The exact arithmetic — including the correlation floor that stops the variance falling to zero no matter how many trees you add — is worked through on ensemble methods and random forest.

Deep reinforcement learning, where the variance was larger than the reported improvements. Henderson et al., Deep Reinforcement Learning that Matters (AAAI 2018, arXiv:1709.06560), ran the same algorithm with the same hyperparameters ten times, changing only the random seed, then split the ten runs into two groups of five and averaged each group. The two averaged learning curves for TRPO on HalfCheetah-v1 were statistically distinguishable: the paper reports "t = −9.0916, p = 0.0016". Nothing differed between the groups except the seed. Their conclusion is the point of this page in one sentence — "the variance between runs is enough to create statistically different distributions just from varying random seeds" — and it means a published gain smaller than that spread carries no information at all. The fix is not a better model but more runs, because the variance of a mean of k runs falls as 1/k.

Manufacturing capability indices measure variance, not averages. The process-capability index Cp is defined as the specification width divided by 6σ, which makes it a statement about spread and nothing else. At Cp = 1 the tolerance band is exactly ±3σ wide, and the normal distribution puts 0.27% of units outside it — 2,700 defects per million. Halve σ at the same specification and Cp becomes 2, the band is ±6σ, and the tail falls to about 2 parts per billion; the familiar 3.4 defects per million associated with "six sigma" is that same calculation with the conventional allowance for the process mean drifting 1.5σ off-centre, leaving 4.5σ on the near side and Φ(−4.5) = 3.4 × 10⁻⁶. A supplier can hit any of those numbers without moving its average at all. Low variance is the entire product.

Online experiments buy statistical power by removing variance rather than adding users. The number of users an A/B test needs to detect a fixed effect is proportional to the variance of the metric, so cutting variance in half halves the required sample. CUPED — introduced in Deng, Xu, Kohavi and Walker's Improving the Sensitivity of Online Controlled Experiments by Utilizing Pre-Experiment Data (WSDM 2013, pp. 123–132) — does this by subtracting a covariate measured before the experiment started, typically the same user's behaviour last week. Replacing the metric Y with Y − θX and choosing θ optimally leaves a variance of Var(Y)(1 − ρ²), where ρ is the correlation between the two. At ρ = 0.7 that removes 51% of the variance, at ρ = 0.9 it removes 81%, and because the covariate is pre-experiment it cannot be affected by the treatment, so the estimate stays unbiased. The same experiment, the same users, half the runtime.

Key Concepts

• Spread of the data versus spread of the estimate: the statistics-class quantity summarises numbers you can see; the machine-learning quantity summarises a model's answers across training sets you never drew. Confusing them produces the belief that clean, tightly clustered inputs guarantee a stable model, which is false in both directions.
• The denominator matters more than it looks: dividing squared deviations by n − 1 rather than n is a correction for having estimated the centre from the same values. It is 25% of the answer at five observations and 0.1% at a thousand.
• Squared units make comparison impossible: this is why the square root is what gets reported, and why the ratio to the mean — 2.8% against 56.6% for the two sets above — is what makes two different quantities comparable at all.
• Averaging attacks this term and nothing else: the mean of k refits has 1/k the scatter of one refit, but the mean of k models each wrong by the same b is still wrong by b. Every ensembling method is this observation.
• The floor is not zero: irreducible noise was exactly 1.00 in the table above and the best achievable total error was 2.02. A stability metric that keeps improving past the point where the error stops improving is measuring refusal to learn.

Challenges

You cannot measure it from one training run, which is why almost nobody measures it. The variance term is defined over the ensemble of training sets you might have drawn, and you drew one. Every practical estimate is a substitute: bootstrap resampling, repeated k-fold cross-validation, or simply retraining under several random seeds and reporting the standard deviation of the result. All of them cost a multiple of the training budget — ten seeds is ten trainings — which is precisely why published results so often report a single run, and why the deep RL case above was possible in the first place. The cheap diagnostic, if nothing else: refit on two disjoint halves of your data and compare the predictions, exactly as the table in this page does.

Benchmark scores have variance too, and it is usually larger than the gap being reported. A model scoring 80% on a 200-question benchmark has a binomial standard error of √(0.8 × 0.2 / 200) = 2.83 percentage points, so a 95% interval spans roughly 5.5 points either side. The difference between two models evaluated on independent samples has a standard error of √2 × 2.83 = 4.0 points — meaning a headline "2 points better" is well inside the noise. Scoring both models on the same questions removes the question-difficulty component of that variance and is the single cheapest improvement available to anyone comparing models, for the same reason CUPED works: the shared term cancels.

Low variance is the failure mode that does not look like one. An overfit model announces itself — the training score and the test score disagree, loudly. An underfit, low-variance model produces a train/test gap of nearly zero, a flat and reassuring learning curve, reproducible numbers, and stable behaviour under retraining. Every symptom that monitoring is built to detect is absent. The only way to see it is to fit something far more flexible than you intend to ship and check whether it does better; if it does, the difference is bias you were carrying and the stability was concealing. Generalization covers why some restriction on the model is nonetheless mandatory — the answer is never zero variance, it is the amount of flexibility your data can support.

Code Example

Two things worth computing rather than asserting: that refitting a flexible model on different samples moves its predictions far more than refitting a rigid one, and that the model with the lowest variance is not the model with the lowest error.

import numpy as np
from sklearn.linear_model import LinearRegression
from sklearn.tree import DecisionTreeRegressor

rng = np.random.default_rng(7)
NOISE = 1.0                      # sd of the noise, so the irreducible term is 1.00


def f(x):                        # the truth: a straight trend with a wave on it
    return 2.0 * x + 3.0 * np.sin(x)


def sample(n):
    x = rng.uniform(0.0, 10.0, n)
    return x.reshape(-1, 1), f(x) + NOISE * rng.standard_normal(n)


GRID = np.linspace(0.0, 10.0, 200).reshape(-1, 1)
TRUTH = f(GRID.ravel())

# --- one pair of retrainings, at a single point ------------------------------
print("two training sets of 40 rows, prediction at x = 5.0 (truth = 7.12)")
for name, make in (("linear fit", LinearRegression),
                   ("full-depth tree", DecisionTreeRegressor)):
    p = []
    for _ in range(2):
        X, y = sample(40)
        p.append(make().fit(X, y).predict([[5.0]])[0])
    print(f"  {name:<16} {p[0]:7.2f}  {p[1]:7.2f}   moved {abs(p[0]-p[1]):6.2f}")

# --- the full decomposition over 500 retrainings -----------------------------
def decompose(make, n=40, trials=500):
    preds = np.empty((trials, len(GRID)))
    for t in range(trials):
        X, y = sample(n)
        preds[t] = make().fit(X, y).predict(GRID)
    variance = preds.var(axis=0).mean()
    bias2 = ((preds.mean(axis=0) - TRUTH) ** 2).mean()
    return bias2, variance, bias2 + variance + NOISE ** 2


class Constant:                  # always predicts 7, whatever it is shown
    def fit(self, X, y):
        return self

    def predict(self, X):
        return np.full(len(X), 7.0)


print("\nmodel               bias^2  variance   noise   total")
for name, make in (("always predicts 7", Constant),
                   ("linear fit", LinearRegression),
                   ("depth-3 tree", lambda: DecisionTreeRegressor(max_depth=3)),
                   ("full-depth tree", DecisionTreeRegressor)):
    b2, v, tot = decompose(make)
    print(f"  {name:<18} {b2:6.2f}    {v:6.2f}    {NOISE**2:.2f}   {tot:6.2f}")

Output:

two training sets of 40 rows, prediction at x = 5.0 (truth = 7.12)
  linear fit         10.46    10.14   moved   0.32
  full-depth tree     7.09     9.99   moved   2.90

model               bias^2  variance   noise   total
  always predicts 7   48.53      0.00    1.00    49.53
  linear fit           3.96      0.22    1.00     5.18
  depth-3 tree         0.10      0.92    1.00     2.02
  full-depth tree      0.02      1.23    1.00     2.25

The variance column is a ranking of stability and the total column is a ranking of usefulness, and they run in almost opposite directions. Change n=40 to n=400 in decompose and watch which numbers move: the variance of the full-depth tree collapses, the bias² of the straight line does not shift at all, and the best model becomes the flexible one. That is the whole practical content of the term — low variance is worth buying when data is scarce and worth selling when it is not.