---
source: 'https://howaiworks.ai/glossary/ensemble-methods'
section: glossary
title: Ensemble Methods
description: >-
  Why averaging models reduces error and where it stops: the variance of a mean
  of m correlated predictors, the correlation floor, and what bagging cannot
  fix.
tags:
  - ensemble learning
  - Machine Learning
  - boosting
  - bias-variance tradeoff
  - tree-based models
category: Machine Learning
datePublished: '2025-08-20'
lastUpdated: '2026-07-22'
---

# Ensemble Methods

> Why averaging models reduces error and where it stops: the variance of a mean of m correlated predictors, the correlation floor, and what bagging cannot fix.

## Definition

Ensemble methods combine several models into a single prediction. They work for a reason that is
one line of arithmetic rather than a hunch: if each model's error has variance σ² and the errors of
any two models correlate at ρ, then the average of m models has error variance

**ρσ² + σ²(1−ρ)/m**

Read the two terms. The right-hand one is the part that cancels, and it goes to zero as you add
members. The left-hand one is the part that does not cancel, and it does not depend on m at all.
**An infinite ensemble of models correlated at ρ = 0.5 still carries half the error variance of one
model.** That single fact explains why the 500th tree in a random forest is worth almost nothing,
why decorrelating the trees matters far more than counting them, and why "just ensemble it" fails
on models that are wrong in the same direction. It also says nothing about bias — averaging leaves
bias exactly where it found it, which is the half of the story most write-ups skip.

## How It Works

### An average is never worse than its average member

Before the correlation argument, there is a cleaner one that needs no probability at all. For
squared error, the ensemble's error decomposes exactly (Krogh and Vedelsby, NIPS 1995):

*error of the average = average of the members' errors − average disagreement among members.*

Three regressors predict 8, 11 and 14 for a point whose true value is 10. Their average is 11.

| Quantity | Calculation | Value |
|---|---|---|
| Average member error | ((8−10)² + (11−10)² + (14−10)²) / 3 = (4 + 1 + 16) / 3 | 7.0 |
| Disagreement (spread around 11) | ((8−11)² + (11−11)² + (14−11)²) / 3 = (9 + 0 + 9) / 3 | 6.0 |
| Ensemble error | (11 − 10)² | 1.0 |

7.0 − 6.0 = 1.0. The identity holds exactly, and it holds for any predictions you care to
substitute. Because a disagreement term is a mean of squares it can never be negative, so the
ensemble is **always at least as good as its average member**, and it is better by precisely the
amount its members disagree. That is the whole content of the word "diversity", which most
explanations leave as an adjective.

Two warnings come free with the identity. The ensemble beats the *average* member, not the *best*
member — averaging a strong model with two weak ones can lose. And disagreement is only helpful
when the members are not disagreeing by being wrong in a shared way, which is where correlation
comes in.

### The formula that sets the ceiling

Take m predictors, each with error variance σ², with every pair of errors correlated at ρ. The
variance of their average expands into m variance terms and m(m−1) covariance terms:

Var(mean) = (1/m²)·[ m·σ² + m(m−1)·ρσ² ] = σ²/m + ((m−1)/m)·ρσ² = **ρσ² + σ²(1−ρ)/m**

At m = 1 it returns σ², as it must. As m → ∞ the second term vanishes and **ρσ² survives**. Set
σ² = 1 and watch what happens at ρ = 0.5 — a plausible figure for trees grown on bootstrap samples
of the same data:

| Members m | Error variance at ρ = 0.5 | Error variance at ρ = 0.1 |
|---|---|---|
| 1 | 1.000 | 1.000 |
| 5 | 0.600 | 0.280 |
| 10 | 0.550 | 0.190 |
| 100 | 0.505 | 0.109 |
| 1,000 | 0.5005 | 0.1009 |
| ∞ | 0.500 | 0.100 |

Going from 1 model to 10 cuts the variance by 45%, from 1.000 to 0.550. Going from 10 to 1,000 —
a hundred times the training and inference cost — cuts it by a further 9.0%, from 0.550 to 0.5005.
The 500th member of that ensemble changes the variance by 0.000002, four parts per million.

The clean way to state the diminishing returns is that the share of the *reducible* variance an
m-member ensemble has captured is exactly **1 − 1/m**, and that fraction does not depend on ρ at
all: 5 members capture 80%, 10 capture 90%, 100 capture 99%, 500 capture 99.8%. This is why
`n_estimators=100` is a sensible default in almost every library and why raising it to 2,000 is
usually cargo cult. It is also why the deep-ensembles paper that popularised ensembling neural
networks for uncertainty ([Lakshminarayanan, Pritzel and Blundell, NIPS
2017](https://arxiv.org/abs/1612.01474)) reported that 5 networks already captured most of the
benefit and more than 10 added little — an empirical finding landing exactly where 1 − 1/m says it
should.

### Decorrelating beats counting, and it is not close

The interesting knob is ρ, because it moves the floor rather than the rate of approach to it.
Compare two 100-member ensembles in the table above. At ρ = 0.5 the variance is 0.505; at ρ = 0.1
it is 0.109. That is **4.6× less error variance, or 2.15× less RMSE**, from decorrelation alone —
and no number of extra members at ρ = 0.5 can reach it, because 0.500 is the floor.

This is the entire design of a [random forest](https://howaiworks.ai/glossary/random-forest). Bootstrap resampling alone
leaves trees quite similar, because they are all fitted greedily on mostly the same rows and will
mostly pick the same strong feature at the root. Restricting each split to a random subset of
features — `max_features='sqrt'`, so 10 of 100 columns are even considered at each node — forces
trees apart by denying them their preferred split most of the time. That is a deliberate,
paid-for reduction in ρ.

The payment is real and is the trade-off nobody states: a tree that may only look at 10 of 100
features is a **worse tree**, so σ² goes up while ρ goes down. The ensemble improves only when the
fall in ρ outweighs the rise in σ². When you have a handful of features and all of them matter,
subsetting can lose, which is why `max_features` is worth tuning rather than accepting.

### Averaging does nothing to bias

Expected squared error splits into three pieces: bias², variance, and irreducible noise. Averaging
attacks exactly one of them. If every member is biased by b, the mean of m members is still biased
by b — the average of ten thermometers that all read 2° high reads 2° high.

Suppose a deep [decision tree](https://howaiworks.ai/glossary/decision-trees) on some problem decomposes as bias² =
0.02, variance = 0.10, noise = 0.05, for a total error of 0.17. Bag 100 of them and push the
correlation down to ρ = 0.05, so the variance term becomes 0.05 × 0.10 + 0.95 × 0.10/100 = 0.00595:

| Model | bias² | variance | noise | total |
|---|---|---|---|---|
| Single deep tree | 0.02 | 0.100 | 0.05 | 0.170 |
| 100 bagged trees, ρ = 0.05 | 0.02 | 0.006 | 0.05 | 0.076 |
| Infinitely many, ρ = 0.05 | 0.02 | 0.005 | 0.05 | 0.075 |
| Infinitely many, ρ = 0 (unreachable) | 0.02 | 0.000 | 0.05 | 0.070 |

A 55% cut in error — and then almost nothing. Read the last two rows together, because the gap
between them is the whole argument: infinitely many trees at ρ = 0.05 still carry 0.005 of
variance, the `ρσ²` term, and the 0.070 row is what you would get only if the trees were
*perfectly* uncorrelated, which bagging cannot deliver. Going from 100 trees to infinity buys
0.001. Dropping ρ from 0.05 to 0 would buy five times that, and is the only move left. Now run the
same
arithmetic on a base learner that is stable but wrong — a depth-1 stump with bias² = 0.30 and
variance = 0.04. Bagging 100 of those moves the total from 0.39 to 0.352, an improvement of 9.7%
for a hundredfold cost. **Bagging a high-bias model is close to a waste of compute**, and that is
the precise reason [gradient boosting](https://howaiworks.ai/glossary/gradient-boosting) exists: it fits each new model
to what the current ensemble still gets wrong, so it drives the bias term down instead. The rule of
thumb "bag low-bias high-variance learners, boost high-bias low-variance ones" is this table, not a
style preference.

### Voting, and why the textbook calculation is a fantasy

For [classification](https://howaiworks.ai/glossary/classification) the members vote rather than average, and the
textbook version is the Condorcet jury theorem: if m *independent* classifiers are each right with
probability 0.6, majority vote is right with probability P(Binomial(m, 0.6) > m/2). That number
climbs fast — 15 voters reach 78.7%, and 101 voters reach 97.9% — and it is the calculation usually
offered as proof that ensembles work.

It is also almost never applicable, because trained models are not independent: they see the same
data, share the same blind spots, and get the same hard examples wrong together. Model correlated
voters as a shared per-example difficulty (a beta-binomial with intra-class correlation ρ, mean
still 0.6) and the ceiling collapses:

| Setting | 15 voters | 101 voters | 1,001 voters |
|---|---|---|---|
| Independent (ρ = 0) | 78.7% | 97.9% | ~100% |
| Correlated at ρ = 0.1 | 69.5% | 72.8% | 73.5% |
| Correlated at ρ = 0.3 | 63.6% | 64.2% | 64.3% |

At ρ = 0.3, going from 15 voters to 1,001 buys 0.7 percentage points. The Condorcet number is not
a forecast, it is an upper bound you will never reach, and the gap between the two is the
correlation you failed to design out.

## Types

Three families are named in every library and are worth telling apart by *what they attack*, which
the arithmetic above has already given you.

**Bagging** (bootstrap aggregating, Breiman 1996) trains members in parallel on resampled data and
averages them. It attacks variance and leaves bias alone, so it wants base learners that are
flexible and unstable — deep, unpruned trees. [Random forests](https://howaiworks.ai/glossary/random-forest) are bagging
plus the feature-subsetting trick that lowers ρ.

**Boosting** trains members sequentially, each fitted to the errors the ensemble has so far. It
attacks bias, so it wants base learners that are weak and stable — depth-3 to depth-6 trees, not
depth-30. [Gradient boosting](https://howaiworks.ai/glossary/gradient-boosting) and its implementations (XGBoost,
LightGBM, CatBoost) are the dominant form. The members here are correlated by construction, so the
variance formula above does not describe boosting; its failure mode is
[overfitting](https://howaiworks.ai/glossary/overfitting) rather than a correlation floor, which is why it needs a
learning rate and early stopping.

**Stacking** (Wolpert 1992) trains a small model — the meta-learner — to combine the base models'
predictions, rather than fixing the weights at 1/m. This is what lets an ensemble beat its *best*
member instead of merely its average one. The one rule that matters: the meta-learner must be
trained on **out-of-fold** predictions, generated by [cross-validation](https://howaiworks.ai/glossary/cross-validation),
because base models fitted on a row predict that row too well and the meta-learner will happily
learn to trust the leakage.

## Real-World Applications

**Operational weather forecasting** is the largest ensemble system in daily use and the one that
takes decorrelation most seriously. ECMWF's medium-range ensemble runs 51 members — one control
forecast plus 50 perturbed ones (configuration as of publication) — where the perturbations are
applied to the initial conditions and the model physics precisely because a single deterministic
run gives a number with no spread around it. The spread across members *is* the product: it is what
turns "12 mm of rain" into a probability of exceeding 12 mm. Note the member count, 51 rather than
5,000: at roughly 1 − 1/m returns, more members would cost linearly and buy almost nothing compared
with the resolution and physics upgrades the same compute funds instead.

**The Netflix Prize** is the canonical example of an ensemble winning and then not shipping.
BellKor's Pragmatic Chaos took the $1M prize in 2009 with a blend of many separately-trained
[recommendation](https://howaiworks.ai/glossary/recommendation-systems) models, improving RMSE by 10.06% over Netflix's
production Cinematch system. Netflix implemented parts of it and publicly declined to deploy the
full blend, citing engineering effort that the measured accuracy gain did not justify. Both halves
of the story are the arithmetic above: the last few percent of accuracy came from many models
contributing very little each, while cost grew linearly in the number of models.

**Uncertainty estimates for neural networks** are the modern reason to ensemble outside the tree
world. Training the same [network](https://howaiworks.ai/glossary/neural-network) 5 times from different random
initialisations and averaging the predicted distributions gives better-calibrated confidence than
any single network, and it beat the Bayesian approximations of the day in the paper that introduced
it (Lakshminarayanan et al., NIPS 2017). The disagreement among members — the same quantity as the
ambiguity term above — is the usable signal: high disagreement flags inputs the model has no
business being confident about, which is the basis of most out-of-distribution detection in
production ML.

**Tabular prediction** remains the ensemble's home ground, and it is where a single model is
routinely the wrong choice. A controlled comparison across 45 tabular datasets ([Grinsztajn,
Oyallon and Varoquaux, NeurIPS 2022](https://arxiv.org/abs/2207.08815)) found tree ensembles still
ahead of tuned deep networks, and ahead by more when the tuning budget was small. Credit scoring,
churn prediction, demand forecasting and fraud detection are all gradient-boosted tree ensembles in
practice, not because ensembling is fashionable but because a single tree has the variance shown on
the [decision trees](https://howaiworks.ai/glossary/decision-trees) page — one changed row can flip its root — and
averaging is the cheapest known fix.

## Key Concepts

- **The correlation floor, ρσ²**: the one number that determines whether an ensemble has a future.
  Estimate it by measuring the correlation between members' residuals on held-out data; if it is
  0.9, stop adding members and go find a genuinely different model instead.
- **Out-of-bag evaluation**: a bootstrap sample of n rows drawn with replacement misses each
  particular row with probability (1 − 1/n)ⁿ, which converges to 1/e ≈ **36.8%**. So every row is
  held out by about a third of the members, and averaging just those members' predictions gives a
  [generalization](https://howaiworks.ai/glossary/generalization) estimate for free — no separate validation split.
- **Weak learner**: a model only slightly better than chance. Boosting theory needs nothing
  stronger, which is why a depth-3 stump-like tree is a sensible base learner there and a terrible
  one for bagging.
- **Soft versus hard voting**: averaging predicted probabilities generally beats counting votes,
  because it keeps the members' confidence. A member that is 51% sure and a member that is 99% sure
  count identically under hard voting, and the information you threw away was the useful part.
- **Snapshot and seed ensembles**: members that differ only by random seed or training checkpoint
  are cheap, and their ρ is correspondingly high. Cheap diversity is usually weak diversity.

## Challenges

**The cost is linear and the benefit is not.** m members cost m× the inference compute and m×
the memory, while returning 1 − 1/m of a fixed pot of reducible variance. Those curves cross early.
Going from 10 members to 100 multiplies your serving bill by ten to collect the last 9% of a
variance term that is already a minority of your total error. In the bias table above, variance was
0.100 of a total 0.170 — 59% — so even a 94% cut in variance was only a 55% cut in error, and every
member past the first ten chased a shrinking fraction of that 59%. Under a latency budget, an
ensemble frequently loses to a single distilled model of similar accuracy.

**Correlated members are the default, not the exception.** Every member sees the same training set,
the same labelling errors and the same distribution shift when it arrives. A dataset that
mislabels 2% of rows imposes a shared error floor no ensemble can average away, because it is not
noise around a common target — it *is* the common target. This is the ensemble-specific form of a
general truth: averaging removes independent error and is blind to systematic error.

**Stacking leaks unless you are careful, and the leak looks like success.** Train the meta-learner
on predictions the base models made about rows they were fitted on, and those predictions are
unrealistically good; the meta-learner learns to weight whichever base model memorised hardest, and
your validation score rises while test performance falls. Out-of-fold generation is not an
optimisation, it is the thing that makes stacking valid at all — and it costs a full k-fold
retraining of every base model.

**You lose the audit trail, and in some domains that is disqualifying.** A depth-3 tree is eight
readable rules; 500 of them averaged is not an explanation in any sense a regulator accepts, and
post-hoc attributions are a reconstruction rather than the actual decision path. This is why
clinical decision rules and adverse-action notices in lending still use single models or heavily
[regularized](https://howaiworks.ai/glossary/regularization) linear ones, even where an ensemble scores better. The
accuracy is available; the accountability is not.

## Code Example

The variance formula is more convincing when you watch it hold against a simulation, and when you
watch adding members stop working.

```python
import numpy as np

rng = np.random.default_rng(0)

def theory(rho, m, s2=1.0):
    """Variance of the mean of m predictors, pairwise-correlated at rho."""
    return rho * s2 + s2 * (1 - rho) / m

def simulate(rho, m, trials=200_000):
    """Equicorrelated errors: one shared shock plus m private ones."""
    shared = rng.standard_normal((trials, 1)) * np.sqrt(rho)
    private = rng.standard_normal((trials, m)) * np.sqrt(1 - rho)
    return (shared + private).mean(axis=1).var()

for rho in (0.0, 0.1, 0.5):
    for m in (1, 10, 100):
        print(f"rho={rho}  m={m:>3}  theory={theory(rho, m):.4f}  simulated={simulate(rho, m):.4f}")

# The floor: more members cannot cross rho * sigma^2.
print("\nrho=0.5, floor = 0.5000")
for m in (10, 100, 1_000, 1_000_000):
    print(f"  m={m:>9,}  variance={theory(0.5, m):.6f}")

# Decorrelating is worth more than counting.
print(f"\n100 members at rho=0.5 : {theory(0.5, 100):.4f}")
print(f"100 members at rho=0.1 : {theory(0.1, 100):.4f}")
print(f"ratio                  : {theory(0.5, 100) / theory(0.1, 100):.2f}x")

# Share of the reducible variance captured, which is 1 - 1/m regardless of rho.
for m in (2, 5, 10, 100, 500):
    print(f"m={m:>3}  captured={1 - 1 / m:.3%}")
```

The simulated column tracks the theoretical one to within Monte-Carlo noise — two to three decimals
at 200,000 trials — and the floor block prints 0.500000 for a million members: a hundred-thousandfold
increase over m = 10 to remove 9.1% of the variance, because everything else was correlation. If
your ensemble has stopped improving, the formula tells you which lever is left. Not `n_estimators`,
but whatever is making your models agree.

## Frequently Asked Questions

### Why does averaging several models reduce error?

Because the errors partly cancel. If each model's error has variance σ² and any two models' errors correlate at ρ, the average of m models has error variance ρσ² + σ²(1−ρ)/m. The second term is the part that cancels and it shrinks as 1/m; the first term does not shrink at all. Averaging buys you the uncorrelated share of the error and nothing else.

### How many models does an ensemble need?

Fewer than most people use. The share of the reducible variance an m-member ensemble captures is exactly 1 − 1/m, whatever the correlation: 5 members get 80%, 10 get 90%, 100 get 99%. Past about 100 you are paying full inference cost for changes in the fourth decimal place.

### Do ensembles reduce bias as well as variance?

Plain averaging does not. The mean of m models each biased by b is still biased by b, so bagging a systematically wrong model gives you a stable wrong answer. Boosting is the method that attacks bias, by fitting each new model to what the current ensemble still gets wrong — which is why gradient boosting works with shallow trees and bagging does not.

### Why do random forests use random feature subsets instead of just more trees?

Because correlation, not headcount, sets the floor. At ρ = 0.5 an infinite forest still carries half the single-tree error variance. Restricting each split to a random subset of features drives ρ down, and dropping ρ from 0.5 to 0.1 cuts a 100-tree ensemble's error variance by 4.6× — far more than any number of extra correlated trees could.

### Is an ensemble always at least as good as its members?

For squared error, yes, and it is an identity rather than a tendency: ensemble error equals the average member error minus the average disagreement among members. Disagreement cannot be negative, so the ensemble is never worse than the average member. It can still be worse than the single best member, which is why stacking learns weights instead of averaging blindly.

### When is an ensemble not worth deploying?

When the accuracy gain does not cover m× the inference cost. Netflix awarded its $1M prize in 2009 to a blend that improved RMSE by 10.06% over its production system and then never shipped the full solution, saying the engineering effort was not justified. Diminishing returns on accuracy meet linear growth in cost, and they cross.

## Related

### Related terms

- [Decision Trees (DT)](https://howaiworks.ai/glossary/decision-trees)
- [Random Forest (RF)](https://howaiworks.ai/glossary/random-forest)
- [Gradient Boosting](https://howaiworks.ai/glossary/gradient-boosting)
- [Overfitting](https://howaiworks.ai/glossary/overfitting)
- [Low Variance](https://howaiworks.ai/glossary/low-variance)
- [Cross-Validation (CV)](https://howaiworks.ai/glossary/cross-validation)

---

Source: https://howaiworks.ai/glossary/ensemble-methods — HowAIWorks.ai