Ensemble Methods

Definition

Ensemble methods combine several models into a single prediction. They work for a reason that is one line of arithmetic rather than a hunch: if each model's error has variance σ² and the errors of any two models correlate at ρ, then the average of m models has error variance

ρσ² + σ²(1−ρ)/m

Read the two terms. The right-hand one is the part that cancels, and it goes to zero as you add members. The left-hand one is the part that does not cancel, and it does not depend on m at all. An infinite ensemble of models correlated at ρ = 0.5 still carries half the error variance of one model. That single fact explains why the 500th tree in a random forest is worth almost nothing, why decorrelating the trees matters far more than counting them, and why "just ensemble it" fails on models that are wrong in the same direction. It also says nothing about bias — averaging leaves bias exactly where it found it, which is the half of the story most write-ups skip.

How It Works

An average is never worse than its average member

Before the correlation argument, there is a cleaner one that needs no probability at all. For squared error, the ensemble's error decomposes exactly (Krogh and Vedelsby, NIPS 1995):

error of the average = average of the members' errors − average disagreement among members.

Three regressors predict 8, 11 and 14 for a point whose true value is 10. Their average is 11.

7.0 − 6.0 = 1.0. The identity holds exactly, and it holds for any predictions you care to substitute. Because a disagreement term is a mean of squares it can never be negative, so the ensemble is always at least as good as its average member, and it is better by precisely the amount its members disagree. That is the whole content of the word "diversity", which most explanations leave as an adjective.

Two warnings come free with the identity. The ensemble beats the average member, not the best member — averaging a strong model with two weak ones can lose. And disagreement is only helpful when the members are not disagreeing by being wrong in a shared way, which is where correlation comes in.

The formula that sets the ceiling

Take m predictors, each with error variance σ², with every pair of errors correlated at ρ. The variance of their average expands into m variance terms and m(m−1) covariance terms:

Var(mean) = (1/m²)·[ m·σ² + m(m−1)·ρσ² ] = σ²/m + ((m−1)/m)·ρσ² = ρσ² + σ²(1−ρ)/m

At m = 1 it returns σ², as it must. As m → ∞ the second term vanishes and ρσ² survives. Set σ² = 1 and watch what happens at ρ = 0.5 — a plausible figure for trees grown on bootstrap samples of the same data:

Going from 1 model to 10 cuts the variance by 45%, from 1.000 to 0.550. Going from 10 to 1,000 — a hundred times the training and inference cost — cuts it by a further 9.0%, from 0.550 to 0.5005. The 500th member of that ensemble changes the variance by 0.000002, four parts per million.

The clean way to state the diminishing returns is that the share of the reducible variance an m-member ensemble has captured is exactly 1 − 1/m, and that fraction does not depend on ρ at all: 5 members capture 80%, 10 capture 90%, 100 capture 99%, 500 capture 99.8%. This is why n_estimators=100 is a sensible default in almost every library and why raising it to 2,000 is usually cargo cult. It is also why the deep-ensembles paper that popularised ensembling neural networks for uncertainty (Lakshminarayanan, Pritzel and Blundell, NIPS 2017) reported that 5 networks already captured most of the benefit and more than 10 added little — an empirical finding landing exactly where 1 − 1/m says it should.

Decorrelating beats counting, and it is not close

The interesting knob is ρ, because it moves the floor rather than the rate of approach to it. Compare two 100-member ensembles in the table above. At ρ = 0.5 the variance is 0.505; at ρ = 0.1 it is 0.109. That is 4.6× less error variance, or 2.15× less RMSE, from decorrelation alone — and no number of extra members at ρ = 0.5 can reach it, because 0.500 is the floor.

This is the entire design of a random forest. Bootstrap resampling alone leaves trees quite similar, because they are all fitted greedily on mostly the same rows and will mostly pick the same strong feature at the root. Restricting each split to a random subset of features — max_features='sqrt', so 10 of 100 columns are even considered at each node — forces trees apart by denying them their preferred split most of the time. That is a deliberate, paid-for reduction in ρ.

The payment is real and is the trade-off nobody states: a tree that may only look at 10 of 100 features is a worse tree, so σ² goes up while ρ goes down. The ensemble improves only when the fall in ρ outweighs the rise in σ². When you have a handful of features and all of them matter, subsetting can lose, which is why max_features is worth tuning rather than accepting.

Averaging does nothing to bias

Expected squared error splits into three pieces: bias², variance, and irreducible noise. Averaging attacks exactly one of them. If every member is biased by b, the mean of m members is still biased by b — the average of ten thermometers that all read 2° high reads 2° high.

Suppose a deep decision tree on some problem decomposes as bias² = 0.02, variance = 0.10, noise = 0.05, for a total error of 0.17. Bag 100 of them and push the correlation down to ρ = 0.05, so the variance term becomes 0.05 × 0.10 + 0.95 × 0.10/100 = 0.00595:

A 55% cut in error — and then almost nothing. Read the last two rows together, because the gap between them is the whole argument: infinitely many trees at ρ = 0.05 still carry 0.005 of variance, the ρσ² term, and the 0.070 row is what you would get only if the trees were perfectly uncorrelated, which bagging cannot deliver. Going from 100 trees to infinity buys 0.001. Dropping ρ from 0.05 to 0 would buy five times that, and is the only move left. Now run the same arithmetic on a base learner that is stable but wrong — a depth-1 stump with bias² = 0.30 and variance = 0.04. Bagging 100 of those moves the total from 0.39 to 0.352, an improvement of 9.7% for a hundredfold cost. Bagging a high-bias model is close to a waste of compute, and that is the precise reason gradient boosting exists: it fits each new model to what the current ensemble still gets wrong, so it drives the bias term down instead. The rule of thumb "bag low-bias high-variance learners, boost high-bias low-variance ones" is this table, not a style preference.

Voting, and why the textbook calculation is a fantasy

For classification the members vote rather than average, and the textbook version is the Condorcet jury theorem: if m independent classifiers are each right with probability 0.6, majority vote is right with probability P(Binomial(m, 0.6) > m/2). That number climbs fast — 15 voters reach 78.7%, and 101 voters reach 97.9% — and it is the calculation usually offered as proof that ensembles work.

It is also almost never applicable, because trained models are not independent: they see the same data, share the same blind spots, and get the same hard examples wrong together. Model correlated voters as a shared per-example difficulty (a beta-binomial with intra-class correlation ρ, mean still 0.6) and the ceiling collapses:

At ρ = 0.3, going from 15 voters to 1,001 buys 0.7 percentage points. The Condorcet number is not a forecast, it is an upper bound you will never reach, and the gap between the two is the correlation you failed to design out.

Types

Three families are named in every library and are worth telling apart by what they attack, which the arithmetic above has already given you.

Bagging (bootstrap aggregating, Breiman 1996) trains members in parallel on resampled data and averages them. It attacks variance and leaves bias alone, so it wants base learners that are flexible and unstable — deep, unpruned trees. Random forests are bagging plus the feature-subsetting trick that lowers ρ.

Boosting trains members sequentially, each fitted to the errors the ensemble has so far. It attacks bias, so it wants base learners that are weak and stable — depth-3 to depth-6 trees, not depth-30. Gradient boosting and its implementations (XGBoost, LightGBM, CatBoost) are the dominant form. The members here are correlated by construction, so the variance formula above does not describe boosting; its failure mode is overfitting rather than a correlation floor, which is why it needs a learning rate and early stopping.

Stacking (Wolpert 1992) trains a small model — the meta-learner — to combine the base models' predictions, rather than fixing the weights at 1/m. This is what lets an ensemble beat its best member instead of merely its average one. The one rule that matters: the meta-learner must be trained on out-of-fold predictions, generated by cross-validation, because base models fitted on a row predict that row too well and the meta-learner will happily learn to trust the leakage.

Real-World Applications

Operational weather forecasting is the largest ensemble system in daily use and the one that takes decorrelation most seriously. ECMWF's medium-range ensemble runs 51 members — one control forecast plus 50 perturbed ones (configuration as of publication) — where the perturbations are applied to the initial conditions and the model physics precisely because a single deterministic run gives a number with no spread around it. The spread across members is the product: it is what turns "12 mm of rain" into a probability of exceeding 12 mm. Note the member count, 51 rather than 5,000: at roughly 1 − 1/m returns, more members would cost linearly and buy almost nothing compared with the resolution and physics upgrades the same compute funds instead.

The Netflix Prize is the canonical example of an ensemble winning and then not shipping. BellKor's Pragmatic Chaos took the $1M prize in 2009 with a blend of many separately-trained recommendation models, improving RMSE by 10.06% over Netflix's production Cinematch system. Netflix implemented parts of it and publicly declined to deploy the full blend, citing engineering effort that the measured accuracy gain did not justify. Both halves of the story are the arithmetic above: the last few percent of accuracy came from many models contributing very little each, while cost grew linearly in the number of models.

Uncertainty estimates for neural networks are the modern reason to ensemble outside the tree world. Training the same network 5 times from different random initialisations and averaging the predicted distributions gives better-calibrated confidence than any single network, and it beat the Bayesian approximations of the day in the paper that introduced it (Lakshminarayanan et al., NIPS 2017). The disagreement among members — the same quantity as the ambiguity term above — is the usable signal: high disagreement flags inputs the model has no business being confident about, which is the basis of most out-of-distribution detection in production ML.

Tabular prediction remains the ensemble's home ground, and it is where a single model is routinely the wrong choice. A controlled comparison across 45 tabular datasets (Grinsztajn, Oyallon and Varoquaux, NeurIPS 2022) found tree ensembles still ahead of tuned deep networks, and ahead by more when the tuning budget was small. Credit scoring, churn prediction, demand forecasting and fraud detection are all gradient-boosted tree ensembles in practice, not because ensembling is fashionable but because a single tree has the variance shown on the decision trees page — one changed row can flip its root — and averaging is the cheapest known fix.

Key Concepts

• The correlation floor, ρσ²: the one number that determines whether an ensemble has a future. Estimate it by measuring the correlation between members' residuals on held-out data; if it is 0.9, stop adding members and go find a genuinely different model instead.
• Out-of-bag evaluation: a bootstrap sample of n rows drawn with replacement misses each particular row with probability (1 − 1/n)ⁿ, which converges to 1/e ≈ 36.8%. So every row is held out by about a third of the members, and averaging just those members' predictions gives a generalization estimate for free — no separate validation split.
• Weak learner: a model only slightly better than chance. Boosting theory needs nothing stronger, which is why a depth-3 stump-like tree is a sensible base learner there and a terrible one for bagging.
• Soft versus hard voting: averaging predicted probabilities generally beats counting votes, because it keeps the members' confidence. A member that is 51% sure and a member that is 99% sure count identically under hard voting, and the information you threw away was the useful part.
• Snapshot and seed ensembles: members that differ only by random seed or training checkpoint are cheap, and their ρ is correspondingly high. Cheap diversity is usually weak diversity.

Challenges

The cost is linear and the benefit is not. m members cost m× the inference compute and m× the memory, while returning 1 − 1/m of a fixed pot of reducible variance. Those curves cross early. Going from 10 members to 100 multiplies your serving bill by ten to collect the last 9% of a variance term that is already a minority of your total error. In the bias table above, variance was 0.100 of a total 0.170 — 59% — so even a 94% cut in variance was only a 55% cut in error, and every member past the first ten chased a shrinking fraction of that 59%. Under a latency budget, an ensemble frequently loses to a single distilled model of similar accuracy.

Correlated members are the default, not the exception. Every member sees the same training set, the same labelling errors and the same distribution shift when it arrives. A dataset that mislabels 2% of rows imposes a shared error floor no ensemble can average away, because it is not noise around a common target — it is the common target. This is the ensemble-specific form of a general truth: averaging removes independent error and is blind to systematic error.

Stacking leaks unless you are careful, and the leak looks like success. Train the meta-learner on predictions the base models made about rows they were fitted on, and those predictions are unrealistically good; the meta-learner learns to weight whichever base model memorised hardest, and your validation score rises while test performance falls. Out-of-fold generation is not an optimisation, it is the thing that makes stacking valid at all — and it costs a full k-fold retraining of every base model.

You lose the audit trail, and in some domains that is disqualifying. A depth-3 tree is eight readable rules; 500 of them averaged is not an explanation in any sense a regulator accepts, and post-hoc attributions are a reconstruction rather than the actual decision path. This is why clinical decision rules and adverse-action notices in lending still use single models or heavily regularized linear ones, even where an ensemble scores better. The accuracy is available; the accountability is not.

Code Example

The variance formula is more convincing when you watch it hold against a simulation, and when you watch adding members stop working.

import numpy as np

rng = np.random.default_rng(0)

def theory(rho, m, s2=1.0):
    """Variance of the mean of m predictors, pairwise-correlated at rho."""
    return rho * s2 + s2 * (1 - rho) / m

def simulate(rho, m, trials=200_000):
    """Equicorrelated errors: one shared shock plus m private ones."""
    shared = rng.standard_normal((trials, 1)) * np.sqrt(rho)
    private = rng.standard_normal((trials, m)) * np.sqrt(1 - rho)
    return (shared + private).mean(axis=1).var()

for rho in (0.0, 0.1, 0.5):
    for m in (1, 10, 100):
        print(f"rho={rho}  m={m:>3}  theory={theory(rho, m):.4f}  simulated={simulate(rho, m):.4f}")

# The floor: more members cannot cross rho * sigma^2.
print("\nrho=0.5, floor = 0.5000")
for m in (10, 100, 1_000, 1_000_000):
    print(f"  m={m:>9,}  variance={theory(0.5, m):.6f}")

# Decorrelating is worth more than counting.
print(f"\n100 members at rho=0.5 : {theory(0.5, 100):.4f}")
print(f"100 members at rho=0.1 : {theory(0.1, 100):.4f}")
print(f"ratio                  : {theory(0.5, 100) / theory(0.1, 100):.2f}x")

# Share of the reducible variance captured, which is 1 - 1/m regardless of rho.
for m in (2, 5, 10, 100, 500):
    print(f"m={m:>3}  captured={1 - 1 / m:.3%}")

The simulated column tracks the theoretical one to within Monte-Carlo noise — two to three decimals at 200,000 trials — and the floor block prints 0.500000 for a million members: a hundred-thousandfold increase over m = 10 to remove 9.1% of the variance, because everything else was correlation. If your ensemble has stopped improving, the formula tells you which lever is left. Not n_estimators, but whatever is making your models agree.