--- source: 'https://howaiworks.ai/glossary/bias-variance-tradeoff' section: glossary title: Bias-Variance Tradeoff description: >- Statistical bias, not fairness. Error splits into bias squared, variance and irreducible noise — derived, worked on numbers, and honest about where it breaks. tags: - bias-variance tradeoff - Machine Learning - overfitting - underfitting - generalization - regularization - Model Training - model evaluation category: Machine Learning datePublished: '2026-07-24' lastUpdated: '2026-07-24' --- # Bias-Variance Tradeoff > Statistical bias, not fairness. Error splits into bias squared, variance and irreducible noise — derived, worked on numbers, and honest about where it breaks. ## Definition The bias-variance tradeoff is the fact that a model's expected error splits into exactly three pieces — **bias², variance and irreducible noise** — and that the first two move in opposite directions as you make the model more flexible. Bias here is *statistical*: the error that remains because the model is too rigid to represent the truth, an error a rainfall model with no people in it can have. It has nothing to do with the discrimination sense of the word, which is a separate subject at [Bias in AI (Algorithmic Bias)](https://howaiworks.ai/glossary/bias). Nor is it the learnable `b` in `wx + b` that a [neuron](https://howaiworks.ai/glossary/neurons) carries. Three unrelated things, one word. Here is the whole idea as one table. Fit polynomials of increasing degree to 30 noisy points drawn from y = sin(3x), where the noise variance is exactly 0.09, and refit each one 400 times to measure how much its predictions move: | Model | bias² | variance | noise | total error | |---|---|---|---|---| | Degree 0 (a constant) | 0.5220 | 0.0209 | 0.0900 | 0.6329 | | **Degree 3** | **0.0031** | **0.0163** | **0.0900** | **0.1094** | | Degree 8 | 0.0250 | 6.7183 | 0.0900 | 6.8333 | The two bad models are bad for opposite reasons and by wildly different amounts. The constant is **83% bias** — it is wrong in the same direction on every refit, and no amount of data changes that. The degree-8 polynomial is **98% variance** — averaged over 400 refits it sits close to the truth, and almost every individual fit is a disaster. It carries **62 times** the error of the model in the middle. Now read the middle row, because it contains the thing most explanations of this term get wrong. At the best model, bias² is 0.0031 and variance is 0.0163: the variance is **five times larger**. "Tradeoff" gets read as "keep the two in balance", and the optimum is nowhere near where they are equal. What is true instead is that **82% of the best model's remaining error is the noise floor** — the 0.09 that nothing can remove. The tradeoff was won not by balancing anything but by pushing the sum down until almost all of what was left was not the model's fault. What breaks if you ignore this is specific: the two failures have **opposite fixes**, and the reflex response to a disappointing score makes one of them worse. "Collect more data" is the standard move and it does nothing at all for bias. "Use a bigger model" is the standard move for underfitting and it buys variance you then have to pay for in the data you did not have. Choosing between them requires knowing which term you are looking at, and the diagnostic most courses teach — the gap between training and test error — does not tell you. ## How It Works ### The decomposition, derived Suppose the world generates labels as `y = f(x) + ε`, where `f` is some true function and `ε` is noise with mean 0 and variance σ². You draw a training set D, fit a model `f̂_D`, and ask what squared error to expect at a fixed input x — averaged over both the noise and over which training set you happened to draw. Write `f̄(x) = E_D[f̂_D(x)]` for the average prediction across all the training sets you might have drawn. Then add and subtract it inside the squared error: ```text E[(y − f̂(x))²] = E[(f(x) + ε − f̂(x))²] = E[ε²] + E[(f(x) − f̂(x))²] = σ² + E[((f(x) − f̄(x)) + (f̄(x) − f̂(x)))²] = σ² + (f(x) − f̄(x))² + E[(f̄(x) − f̂(x))²] ``` Every cross term vanishes. The first one goes because ε is independent of the training set and has mean zero. The second goes because `f(x) − f̄(x)` is a constant once you have fixed x, and the other factor `f̄(x) − f̂(x)` has mean zero by the definition of `f̄`. What is left has names: **expected error = bias² + variance + irreducible noise** - **bias²** = `(f(x) − f̄(x))²`, how far the *average* model sits from the truth. Nothing about your sample appears in it, which is why more data cannot move it. - **variance** = `E[(f̄(x) − f̂(x))²]`, how far an *individual* fit scatters around that average. - **σ²**, what the label does that no input explains. Two things follow from the shape of that expression alone. The first is that this is an **identity**, not a theory — it is three lines of algebra and it cannot be wrong. Anything you later read about the tradeoff "breaking down" is a claim about how the terms *move*, never about whether they add up. The second is that bias is squared and variance is not, so they are not symmetric quantities: halving a bias divides its contribution by four, while halving a variance divides its contribution by two. ### Both terms, measured, as the model gets more flexible The `## Code Example` runs the sweep behind the three-row table above. Full output: | Degree | Parameters | bias² | variance | noise | sum | measured | |---|---|---|---|---|---|---| | 0 | 1 | 0.5220 | 0.0209 | 0.0900 | 0.6329 | 0.6349 | | 1 | 2 | 0.1667 | 0.0215 | 0.0900 | 0.2782 | 0.2781 | | 2 | 3 | 0.1683 | 0.0548 | 0.0900 | 0.3132 | 0.3131 | | 3 | 4 | **0.0031** | 0.0163 | 0.0900 | **0.1094** | 0.1094 | | 4 | 5 | 0.0034 | 0.0302 | 0.0900 | 0.1236 | 0.1236 | | 5 | 6 | **0.0003** | 0.1027 | 0.0900 | 0.1930 | 0.1935 | | 6 | 7 | 0.0006 | 0.1435 | 0.0900 | 0.2341 | 0.2332 | | 7 | 8 | 0.0059 | 2.4072 | 0.0900 | 2.5031 | 2.5053 | | 8 | 9 | 0.0250 | 6.7183 | 0.0900 | 6.8333 | 6.8322 | The last two columns are the identity checked rather than asserted. `bias² + variance + σ²` is computed from 400 refits; `measured` is the mean squared error of those same fits against fresh noisy labels they never saw. **They agree to within 0.002 in all nine rows.** Read the two middle columns in opposite directions. Bias² falls from 0.5220 to 0.0003, a factor of roughly **1,740**. Variance rises from 0.0209 to 6.7183, a factor of **321**. Their sum bottoms out at degree 3, and the two arms around it are not symmetric. Near the bottom, being one step too flexible is cheap — degree 4 costs 13% (0.1236 against 0.1094) — while being one step too simple costs 186% (0.3132). Further out the asymmetry reverses hard: the simplest model on the table is **5.8×** the optimum and the most flexible is **62.5×**. Erring simple is expensive immediately and then stops getting worse; erring flexible is nearly free at first and then falls off a cliff, which is the worse shape to be walking blindfolded. Two rows are worth more than the trend. **Degree 1 to degree 2 buys almost nothing**: bias² goes from 0.1667 to 0.1683, a fraction *worse*, while variance more than doubles. The reason is that sin(3x) is an odd function and a quadratic term is even, so the extra parameter can represent nothing the truth actually contains. Capacity only reduces bias when it is capacity the target can use — which is why "the model is underfitting, make it bigger" so often fails and "add the missing feature" so often works. And **degree 5, not degree 3, is the lowest-bias model on the table** (0.0003 against 0.0031). It is also 76% worse overall. Selecting on bias alone picks it; selecting on variance alone picks the constant, which is 5.8× worse than the best. Only the sum is the thing to minimise, and neither half of the tradeoff can be optimised on its own without producing a model nobody should ship. ### Why the two terms move against each other The mechanism is that a fixed number of training rows has to pay for every parameter. For ordinary least squares with p parameters fitted on n rows of Gaussian inputs, the variance term is **variance ≈ σ² · p / (n − p − 1)** and this is exact enough to check. In the second experiment below — a linear model with 50 training rows and noise variance 0.25 — the formula predicts 0.3947 at p = 30, 1.1111 at p = 40 and 12.0000 at p = 48. The simulation measured **0.3990, 1.1209 and 12.1171**: about 1% high at all three, the residual being Monte Carlo error over 200 refits. That denominator is the entire story. At p = 5 out of 50 rows the marginal parameter costs σ²/44 of variance and is nearly free. At p = 48 out of 50 the same parameter costs σ²/1 — **44 times as much** — and the term as a whole has grown from 0.028 to 12.0. **As p approaches n the denominator approaches zero and the variance diverges**, which is not a metaphor: it is the arithmetic reason the classical curve has a right arm at all, and, as it turns out, the reason for something stranger. ### Where the textbook curve stops describing modern practice Everything above assumed you stop before p = n. Modern models do not. A frontier network has more parameters than it has training tokens to constrain them, is trained until its training loss is essentially zero, and generalizes anyway. The naive U-curve says this is impossible; it happens every day, so the naive U-curve is not the whole story. Push the same linear experiment straight through p = n and out the other side, using the minimum-norm solution that least squares returns once the system is underdetermined. There are 50 training rows, 30 columns carrying real signal, and every column past the 30th is pure noise: | Columns | bias² | variance | test MSE | Regime | |---|---|---|---|---| | 10 | 0.7429 | 0.2474 | 0.9903 | too few columns to carry the signal | | **30** | 0.0022 | 0.3990 | **0.4012** | classical optimum | | 48 | 0.0824 | 12.1171 | 12.1995 | variance running away | | **50** | 7.3746 | **1497.7829** | **1505.1575** | interpolation threshold, p = n | | 60 | 0.0445 | 1.5119 | 1.5564 | past the peak | | 150 | 0.5996 | 0.4353 | 1.0348 | over-parameterized | | 800 | 1.2326 | 0.0931 | 1.3258 | 16× the rows, 1/16,000 the variance | The peak at p = n is **3,752× worse** than the classical optimum, and it is almost entirely the variance term — 99.5% of it. Then the direction reverses. Going from 50 columns to 800 multiplies the parameter count by 16 and **divides the variance by 16,088**. Adding parameters that carry no signal whatsoever made the model more stable, not less. That shape has a name and a literature. Belkin, Hsu, Ma and Mandal ([PNAS 2019](https://arxiv.org/abs/1812.11118)) fitted random Fourier features to a 10,000-example subset of MNIST, hit the interpolation threshold at exactly 10⁴ features, and wrote that "the model class that achieves interpolation with fewest parameters (N = n random features) yields the least accurate predictor. (In fact, it has no predictive ability for classification.) But as the number of features increases beyond n, the accuracy improves dramatically, exceeding that of the predictor corresponding to the bottom of the U-shaped curve." They named the curve **double descent**, and described it as one that "subsumes the textbook U-shaped bias-variance trade-off curve". The stronger result is that on real networks the variance term does not merely recover — it never rose in the first place. Neal et al., *A Modern Take on the Bias-Variance Tradeoff in Neural Networks* ([arXiv:1810.08591](https://arxiv.org/abs/1810.08591)), took 50 bootstrap resamples of each training set, trained 50 single-hidden-layer networks per width, and measured the two terms directly on MNIST, CIFAR-10, SVHN and a regression task. Their finding: "for all tasks that we study, bias and variance both decrease as we scale network width." They also went back to the paper that established the folklore — Geman, Bienenstock and Doursat's *Neural Networks and the Bias/Variance Dilemma* (*Neural Computation* 4(1):1–58, 1992), which asserted that "the basic trend is what we expect: bias falls and variance increases with the number of hidden units" — and observed that Geman et al.'s own figures do not support their claim. Be careful about what this does and does not overturn, because the folklore reading of double descent is as wrong as the folklore it replaced. The decomposition is untouched: it is algebra, and the tables above are it, measured, on both sides of the threshold. What is false is the *empirical generalisation* that variance increases with parameter count. It rises up to p = n and falls after, so parameter count is not a monotone proxy for capacity and was never the right axis to plot. And "bigger is always better" does not follow. **The peak is real, it is catastrophic, and it sits exactly where a classically-sized model sits** — at p = 50 the model here was three thousand times worse than at p = 30. Nor did the second descent win: in this experiment the best over-parameterized model scored 0.9577 at 80 columns, still 2.4× worse than the classical optimum of 0.4012. Belkin's MNIST setting did exceed its classical optimum; this one does not. Which of the two minima is lower is a property of the problem, not a law. [Generalization](https://howaiworks.ai/glossary/generalization) goes further into what replaced capacity as the explanation. ## Real-World Applications **Self-consistency in language models, and why it works on some errors and not others.** Wang et al., *Self-Consistency Improves Chain of Thought Reasoning in Language Models* (ICLR 2023, [arXiv:2203.11171](https://arxiv.org/abs/2203.11171)), sample many [chain-of-thought](https://howaiworks.ai/glossary/chain-of-thought) traces at non-zero temperature instead of greedily decoding one, then take the majority answer. On GSM8K it added **17.9 percentage points** over chain-of-thought prompting. What makes it a bias-variance result rather than a prompting trick is their Table 7, which puts it beside two other things you could average over. On LaMDA-137B: plain chain-of-thought scores **17.1**, ensembling over **40 prompt permutations** scores **19.2**, and self-consistency over **40 sampled reasoning paths** scores **27.7**. Same model, same budget of 40 forward passes, **five times** the gain (+10.6 against +2.1) — because decoding stochasticity is variance and averages away, while the model's systematic errors are bias and survive any vote. The operational consequence is sharp: on a task where a [model](https://howaiworks.ai/glossary/large-language-model) is confidently and consistently wrong, forty samples buy nothing and cost forty times the tokens. **Shrinkage covariance estimation, where the sample estimate is not merely noisy but singular.** An S&P 500-sized portfolio has 500 assets, so its covariance matrix has 500 × 501 / 2 = **125,250** free parameters. Ledoit and Wolf's evaluation in *Honey, I Shrunk the Sample Covariance Matrix* (*Journal of Portfolio Management* 30(4):110–119, 2004) estimates it from the last **T = 60** monthly returns, across benchmark sizes of N = 30, 50, 100, 225 and 500. At N = 500 that is 30,000 observed numbers for 125,250 parameters — **4.2 parameters per number** — and the sample covariance matrix has rank at most 59, so **441 of its 500 eigenvalues are exactly zero**. A mean-variance optimiser reads a zero eigenvalue as a riskless direction and levers into it. Their conclusion is the bluntest sentence in the bias-variance literature: "nobody should be using the sample covariance matrix for the purpose of portfolio optimization." The fix is to pull the estimate toward a rigid, structured target — deliberately biased, dramatically less variable, and invertible. It is standard enough to ship in scikit-learn as `sklearn.covariance.LedoitWolf`. **The ten-events-per-variable rule, a variance budget written as a headcount.** Clinical prediction models are built under a rule of thumb that comes from one simulation: Peduzzi, Concato, Kemper, Holford and Feinstein, *A simulation study of the number of events per variable in logistic regression analysis* (*Journal of Clinical Epidemiology* 49(12):1373–9, 1996). They took a cardiac trial of **673 patients with 252 deaths and 7 predictors** — 252 / 7 = **36 events per variable** — and resampled it 500 times at each of EPV = 2, 5, 10, 15, 20 and 25. Their result: "for EPV values of 10 or greater, no major problems occurred", while below 10 the coefficients were "biased in both positive and negative directions", the 90% confidence limits lost their coverage, and significance appeared in the wrong direction. That is variance in the estimated coefficients, converted into a rule anyone can apply without simulating anything: an outcome that occurs 80 times in your data supports about **8 predictors**. Note also what the rule cannot do — it says nothing about whether those 8 are the *right* predictors, which is the other term entirely. ## Key Concepts - **Bias is squared and variance is not.** Halving a bias divides its contribution by four; halving a variance divides it by two. So at the top of the U-curve's left arm, a small reduction in rigidity is worth far more than the same-sized reduction in scatter, and the reverse is true on the right arm. - **The optimum is not where the two are equal.** At the best degree in the table, variance is 5.3× bias² and 82% of the total error is irreducible noise. Reading "tradeoff" as "balance" produces the wrong target; the target is the sum, and the sum's minimum has no interest in the ratio. - **σ² is what you are aiming at, and you do not know it.** In a simulation it was set to 0.09; on real data nothing tells you what it is. The workable substitute is comparative — fit something far more flexible than you intend to ship and treat its plateau as a stand-in floor. That gives a lower bound on your bias, never a measurement of it. - **Both terms are properties of a procedure at a sample size, not of a saved model.** The same architecture is a high-variance choice on 30 rows and a low-variance one on 30,000. A model file has no variance; a training pipeline does, which is why the question is always "at what n". - **Parameter count is not the capacity axis.** Variance rose with p up to p = n and fell by four orders of magnitude after it. Anything that assumes "more parameters, more variance" is describing only the left half of a curve whose right half is where every model this site covers lives. ## Challenges **You cannot measure the split on your own data, which is why almost nobody does.** Both terms are defined as expectations over training sets you might have drawn, and you drew one; bias additionally needs the true function `f`, which is the thing you were trying to estimate. Every number in the tables on this page exists because the truth was known and the sampling could be repeated 400 times. On a real problem you get neither. What is actually available is indirect: repeated [cross-validation](https://howaiworks.ai/glossary/cross-validation) or several random seeds to see the scatter, and a deliberately over-powered reference model to bound the bias. Both cost a multiple of the training budget, which is why published results usually report one run and one number that contains both terms fused together. **The diagnostic everyone learns does not distinguish the two failures.** The train/test gap is taught as the signature of variance, and it is not: for any fixed model class the gap shrinks toward zero as n grows, so a large gap means "not enough data yet" and a small one means "enough", in both regimes. The [high bias](https://howaiworks.ai/glossary/high-bias) and [low variance](https://howaiworks.ai/glossary/low-variance) pages each show a version of this on numbers. What separates the two is the *level* the curves converge to relative to the noise floor, and that requires the reference model above — which is exactly the measurement people skip. **The dial has half a dozen handles and they compound silently.** λ in a ridge penalty, weight decay, max_depth, min_samples_leaf, dropout rate, [early stopping](https://howaiworks.ai/glossary/early-stopping), and an aggressive [feature-selected](https://howaiworks.ai/glossary/feature-selection) subset are all the same purchase: bias bought to shed variance. Each was set separately, often by a default, and their effects multiply. A model that plateaus above the noise floor is at least as likely to be carrying four accumulated hyperparameter decisions as to be the wrong model class — and the first experiment is to turn them off, not to make the model bigger. Everything a [regularizer](https://howaiworks.ai/glossary/regularization) does is on this axis, and every ensemble is the same trade run in the other direction, which [ensemble methods](https://howaiworks.ai/glossary/ensemble-methods) works out on numbers. **Nothing here tells you what to do when the tradeoff is between models, not within one.** The decomposition assumes one model class and one flexibility dial. Real decisions are not shaped that way: a gradient-boosted ensemble and a fine-tuned transformer do not sit on a common capacity axis, and neither is a more-flexible version of the other. The decomposition explains why any given dial has an interior optimum. It does not rank two dials against each other, and treating it as if it does is how "we need a bigger model" gets said in rooms where the real problem was a missing feature. ## Code Example Two things worth computing rather than asserting: that the three terms actually add up to the error you measure, and that the U-shaped curve they produce reverses if you keep going. ```python import numpy as np rng = np.random.default_rng(0) NOISE = 0.3 # sd of the label noise, so the irreducible term is 0.09 N_TRAIN, TRIALS = 30, 400 def f(x): # the truth every model is trying to recover return np.sin(3.0 * x) x_test = np.linspace(-1.0, 1.0, 400) truth = f(x_test) def fit_predict(degree): """Refit a degree-d polynomial TRIALS times; return one row of predictions each.""" preds = np.empty((TRIALS, len(x_test))) noisy_mse = np.empty(TRIALS) for t in range(TRIALS): x = rng.uniform(-1.0, 1.0, N_TRAIN) y = f(x) + NOISE * rng.standard_normal(N_TRAIN) c = np.polyfit(x, y, degree) preds[t] = np.polyval(c, x_test) y_fresh = truth + NOISE * rng.standard_normal(len(x_test)) # unseen labels noisy_mse[t] = np.mean((preds[t] - y_fresh) ** 2) return preds, noisy_mse.mean() print("polynomial degree sweep, 30 training rows, truth = sin(3x)") print(f"{'degree':>6} {'bias^2':>8} {'variance':>9} {'noise':>7} {'sum':>8} {'measured':>9}") for d in range(0, 9): preds, measured = fit_predict(d) bias2 = np.mean((preds.mean(axis=0) - truth) ** 2) var = np.mean(preds.var(axis=0)) print(f"{d:>6} {bias2:>8.4f} {var:>9.4f} {NOISE**2:>7.4f}" f" {bias2 + var + NOISE**2:>8.4f} {measured:>9.4f}") # --- past the interpolation threshold --------------------------------------- D_SIG, N, SD, TRIALS2, PMAX = 30, 50, 0.5, 200, 800 beta = np.zeros(PMAX) beta[:D_SIG] = rng.standard_normal(D_SIG) / np.sqrt(D_SIG) # only 30 columns carry signal Xte = rng.standard_normal((3_000, PMAX)) yte = Xte @ beta print("\nlinear model, 50 training rows, 30 useful columns, the rest pure noise") print(f"{'columns':>8} {'bias^2':>9} {'variance':>12} {'test MSE':>12}") for p in (1, 5, 10, 20, 30, 40, 48, 50, 52, 60, 80, 150, 300, 800): preds = np.empty((TRIALS2, len(Xte))) for t in range(TRIALS2): X = rng.standard_normal((N, PMAX)) y = X @ beta + SD * rng.standard_normal(N) w, *_ = np.linalg.lstsq(X[:, :p], y, rcond=None) # min-norm when p > N preds[t] = Xte[:, :p] @ w bias2 = np.mean((preds.mean(axis=0) - yte) ** 2) var = np.mean(preds.var(axis=0)) print(f"{p:>8} {bias2:>9.4f} {var:>12.4f} {np.mean((preds - yte) ** 2):>12.4f}") ``` Output: ```text polynomial degree sweep, 30 training rows, truth = sin(3x) degree bias^2 variance noise sum measured 0 0.5220 0.0209 0.0900 0.6329 0.6349 1 0.1667 0.0215 0.0900 0.2782 0.2781 2 0.1683 0.0548 0.0900 0.3132 0.3131 3 0.0031 0.0163 0.0900 0.1094 0.1094 4 0.0034 0.0302 0.0900 0.1236 0.1236 5 0.0003 0.1027 0.0900 0.1930 0.1935 6 0.0006 0.1435 0.0900 0.2341 0.2332 7 0.0059 2.4072 0.0900 2.5031 2.5053 8 0.0250 6.7183 0.0900 6.8333 6.8322 linear model, 50 training rows, 30 useful columns, the rest pure noise columns bias^2 variance test MSE 1 1.3670 0.0418 1.4089 5 1.1205 0.1592 1.2798 10 0.7429 0.2474 0.9903 20 0.2981 0.3914 0.6895 30 0.0022 0.3990 0.4012 40 0.0040 1.1209 1.1249 48 0.0824 12.1171 12.1995 50 7.3746 1497.7829 1505.1575 52 0.0566 12.5329 12.5895 60 0.0445 1.5119 1.5564 80 0.2005 0.7573 0.9577 150 0.5996 0.4353 1.0348 300 0.9657 0.2417 1.2074 800 1.2326 0.0931 1.3258 ``` The first block is the identity, checked: the `sum` column is built from bias and variance measured across 400 refits, the `measured` column is error against labels those fits never saw, and they agree everywhere. The second block is what the classical picture leaves out. Test error falls to 0.4012 at 30 columns, is destroyed at 50 — one column per training row, 1505.1575, of which 99.5% is variance — and then falls back to 0.9577 by 80 columns, while the variance term keeps dropping all the way to 0.0931 at 800. Change `SD = 0.5` to `SD = 0.0` and the peak does not shrink — it disappears. Every row from 30 columns through 50 then prints bias², variance and test MSE of exactly 0.0000, because with no noise in the labels there is nothing left for a model with one parameter per row to chase, and it recovers the true function outright. That is the shortest statement of what the variance term is. It was never about how many parameters a model has; it is about how much of the *sample* those parameters are free to memorise, and at exactly one parameter per row they are free to memorise all of it. ## Frequently Asked Questions ### Is the 'bias' in bias-variance about fairness or discrimination? No. This is statistical bias — the error left over because a model is too rigid to represent the truth — and it applies to a rainfall model with no people anywhere in it. Unfair treatment of groups by an AI system is an unrelated subject that happens to share the word, covered at [Bias in AI](https://howaiworks.ai/glossary/bias). A model can have zero statistical bias and be grossly unfair, or high statistical bias and be perfectly fair. ### What is the bias-variance tradeoff in simple terms? A model that is too rigid misses the real pattern the same way every time; a model that is too flexible chases whatever noise happened to be in the sample it saw. Making the model more flexible reduces the first error and increases the second, so the total error falls, bottoms out, and rises again. The tradeoff is that you cannot reduce both at once by turning the flexibility dial — you have to find where their sum is smallest. ### What is the bias-variance decomposition formula? Expected squared error = bias² + variance + irreducible noise. Bias is how far the average prediction across many refits sits from the truth; variance is how much individual refits scatter around that average; noise is what nothing in the input explains. It is an algebraic identity, not a hypothesis — in the simulation on this page, the three terms add to the measured test error in all nine rows to within 0.002. ### How do I tell whether I have high bias or high variance? Fit something far more flexible than you intend to ship on all the data you have. If it lands meaningfully lower, the difference is bias and it is available to you. If it lands in the same place, you are at the noise floor. The train/test gap, which is what most courses teach, only tells you whether you have enough data — both failures show a large gap on small samples and neither shows one on large samples. ### Should bias and variance be equal at the best model? No, and expecting that is the most common misreading of the word 'tradeoff'. At the optimum in the table on this page bias² is 0.0031 and variance is 0.0163 — five times larger — and 82% of the error that remains is irreducible noise. The quantity to minimise is the sum, and nothing makes the sum's minimum land where the two parts are equal. ### Is the bias-variance tradeoff still true for large models? The decomposition is an identity and cannot stop being true. The empirical claim built on top of it — that variance rises with parameter count — is false for modern networks. Neal et al. (2019) measured bias and variance both falling as network width grew, and Belkin et al. (PNAS 2019) showed test error peaks at the interpolation threshold and then descends again. Parameter count was never the right axis to plot capacity on. ## Related ### Related terms - [High Bias](https://howaiworks.ai/glossary/high-bias) - [Low Variance](https://howaiworks.ai/glossary/low-variance) - [Overfitting](https://howaiworks.ai/glossary/overfitting) - [Underfitting](https://howaiworks.ai/glossary/underfitting) - [Regularization](https://howaiworks.ai/glossary/regularization) - [Generalization](https://howaiworks.ai/glossary/generalization) --- Source: https://howaiworks.ai/glossary/bias-variance-tradeoff — HowAIWorks.ai