Term tag: bias-variance tradeoff

Bias-Variance Tradeoff

Statistical bias, not fairness. Error splits into bias squared, variance and irreducible noise — derived, worked on numbers, and honest about where it breaks.

Ensemble Methods

Why averaging models reduces error and where it stops: the variance of a mean of m correlated predictors, the correlation floor, and what bagging cannot fix.

Generalization

Why a model that scores 99% in training fails in the world: the generalization gap, the test-set size needed to measure it, and why classical theory is wrong.

Gradient Boosting

Each tree is fitted to the negative gradient of the loss, not to the residuals — a boosting round worked by hand, and what changes under log loss.

High Bias

Statistical bias, not AI fairness: the error a model class cannot escape even with infinite data — worked in closed form on a line fitted to a parabola.

Low Variance

Low variance means values sit close to their average — and for a model, that retraining it on fresh data barely moves its predictions. Worked on numbers.

Overfitting

Overfitting is when a model memorizes its training data, including the noise, so it scores high on training data but fails on new, unseen data.

Regularization

Regularization deliberately constrains a model so it fits training data less perfectly, trading a little training accuracy for much better generalization.

Underfitting

Underfitting is when a model is too simple to capture the real pattern, so it does poorly on both training and new data — the mirror of overfitting.