Term tag: overfitting

Bias-Variance Tradeoff

Statistical bias, not fairness. Error splits into bias squared, variance and irreducible noise — derived, worked on numbers, and honest about where it breaks.

Data Leakage

Data leakage is training a model on information it will not have at prediction time. It pushes scores up, not down, which is why review never catches it.

Early Stopping

Early stopping halts training once validation loss stops improving, so the model generalises instead of memorising. How patience and best-weight restore work.

Feature Selection (FS)

Which columns to keep and how to decide: filter vs wrapper vs embedded, why 2^n subsets rules out brute force, and the CV mistake that fakes accuracy.

Generalization

Why a model that scores 99% in training fails in the world: the generalization gap, the test-set size needed to measure it, and why classical theory is wrong.

High Bias

Statistical bias, not AI fairness: the error a model class cannot escape even with infinite data — worked in closed form on a line fitted to a parabola.

Low Variance

Low variance means values sit close to their average — and for a model, that retraining it on fresh data barely moves its predictions. Worked on numbers.

Machine Learning (ML)

How a machine finds a rule nobody wrote: a line fitted to four points by hand, loss falling from 433.5 to 6.38 in three steps, and why 98% accuracy lies.

Membership Inference

A privacy attack that decides whether a specific record was in a model's training set, exploiting that models are more confident on data they memorized.

Overfitting

Overfitting is when a model memorizes its training data, including the noise, so it scores high on training data but fails on new, unseen data.

Regularization

Regularization deliberately constrains a model so it fits training data less perfectly, trading a little training accuracy for much better generalization.