Term tag: Model Training

Bias-Variance Tradeoff

Statistical bias, not fairness. Error splits into bias squared, variance and irreducible noise — derived, worked on numbers, and honest about where it breaks.

Catastrophic Forgetting

Why fine-tuning makes a model worse at everything it used to do well — the mechanism inside shared weights, and what each fix actually costs.

Continuous Learning (CL)

Keeping a deployed model current as the world moves — why teams schedule batch retraining instead of true online learning, and what breaks if they don't.

High Bias

Statistical bias, not AI fairness: the error a model class cannot escape even with infinite data — worked in closed form on a line fitted to a parabola.

Overfitting

Overfitting is when a model memorizes its training data, including the noise, so it scores high on training data but fails on new, unseen data.

Scaling Laws

The empirical curves that make AI capex rational: model loss falls predictably with compute, data and parameters — and what they do not promise.

Simple Continual Pretraining (SCP)

Method for turning a pretrained autoregressive LLM into a diffusion language model by continuing pretraining with a bidirectional attention mask.

Supervised Fine-Tuning (SFT)

Supervised fine-tuning trains a pretrained language model on curated prompt-response pairs with next-token loss, teaching it to follow instructions.

Synthetic Data

Data generated by a model instead of collected from the world: what synthetic data is, where it helps, and why recursive use triggers model collapse.

Underfitting

Underfitting is when a model is too simple to capture the real pattern, so it does poorly on both training and new data — the mirror of overfitting.