Term tag: model evaluation

A/B Testing

A/B testing is the randomized experiment that decides whether a model, prompt, or feature change truly improves a live metric — or just moved on noise.

Benchmark

An AI benchmark is a standardized test or dataset used to evaluate and compare the performance of different AI models across tasks like reasoning and coding.

Bias-Variance Tradeoff

Statistical bias, not fairness. Error splits into bias squared, variance and irreducible noise — derived, worked on numbers, and honest about where it breaks.

Calibration

Calibration means a model's confidence matches reality: of everything it calls 80% likely, about 80% should happen — a property separate from raw accuracy.

Class Imbalance

Why a fraud model can score 99% accuracy and catch nothing — the imbalance trap, why precision/recall/PR-AUC replace accuracy, and how to fix it.

Concept Drift

Concept drift is when the relationship a deployed model learned changes over time, so accuracy silently decays until you detect the drift and retrain.

Conformal Prediction

A model-agnostic, distribution-free method that turns any prediction into a set guaranteed to contain the true answer at a chosen rate, such as 90%.

Cross-Validation (CV)

How cross-validation turns one lucky train/test split into a stable estimate, why 5 or 10 folds, and the leakage that quietly inflates every score.

Data Leakage

Data leakage is training a model on information it will not have at prediction time. It pushes scores up, not down, which is why review never catches it.

Early Stopping

Early stopping halts training once validation loss stops improving, so the model generalises instead of memorising. How patience and best-weight restore work.

Educational AI

AI that tutors, adapts practice and grades work — and what forty years of effect-size research actually shows about whether it improves learning.

Low Variance

Low variance means values sit close to their average — and for a model, that retraining it on fresh data barely moves its predictions. Worked on numbers.

Model Card

A model card is a short document shipped with a released AI model stating its intended use, training data, and evaluation broken down by group.

Precision and Recall

Precision, recall, F1 and the confusion matrix worked through one imbalanced example — and why the model with the best accuracy can catch nothing at all.

Robustness

A model's ability to keep performing when inputs are noisy, out of distribution, or crafted by an attacker — not just accurate on clean test data.