Definition
Classification is the machine learning task of assigning each input to one of a fixed set of labels — spam or not spam, which of ten handwritten digits, which of forty product categories — learned from examples that already carry the right answer. Whether a classifier is any good is a separate question with an uncomfortable answer: accuracy, the metric almost everyone reaches for first, is the wrong one on most real problems, because a model that blindly predicts the majority class can score 99.9% and catch nothing at all.
The other thing worth knowing before anything else is that a classifier does not really output a label. It outputs a score — a number expressing how strongly it believes each class — and a label only appears when someone picks a cutoff. That cutoff is not part of the model. It is a business decision, and it is the knob that moves your results the most.
How It Works
Training is the familiar part. You hand a learning algorithm a table of examples, each with input features and a known label, and it searches for a decision boundary: a surface through feature space that puts most of one class on one side and most of the other on the other. Logistic regression draws a straight one; a decision tree draws a staircase of axis-aligned cuts; a support vector machine pushes for the widest possible margin between classes; a neural network bends the boundary into whatever shape the data demands. A loss function measures how badly the current boundary misclassifies the training examples, and the algorithm adjusts until the loss stops falling. Everything from that point on — the entire rest of this page — is evaluation, and it is where most beginners lose the most value.
The confusion matrix is where every metric comes from
Every classification metric you have ever heard of is arithmetic on four counts. Suppose you evaluate a spam filter on 10,000 held-out emails, 2,000 of which are genuinely spam. The filter flags 1,800 messages, and 1,600 of those flags are correct. That gives you:
- 1,600 true positives — spam, correctly flagged;
- 200 false positives — real mail, wrongly flagged (1,800 flagged − 1,600 correct);
- 400 false negatives — spam that slipped through (2,000 spam − 1,600 caught);
- 7,800 true negatives — real mail, correctly left alone (10,000 − 1,600 − 200 − 400).
Those four numbers are the confusion matrix, and they sum to your test set. Accuracy is (1,600 + 7,800) / 10,000 = 94%. Hold on to the four counts rather than the headline, because every disagreement about whether a model is good is really a disagreement about which of the four you are allowed to ignore.
Why accuracy lies, in one calculation
Take a payment stream where 1 transaction in 1,000 is fraudulent. Build the laziest possible classifier: it prints "not fraud" for every transaction and contains no model at all. Over a million transactions there are 1,000 frauds and 999,000 legitimate payments, so it gets 999,000 predictions right and 1,000 wrong. Accuracy: 99.9%. Frauds caught: zero.
You can check that arithmetic in your head, and it is the whole argument. Accuracy is a weighted average in which each class contributes in proportion to how common it is, so when one class is 99.9% of the data, accuracy is essentially a measurement of that class alone. Every honest evaluation of an imbalanced problem therefore starts by writing down the base rate. When the positive class gets rarer still — one in ten thousand, one in a million — the arithmetic turns genuinely hostile and the problem becomes a different one; anomaly detection works that case through in detail.
Precision and recall: the two questions you actually have
Replace accuracy with two questions a non-technical colleague would ask anyway.
Precision — of the things I flagged, how many were real? That is 1,600 / 1,800 = 88.9% for the spam filter. It is the metric your users feel when the system cries wolf, and its denominator is your own output.
Recall — of the real things, how many did I catch? That is 1,600 / 2,000 = 80%. It is the metric that describes what you missed, and its denominator is the world.
Notice that neither one mentions true negatives. That is exactly why they survive imbalance while accuracy does not: the lazy fraud classifier above has a recall of 0 / 1,000 = 0%, and no amount of correctly ignoring legitimate payments can hide it.
The threshold is a decision, not a property of the model
Precision and recall trade against each other, and they do so for a concrete reason: both are computed from the same set of flags, and the flags come from one threshold. Lower the threshold and you flag more things — recall rises because you catch more of the real cases, precision falls because more of what you flagged is junk. Raise it and the movement reverses exactly.
This is the single most useful idea on this page, because it reframes the whole job. "The model isn't good enough" is usually the wrong diagnosis; "we are operating it at the wrong point" is usually the right one, and moving the threshold costs nothing and requires no retraining. What sets the correct point is the relative cost of the two errors, which no dataset can tell you:
- A cancer screening tool's false negative is a missed tumour and its false positive is a follow-up scan. The costs differ by orders of magnitude, so you run it at a low threshold and accept the extra scans.
- A spam filter's false positive is a job offer in the junk folder and its false negative is one more advert in the inbox. The asymmetry runs the other way, so it runs at a high threshold.
Same mathematics, opposite settings, and the difference has nothing to do with machine learning.
F1, and why it is the harmonic mean
Reporting two numbers is awkward when you want to rank models, so F1 collapses precision and
recall into one: F1 = 2PR / (P + R), their harmonic mean. For the spam filter,
2 × 0.889 × 0.80 / (0.889 + 0.80) = 0.842, which is barely below the ordinary average of 0.844
— when the two values are close, the choice of mean hardly matters.
It matters enormously when they are not. A model with precision 0.95 and recall 0.05 — superb at being right about the handful it flags, useless because it flags almost nothing — has an ordinary average of 0.50, a perfectly respectable-looking score. Its F1 is 2 × 0.95 × 0.05 / 1.00 = 0.095. The harmonic mean is dominated by the smaller number, which is precisely the behaviour you want from a summary metric: it refuses to let one good number cover for one bad one.
ROC-AUC versus the precision-recall curve
Rather than evaluate at one threshold, you can sweep all of them and plot the result. The ROC curve plots the true positive rate (recall) against the false positive rate, FP / (FP + TN), and the area under it — ROC-AUC — has a clean interpretation: the probability that a random positive is scored higher than a random negative. A coin flip scores 0.5, a perfect ranker 1.0.
The trap is that the false positive rate's denominator is the entire negative class, so on imbalanced data it is almost impossible to make look bad. Return to the million transactions with 1,000 frauds. Suppose a detector catches 500 frauds while raising 10,000 false alarms. Recall is 500 / 1,000 = 50%, and the false positive rate is 10,000 / 999,000 = 1.0% — a point far above the diagonal that flatters the AUC. Precision, meanwhile, is 500 / 10,500 = 4.8%: the review team is looking at twenty items to find one fraud. The precision-recall curve plots precision against recall and shows that immediately, because precision's denominator is what you flagged rather than what exists. Use ROC-AUC to compare rankers on roughly balanced data, and a precision-recall curve whenever the positive class is scarce.
Types
Binary classification picks between two labels and is the case all the arithmetic above describes. Nearly everything else reduces to it.
Multi-class classification picks exactly one label from k mutually exclusive options — which of ten digits, which of a thousand ImageNet categories. The model emits k scores that a softmax forces to sum to 1, and the prediction is the largest. The confusion matrix becomes k × k, so precision and recall are computed per class and then averaged: macro-averaging treats all classes equally and lets a rare class drag the number down, micro-averaging weights by class frequency and lets the common ones dominate. Reporting one without saying which is a common way to mislead by accident.
Multi-label classification is the one people conflate with multi-class, and the distinction is real. Here each input may carry any number of labels at once — an article tagged politics, economics and Europe; an X-ray showing two findings. It is not a choice among k options but k independent yes/no decisions, each with its own threshold, and the output space grows accordingly: with 10 possible tags, a multi-class model has 10 possible answers while a multi-label model has 2^10 = 1,024. Softmax is the wrong output layer for it, because forcing the scores to sum to 1 makes the labels compete when they should not — you want 10 independent sigmoids instead. If you find your tagger can never assign two tags at once, this is why.
Real-World Applications
Autonomous diabetic retinopathy screening. IDx-DR was authorised by the US FDA in April 2018 as the first autonomous diagnostic device permitted to return a result without a physician reviewing the image. What makes it a classification story is that the FDA set the operating point in advance: the pivotal trial had to clear pre-specified endpoints of 85% sensitivity and 82.5% specificity, and reported 87.2% and 90.7%. A regulator chose the exchange rate between missed disease and unnecessary referrals, and the developer built to it — the clearest possible illustration that the threshold is a decision rather than a model property.
Email spam filtering. Providers such as Gmail run their filters at a deliberately conservative threshold because the two errors are wildly unequal in cost: one more advert in your inbox is an irritation, one lost invoice in the spam folder is a support ticket and a lost customer. This is also why spam filters lean on a quarantine tier rather than a hard binary — a third bucket for scores in the uncertain middle is often better engineering than agonising over where to put a single line.
Credit and lending decisions. A scoring model outputs a probability of default; the approval cutoff is set by the lender's risk appetite and by regulation, not by the model. Two banks running the identical model will approve different applicants, and adverse-action rules mean the same system must also explain the decision — which is why interpretable families such as logistic regression and shallow decision trees remain common in this domain long after they stopped being the most accurate option available.
Key Concepts
- Base rate: the proportion of your data that is positive. Every metric except precision and recall changes meaning when it changes, so write it at the top of any evaluation.
- Calibration: whether a score of 0.8 means the event really happens 80% of the time. A model can rank perfectly (AUC 1.0) and still be badly calibrated, which matters the moment you multiply a score by a cost.
- Cost-sensitive learning: telling the loss function that a mistake on the rare class costs more. It reshapes the boundary during training, whereas moving the threshold reshapes only the decision — try the free one first.
- Stratified splits: preserving the class ratio in every fold of cross-validation. With 2% positives and 5 random folds, one fold can easily end up with almost none, and your variance estimate becomes noise.
Challenges
Your labels set the ceiling. If two human annotators disagree on 8% of examples, no classifier can be more than about 92% accurate on that task — the remaining disagreement is not learnable signal. Teams routinely spend months chasing a gap that was in the labelling guidelines all along. Measure inter-annotator agreement before you measure the model.
The threshold expires. A cutoff chosen when fraud was 1% of transactions is silently wrong when a new attack pushes it to 3%, because the score distribution underneath it has moved. Thresholds are monitored and periodically re-fit; a model deployed with a frozen cutoff and no monitoring quietly degrades in a way accuracy dashboards are especially bad at revealing.
Optimising the threshold on the test set is cheating. Choosing the cutoff that maximises F1 on the same data you report F1 on is a form of overfitting, and it is extremely common. Pick the threshold on a validation split, then report on data neither the model nor the threshold has seen.
A single number hides who is failing. An aggregate 94% can conceal 99% on the majority group and 61% on a minority one. Per-class and per-segment confusion matrices are the minimum honest reporting for any system that makes decisions about people.
Future Trends
The most visible shift is that classification increasingly happens without a training set at all: you describe the categories to a large language model in a prompt and it labels the data, the zero-shot approach that has largely replaced the fine-tune-a-small-model workflow for one-off text labelling. The evaluation problem it creates is subtle and worth watching. A generative classifier hands back the word "spam", not a score, which quietly removes the threshold knob — you lose the ability to slide the operating point between precision and recall, and you cannot draw a precision-recall curve without extracting token probabilities or asking for a numeric confidence. Teams that migrated to LLM classifiers and then found they could not tune the precision/recall balance had not hit a limit of the model; they had given up the control surface described above.
Code Example
Three thresholds, one model, one set of scores. Only the last line of the pipeline changes:
import random
# 5,000 transactions, ~4% of them fraud. The model scores each one in [0, 1]:
# fraud scores higher on average, but the two distributions overlap.
rng = random.Random(7)
data = []
for _ in range(5000):
is_fraud = rng.random() < 0.04
score = min(max(rng.gauss(0.65 if is_fraud else 0.30, 0.13), 0.0), 1.0)
data.append((score, is_fraud))
def report(t):
tp = sum(1 for s, y in data if s >= t and y)
fp = sum(1 for s, y in data if s >= t and not y)
fn = sum(1 for s, y in data if s < t and y)
tn = len(data) - tp - fp - fn
precision, recall = tp / (tp + fp), tp / (tp + fn)
f1 = 2 * precision * recall / (precision + recall)
print(f"threshold {t:.2f} TP {tp:3d} FP {fp:4d} FN {fn:3d} "
f"precision {precision:.2f} recall {recall:.2f} "
f"F1 {f1:.2f} accuracy {(tp + tn) / len(data):.3f}")
for t in (0.40, 0.50, 0.60):
report(t)
threshold 0.40 TP 196 FP 1080 FN 5 precision 0.15 recall 0.98 F1 0.27 accuracy 0.783
threshold 0.50 TP 185 FP 314 FN 16 precision 0.37 recall 0.92 F1 0.53 accuracy 0.934
threshold 0.60 TP 133 FP 52 FN 68 precision 0.72 recall 0.66 F1 0.69 accuracy 0.976
Precision nearly quintuples from 0.15 to 0.72 while recall falls from 0.98 to 0.66, and the model was never retrained. Note the last column: accuracy climbs steadily to 0.976 as the threshold rises, right past the setting where the detector starts missing a third of all fraud. That is the failure mode this page exists to prevent.