Precision and Recall

Definition

Precision and recall are the two halves of "how good is this model?" that plain accuracy throws away. Precision is the share of the things a model flagged that turned out to be real. Recall is the share of the real things that the model flagged. Precision's denominator is your own output; recall's denominator is the world.

The reason they deserve a page rather than a footnote is one piece of arithmetic that surprises almost everyone the first time. Take an end-of-line inspection system on a factory line: 50,000 parts go past the camera and 500 of them are genuinely defective. Tune the classifier to its best accuracy — 99.00% — and it is exactly as accurate as a machine that stamps every part "fine" without looking, while letting three defects in five reach the customer. Accuracy is a weighted average across classes in which each class counts in proportion to how common it is, so when 99% of the parts are good, accuracy is a measurement of good parts and nothing else. Precision and recall do not have this defect, for a simple structural reason: neither of them counts the true negatives at all.

Which of the two you should care about is not a matter of preference. It is set by which of your two mistakes costs more, and the answer is usually lopsided by an order of magnitude or more. A cancer screen's false negative is a tumour found a year later; its false positive is a follow-up scan. A résumé filter's false positive is an hour of an interviewer's time; its false negative is a hire you never knew you missed. Write down both prices before you choose a metric, because the metric you choose will silently pick an operating point for you — and as the worked numbers below show, F1 and accuracy pick the most expensive setting of the three on this inspection line.

How It Works

The four counts everything is made of

Every classification metric you have ever seen — accuracy, precision, recall, F1, specificity, ROC-AUC — is arithmetic on four numbers. Run the inspection model over the 50,000 parts at its default cutoff of 0.50. It flags 1,200 parts for re-inspection, and 300 of those flags are real defects:

That table is the confusion matrix. The four cells sum to the test set, and the whole rest of this page is division. True positives are defects caught; false positives are good parts pulled off the line for nothing; false negatives are defects shipped; true negatives are the enormous, boring mass of good parts correctly ignored.

Now read four metrics off the same four numbers:

Precision = TP / (TP + FP) = 300 / 1,200 = 0.250. One flag in four is a real defect; the re-inspection team throws away three quarters of its effort.

Recall = TP / (TP + FN) = 300 / 500 = 0.600. Two defects in five are shipped to customers.

Accuracy = (TP + TN) / 50,000 = 48,900 / 50,000 = 0.978.

F1 = 2PR / (P + R) = 2 × 0.250 × 0.600 / 0.850 = 0.353.

Notice the spread. The same model, on the same data, in the same minute, is either a 97.8% success or a 25%-precision failure depending entirely on which division you perform. Nothing about the model changed between those two sentences.

Why accuracy is a liar here, in one line

Build the laziest possible competitor: a classifier that never flags anything. It has no camera and no model. TP = 0, FP = 0, FN = 500, TN = 49,500, so its accuracy is 49,500 / 50,000 = 0.990 — higher than the real model's 0.978. Its recall is 0 / 500 = 0.000. Its precision is not low; it is undefined, because it flagged nothing to be precise about.

This is the base-rate argument, and it is the reason the page exists. Any metric whose formula contains TN inherits the size of the majority class, and on a 1% positive rate the majority class is 99 times louder than the thing you are trying to detect. Precision and recall are the two ratios that omit TN, which is exactly why they survive imbalance. The same arithmetic gets worse as the positive class gets rarer: at 1 in 10,000, doing nothing scores 99.99%, and the problem shades into anomaly detection, where the queue rather than the model is the subject.

Which one do I care about? Put a price on each error

The honest answer to "precision or recall" is not a rule of thumb, it is a ratio. Suppose that on this line a needlessly re-inspected part costs $8 of an operator's time, and a defect that escapes to a customer costs $400 in warranty, shipping and goodwill. The ratio is 50 to 1, so the rule follows immediately: keep loosening the threshold until you are raising more than 50 false alarms for each additional defect you catch. Past that point the extra inspections cost more than the defects they prevent; before it, you are saving money by flagging more.

That single ratio dissolves most arguments about which metric to report. It is also the reason no dataset can answer the question for you — the exchange rate between your two errors lives in your business, not in your data, and two companies running the identical model will correctly choose different cutoffs.

Precision and recall belong to a threshold, not to a model

A classifier does not really emit a label. It emits a score, and a label appears only when somebody picks a cutoff. Move the cutoff and both numbers move, in opposite directions, with no retraining and no new data. Here is the same model at three cutoffs, plus the do-nothing baseline, with the $8/$400 cost attached:

Three things in that table are worth more than the rest of this page.

First, the strict cutoff's accuracy of 0.9900 is identical, to four decimal places, to the accuracy of the model that flags nothing at all. Not similar — equal. Accuracy cannot distinguish a detector that catches 40% of defects from a detector that does not exist.

Second, F1 and accuracy both rank the cutoffs in exactly the wrong order. F1 rises 0.200 → 0.353 → 0.444 as the threshold tightens; cost rises $48,400 → $87,200 → $121,600 alongside it. The setting F1 likes best is the one that loses two and a half times as much money as the setting it likes least. Check it against the exchange rate: going from cutoff 0.80 to 0.50 catches 100 more defects for 700 more false alarms — 7 alarms per defect, comfortably under the break-even 50 — and going from 0.50 to 0.20 catches 150 more for 2,650 more, or 17.7 per defect, still worth paying.

Third, none of this involved a better model. It is the same score column read four ways. "The model isn't good enough" is usually the wrong diagnosis; "we are operating it at the wrong point" is usually the right one, and it is free to fix.

F1, and why it is a harmonic mean

Two numbers are awkward when you want to rank models on a leaderboard, so F1 collapses them into one: the harmonic mean of precision and recall, F1 = 2PR / (P + R). At the default cutoff that is 0.353, against an ordinary average of (0.250 + 0.600) / 2 = 0.425. The harmonic mean sits below the arithmetic mean always, and equals it only when P and R are identical.

The gap is the whole point, and it grows as the two numbers diverge. Consider a model that flags a single part in the whole run and is right about it: precision 1.00, recall 0.002. The ordinary average is a respectable-looking 0.501. Its F1 is 2 × 1.00 × 0.002 / 1.002 = 0.004. Because the harmonic mean is the reciprocal of the average of the reciprocals, a small value contributes an enormous reciprocal and dominates the result — so F1 refuses to let one excellent number cover for one terrible one. That is precisely the behaviour you want from a single summary of two quantities that can be traded against each other, since any metric that could be gamed by pushing one to 1.0 would be gamed exactly that way.

Where F1 is the wrong summary, the fix is usually F-beta, which the CoNLL-2003 shared task writes as Fβ = (β² + 1) · P · R / (β² · P + R). Setting β = 1 recovers F1; β = 2 weights recall twice as heavily as precision, β = 0.5 does the reverse. On the inspection line, F2 scores the three cutoffs 0.375 / 0.469 / 0.417 — it prefers the middle setting, whereas F1 preferred the strictest. Changing β changed which model you would ship. Choosing β is the same decision as choosing the threshold, made in a different notation, and it has the same answer: the relative cost of the two errors.

The same matrix, a different "positive"

Here is a property of F1 that surprises people who have used it for years: it is not symmetric. Take the confusion matrix above, change nothing about the predictions, and merely decide that a good part is the positive class. TP becomes 48,600, FP becomes 200, FN becomes 900, and:

• calling a defect the positive class: precision 0.250, recall 0.600, F1 = 0.353
• calling a good part the positive class: precision 0.996, recall 0.982, F1 = 0.989

Same model, same predictions, same test set, F1 nearly tripled. F1 ignores TN, so which cell counts as "the big boring one" is a choice, and the metric changes drastically with it. Always say which class is positive when you report an F1, and be suspicious of any leaderboard that does not. If you want a single number that does not have this behaviour, the Matthews correlation coefficient uses all four cells and returns 0.378 for this matrix whichever class you nominate.

Curves, and where PR beats ROC

Rather than judge at one cutoff, sweep every cutoff and plot the result. The precision-recall curve plots precision against recall; the ROC curve plots recall (also called the true positive rate, or sensitivity) against the false positive rate FP / (FP + TN). The area under either is a threshold-free summary of the ranking.

The trap is that the false positive rate's denominator is the entire negative class, which on imbalanced data is enormous and forgiving. Read the loose cutoff both ways: 3,550 false alarms out of 49,500 good parts is a false positive rate of 7.2%, a point that looks strong on an ROC plot and flatters the AUC. The same 3,550 false alarms against 450 catches is a precision of 11.3% — nearly nine parts pulled for every real defect. The PR curve shows the second reading immediately because precision's denominator is what you flagged rather than what exists.

The baselines differ too, and this is the most practical way to remember which to use. A random ranker always scores ROC-AUC 0.5, whatever the class balance. Its PR-AUC is the base rate — 0.01 here — so a PR-AUC of 0.30 on this data is a thirtyfold improvement over chance, while an ROC-AUC of 0.85 tells you much less than it appears to. Davis and Goadrich showed at ICML 2006 that the two spaces are not in conflict — one curve dominates another in ROC space if and only if it dominates in PR space — so they cannot disagree about which model is better; they disagree about how large the difference looks, and on skewed data PR is the space where the difference is visible.

How much should you trust these two numbers?

Recall is estimated only from the positives in your test set, so its uncertainty is governed by how many positives you have, not by how large the test set is. With 500 defects and a recall of 0.600, the 95% interval is roughly ±4.3 percentage points. Cut the test set down to 50 defects — still 5,000 parts at this base rate — and the same estimate carries ±13.6 points, which is wide enough that a model that genuinely improved recall from 0.60 to 0.70 would be indistinguishable from noise. Precision is estimated from the flags instead: 1,200 of them here, giving about ±2.5 points.

The practical consequence is that on rare-positive problems your test set must be sized by the count of positives you need, not by the fraction of data you feel like holding out — and that cross-validation folds must be stratified, or one fold will end up with almost no positives and a recall estimate that is pure noise.

Real-World Applications

Two-stage retrieval in search and RAG. Production retrieval-augmented generation systems split the job in two precisely along this axis. Stage one — a vector search or keyword index — is tuned for recall@k: fetch the top 100 or 500 candidates and make sure the right passage is somewhere in there, because nothing downstream can recover a document that was never retrieved. Stage two, a cross-encoder reranker, is tuned for precision@10, since the language model will only read a handful of passages and each irrelevant one spends context and invites a wrong answer. Teams that report a single "retrieval accuracy" for the pipeline cannot see which stage is failing; recall@100 and precision@10 are two different bug reports.

Named-entity recognition. The CoNLL-2003 shared task, still the standard NER benchmark, publishes 23,499 entity mentions across 203,621 tokens in its English training split — roughly one mention every nine tokens. A tagger that labels every token "not an entity" is therefore right about the overwhelming majority of them, which is why the task specifies Fβ=1 over exactly-matched entities rather than token accuracy, and why every NER paper since reports precision, recall and F1 as a triple. The same logic carries into extractive question answering, where SQuAD scores token overlap between the predicted and reference span with an F1 rather than demanding exact matches.

Object detection. The COCO benchmark's headline metric, AP, is literally the area under a precision-recall curve — computed at 101 recall thresholds, averaged across 10 intersection-over-union thresholds from .50 to .95 in steps of .05, and then across all 80 object categories. There is no accuracy figure on the leaderboard, and there could not be: an image contains an unbounded number of places where an object is not, so the true-negative count is undefined and every metric that needs it is unavailable. Computer vision had to become precision-and-recall native, and the practice spread from there.

Key Concepts

• Precision has other names. In clinical medicine, recall is sensitivity, precision is positive predictive value, and TN / (TN + FP) is specificity. The vocabularies matter because the field that says "PPV" also insists you state the prevalence next to it — sensitivity is a property of the test, PPV is a property of the test and the population, which is the base-rate point wearing different clothes.
• Micro-averaging over classes is just accuracy. Pool the TP, FP and FN counts across all classes of a single-label multi-class problem and micro-precision, micro-recall and accuracy all come out identical. Macro-averaging — compute per class, then average — is the one that lets a rare class pull the number down, which is usually what you wanted. Reporting either without saying which is a common way to mislead by accident.
• Precision@k for ranked output. When a system produces a ranked list and a human reviews a fixed number of items per shift, the operational metric is precision within that budget, not at a threshold. Twenty-item queues and score cutoffs are different products.
• Calibration is a separate question. A model can rank perfectly — every positive scored above every negative, ROC-AUC 1.0 — while its scores are nowhere near probabilities. The moment you multiply a score by a cost, as the $8/$400 rule does, you need the score to mean what it says.

Challenges

You usually cannot measure recall, only estimate it. Precision is directly observable: read your flag queue and count. Recall needs the positives you missed, which by construction you did not see. The standard fix is to audit a random sample of the unflagged pool, and the arithmetic is discouraging. At the default cutoff, 48,800 parts go unflagged and 200 of them are defective — 0.41%. Audit 2,000 of them and you expect about 8 defects, whose 95% interval runs from roughly 3 to 16; scaled back up, the missed-defect count lands somewhere between 84 and 385, so your measured recall is somewhere between 44% and 78%. That is a range wide enough to contain both "acceptable" and "fire the vendor", from an audit ten times larger than most teams run.

Precision does not transfer between datasets, and neither does F1. Recall is a property of the model on the positive class alone, so it is roughly stable if the positives look the same. Precision depends on how many negatives are in the pool, so the identical model scores 25% precision on this line and something quite different on a line with a 3% defect rate. Comparing your F1 to a published F1 from another dataset is therefore close to meaningless unless the base rates match, and they almost never do. Compare against your own base rate and your own previous release.

Choosing the threshold on the test set is a form of overfitting. Picking the cutoff that maximises F1 and then reporting that F1 on the same data leaks the answer into the result, and it is extremely common because the sweep is so easy to run. Pick the operating point on a validation split, then report on data that neither the model nor the threshold has seen.

The base rate moves and takes precision with it. A cutoff chosen when the defect rate was 1% is silently wrong when a supplier change pushes it to 3%: recall may hold, but precision will jump and the review team's workload with it, and nobody will be alerted because accuracy barely twitches. Precision and recall are monitored quantities, not launch-day numbers, and the base rate belongs on the same dashboard.

A single number hides who is failing. An aggregate recall of 0.600 can be 0.85 on one product line and 0.20 on another, or — where the classifier makes decisions about people — 0.85 for one demographic group and 0.20 for another. Per-segment confusion matrices are the minimum honest reporting for any deployed classification system.

Code Example

Every number on this page comes out of four counts and some division. This runs the whole argument, including the do-nothing baseline and the class-swap demonstration:

# 50,000 inspected parts, 500 of them genuinely defective: a 1% base rate.
# One model, one set of scores, three cutoffs, plus the do-nothing baseline.
# Counts are (true positives, false positives, false negatives, true negatives).
points = {
    "cutoff 0.20": (450, 3550, 50, 45950),
    "cutoff 0.50": (300, 900, 200, 48600),
    "cutoff 0.80": (200, 200, 300, 49300),
    "flag nothing": (0, 0, 500, 49500),
}

def pr(tp, fp, fn):
    precision = tp / (tp + fp) if tp + fp else float("nan")
    return precision, tp / (tp + fn)

def fbeta(precision, recall, beta):
    b2 = beta ** 2
    return (1 + b2) * precision * recall / (b2 * precision + recall)

for label, (tp, fp, fn, tn) in points.items():
    precision, recall = pr(tp, fp, fn)
    accuracy = (tp + tn) / (tp + fp + fn + tn)
    cost = 8 * fp + 400 * fn    # $8 per needless re-inspection, $400 per escaped defect
    print(f"{label:<13} P {precision:5.3f}  R {recall:5.3f}"
          f"  F1 {fbeta(precision, recall, 1):5.3f}"
          f"  F2 {fbeta(precision, recall, 2):5.3f}"
          f"  acc {accuracy:.4f}  cost ${cost:>7,}")

# Same 50,000 parts, same predictions - only the word "positive" moves.
tp, fp, fn, tn = points["cutoff 0.50"]
print()
for label, counts in [("defect = positive", (tp, fp, fn)), ("good part = positive", (tn, fn, fp))]:
    precision, recall = pr(*counts)
    print(f"{label:<21} P {precision:.3f}  R {recall:.3f}  F1 {fbeta(precision, recall, 1):.3f}")

cutoff 0.20   P 0.113  R 0.900  F1 0.200  F2 0.375  acc 0.9280  cost $ 48,400
cutoff 0.50   P 0.250  R 0.600  F1 0.353  F2 0.469  acc 0.9780  cost $ 87,200
cutoff 0.80   P 0.500  R 0.400  F1 0.444  F2 0.417  acc 0.9900  cost $121,600
flag nothing  P   nan  R 0.000  F1   nan  F2   nan  acc 0.9900  cost $200,000

defect = positive     P 0.250  R 0.600  F1 0.353
good part = positive  P 0.996  R 0.982  F1 0.989

Read the columns against each other rather than down. Accuracy is monotone in the threshold and peaks at 0.9900, tying a classifier with no model in it. F1 is monotone the same way and picks the $121,600 setting. F2 picks the $87,200 one. Cost — the only column that knows what the errors are worth — picks the $48,400 one, the row where precision looks worst. And the last two lines are the identical prediction set scored twice, with F1 moving from 0.353 to 0.989 because someone changed which class was called positive.

The lesson is not that F1 is bad. It is that a summary metric is a guess at your cost function, and when you know the actual costs you should use them and treat precision and recall as the two quantities the guess was summarising.