Naive Bayes

Definition

Naive Bayes is a family of probabilistic classifiers that use Bayes' theorem to work out the probability that an item belongs to each class, under one deliberately unrealistic assumption: that every feature is independent of every other feature once the class is known. It reads an email as a bag of words and asks, "given that these particular words appeared, is this message more likely to be spam or not?"

The "naive" part is that independence assumption. In real text, words are strongly correlated — "free" and "prize" travel together, and "meeting" and "agenda" do too — so treating each word as separate, unrelated evidence is provably wrong. The surprise, and the reason the method has survived since the 1960s, is that being wrong about the probabilities barely hurts the decision. A classifier only has to make the correct class score highest; it does not have to report the correct probability. Naive Bayes routinely reports a nonsense confidence of 0.9999 and still picks the right label.

That combination — a crude model of the world that nevertheless decides well — is what makes it worth understanding. It trains in a single pass over the data, needs very little of it, and runs fast enough to score a message while it is still being typed. If you ignore it and reach straight for a heavier model, you skip the cheapest baseline you have, and you lose the number that tells you whether the heavier model is actually earning its cost.

How It Works

Bayes' theorem relates the thing you want to know to things you can count. You want the probability of a class given the observed features; you can measure the reverse — how often those features show up in each class — from labelled training data. For a spam filter:

                P(words | spam) x P(spam)
P(spam | words) = -------------------------
                        P(words)

P(spam) is the prior: the fraction of your training emails that were spam, before you read a single word. P(words | spam) is the likelihood: how probable this exact combination of words is among spam messages. P(words) in the denominator is the same for every class, so it does not affect which class wins — but you need it to turn the raw scores into probabilities that sum to one, which is the normalisation step at the end.

The obstacle is P(words | spam) for a whole message. The number of possible word combinations is astronomically large, so you could never count them all directly. This is where the naive assumption does its work: assume the words are conditionally independent given the class, and the joint likelihood becomes a simple product of each word's individual likelihood.

P(w1, w2, ..., wn | spam) = P(w1 | spam) x P(w2 | spam) x ... x P(wn | spam)

Now every term is something you can count. To classify, compute prior x product-of-likelihoods for each class and pick the larger — that is the maximum a posteriori decision.

A worked spam example

Take a tiny training set: 4 spam messages and 6 ham (legitimate) ones.

spam: "free money now"          ham: "meeting at noon"
      "free money free"              "project meeting today"
      "claim your free prize"        "lunch with the team"
      "win money money"              "see you at noon"
                                     "the project deadline is today"
                                     "team lunch today"

The priors come straight from the class counts: P(spam) = 4/10 = 0.4 and P(ham) = 6/10 = 0.6. Counting tokens gives 13 words in the spam class, 22 in ham, and a shared vocabulary of 20 distinct words. Now classify the new message "free money".

The word "free" appears 4 times in spam and 0 times in ham; "money" is identical, 4 and 0. A raw likelihood for ham would therefore be zero, which is the zero-frequency trap — one unseen word would veto the entire class. Laplace smoothing fixes this by adding 1 to every count and adding the vocabulary size to every denominator, so nothing is ever impossible:

P(free  | spam) = (4 + 1) / (13 + 20) = 5/33  = 0.1515
P(money | spam) = (4 + 1) / (13 + 20) = 5/33  = 0.1515
P(free  | ham)  = (0 + 1) / (22 + 20) = 1/42  = 0.0238
P(money | ham)  = (0 + 1) / (22 + 20) = 1/42  = 0.0238

Multiply each prior by its two likelihoods:

score(spam) = 0.4 x 0.1515 x 0.1515 = 0.009183
score(ham)  = 0.6 x 0.0238 x 0.0238 = 0.000340

These are not probabilities yet — they do not sum to one. Normalise by dividing each by their total Z = 0.009183 + 0.000340 = 0.009523:

P(spam | "free money") = 0.009183 / 0.009523 = 0.964
P(ham  | "free money") = 0.000340 / 0.009523 = 0.036

So the message is classified spam with 96.4% posterior probability. Notice what smoothing bought: without it, P(free | ham) would have been exactly 0, driving score(ham) to 0 and the spam probability to a flat 100% — an unjustified certainty produced by a single missing word. Smoothing pulls that back to a defensible 96.4%.

Why real implementations work in logs

Multiply a few hundred probabilities, each well below 1, and the product underflows to zero in floating-point arithmetic long before you reach the end of a document. Every practical implementation therefore adds the logarithms of the probabilities instead of multiplying them — log(a x b) = log(a) + log(b) — and compares the sums. The decision is identical because the logarithm preserves order; only the numerical stability changes. The Code Example below does exactly this and lands on the same 0.964.

Types

The variants differ only in how they model P(feature | class) — the likelihood — and the right choice is dictated by what your features actually are:

• Multinomial Naive Bayes models features as counts drawn from a multinomial distribution, which is the natural fit for word frequencies. It is the standard choice for text classification and the variant used in the worked example above. A word that appears three times contributes its likelihood three times.
• Bernoulli Naive Bayes models each feature as a binary present/absent flag and, crucially, includes a term for words that are absent. When the absence of a word is itself informative — a short message that lacks the vocabulary of a legitimate one — Bernoulli can beat Multinomial, particularly on very short documents.
• Gaussian Naive Bayes handles continuous features by assuming each one follows a bell curve (a normal distribution) within each class, storing only a mean and variance per feature per class. It is the version to reach for when features are real numbers — sensor readings, measurements, pixel intensities — rather than counts.

These three are a genuine taxonomy, not an invented one: they correspond to three standard probability distributions, and scikit-learn ships them as MultinomialNB, BernoulliNB, and GaussianNB precisely because the distinction is real and load-bearing.

Real-World Applications

The defining application is spam filtering. Paul Graham's widely cited 2002 essay "A Plan for Spam" popularised the Bayesian approach for email, and Bayesian classifiers went on to ship inside filters such as SpamAssassin and the spam engines built into desktop mail clients of that era. The appeal was practical: the filter learns from the individual user's own marked messages, adapts as their mail changes, and runs cheaply on a mail server handling large volumes.

Beyond spam, it remains a workhorse for document and text classification — routing support tickets to the right queue, tagging news articles by topic, filtering content — anywhere the input is text and a fast, interpretable baseline is wanted. It is a common first pass for sentiment analysis, deciding whether a review is positive or negative from its word mix, and it appears in medical and diagnostic screening contexts where a transparent probability from a handful of independent-ish indicators is more defensible than a black box.

Its most honest role, though, is as the baseline everyone should run first. Because it trains in one pass and needs almost no tuning, it produces a number in minutes that any more expensive model — a support vector machine, a fine-tuned transformer — has to clearly beat to justify its extra compute, data, and maintenance. Skipping it means never knowing how much your heavy model actually added.

Key Concepts

Conditional independence is the whole engine and the whole compromise. It says that once you know the class, learning one feature's value tells you nothing about any other's. This is what collapses an intractable joint probability into a product of countable pieces, and it is also exactly what makes the reported probabilities untrustworthy.

Generative, not discriminative. Naive Bayes learns a model of how each class generates features — it could, in principle, invent a plausible spam message — and then inverts that model with Bayes' theorem to classify. Logistic regression, its closest cousin, instead learns the decision boundary directly. The practical consequence is a well-known trade-off: Naive Bayes reaches its (lower) accuracy ceiling with far less data, so it tends to win on small training sets and lose on large ones, where logistic regression's higher ceiling pays off.

Ranking survives, calibration does not. Because correlated features get counted as if they were independent, their evidence is double-counted and the scores are pushed toward the extremes — the 0.964 above would drift toward 0.9999 on a longer message full of co-occurring spam words. The class ordering usually survives this, which is why the decision is reliable; the probability is not, and anything downstream that consumes the number as a real confidence needs a separate calibration step to correct it.

Challenges

The probabilities are overconfident. This is the failure that surprises people who trust the output number. On correlated features the posterior collapses to near-0 or near-1, so "96% spam" and "99.99% spam" carry almost the same real meaning. If you threshold decisions on the raw probability, or feed it into a cost-sensitive system, you are acting on a miscalibrated figure — the reason calibration is a standard follow-up step for Naive Bayes specifically.

Correlated and redundant features hurt. Duplicating an informative feature effectively lets it vote twice. Adding "FREE", "free", and "F-R-E-E" as three separate features triples the weight of one underlying signal, skewing the result in a way a model that accounts for feature interactions would not.

Distribution shift breaks the counts. A spam filter's word statistics are only as current as its training mail, and spammers deliberately change their vocabulary to evade it — a moving target that a static count-based model cannot track without retraining. This concept drift is why Bayesian filters were designed to keep learning from newly labelled messages rather than being trained once.

Skewed class balance skews the prior. If 99% of training messages are ham, the prior alone makes ham the default, and rare-class detection suffers unless the class imbalance is addressed — by reweighting the priors or resampling — rather than trusting the raw frequencies.

Continuous features need the right variant. Gaussian Naive Bayes assumes each feature is bell-shaped within a class; feed it a skewed or multi-modal feature and the likelihood is simply wrong. Note that, unlike distance-based methods such as k-nearest-neighbours, Naive Bayes is unaffected by the scale of features — each is modelled in its own units — so feature scaling neither helps nor hurts it.

Code Example

A from-scratch multinomial classifier over the worked example above. It counts words per class, applies Laplace smoothing, works in log-space to avoid underflow, and normalises the two class scores back into a probability. Run as shown it prints the same result as the hand calculation.

from collections import Counter
from math import log, exp

spam = ["free money now", "free money free",
        "claim your free prize", "win money money"]
ham  = ["meeting at noon", "project meeting today",
        "lunch with the team", "see you at noon",
        "the project deadline is today", "team lunch today"]

def tokens(msgs):
    c = Counter()
    for m in msgs:
        c.update(m.split())
    return c, sum(len(m.split()) for m in msgs)

cs, Ns = tokens(spam)          # spam word counts, total spam tokens
ch, Nh = tokens(ham)           # ham word counts, total ham tokens
V = len(set(cs) | set(ch))     # vocabulary size
alpha = 1                      # Laplace smoothing

def loglik(word, c, N):
    return log((c[word] + alpha) / (N + alpha * V))

def score(msg, prior, c, N):
    s = log(prior)
    for w in msg.split():
        s += loglik(w, c, N)   # add logs instead of multiplying
    return s

prior_spam = len(spam) / (len(spam) + len(ham))
prior_ham  = len(ham)  / (len(spam) + len(ham))

msg = "free money"
s_spam = score(msg, prior_spam, cs, Ns)
s_ham  = score(msg, prior_ham,  ch, Nh)

# turn the two log-scores back into a normalised probability
p_spam = exp(s_spam) / (exp(s_spam) + exp(s_ham))
print(f"P(spam | '{msg}') = {p_spam:.3f}")

Output:

P(spam | 'free money') = 0.964

The log-space arithmetic makes no difference on a two-word message, but on a real document of several hundred words it is the difference between a stable answer and a product that has silently underflowed to zero.