Bayes' Theorem

Introduction

Bayes' theorem is a rule for updating a belief when new evidence arrives. It is named after the statistician Thomas Bayes and it is one of the most important formulas in machine learning, medicine, spam filtering, search and scientific reasoning.

It answers this question: "I know how likely the evidence is if my hypothesis is true. How likely is the hypothesis, now that I have seen the evidence?"

Where the formula comes from

From the multiplication rule, the joint probability can be written in two ways:

P(A ∩ B) = P(A | B) · P(B) = P(B | A) · P(A)

Divide both sides by P(B):

                 P(B | A) · P(A)
P(A | B)  =  ─────────────────────
                      P(B)

That is Bayes' theorem.

The four ingredients

Let H be a hypothesis (for example "the patient has the disease") and E be the evidence (for example "the test is positive"):

                 Likelihood × Prior
Posterior  =  ────────────────────────      →      P(H | E) = P(E | H) · P(H) / P(E)
                    Evidence

Term	Symbol	Meaning
Prior	P(H)	Your belief in H before seeing the evidence
Likelihood	P(E | H)	How probable the evidence is if H is true
Evidence (marginal likelihood)	P(E)	How probable the evidence is overall
Posterior	P(H | E)	Your updated belief after seeing the evidence

Computing the evidence

The denominator P(E) usually comes from the law of total probability:

P(E) = P(E | H) · P(H)  +  P(E | not H) · P(not H)

Bayes' theorem for many hypotheses

If H₁, H₂, ..., Hₙ are mutually exclusive and exhaustive:

                          P(E | Hᵢ) · P(Hᵢ)
P(Hᵢ | E)  =  ─────────────────────────────────
                    Σⱼ  P(E | Hⱼ) · P(Hⱼ)

The denominator simply makes all the posteriors add up to 1.

A famous example: the medical test

• 1% of people have a certain disease (prior).
• If a person has the disease, the test is positive 95% of the time (sensitivity).
• If a person does not have the disease, the test is still positive 5% of the time (false-positive rate).

A person tests positive. What is the probability that they actually have the disease?

Step 1: Evidence.

P(positive) = 0.95 × 0.01 + 0.05 × 0.99 = 0.0095 + 0.0495 = 0.059

Step 2: Posterior.

P(disease | positive) = (0.95 × 0.01) / 0.059 = 0.0095 / 0.059 ≈ 0.161

Only about 16%, even though the test is "95% accurate". Most people are healthy, so the small false-positive rate applied to a very large healthy group produces more false alarms than the true positives.

The natural-frequency view

Imagine 10,000 people:

• 100 have the disease → 95 test positive.
• 9,900 are healthy → 495 test positive (5%).
• Total positives = 95 + 495 = 590.
• Chance that a positive person is truly sick = 95 / 590 ≈ 16.1%.

This is the base rate effect: ignoring the prior (how rare something is) leads to badly wrong conclusions. It is also why accuracy is a poor metric for imbalanced datasets in machine learning.

Bayesian updating: today's posterior is tomorrow's prior

After the first positive test the probability is 0.161. If the patient takes a second, independent test and it is also positive, use 0.161 as the new prior:

P(positive) = 0.95 × 0.161 + 0.05 × 0.839 ≈ 0.1949
P(disease | 2nd positive) = (0.95 × 0.161) / 0.1949 ≈ 0.785

The probability jumps to about 78.5%. Each new piece of evidence refines the belief. This step-by-step learning is the heart of Bayesian inference.

The Naive Bayes classifier

Naive Bayes applies Bayes' theorem to classification. Given features x₁, x₂, ..., xₙ (for example the words in an email), it picks the class C with the highest posterior:

P(C | x₁, ..., xₙ)  ∝  P(C) · P(x₁ | C) · P(x₂ | C) · ... · P(xₙ | C)

It is called naive because it assumes the features are conditionally independent given the class. That assumption is rarely exactly true, yet the classifier works surprisingly well, trains extremely fast, and needs little data.

Common variants:

• Multinomial Naive Bayes: word counts in text classification.
• Bernoulli Naive Bayes: binary word presence or absence.
• Gaussian Naive Bayes: continuous features assumed to be normally distributed within each class.

Laplace smoothing

If a word never appears in the spam training emails, P(word | spam) = 0 and the whole product becomes 0. Laplace (add-one) smoothing avoids this by adding 1 to every count so no probability is ever exactly zero.

Real-life applications

• Spam filtering: the classic use of Naive Bayes.
• Medical diagnosis: interpreting screening and test results.
• Fraud detection: how likely a flagged transaction is truly fraudulent.
• Search and recommendation: ranking based on updated beliefs.
• Legal reasoning and forensics: weighing evidence.
• Self-driving and robotics: Bayesian filters track position from noisy sensors.

Where it appears in machine learning

• Naive Bayes text and document classifiers.
• Bayesian inference: estimating model parameters as distributions rather than single numbers.
• Bayesian optimisation: choosing the next hyperparameters to try.
• MAP estimation: many regularisers (such as L2) correspond to a prior over the weights.
• Bayesian neural networks and Gaussian processes for uncertainty estimates.

Python implementation

The medical test

prior = 0.01            # P(disease)
sensitivity = 0.95      # P(positive | disease)
false_positive = 0.05   # P(positive | no disease)

evidence = sensitivity * prior + false_positive * (1 - prior)
posterior = sensitivity * prior / evidence
print(round(posterior, 3))     # 0.161

# Sequential update with a second positive test
prior2 = posterior
evidence2 = sensitivity * prior2 + false_positive * (1 - prior2)
posterior2 = sensitivity * prior2 / evidence2
print(round(posterior2, 3))    # 0.785

A Naive Bayes spam classifier with scikit-learn

from sklearn.feature_extraction.text import CountVectorizer
from sklearn.naive_bayes import MultinomialNB

texts = [
    "win free money now", "claim your free prize", "limited offer win cash",
    "meeting at noon tomorrow", "lunch with the team", "project report attached",
]
labels = [1, 1, 1, 0, 0, 0]          # 1 = spam, 0 = not spam

vec = CountVectorizer()
X = vec.fit_transform(texts)

model = MultinomialNB(alpha=1.0)     # alpha=1.0 is Laplace smoothing
model.fit(X, labels)

new = vec.transform(["free cash prize"])
print(model.predict(new))            # [1]
print(model.predict_proba(new))      # probabilities for each class

Solved examples

Example 1: Which machine made the defective item?

Using the factory from the previous lesson: Machine 1 makes 50% of items (1% defective), Machine 2 makes 30% (2% defective), Machine 3 makes 20% (4% defective). An item is found to be defective. What is the probability it came from Machine 3?

Solution.

P(defect) = 0.005 + 0.006 + 0.008 = 0.019.

P(M3 | defect) = (0.04 × 0.20) / 0.019 = 0.008 / 0.019 ≈ 0.421.

Machine 3 makes only 20% of items, but it is responsible for about 42% of the defects.

Example 2: Spam and the word "offer"

30% of all emails are spam. The word "offer" appears in 40% of spam emails and in 5% of legitimate emails. An email contains "offer". What is the probability it is spam?

Solution.

P(offer) = 0.40 × 0.30 + 0.05 × 0.70 = 0.12 + 0.035 = 0.155.

P(spam | offer) = 0.12 / 0.155 ≈ 0.774.

Example 3: Two bags

Bag A contains 3 red and 2 blue balls. Bag B contains 1 red and 4 blue balls. A bag is chosen at random (each with probability 1/2) and one ball is drawn. It is red. What is the probability that it came from Bag A?

Solution.

P(red | A) = 3/5 = 0.6 and P(red | B) = 1/5 = 0.2.

P(A | red) = (0.6 × 0.5) / (0.6 × 0.5 + 0.2 × 0.5) = 0.3 / 0.4 = 0.75.

Example 4: Naive Bayes by hand

A message contains the words "free" and "win". Prior probabilities: P(spam) = 0.4, P(ham) = 0.6.

	P(free | class)	P(win | class)
Spam	0.5	0.4
Ham	0.1	0.05

Solution.

Spam score = 0.4 × 0.5 × 0.4 = 0.08 Ham score = 0.6 × 0.1 × 0.05 = 0.003

Normalise: P(spam | message) = 0.08 / (0.08 + 0.003) = 0.08 / 0.083 ≈ 0.964.

The message is classified as spam with about 96% confidence.

Example 5: Fraud detection and the base rate

0.1% of transactions are fraudulent. A detector flags 90% of fraudulent transactions and also wrongly flags 2% of legitimate ones. A transaction is flagged. What is the probability it is fraud?

Solution.

P(flag) = 0.9 × 0.001 + 0.02 × 0.999 = 0.0009 + 0.01998 = 0.02088.

P(fraud | flag) = 0.0009 / 0.02088 ≈ 0.043.

Only about 4.3% of flagged transactions are truly fraudulent. The detector looks strong but produces mostly false alarms because fraud is so rare.

Common mistakes

• Ignoring the base rate (prior) and trusting only the test accuracy.
• Confusing sensitivity P(positive | disease) with P(disease | positive).
• Forgetting to include all hypotheses when computing the evidence.
• Assuming Naive Bayes features are truly independent and trusting its probabilities as perfectly calibrated (its class ranking is usually good; its exact probabilities often are not).

Practice problems

1. A disease affects 2% of a population. A test has 90% sensitivity and a 10% false-positive rate. If a person tests positive, what is the probability of having the disease?
2. Box 1 contains 2 gold and 1 silver coin. Box 2 contains 1 gold and 3 silver coins. A box is picked at random and a coin drawn is gold. What is the probability that it came from Box 1?
3. It rains on 20% of days. A forecast predicts rain on 90% of rainy days and on 15% of dry days. Given that rain is forecast, what is the probability that it actually rains?
4. Three suppliers provide 50%, 30% and 20% of parts, with defect rates of 2%, 3% and 5%. A part is defective. What is the probability that it came from Supplier 3?
5. A review classifier has P(positive) = P(negative) = 0.5. The word "great" has probability 0.3 in positive reviews and 0.02 in negative ones. The word "boring" has probability 0.01 in positive and 0.25 in negative reviews. A review contains both words. Using Naive Bayes, what is P(positive | review)?

Answers

1. 0.018 / (0.018 + 0.098) = ≈ 0.155
2. (1/2 × 2/3) / (1/2 × 2/3 + 1/2 × 1/4) = 8/11 ≈ 0.727
3. 0.18 / (0.18 + 0.12) = 0.60
4. 0.01 / (0.01 + 0.009 + 0.01) = 0.01 / 0.029 ≈ 0.345
5. Positive score = 0.5 × 0.3 × 0.01 = 0.0015. Negative score = 0.5 × 0.02 × 0.25 = 0.0025. P = 0.0015 / 0.004 = 0.375

Key takeaways

• Bayes' theorem: Posterior = Likelihood × Prior / Evidence.
• It reverses conditional probabilities: from P(evidence | hypothesis) to P(hypothesis | evidence).
• Rare events stay unlikely even after a "positive" result unless the evidence is very strong. Always account for the base rate.
• Beliefs can be updated repeatedly; each posterior becomes the next prior.
• Naive Bayes turns Bayes' theorem into a fast, simple classifier by assuming conditional independence of features.

Further reading

• Bayes' theorem in machine learning