Conditional Probability Explained with Examples for ML

Introduction

Probabilities change when we get new information. The probability that a randomly chosen person owns an umbrella is low, but the probability that they own an umbrella given that they are standing in a rainy city is much higher.

Conditional probability is the probability of an event A occurring given that another event B has already occurred. It is written P(A | B) and read "the probability of A given B".

This idea is central to machine learning: a classifier estimates P(class | features), a language model estimates P(next word | previous words), and a recommender estimates P(user clicks | user history).

The formula

                 P(A ∩ B)
P(A | B)  =  ─────────────── ,      provided P(B) > 0
                   P(B)

P(A ∩ B): the probability that both A and B happen (the joint probability).
P(B): the probability of the condition (the marginal probability).

The intuition: a smaller sample space

Once we know B has happened, B becomes our new sample space. We only ask: within B, what fraction also belongs to A?

Example. Roll a die. What is P(the roll is 6 | the roll is even)?

The even outcomes are {2, 4, 6}. Only one of the three is a 6, so P = 1/3, not 1/6. Knowing the roll was even raised the probability of a 6.

Joint, marginal and conditional probability

Type	Notation	Question it answers
Marginal	P(A)	How likely is A, ignoring everything else?
Joint	P(A ∩ B)	How likely are A and B together?
Conditional	P(A \| B)	How likely is A once we know B?

The multiplication rule

Rearranging the definition gives a rule for finding "and" probabilities when events are dependent:

P(A ∩ B) = P(B) × P(A | B) = P(A) × P(B | A)

For independent events, knowing B does not change A, so P(A | B) = P(A) and the rule reduces to P(A) × P(B).

The chain rule for many events

P(A₁ ∩ A₂ ∩ A₃ ∩ ... ∩ Aₙ) = P(A₁) · P(A₂ | A₁) · P(A₃ | A₁ ∩ A₂) · ... · P(Aₙ | A₁ ∩ ... ∩ Aₙ₋₁)

Language models use exactly this rule: the probability of a sentence is the product of the probability of each word given the words before it.

Testing for independence

Events A and B are independent if any one of these is true (they are equivalent):

P(A | B) = P(A)
P(B | A) = P(B)
P(A ∩ B) = P(A) × P(B)

If none holds, the events are dependent.

Tree diagrams

A tree diagram lays out sequential events. Each branch carries a (conditional) probability, and you multiply along a path to get the probability of that whole sequence.

Draw 1            Draw 2 (no replacement)          Path probability
                  ┌── Ace   (3/51)                 4/52 × 3/51 = 12/2652
   ┌── Ace (4/52)┤
   │              └── Not ace (48/51)              4/52 × 48/51
Start
   │              ┌── Ace   (4/51)                 48/52 × 4/51
   └── Not ace ──┤
      (48/52)     └── Not ace (47/51)              48/52 × 47/51

The four path probabilities add up to 1.

The law of total probability

Suppose events B₁, B₂, ..., Bₙ are mutually exclusive and exhaustive (they split the sample space into non-overlapping pieces). Then for any event A:

P(A) = P(A | B₁)·P(B₁) + P(A | B₂)·P(B₂) + ... + P(A | Bₙ)·P(Bₙ)

In words: to find the overall probability of A, work out its probability inside each scenario, weight each by how likely that scenario is, and add. This rule provides the evidence term in Bayes' theorem.

A very common trap: P(A | B) is not P(B | A)

P(a person is a professional basketball player | they are very tall) is tiny.
P(a person is very tall | they are a professional basketball player) is large.

The two are related but generally very different. Confusing them is called the transposed conditional fallacy (a version of it is the "prosecutor's fallacy"). Bayes' theorem, in the next lesson, is the correct way to convert one into the other.

Conditional independence

Events A and B are conditionally independent given C if P(A ∩ B | C) = P(A | C) × P(B | C). The Naive Bayes classifier is built on this idea: it assumes features are independent given the class, even though they are not independent overall.

Real-life applications

Medical diagnosis: probability of a disease given a symptom or test result.
Finance and insurance: probability of default given a credit history; probability of a claim given driver age.
Weather: probability of rain given today's cloud cover.
Search and recommendation: probability that you click given what you watched before.
Quality control: probability that an item is defective given it came from a certain machine.

Where it appears in machine learning

Classifiers model P(y | x).
Language models and speech recognition use the chain rule of conditional probabilities.
Hidden Markov models and Markov chains are defined by conditional transition probabilities.
Bayesian networks encode a full joint distribution as a graph of conditional probabilities.
Logistic regression directly models a conditional probability.

Python implementation

From a table of counts

import pandas as pd

df = pd.DataFrame({
    "attended": ["yes"] * 100 + ["no"] * 100,
    "passed":   ["yes"] * 90 + ["no"] * 10 + ["yes"] * 40 + ["no"] * 60,
})

table = pd.crosstab(df["attended"], df["passed"])
print(table)

# P(passed | attended)
p = table.loc["yes", "yes"] / table.loc["yes"].sum()
print(p)       # 0.9

# All conditional probabilities at once (each row sums to 1)
print(pd.crosstab(df["attended"], df["passed"], normalize="index"))

By simulation

import random

trials = 200_000
count_b = count_ab = 0

for _ in range(trials):
    d1, d2 = random.randint(1, 6), random.randint(1, 6)
    if 6 in (d1, d2):               # condition B: at least one six
        count_b += 1
        if d1 + d2 >= 10:           # event A: sum at least 10
            count_ab += 1

print(count_ab / count_b)   # about 5/11 = 0.4545

Solved examples

Example 1: Reduced sample space with dice

Two fair dice are rolled. Given that at least one die shows a 6, what is the probability that the sum is at least 10?

Solution. Outcomes with at least one 6: there are 11 of them ((1,6) through (6,6) and (6,1) through (6,5)).

Among those, the sum is at least 10 for (4,6), (6,4), (5,6), (6,5), (6,6), which is 5 outcomes.

P = 5/11 ≈ 0.4545.

Example 2: Cards without replacement

Two cards are drawn from a deck without replacement. What is the probability that both are aces?

Solution.

P(first ace) = 4/52. P(second ace | first ace) = 3/51.

P(both aces) = 4/52 × 3/51 = 12/2652 = 1/221 ≈ 0.0045.

Example 3: A contingency table

200 students were surveyed about whether they attended a revision class and whether they passed.

	Passed	Failed	Total
Attended	90	10	100
Did not attend	40	60	100
Total	130	70	200

Find (a) P(pass | attended), (b) P(pass | not attended), (c) P(attended | passed).

Solution.

(a) 90/100 = 0.90 (b) 40/100 = 0.40 (c) 90/130 = 0.692

Note that P(pass | attended) = 0.90 but P(attended | pass) = 0.692. They are different questions.

Example 4: Law of total probability

A factory has three machines. Machine 1 makes 50% of the items with a 1% defect rate, Machine 2 makes 30% with a 2% defect rate, and Machine 3 makes 20% with a 4% defect rate. What is the probability that a randomly chosen item is defective?

Solution.

P(defect) = 0.01 × 0.50 + 0.02 × 0.30 + 0.04 × 0.20 = 0.005 + 0.006 + 0.008 = 0.019 (1.9%).

Example 5: Checking independence

For two events, P(A) = 0.5, P(B) = 0.4 and P(A ∩ B) = 0.2. Are A and B independent?

Solution. P(A) × P(B) = 0.5 × 0.4 = 0.2, which equals P(A ∩ B). So A and B are independent. Confirm with the conditional form: P(A | B) = 0.2/0.4 = 0.5 = P(A).

Example 6: The chain rule

Three cards are drawn without replacement. What is the probability that all three are hearts?

Solution.

P = 13/52 × 12/51 × 11/50 = 1716/132600 = 11/850 ≈ 0.0129.

Common mistakes

Swapping P(A | B) and P(B | A).
Treating dependent events (sampling without replacement) as independent.
Forgetting that the denominator is P(B), the condition, not P(A).
Applying the law of total probability when the scenarios overlap or do not cover every case.

Practice problems

A bag has 5 red and 3 blue balls. Two balls are drawn without replacement. What is the probability that both are red?
Given P(A) = 0.6, P(B) = 0.5 and P(A ∩ B) = 0.3, find P(A | B). Are A and B independent?
If P(B) = 0.4 and P(A ∩ B) = 0.1, find P(A | B).
A fair die is rolled. What is P(6 | the number is even)?
Machine A produces 70% of a company's items, with a 3% defect rate. Machine B produces 30%, with a 5% defect rate. What is the overall defect rate?
A family has two children. Given that at least one is a boy, what is the probability that both are boys? (Assume boys and girls are equally likely.)

Answers

5/8 × 4/7 = 20/56 = 5/14
P(A | B) = 0.3/0.5 = 0.6. P(A) × P(B) = 0.3 = P(A ∩ B), so they are independent.
0.1/0.4 = 0.25
1/3
0.7 × 0.03 + 0.3 × 0.05 = 0.021 + 0.015 = 0.036 (3.6%)
The equally likely outcomes with at least one boy are BB, BG, GB, so the answer is 1/3 (not 1/2).

Key takeaways

P(A | B) = P(A ∩ B) / P(B): restrict attention to the cases where B happened.
Multiplication rule: P(A ∩ B) = P(B) · P(A | B).
Independence means P(A | B) = P(A).
The law of total probability combines probabilities across scenarios.
P(A | B) and P(B | A) are different, and Bayes' theorem connects them.

Conditional Probability

Introduction

The formula

The intuition: a smaller sample space

Joint, marginal and conditional probability

The multiplication rule

The chain rule for many events

Testing for independence

Tree diagrams

The law of total probability

A very common trap: P(A | B) is not P(B | A)

Conditional independence

Real-life applications

Where it appears in machine learning

Python implementation

From a table of counts

By simulation

Solved examples

Example 1: Reduced sample space with dice

Example 2: Cards without replacement

Example 3: A contingency table

Example 4: Law of total probability

Example 5: Checking independence

Example 6: The chain rule

Common mistakes

Practice problems

Answers

Key takeaways

Further reading