Introduction
Probability is the branch of mathematics that measures how likely it is that something will happen. It gives a numerical value to the chance of an event. We write the probability of an event E as P(E).
Every probability is a number between 0 and 1:
- P(E) = 0: the event is impossible (rolling a 7 on a normal die).
- P(E) = 1: the event is certain (a coin lands either heads or tails).
- 0 < P(E) < 1: the event is possible but not guaranteed. The closer to 1, the more likely it is.
Probabilities are often written as fractions (1/6), decimals (0.1667) or percentages (16.67%). All three mean the same thing.
Probability is the foundation of machine learning. A spam filter says "this email is 97% likely to be spam", a medical model says "this scan has a 12% chance of showing disease", and a language model picks the next word by assigning a probability to every word in its vocabulary. To understand how these models work and how to judge them, you first need the basic language of probability.
Key terms
| Term | Meaning | Example |
|---|---|---|
| Experiment | A process whose result is uncertain | Rolling a die |
| Trial | One performance of the experiment | One roll |
| Outcome | One possible result of a trial | Getting a 4 |
| Sample space (S) | The set of all possible outcomes | S = {1, 2, 3, 4, 5, 6} |
| Event (E) | Any subset of the sample space | E = "even number" = {2, 4, 6} |
| Favourable outcome | An outcome that belongs to the event | 2, 4 and 6 for "even" |
| Simple event | An event with exactly one outcome | Getting a 5 |
| Compound event | An event with more than one outcome | Getting an even number |
Sample spaces you should know
- One coin: S = {H, T}, so 2 outcomes.
- Two coins: S = {HH, HT, TH, TT}, so 4 outcomes.
- Three coins: 8 outcomes (2³).
- One die: 6 outcomes.
- Two dice: 36 outcomes (6 × 6), written as ordered pairs (first die, second die).
- One deck of cards: 52 outcomes (13 ranks × 4 suits).
In general, if one step has a outcomes and another independent step has b outcomes, together they have a × b outcomes. This is the counting principle.
The classical formula
When every outcome in the sample space is equally likely:
Number of favourable outcomes n(E)
P(E) = ─────────────────────────────────────── = ───────
Total number of outcomes n(S)
Example. A fair die is rolled. What is the probability of getting a number greater than 4?
Favourable outcomes = {5, 6}, so n(E) = 2. Total outcomes n(S) = 6.
P(E) = 2/6 = 1/3.
The sum of the probabilities of all outcomes in a sample space is always 1. For a fair die: 1/6 + 1/6 + 1/6 + 1/6 + 1/6 + 1/6 = 1.
Three ways to think about probability
-
Classical (theoretical) probability: based on equally likely outcomes, using the formula above. Works for fair coins, dice and cards.
-
Empirical (experimental) probability: based on what actually happened in repeated trials.
P(E) = (number of times E occurred) / (total number of trials)Example: in 500 emails, 120 were spam, so the empirical probability that an email is spam is 120/500 = 0.24.
-
Subjective probability: a personal degree of belief, used when no repeatable experiment exists ("I think there is a 70% chance our team wins").
The law of large numbers connects the first two: as the number of trials grows, the empirical probability gets closer and closer to the theoretical probability.
Types of events
| Type | Definition | Example |
|---|---|---|
| Impossible event | Cannot happen, P = 0 | Rolling a 9 on one die |
| Sure (certain) event | Always happens, P = 1 | Rolling a number from 1 to 6 |
| Mutually exclusive events | Cannot happen at the same time | "Even" and "odd" on one roll |
| Exhaustive events | Together they cover the whole sample space | "Even" and "odd" |
| Independent events | One event does not affect the other's probability | Two separate coin tosses |
| Dependent events | One event changes the other's probability | Drawing two cards without replacement |
| Complementary events | A and "not A" | "Rain" and "no rain" |
The basic probability rules
1. Complement rule
P(not A) = P(A′) = 1 − P(A)
Use it when "the opposite" is easier to count. "At least one head in 3 tosses" is easiest as 1 − P(no heads).
2. Addition rule
For any two events A and B:
P(A ∪ B) = P(A) + P(B) − P(A ∩ B)
A ∪ B means "A or B (or both)". We subtract P(A ∩ B) because outcomes in both events would otherwise be counted twice.
If A and B are mutually exclusive, then P(A ∩ B) = 0 and the rule simplifies to:
P(A ∪ B) = P(A) + P(B)
3. Multiplication rule for independent events
P(A ∩ B) = P(A) × P(B)
A ∩ B means "A and B". The rule for dependent events uses conditional probability and is covered in the next lesson.
4. Odds versus probability
Odds compare success to failure: odds = P / (1 − P). A probability of 0.75 corresponds to odds of 3 to 1. Odds appear in logistic regression as the "log-odds".
The sample space of two dice
Rolling two dice gives 36 equally likely outcomes. Counting how many produce each sum:
| Sum | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Ways | 1 | 2 | 3 | 4 | 5 | 6 | 5 | 4 | 3 | 2 | 1 |
| Probability | 1/36 | 2/36 | 3/36 | 4/36 | 5/36 | 6/36 | 5/36 | 4/36 | 3/36 | 2/36 | 1/36 |
A sum of 7 is the most likely (6/36 = 1/6), which is why it matters so much in board games.
Real-life applications
- Weather forecasting: "70% chance of rain" comes from probability models built on past data.
- Insurance: companies estimate the probability of accidents, illness or theft to set premiums.
- Finance: analysts estimate the probability of price movements and default risk.
- Medicine: the chance that a treatment works or that a test result is correct.
- Games and lotteries: cards, dice, raffles and lucky draws.
- Quality control: the probability that a manufactured item is defective.
Where probability appears in machine learning
- Classifier outputs: softmax and sigmoid layers turn raw scores into probabilities that sum to 1.
- Loss functions: cross-entropy loss measures how much probability a model assigned to the correct answer.
- Naive Bayes multiplies probabilities of features.
- Data splitting and sampling: random train/test splits and mini-batches depend on random selection.
- Evaluation: precision, recall and the ROC curve are all built from probabilities of correct and incorrect predictions.
Python implementation
Enumerating a sample space
from itertools import product
# All outcomes of rolling two dice
space = list(product(range(1, 7), repeat=2))
print(len(space)) # 36
# Event: sum equals 8
event = [o for o in space if sum(o) == 8]
print(len(event), len(event) / len(space)) # 5 0.1388...
Simulating with the law of large numbers
import random
def estimate_p_even(n):
rolls = [random.randint(1, 6) for _ in range(n)]
return sum(r % 2 == 0 for r in rolls) / n
for n in [10, 100, 10_000, 1_000_000]:
print(n, estimate_p_even(n))
# The estimate approaches 0.5 as n grows.
Solved examples
Example 1: Simple probability
A bag has 5 red, 3 green and 2 blue balls. One ball is drawn at random. Find (a) P(green) and (b) P(not red).
Solution. Total balls = 5 + 3 + 2 = 10.
(a) P(green) = 3/10 = 0.3.
(b) P(not red) = 1 − P(red) = 1 − 5/10 = 1/2. (Check: green + blue = 5 balls, so 5/10.)
Example 2: Sum of two dice
Two fair dice are rolled. What is the probability that the sum is 8?
Solution. The sample space has 36 outcomes. The favourable outcomes are (2,6), (3,5), (4,4), (5,3), (6,2), which is 5 outcomes.
P(sum = 8) = 5/36 ≈ 0.139.
Example 3: The addition rule with cards
A card is drawn from a standard 52-card deck. What is the probability that it is a king or a heart?
Solution.
- P(king) = 4/52
- P(heart) = 13/52
- P(king and heart) = 1/52 (the king of hearts)
P(king or heart) = 4/52 + 13/52 − 1/52 = 16/52 = 4/13 ≈ 0.308.
Example 4: Empirical probability
Out of 500 emails received last month, 120 were spam. (a) What is the empirical probability that a new email is spam? (b) How many spam emails would you expect out of 2000?
Solution.
(a) P(spam) = 120/500 = 0.24.
(b) Expected spam = 0.24 × 2000 = 480 emails.
Example 5: Dependent draws
An urn contains 3 red, 4 blue and 5 green balls. Three balls are drawn one after another without replacement. What is the probability that the first is red, the second is blue and the third is green?
Solution. After each draw, one ball fewer remains.
- P(first red) = 3/12
- P(second blue | first was red) = 4/11
- P(third green | first two drawn) = 5/10
P = 3/12 × 4/11 × 5/10 = 60/1320 = 1/22 ≈ 0.0455 (about 4.5%).
Example 6: The complement rule
A fair coin is tossed 3 times. What is the probability of getting at least one head?
Solution. The sample space has 2³ = 8 outcomes. The only outcome with no heads is TTT.
P(at least one head) = 1 − P(no heads) = 1 − 1/8 = 7/8 = 0.875.
Common mistakes
- Forgetting to subtract the overlap in the addition rule.
- Using P(A) × P(B) when the events are not independent.
- Assuming outcomes are equally likely when they are not (the sum of two dice is not uniform).
- Believing in the "gambler's fallacy": after five heads in a row, the next toss is still 50/50.
Practice problems
- A box holds 6 white, 4 black and 10 grey marbles. If one marble is picked at random, what is the probability that it is black?
- Two fair dice are rolled. What is the probability that the sum is 10?
- In a survey of 500 people, 300 like tea, 250 like coffee and 100 like both. What is the probability that a randomly chosen person likes tea or coffee?
- The probability that it rains tomorrow is 0.35. What is the probability that it does not rain?
- A fair coin is tossed three times. What is the probability of getting exactly two heads?
- A card is drawn from a deck. What is the probability that it is neither a face card (J, Q, K) nor a heart?
Answers
- 4/20 = 1/5
- 3/36 = 1/12
- (300 + 250 − 100)/500 = 0.9
- 0.65
- Outcomes HHT, HTH, THH give 3/8 = 0.375
- Face cards = 12, hearts = 13, overlap = 3, so union = 22. Neither = (52 − 22)/52 = 30/52 = 15/26
Key takeaways
- Probability is a number from 0 to 1 that measures how likely an event is.
- Classical probability = favourable outcomes ÷ total outcomes, if outcomes are equally likely.
- Use the complement rule for "at least one" problems, the addition rule for "or", and the multiplication rule for "and".
- Empirical probability approaches the true probability as you collect more data.
- Machine learning models output and reason with probabilities everywhere.