Introduction
A probability distribution describes all the possible values a random quantity can take and how likely each one is. If probability tells you the chance of one event, a distribution gives you the whole picture at once.
Distributions are the language of data. Heights, exam scores, website visits, the number of defects in a batch, and the errors a model makes all follow some distribution. Choosing the right one is often the first modelling decision in machine learning and statistics.
Random variables
A random variable (usually written X) is a rule that assigns a number to each outcome of a random experiment.
- Toss 3 coins: let X = number of heads. X can be 0, 1, 2 or 3.
- Measure a person's height: let X = height in cm.
There are two kinds:
| Discrete | Continuous | |
|---|---|---|
| Values | Countable (0, 1, 2, ...) | Any value in a range |
| Examples | Number of emails per hour, dice rolls | Height, temperature, time |
| Described by | PMF: probability mass function | PDF: probability density function |
| Probability of one exact value | P(X = x) can be positive | Always 0; use intervals instead |
| Total | Σ P(x) = 1 | Total area under the curve = 1 |
PMF, PDF and CDF
- PMF P(X = x): the probability that a discrete variable equals x.
- PDF f(x): the height of the curve. The probability that X lies between a and b is the area under the curve from a to b.
- CDF F(x) = P(X ≤ x): the accumulated probability up to x. It works for both types and always rises from 0 to 1.
Expected value and variance
The expected value (mean) is the long-run average:
E[X] = Σ x · P(x) (discrete)
The variance measures spread, and the standard deviation is its square root:
Var(X) = E[(X − μ)²] = E[X²] − (E[X])²
σ = √Var(X)
Example. For one fair die, E[X] = (1 + 2 + 3 + 4 + 5 + 6)/6 = 3.5, and Var(X) = 35/12 ≈ 2.917.
Discrete distributions
1. Bernoulli distribution
A single trial with two outcomes: success (1) with probability p, failure (0) with probability 1 − p.
P(X = 1) = p, P(X = 0) = 1 − p
Mean = p, Variance = p(1 − p)
Examples: one coin toss, whether an email is spam, whether a customer clicks an ad. This is the distribution behind binary classification.
2. Binomial distribution
The number of successes in n independent Bernoulli trials, each with success probability p.
P(X = k) = C(n, k) · pᵏ · (1 − p)ⁿ⁻ᵏ , where C(n, k) = n! / (k!(n − k)!)
Mean = np, Variance = np(1 − p)
Examples: the number of heads in 10 tosses, the number of defective items in a batch of 50, the number of correct answers when guessing on a quiz.
3. Poisson distribution
The number of events in a fixed interval of time or space, when events occur independently at a constant average rate λ.
P(X = k) = e^(−λ) · λᵏ / k!
Mean = λ, Variance = λ
Examples: calls arriving at a call centre per minute, typos per page, server requests per second.
Continuous distributions
4. Uniform distribution
Every value in the interval [a, b] is equally likely.
f(x) = 1 / (b − a) for a ≤ x ≤ b
Mean = (a + b) / 2, Variance = (b − a)² / 12
Examples: a random number generator, arrival time when you know only the interval.
5. Normal (Gaussian) distribution
The famous bell curve, completely described by its mean μ and standard deviation σ.
f(x) = 1 / (σ√(2π)) · exp( −(x − μ)² / (2σ²) )
Properties:
- Symmetric around μ; mean = median = mode.
- 68–95–99.7 rule: about 68% of values lie within 1σ of the mean, 95% within 2σ, and 99.7% within 3σ.
- The standard normal has μ = 0 and σ = 1.
A z-score converts any normal value into standard units:
z = (x − μ) / σ
It tells you how many standard deviations a value is from the mean. Probabilities are then read from the standard normal CDF (a "z-table") or computed with software.
Why is it everywhere? Because of the Central Limit Theorem: sums and averages of many independent effects tend towards a normal distribution.
6. Exponential distribution
The waiting time between events of a Poisson process, with rate λ.
f(x) = λ · e^(−λx) for x ≥ 0
CDF: P(X ≤ x) = 1 − e^(−λx)
Mean = 1/λ, Variance = 1/λ²
Examples: time until the next customer arrives, lifetime of a component. It has the memoryless property: the time you have already waited does not change how much longer you must wait.
Choosing a distribution
| Situation | Distribution |
|---|---|
| One yes/no trial | Bernoulli |
| Count successes in n fixed trials | Binomial |
| Count events per interval, rare and independent | Poisson |
| Continuous, equally likely over a range | Uniform |
| Measurements clustering around an average | Normal |
| Time between independent events | Exponential |
Real-life applications
- Quality control: binomial for the number of defects in a sample.
- Telecom and web servers: Poisson for arrivals; exponential for waiting times.
- Human measurements: heights, weights, test scores are often approximately normal.
- Finance: returns are often modelled with normal (or heavier-tailed) distributions.
- Reliability engineering: exponential lifetimes of machines and components.
Where distributions appear in machine learning
- Bernoulli / categorical distributions underlie logistic regression and softmax classifiers.
- Normal distributions: weight initialisation, noise in linear regression (minimising squared error is equivalent to maximum likelihood under Gaussian noise), Gaussian Naive Bayes, Gaussian mixture models and VAEs.
- Poisson regression models count data.
- Uniform distributions: random initialisation and random sampling.
- Cross-entropy and KL divergence measure how far one distribution is from another.
Python implementation
import numpy as np
from scipy import stats
# Binomial: P(exactly 3 heads in 10 fair tosses)
print(stats.binom.pmf(k=3, n=10, p=0.5)) # 0.1171875
# Poisson: P(exactly 2 calls when the average is 4 per minute)
print(stats.poisson.pmf(k=2, mu=4)) # 0.1465
# Normal: P(X > 85) when mu=70, sigma=10
print(1 - stats.norm.cdf(85, loc=70, scale=10)) # 0.0668
# Exponential: P(wait > 10 min) when the mean wait is 5 min
print(1 - stats.expon.cdf(10, scale=5)) # 0.1353
# Draw random samples
rng = np.random.default_rng(0)
samples = rng.normal(loc=50, scale=10, size=5)
print(samples)
Visualising a distribution
import matplotlib.pyplot as plt
x = np.linspace(-4, 4, 400)
plt.plot(x, stats.norm.pdf(x))
plt.title("Standard normal PDF")
plt.show()
Solved examples
Example 1: Binomial
A fair coin is tossed 5 times. What is the probability of exactly 2 heads?
Solution. n = 5, p = 0.5, k = 2.
P(X = 2) = C(5, 2) × 0.5² × 0.5³ = 10 × 0.03125 = 0.3125.
Example 2: Guessing on a quiz
A quiz has 10 multiple-choice questions, each with 4 options, and a student guesses every answer (p = 0.25). Find (a) P(exactly 3 correct) and (b) P(at least one correct).
Solution.
(a) P(X = 3) = C(10, 3) × 0.25³ × 0.75⁷ = 120 × 0.015625 × 0.13348 ≈ 0.2503.
(b) P(X ≥ 1) = 1 − P(X = 0) = 1 − 0.75¹⁰ ≈ 1 − 0.0563 = 0.9437.
Example 3: Poisson
A call centre receives on average 4 calls per minute. What is the probability of (a) exactly 2 calls in a minute and (b) no calls in a minute?
Solution. λ = 4.
(a) P(X = 2) = e⁻⁴ × 4² / 2! = 8e⁻⁴ ≈ 0.1465.
(b) P(X = 0) = e⁻⁴ ≈ 0.0183.
Example 4: Normal distribution
Exam scores are normally distributed with mean 70 and standard deviation 10. Find (a) P(score > 85) and (b) P(60 < score < 80).
Solution.
(a) z = (85 − 70)/10 = 1.5. P(Z > 1.5) = 1 − 0.9332 = 0.0668 (about 6.7%).
(b) The limits 60 and 80 are exactly one standard deviation below and above the mean, so by the 68–95–99.7 rule the probability is about 0.6827.
Example 5: Uniform
A bus arrives at a stop at a time uniformly distributed between 0 and 20 minutes from now. What is the probability of waiting less than 5 minutes? What is the average wait?
Solution. f(x) = 1/20.
P(X < 5) = 5/20 = 0.25. Mean = (0 + 20)/2 = 10 minutes.
Example 6: Exponential
Customers arrive on average every 5 minutes (λ = 1/5 = 0.2 per minute). What is the probability that the next customer takes more than 10 minutes?
Solution.
P(T > 10) = e^(−0.2 × 10) = e⁻² ≈ 0.1353.
Common mistakes
- Using a normal distribution for data that is clearly skewed or bounded (such as income or counts of rare events).
- Treating a continuous value as if P(X = exact value) were positive.
- Using the binomial distribution when the trials are not independent or p changes between trials.
- Mixing up the PDF (a density, can be greater than 1) with a probability.
Practice problems
- A Bernoulli variable has p = 0.3. Find its mean and variance.
- A fair coin is tossed 6 times. What is P(exactly 3 heads)?
- Emails arrive at an average rate of 2 per hour (Poisson). What is the probability of exactly 1 email in an hour?
- IQ scores are normal with mean 100 and standard deviation 15. What is the probability that a person has an IQ above 130?
- X is uniform on [2, 8]. Find the mean and variance.
- A component's lifetime is exponential with rate λ = 0.5 per year. What is the probability it fails within 2 years?
Answers
- Mean = 0.3, variance = 0.3 × 0.7 = 0.21
- C(6, 3)/2⁶ = 20/64 = 0.3125
- 2e⁻² ≈ 0.2707
- z = 2, so P ≈ 1 − 0.9772 = 0.0228
- Mean = 5, variance = 36/12 = 3
- 1 − e⁻¹ ≈ 0.6321
Key takeaways
- A distribution tells you every possible value of a random variable and how likely it is.
- Discrete variables use a PMF; continuous variables use a PDF and probabilities are areas.
- Know the six workhorses: Bernoulli, Binomial, Poisson, Uniform, Normal, Exponential.
- The z-score standardises normal values; the 68–95–99.7 rule gives quick estimates.
- Many ML models are built by assuming a distribution for the data or the noise.