Introduction
We rarely have data on an entire group. A company cannot ask every customer, a hospital cannot test every patient, and a machine-learning engineer cannot evaluate a model on every possible future input.
Inferential statistics uses data from a sample to draw conclusions about the larger population, while measuring how uncertain those conclusions are.
- Descriptive statistics: "The average of these 100 customers is 4.2 stars."
- Inferential statistics: "The average of all customers is between 4.0 and 4.4 stars, with 95% confidence."
Population, sample, parameter and statistic
| Term | Meaning | Symbol (example) |
|---|---|---|
| Population | The complete group of interest | N |
| Sample | A subset actually observed | n |
| Parameter | A fixed (usually unknown) number describing the population | μ, σ, p |
| Statistic | A number computed from the sample, used to estimate a parameter | x̄, s, p̂ |
The sample mean x̄ is a statistic used to estimate the population mean μ, which is a parameter.
Sampling
A good inference needs a good sample. The sample should be representative and, ideally, chosen randomly.
| Method | How it works |
|---|---|
| Simple random sampling | Every member has an equal chance of being selected |
| Stratified sampling | Split the population into groups (strata) and sample from each |
| Cluster sampling | Randomly choose entire groups (such as schools or villages) |
| Systematic sampling | Select every k-th member from a list |
Sampling bias happens when some members are more likely to be selected than others (for example, surveying only people who answer phone calls). More data does not fix a biased sample.
Sampling distribution and standard error
If you took many different samples of size n and calculated the mean of each, those means would vary. The distribution of a statistic across all possible samples is its sampling distribution.
The standard error (SE) is the standard deviation of that sampling distribution. For the sample mean:
SE = σ / √n (or s / √n when σ is unknown)
Key insight: larger samples give smaller standard error, so more precise estimates. To halve the error you need four times as much data.
The Central Limit Theorem (CLT)
For a sufficiently large sample size n (a common rule of thumb is n ≥ 30), the sampling distribution of the sample mean is approximately normal with mean μ and standard deviation σ/√n, regardless of the shape of the original population.
This is why the normal distribution appears so often, and why so many statistical tests work even on non-normal data.
Estimation
- Point estimate: a single best guess (for example x̄ = 72).
- Interval estimate (confidence interval): a range of plausible values (for example 69 to 75).
Confidence intervals
Confidence interval = point estimate ± margin of error
Margin of error = critical value × standard error
For a mean:
x̄ ± z* · (σ / √n) when σ is known or n is large
x̄ ± t* · (s / √n) when σ is unknown and n is small (t-distribution, df = n − 1)
Common critical values: z* = 1.645 (90%), 1.96 (95%), 2.576 (99%).
How to interpret it
A 95% confidence interval does not mean "there is a 95% probability that μ lies in this interval". It means: if we repeated the sampling procedure many times, about 95% of the intervals constructed this way would contain the true value.
Sample size for a desired margin of error
n = (z* · σ / E)² (round up)
where E is the margin of error you can tolerate.
Hypothesis testing
A hypothesis test decides whether an observed effect is real or could easily be random noise.
The steps
- State the hypotheses.
- Null hypothesis H₀: no effect or no difference ("the new model is no better").
- Alternative hypothesis H₁: there is an effect.
- Choose the significance level α (commonly 0.05).
- Compute the test statistic from the sample.
- Find the p-value (or compare with a critical value).
- Decide: if p < α, reject H₀; otherwise, fail to reject H₀.
The p-value
The p-value is the probability of getting a result at least as extreme as the one observed, assuming H₀ is true. A small p-value means the data would be surprising if H₀ were true.
It is not the probability that H₀ is true, and it does not measure the size or importance of an effect.
One-tailed versus two-tailed tests
- Two-tailed: H₁ says "different" (μ ≠ 100). Look at both ends of the distribution.
- One-tailed: H₁ says "greater" or "less" (μ > 100). Look at one end only.
Errors and power
| H₀ is true | H₀ is false | |
|---|---|---|
| Reject H₀ | Type I error (false positive), probability α | Correct decision; probability = power (1 − β) |
| Fail to reject H₀ | Correct decision | Type II error (false negative), probability β |
There is a trade-off: making α smaller reduces Type I errors but increases Type II errors. Power (typically aimed at 80% or more) rises with larger samples and larger true effects.
Common tests
| Test | Purpose |
|---|---|
| z-test | Compare a mean to a known value when σ is known or n is large |
| One-sample t-test | Compare a sample mean to a value when σ is unknown |
| Two-sample t-test | Compare the means of two independent groups |
| Paired t-test | Compare before/after measurements on the same subjects |
| Chi-square test | Test relationships or fit for categorical data |
| ANOVA | Compare the means of three or more groups |
Statistical significance versus practical significance
With a huge dataset, even a tiny, useless difference can be "statistically significant". Always look at the effect size and the confidence interval, not only the p-value.
Multiple testing
If you run 20 independent tests at α = 0.05, you expect about one false positive by chance alone. Use corrections such as Bonferroni (use α / number of tests) when running many tests.
Real-life applications
- Clinical trials: does a new drug outperform a placebo?
- A/B testing: does a new web-page design increase sign-ups?
- Opinion polls and surveys: estimate voter preferences from a sample, with a margin of error.
- Manufacturing: is the mean weight of packages within specification?
- Education: does a new teaching method improve scores?
Where it appears in machine learning
- Comparing models: is model A really better than model B, or is the difference just noise across cross-validation folds?
- A/B testing new models and features in production.
- Confidence intervals for metrics such as accuracy, or for coefficients in regression.
- Feature testing: chi-square and ANOVA tests can rank features by association with the target.
- Data drift detection: statistical tests compare training and production data distributions.
Python implementation
Confidence interval for a mean
import numpy as np
from scipy import stats
data = np.array([72, 75, 68, 80, 77, 74, 71, 79, 73, 76])
mean = data.mean()
sem = stats.sem(data) # standard error using s / sqrt(n)
ci = stats.t.interval(0.95, df=len(data) - 1, loc=mean, scale=sem)
print(mean, ci)
One-sample t-test
# H0: the true mean is 70
t_stat, p_value = stats.ttest_1samp(data, popmean=70)
print(t_stat, p_value)
Comparing two models
model_a = np.array([0.81, 0.79, 0.83, 0.80, 0.82]) # accuracy on 5 folds
model_b = np.array([0.78, 0.77, 0.80, 0.76, 0.79])
t_stat, p_value = stats.ttest_ind(model_a, model_b, equal_var=False) # Welch's t-test
print(p_value) # compare with alpha = 0.05
# For folds evaluated on the same data splits, a paired test is more appropriate:
print(stats.ttest_rel(model_a, model_b))
Chi-square goodness-of-fit
observed = [8, 9, 12, 11, 6, 14] # 60 die rolls
print(stats.chisquare(observed)) # expected counts default to equal (10 each)
Solved examples
Example 1: Standard error
A population has σ = 20. What is the standard error of the mean for a sample of n = 100?
Solution. SE = 20 / √100 = 20 / 10 = 2.
Example 2: Confidence interval
A sample of n = 64 has mean 50 and standard deviation 16. Build a 95% confidence interval for the population mean.
Solution. SE = 16 / √64 = 2. Margin of error = 1.96 × 2 = 3.92.
CI = 50 ± 3.92 = (46.08, 53.92).
Example 3: One-sample z-test
A machine is supposed to fill bags with a mean of 100 g (σ = 12 g). A sample of 36 bags has a mean of 104 g. Test at α = 0.05 whether the mean has changed.
Solution.
- H₀: μ = 100, H₁: μ ≠ 100.
- SE = 12 / √36 = 2.
- z = (104 − 100)/2 = 2.0.
- Two-tailed p-value = 2 × P(Z > 2) ≈ 0.0455.
Since 0.0455 < 0.05, we reject H₀: there is evidence that the mean fill has changed.
Example 4: Two-sample (Welch) t-test
Two groups took the same test. Group A: n = 10, mean 82, standard deviation 4. Group B: n = 10, mean 78, standard deviation 5. Is there a significant difference at α = 0.05?
Solution.
SE = √(4²/10 + 5²/10) = √4.1 ≈ 2.025.
t = (82 − 78)/2.025 ≈ 1.975.
With roughly 17 degrees of freedom the two-tailed p-value is about 0.065. Since 0.065 > 0.05, we fail to reject H₀. A 4-point difference in means is not statistically distinguishable from noise with samples this small.
Example 5: Chi-square goodness of fit
A die is rolled 60 times with counts 8, 9, 12, 11, 6, 14 for faces 1 to 6. Is the die fair (α = 0.05)?
Solution. Expected count for each face = 60/6 = 10.
χ² = Σ (O − E)²/E = (4 + 1 + 4 + 1 + 16 + 16)/10 = 4.2, with 5 degrees of freedom.
The 5% critical value is 11.07 and the p-value is about 0.52. Since 4.2 < 11.07, we fail to reject H₀: no evidence the die is unfair.
Example 6: Sample size
How large a sample is needed to estimate a mean to within ±2 units at 95% confidence, if σ = 10?
Solution. n = (1.96 × 10 / 2)² = 9.8² = 96.04, so n = 97.
Example 7: Interpreting errors
A spam filter is tested with H₀: "the email is legitimate". What are the two error types?
Solution. A Type I error (false positive) is marking a legitimate email as spam. A Type II error (false negative) is letting a spam email through.
Common mistakes
- Reading the p-value as "the probability that the null hypothesis is true".
- Saying "accept H₀" instead of "fail to reject H₀".
- Claiming importance from significance alone.
- Running many tests and reporting only the significant one (p-hacking).
- Using a non-random or biased sample and expecting large n to rescue it.
- Using an independent t-test on paired data.
Practice problems
- σ = 15 and n = 225. Find the standard error of the mean.
- n = 100, x̄ = 200 and s = 30. Build a 95% confidence interval (use z = 1.96).
- Test H₀: μ = 50 against H₁: μ ≠ 50 using n = 25, x̄ = 53 and σ = 10 at α = 0.05.
- What type of error occurs when you reject a null hypothesis that is actually true?
- What sample size is needed for a margin of error of 1.5 at 95% confidence when σ = 6?
- A test gives p = 0.03. What is your decision at α = 0.01?
Answers
- 15 / 15 = 1
- SE = 3, margin = 5.88, CI = (194.12, 205.88)
- SE = 2, z = 1.5, two-tailed p ≈ 0.134 > 0.05, so fail to reject H₀
- Type I error
- (1.96 × 6 / 1.5)² ≈ 61.47, so n = 62
- 0.03 > 0.01, so fail to reject H₀
Key takeaways
- Inferential statistics generalises from a sample to a population and quantifies the uncertainty.
- The standard error shrinks with √n; the CLT makes the sample mean approximately normal for large n.
- A confidence interval gives a plausible range for the parameter.
- In a hypothesis test, a small p-value means the data are unlikely under H₀, but it says nothing about effect size or importance.
- Balance Type I and Type II errors, and beware of multiple testing.