1. Z-score method
The z-score tells how many standard deviations a value is away from the mean. It works best when the data is roughly normal.
z = (x - mean) / std
A common rule: treat values with |z| > 3 as outliers (some people use 2 for stricter detection).
import numpy as np
from scipy import stats
fare = df["Fare"].dropna()
z = np.abs(stats.zscore(fare))
outliers_z = fare[z > 3]
print(len(outliers_z), "outliers by z-score")
Limitation: the mean and standard deviation are themselves pulled by outliers, so extreme values can hide themselves.
2. IQR (Interquartile Range) method
The IQR method is robust and does not assume a normal distribution.
IQR = Q3 - Q1- Lower bound =
Q1 - 1.5 x IQR - Upper bound =
Q3 + 1.5 x IQR
Values outside these bounds are outliers (this is exactly what a box plot draws).
Q1 = df["Fare"].quantile(0.25)
Q3 = df["Fare"].quantile(0.75)
IQR = Q3 - Q1
lower = Q1 - 1.5 * IQR
upper = Q3 + 1.5 * IQR
outliers_iqr = df[(df["Fare"] < lower) | (df["Fare"] > upper)]
print(f"Bounds: {lower:.2f} to {upper:.2f} -> {len(outliers_iqr)} outliers")
Use 3 x IQR instead of 1.5 x IQR to flag only extreme outliers.
3. Percentile method
Treat the lowest and highest 1% (or 5%) of values as outliers or cap them.
low, high = df["Fare"].quantile([0.01, 0.99])
outliers_pct = df[(df["Fare"] < low) | (df["Fare"] > high)]
print(low, high, len(outliers_pct))
4. Modified z-score (robust)
Uses the median and the MAD (median absolute deviation) instead of mean and std.
median = fare.median()
mad = np.median(np.abs(fare - median))
modified_z = 0.6745 * (fare - median) / mad
outliers_mz = fare[np.abs(modified_z) > 3.5]
print(len(outliers_mz), "outliers by modified z-score")
Comparison
| Method | Assumption | Robust to outliers? | Best for |
|---|---|---|---|
| Z-score | Roughly normal data | No | Symmetric numeric data |
| IQR | None | Yes | Skewed data |
| Percentile | None | Yes | Quick capping |
| Modified z-score | None | Yes | Small or skewed samples |
Compute the thresholds from the training set only and reuse them for validation and test data.