Identifying Outliers and Anomalies

An outlier is a data point that differs significantly from the rest. Outliers distort means, standard deviations, correlations and many models, so finding them is a key part of EDA.

Why do outliers occur?

Cause	Example	Action
Data-entry / sensor error	age = 250, temperature = -999	fix or remove
Measurement unit mix-up	some heights in cm, others in metres	convert
Natural but rare event	a billionaire in income data	keep, but use robust methods
Real anomaly of interest	fraudulent transaction	study it - it may be the goal

Never delete an outlier just because it is inconvenient. First ask: is it an error, or is it real information?

from scipy import stats
from sklearn.ensemble import IsolationForest
from sklearn.neighbors import LocalOutlierFactor

col = "alcohol"

# 1) Z-score rule
z = np.abs(stats.zscore(df[col]))
print("Z-score outliers:", (z > 3).sum())

# 2) IQR rule
q1, q3 = df[col].quantile([0.25, 0.75])
iqr = q3 - q1
mask = (df[col] < q1 - 1.5 * iqr) | (df[col] > q3 + 1.5 * iqr)
print("IQR outliers:", mask.sum())

# 3) Isolation Forest on several features at once
X = df.select_dtypes("number").drop(columns=["quality"])
iso = IsolationForest(contamination=0.02, random_state=42)
df["iso_flag"] = iso.fit_predict(X)          # -1 = outlier
print("Isolation Forest outliers:", (df["iso_flag"] == -1).sum())

# 4) Local Outlier Factor
lof = LocalOutlierFactor(n_neighbors=20, contamination=0.02)
df["lof_flag"] = lof.fit_predict(X)
print("LOF outliers:", (df["lof_flag"] == -1).sum())

1. Z-score

The Z-score tells how many standard deviations a value is from the mean:

z = (x - mean) / std

• Best for: roughly normally distributed data. |z| above 3 is usually an outlier.
• Not suitable: non-normal or heavily skewed data, or small samples; the mean and std themselves are affected by the outliers ("masking"). A modified Z-score based on the median and MAD is more robust.
• Example: a company flags days when sales deviate hugely from average, e.g. a promotion spike.

2. IQR rule (box-plot rule)

A value is an outlier if it lies below Q1 - 1.5 x IQR or above Q3 + 1.5 x IQR.

• Robust and needs no normality assumption; the same rule drawn by box-plot whiskers.
• Use 3 x IQR to flag only extreme outliers.

3. Isolation Forest

A tree-based algorithm that isolates points by random splits. Outliers are isolated in very few splits (short paths), normal points need many.

• Best for: large, complex, high-dimensional datasets, e.g. fraud detection in financial transactions.
• Not suitable: small data or simple distributions where Z-score or box plots suffice.
• Key parameter: contamination = expected share of outliers.
• Example: an e-commerce platform flags purchases that deviate from typical patterns.

4. Local Outlier Factor (LOF)

Compares the local density of a point with that of its neighbours. A point in a much sparser area than its neighbours has a high LOF score.

• Best for: data with clusters of different densities, where a globally defined threshold fails.
• Not suitable: very large data (distance computations are costly), or when n_neighbors cannot be tuned sensibly.
• Use lof.negative_outlier_factor_ to see scores; lower (more negative) = more abnormal.

Comparison

Method	Dimensions	Assumption	Speed	Interpretability
Z-score	1	approx. normal	very fast	very easy
IQR	1	none	very fast	easy
Isolation Forest	many	none	fast	moderate
LOF	many	local density meaningful	slower	moderate

What to do with outliers

1. Investigate - look at the actual rows.
2. Correct obvious errors if the true value is known.
3. Remove only if clearly invalid and few.
4. Cap / winsorise - replace extremes with the 1st / 99th percentile:

low, high = df["alcohol"].quantile([0.01, 0.99])
df["alcohol_capped"] = df["alcohol"].clip(low, high)

5. Transform - log or Box-Cox reduces the effect of extremes.
6. Use robust models / statistics - median, IQR, tree-based models, Huber regression.
7. Keep and flag - add an is_outlier column so the model can learn from it.

Always document what you did and why.