Handling Outliers

After detecting an outlier, decide what to do with it. Properly handling outliers gives more reliable insights.

Option 1 — Remove

Delete rows that come from errors or do not represent the population.

df_clean = df[(df["Fare"] >= lower) & (df["Fare"] <= upper)]
print("Rows removed:", len(df) - len(df_clean))

Option 2 — Cap (winsorize / clip)

Replace extreme values with a boundary value. The row is kept.

df["Fare_capped"] = df["Fare"].clip(lower=lower, upper=upper)

# or cap at percentiles
df["Fare_capped_pct"] = df["Fare"].clip(lower=low, upper=high)

Option 3 — Transform

Compress large values with a mathematical function. Useful for extreme but valid values and skewed data.

import numpy as np
from sklearn.preprocessing import PowerTransformer

df["Fare_log"] = np.log1p(df["Fare"])       # log(1 + x), safe for zeros
df["Fare_sqrt"] = np.sqrt(df["Fare"])

pt = PowerTransformer(method="yeo-johnson")
df["Fare_power"] = pt.fit_transform(df[["Fare"]])

Option 4 — Impute

Treat the outlier as missing and replace it with the median or a model prediction.

Option 5 — Keep it and use robust methods

• Tree-based models (Random Forest, XGBoost) are relatively insensitive to outliers.
• Use RobustScaler, median-based statistics, Huber loss or quantile regression.

Option 6 — Analyse separately

In fraud or anomaly detection the outliers are the target. Model them; do not delete them.

Decision guide

Question	If yes	If no
Is it clearly an error?	Fix or remove	Continue
Is it a real but rare event?	Keep, cap or transform	Continue
Is the model sensitive to it (linear, distance-based)?	Cap, transform or RobustScaler	Keep as is
Is detecting it the goal?	Keep and model it	Continue

Good practices

1. Always visualize before deciding.
2. Compute bounds on the training set and apply the same bounds to test data.
3. Compare model performance with and without the treatment.
4. Document how many rows were changed and why.
5. Use domain knowledge: is Age = 110 impossible, or just rare?