Choosing an Approach and Common Mistakes

Which technique should I use?

Situation	Suggested approach
Mild imbalance	Stratified split + class weights
Large dataset, moderate imbalance	Random undersampling or class weights
Small dataset, moderate to severe imbalance	SMOTE (inside a pipeline) + class weights
Extreme imbalance	Anomaly detection, PR-AUC, threshold tuning
Tree-based boosting	scale_pos_weight / class weights + threshold tuning
Need calibrated probabilities	Class weights or resampling, then calibrate
Very costly false negatives	Optimise recall, lower the threshold

A practical recipe

1. Check the class distribution.
2. Split with stratification.
3. Choose the right metric (PR-AUC, F1, recall at fixed precision).
4. Build a naive baseline (DummyClassifier) and a plain model.
5. Try class weights first (simple and safe).
6. Try resampling (SMOTE, undersampling) inside a pipeline.
7. Try balanced ensembles or boosting.
8. Tune the threshold on validation data.
9. Evaluate once on the untouched test set.
10. Monitor the class ratio and metrics in production.

Compare approaches side by side

from imblearn.pipeline import Pipeline as ImbPipeline
from imblearn.over_sampling import SMOTE
from imblearn.under_sampling import RandomUnderSampler
from sklearn.dummy import DummyClassifier
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import StratifiedKFold, cross_validate
import pandas as pd

candidates = {
    "Baseline (dummy)": DummyClassifier(strategy="most_frequent"),
    "Plain RF": RandomForestClassifier(random_state=42),
    "RF + class weight": RandomForestClassifier(class_weight="balanced_subsample", random_state=42),
    "SMOTE + RF": ImbPipeline([("s", SMOTE(random_state=42)), ("m", RandomForestClassifier(random_state=42))]),
    "Undersample + RF": ImbPipeline([("u", RandomUnderSampler(random_state=42)), ("m", RandomForestClassifier(random_state=42))]),
}

cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
rows = []
for name, est in candidates.items():
    res = cross_validate(est, X_train, y_train, cv=cv,
                         scoring=["average_precision", "recall", "f1"])
    rows.append({"model": name,
                 "PR-AUC": res["test_average_precision"].mean(),
                 "recall": res["test_recall"].mean(),
                 "F1": res["test_f1"].mean()})

print(pd.DataFrame(rows).round(3))

Common mistakes

Mistake	Why it is a problem
Using accuracy as the main metric	Hides the failure to detect the minority class
Resampling before the train/test split	Data leakage, inflated scores
Oversampling the test set	The test set no longer reflects reality
Forgetting stratify=y	The test set may contain almost no minority samples
Using SMOTE on categorical features without SMOTENC	Meaningless synthetic categories
Resampling to 1:1 blindly	Can hurt precision badly; try smaller ratios
Tuning the threshold on the test set	Overly optimistic results
Ignoring business cost	The "best" F1 may not be the cheapest outcome
Trusting probabilities after resampling	They are distorted; calibrate if needed

Checklist

• Class distribution inspected
• Stratified train/test split
• Metric chosen from business cost (PR-AUC, recall, F1)
• Baseline built for comparison
• Resampling done only on training data, inside a pipeline
• Class weights and ensembles compared
• Threshold tuned on validation data
• Final evaluation on an untouched test set

Key takeaways

• Imbalanced data makes accuracy meaningless; use precision, recall, F1 and PR-AUC.
• Fix it at the data level (resampling), the algorithm level (class weights) or the decision level (threshold tuning).
• Always split first, stratify, and resample only the training data inside a pipeline.