Instead of changing the data, you can change how the model learns or how it decides.
1. Class weights (cost-sensitive learning)
Tell the model that mistakes on the minority class cost more. No data is added or removed.
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import RandomForestClassifier
from sklearn.svm import SVC
lr = LogisticRegression(class_weight="balanced", max_iter=1000).fit(X_train, y_train)
rf = RandomForestClassifier(class_weight="balanced_subsample", random_state=42).fit(X_train, y_train)
svc = SVC(class_weight={0: 1, 1: 10}, probability=True).fit(X_train, y_train) # custom weights
| Value | Meaning |
|---|---|
"balanced" | Weight = n_samples / (n_classes x class_count) |
"balanced_subsample" | Same, recomputed for each tree's bootstrap sample (Random Forest) |
{0: 1, 1: 10} | Custom weights, for example when a miss costs 10x more |
For XGBoost use scale_pos_weight = n_negative / n_positive. For LightGBM use is_unbalance=True or scale_pos_weight.
2. Sample weights
from sklearn.utils.class_weight import compute_sample_weight
weights = compute_sample_weight(class_weight="balanced", y=y_train)
model = RandomForestClassifier(random_state=42).fit(X_train, y_train, sample_weight=weights)
3. Threshold tuning
Most classifiers predict the positive class when the probability is at least 0.5. With imbalanced data a lower threshold can catch more minority cases.
import numpy as np
from sklearn.metrics import precision_recall_curve
y_prob = rf.predict_proba(X_test)[:, 1]
prec, rec, thresholds = precision_recall_curve(y_test, y_prob)
f1 = 2 * prec * rec / (prec + rec + 1e-12)
best = np.argmax(f1[:-1])
print("Best threshold:", round(thresholds[best], 3), "| F1:", round(f1[best], 3))
y_pred_tuned = (y_prob >= thresholds[best]).astype(int)
Choose the threshold on a validation set (or with cross-validation), never on the test set.
You can also pick a threshold from a business rule, for example "recall must be at least 90%".
# smallest threshold that still gives at least 90% recall
target_recall = 0.90
ok = np.where(rec[:-1] >= target_recall)[0]
threshold = thresholds[ok[-1]] if len(ok) else 0.5
print("Threshold for 90% recall:", round(threshold, 3))
4. Probability calibration
Class weights and resampling distort predicted probabilities. If you need real probabilities, calibrate them.
from sklearn.calibration import CalibratedClassifierCV
calibrated = CalibratedClassifierCV(RandomForestClassifier(class_weight="balanced", random_state=42),
method="isotonic", cv=5)
calibrated.fit(X_train, y_train)
proba = calibrated.predict_proba(X_test)[:, 1]
5. Ensemble methods designed for imbalance
from imblearn.ensemble import BalancedRandomForestClassifier, EasyEnsembleClassifier
brf = BalancedRandomForestClassifier(n_estimators=300, random_state=42).fit(X_train, y_train)
print("BalancedRF PR-AUC:", round(average_precision_score(y_test, brf.predict_proba(X_test)[:, 1]), 3))
# XGBoost example
# from xgboost import XGBClassifier
# spw = (y_train == 0).sum() / (y_train == 1).sum()
# xgb = XGBClassifier(scale_pos_weight=spw, eval_metric="aucpr").fit(X_train, y_train)
| Method | Idea |
|---|---|
| BalancedRandomForest | Each tree is trained on a balanced bootstrap sample |
| EasyEnsemble | Boosted ensemble of models trained on balanced undersampled subsets |
| RUSBoost | Random undersampling plus boosting |
| Boosting with class weights | XGBoost, LightGBM, CatBoost with weighting parameters |
6. Anomaly detection
When the minority class is extremely rare (for example 0.01%) or has no clear pattern, treat it as an anomaly: model normal behaviour and flag deviations with Isolation Forest, One-Class SVM or autoencoders.
7. Cost-sensitive decisions
Assign real costs to errors and choose the model or threshold that minimizes total cost.
cost_fn, cost_fp = 100, 5 # a missed fraud costs far more than a false alarm
def total_cost(y_true, y_pred):
fn = ((y_true == 1) & (y_pred == 0)).sum()
fp = ((y_true == 0) & (y_pred == 1)).sum()
return fn * cost_fn + fp * cost_fp
for t in [0.1, 0.2, 0.3, 0.5]:
print(t, total_cost(y_test, (y_prob >= t).astype(int)))