Evaluation Metrics for Imbalanced Data

With imbalanced classes you need metrics that focus on the minority (positive) class.

The confusion matrix

	Predicted Positive	Predicted Negative
Actual Positive	True Positive (TP)	False Negative (FN)
Actual Negative	False Positive (FP)	True Negative (TN)

Key metrics

Metric	Formula	Meaning	Use when
Precision	TP / (TP + FP)	Of predicted positives, how many are correct	False alarms are costly
Recall (Sensitivity)	TP / (TP + FN)	Of actual positives, how many were found	Missing a positive is costly (disease, fraud)
F1-score	2PR / (P + R)	Harmonic mean of precision and recall	You need a balance
F-beta	Weighted F-score	Weights recall (beta > 1) or precision (beta < 1)	Custom trade-off
Specificity	TN / (TN + FP)	Of actual negatives, how many are correct	Negatives matter too
ROC-AUC	Area under ROC curve	Ranking ability across thresholds	Mild imbalance
PR-AUC	Area under precision-recall curve	Precision/recall trade-off	Severe imbalance
MCC	Correlation of prediction and truth	A single balanced score	Robust summary
Balanced accuracy	Mean of recall per class	Accuracy that treats classes equally	Quick fix for accuracy

Computing the metrics

from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import (classification_report, confusion_matrix, average_precision_score,
                             roc_auc_score, f1_score, matthews_corrcoef, balanced_accuracy_score)

model = RandomForestClassifier(random_state=42).fit(X_train, y_train)
y_pred = model.predict(X_test)
y_prob = model.predict_proba(X_test)[:, 1]

print(confusion_matrix(y_test, y_pred))
print(classification_report(y_test, y_pred, digits=3))

print("F1      :", round(f1_score(y_test, y_pred), 3))
print("PR-AUC  :", round(average_precision_score(y_test, y_prob), 3))
print("ROC-AUC :", round(roc_auc_score(y_test, y_prob), 3))
print("MCC     :", round(matthews_corrcoef(y_test, y_pred), 3))
print("Bal. acc:", round(balanced_accuracy_score(y_test, y_pred), 3))

Precision-recall curve

from sklearn.metrics import precision_recall_curve
import matplotlib.pyplot as plt

prec, rec, thr = precision_recall_curve(y_test, y_prob)
plt.plot(rec, prec)
plt.xlabel("Recall")
plt.ylabel("Precision")
plt.title("Precision-Recall curve")
plt.show()

Why PR-AUC over ROC-AUC for severe imbalance

ROC-AUC includes the huge number of true negatives, so it can look excellent even when the model finds few positives. PR-AUC ignores true negatives and reveals the real performance on the minority class.

Averaging in multi-class problems

Averaging	Behaviour
macro	Average of per-class scores; every class counts equally
weighted	Average weighted by class size
micro	Global counts; dominated by large classes

Practical advice

• Decide the metric before modeling, based on business cost.
• Always look at the confusion matrix, not only a single number.
• Use stratified cross-validation and report the mean and std of the chosen metric.
• Compare against a naive baseline (DummyClassifier).