Nearly every modeling decision in machine learning comes back to managing the tension between a model being too simple and too complex — understanding this tradeoff is essential for diagnosing why a model performs poorly.
from sklearn.tree import DecisionTreeClassifier
from sklearn.model_selection import cross_val_score
import numpy as np
for depth in [1, 3, 5, 10, None]: # None = unlimited depth
model = DecisionTreeClassifier(max_depth=depth, random_state=42)
train_score = model.fit(X_train, y_train).score(X_train, y_train)
cv_scores = cross_val_score(model, X_train, y_train, cv=5)
print(f"depth={depth}: train={train_score:.3f}, cv_mean={cv_scores.mean():.3f}")
# a large gap between train and cv scores signals overfitting
Underfitting happens when a model is too simple to capture the real pattern in the data — both training and test/validation performance are poor, because the model genuinely lacks the capacity to represent the relationship (a linear model trying to fit a strongly curved relationship, for instance). The fix is usually a more expressive model, better features, or fewer overly-strong regularization constraints.
Overfitting happens when a model is complex enough to memorize noise and idiosyncrasies specific to the training data rather than the genuine underlying pattern — training performance looks excellent, but test/validation performance is meaningfully worse, since the memorized noise doesn't generalize to new data. This is exactly the gap the loop above is designed to reveal: as max_depth increases, training accuracy climbs toward 100%, while cross-validation accuracy eventually plateaus or even declines once the tree starts fitting noise instead of signal.
This tension is formalized as the bias-variance tradeoff: a high-bias (too-simple) model makes overly strong assumptions and underfits; a high-variance (too-complex) model is overly sensitive to the specific training data and overfits. The goal is the sweet spot in between — found through techniques like cross-validation (previous lesson), regularization, and hyperparameter tuning (covered later in this notebook) — rather than either extreme.