Having too many irrelevant or redundant features can hurt a model as much as having too few useful ones — feature selection identifies which inputs actually earn their place.
from sklearn.feature_selection import SelectKBest, f_classif
from sklearn.ensemble import RandomForestClassifier
import pandas as pd
# Filter method: score each feature independently, keep the top k
selector = SelectKBest(score_func=f_classif, k=5)
X_selected = selector.fit_transform(X_train, y_train)
print(X_train.columns[selector.get_support()]) # which columns were kept
# Model-based: use a trained model's own feature importances
model = RandomForestClassifier(random_state=42)
model.fit(X_train, y_train)
importances = pd.Series(model.feature_importances_, index=X_train.columns)
print(importances.sort_values(ascending=False))
correlation = X_train.corr() # check for highly correlated (redundant) feature pairs
Filter methods (SelectKBest and similar) score each feature's individual statistical relationship with the target, independent of any model, and keep only the top-scoring ones — fast, but they miss interactions between features (two individually weak features that are jointly very predictive together).
Model-based selection trains a model capable of reporting feature importance directly — tree-based models like RandomForestClassifier naturally expose .feature_importances_, ranking how much each feature actually contributed to the model's decisions — often more reliable than filter methods since it accounts for how the features are actually used together, not just individually.
Checking a correlation matrix among features (not to be confused with correlation to the target) helps spot redundant, highly-correlated pairs — if two features are 95% correlated with each other, they're carrying almost the same information, and keeping both usually adds noise and complexity without meaningfully improving predictions. Fewer, well-chosen features generally produce a model that's faster to train, easier to interpret, and less prone to overfitting on noise — "more features" is not automatically "better.