Filter methods score each feature with a statistical measure, without training a model. They are fast and a good first pass, especially when you have hundreds or thousands of features.
1. Remove constant and near-constant features
A feature with almost no variance carries almost no information.
from sklearn.feature_selection import VarianceThreshold
selector = VarianceThreshold(threshold=0.01)
X_high_var = selector.fit_transform(X_train)
print("Kept:", X_train.columns[selector.get_support()].tolist())
2. Remove highly correlated (redundant) features
If two features are almost the same, keep one.
import numpy as np
corr = X_train.corr().abs()
upper = corr.where(np.triu(np.ones(corr.shape), k=1).astype(bool))
to_drop = [col for col in upper.columns if any(upper[col] > 0.9)]
X_train_reduced = X_train.drop(columns=to_drop)
print("Dropped:", to_drop)
3. Statistical tests with SelectKBest
Choose the scoring function based on the feature type and target type.
| Feature type | Target type | Score function |
|---|---|---|
| Numeric | Categorical (classification) | f_classif (ANOVA F-test) |
| Numeric | Categorical | mutual_info_classif |
| Non-negative numeric or counts | Categorical | chi2 |
| Numeric | Numeric (regression) | f_regression, mutual_info_regression |
import pandas as pd
from sklearn.feature_selection import SelectKBest, f_classif
skb = SelectKBest(score_func=f_classif, k=3)
X_new = skb.fit_transform(X_train.fillna(0), y_train)
scores = pd.Series(skb.scores_, index=X_train.columns).sort_values(ascending=False)
print(scores)
print("Selected:", X_train.columns[skb.get_support()].tolist())
4. Mutual information
Captures non-linear relationships as well as linear ones.
from sklearn.feature_selection import mutual_info_classif
mi = mutual_info_classif(X_train.fillna(0), y_train, random_state=42)
print(pd.Series(mi, index=X_train.columns).sort_values(ascending=False))
5. Chi-square test for categorical features
from sklearn.feature_selection import chi2
from sklearn.preprocessing import MinMaxScaler
X_pos = MinMaxScaler().fit_transform(X_train.fillna(0)) # chi2 needs non-negative values
chi_scores, p_values = chi2(X_pos, y_train)
print(pd.DataFrame({"feature": X_train.columns, "chi2": chi_scores, "p_value": p_values}))
Pros and cons
| Pros | Cons |
|---|---|
| Very fast and scalable | Ignores feature interactions |
| Model-agnostic | Ignores the specific model you will use |
| Low risk of overfitting | Univariate scores can miss features that are useful only in combination |