Wrapper methods evaluate subsets of features by actually training a model and measuring its performance. They find feature combinations tailored to the chosen model, at a higher computational cost.
1. Forward selection
Start with no features. Add the feature that improves the model most, one at a time, until no significant improvement remains.
2. Backward elimination
Start with all features. Remove the least useful feature one at a time.
3. Sequential Feature Selector
from sklearn.feature_selection import SequentialFeatureSelector
from sklearn.linear_model import LogisticRegression
sfs = SequentialFeatureSelector(
LogisticRegression(max_iter=1000),
n_features_to_select=3,
direction="forward", # or "backward"
cv=5,
)
sfs.fit(X_train.fillna(0), y_train)
print("Selected:", X_train.columns[sfs.get_support()].tolist())
4. Recursive Feature Elimination (RFE)
- Train a model on all features.
- Rank features by importance (coefficients or feature importances).
- Remove the weakest feature(s).
- Repeat until the desired number of features remains.
from sklearn.feature_selection import RFE
rfe = RFE(estimator=LogisticRegression(max_iter=1000), n_features_to_select=3)
rfe.fit(X_train.fillna(0), y_train)
print(dict(zip(X_train.columns, rfe.ranking_))) # 1 = selected
print("Selected:", X_train.columns[rfe.support_].tolist())
5. RFECV — choose the number of features automatically
RFECV uses cross-validation to find the best number of features.
from sklearn.feature_selection import RFECV
rfecv = RFECV(estimator=LogisticRegression(max_iter=1000), step=1, cv=5, scoring="accuracy")
rfecv.fit(X_train.fillna(0), y_train)
print("Optimal number of features:", rfecv.n_features_)
print("Selected:", X_train.columns[rfecv.support_].tolist())
Exhaustive search
Trying all possible subsets guarantees the best subset but needs 2^n model fits, so it is only practical for a handful of features.
Pros and cons
| Pros | Cons |
|---|---|
| Considers feature interactions | Computationally expensive |
| Tuned to the chosen model | Risk of overfitting on small datasets |
| Often gives the best subset | Results depend on the estimator used |
Tips
- Reduce the feature set first with cheap filter methods, then apply a wrapper.
- Always run wrappers inside cross-validation and on the training data only.
- Use a simple, fast estimator (linear model, small tree) inside the wrapper.