Filter vs. wrapper vs. embedded
| Aspect | Filter | Wrapper | Embedded |
|---|---|---|---|
| Uses a model? | No | Yes (repeatedly) | Yes (built in) |
| Computation | Low | High | Medium |
| Captures interactions | No | Yes | Partly |
| Overfitting risk | Low | Higher | Medium |
| Best for | Huge feature sets, first pass | Small or medium feature sets | General purpose |
A practical recipe
- Clean and split the data first.
- Remove constant and duplicated features.
- Drop one of each highly correlated pair.
- Use a filter method to cut the feature set to a manageable size.
- Use an embedded method (Lasso or tree importances) to rank features.
- Optionally use a wrapper (RFE) for the final subset.
- Validate with cross-validation on the training set.
- Confirm the final choice on the untouched test set.
Put selection inside a pipeline
Doing selection inside the training folds avoids leakage.
from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.feature_selection import SelectKBest, f_classif
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score
pipe = Pipeline([
("impute", SimpleImputer(strategy="median")),
("scale", StandardScaler()),
("select", SelectKBest(f_classif, k=3)),
("model", LogisticRegression(max_iter=1000)),
])
print(cross_val_score(pipe, X_train, y_train, cv=5).mean())
pipe.fit(X_train, y_train)
print("Test accuracy:", pipe.score(X_test, y_test))
Best practices
- Do selection on the training data only.
- Start with cheap filters, then refine with embedded or wrapper methods.
- Combine statistical scores with domain knowledge.
- Compare model performance before and after selection.
- Remember that correlation does not imply causation.
- Prefer stable selections: check that the same features are chosen across folds.
- Record which features were removed and why.
When not to select features
- The dataset is small and already has few features.
- The model already regularizes strongly (for example, gradient boosting or ridge) and performs well.
- Interpretability requires keeping all domain-relevant features.
Key takeaways
- Fewer, better features generally give simpler, faster and more robust models.
- Filter = fast screening, wrapper = accurate but costly, embedded = built into training.
- Always fit selectors on the training data only.