Embedded methods perform feature selection as part of model training. They balance the speed of filters and the accuracy of wrappers.
1. L1 regularization (Lasso)
L1 regularization adds a penalty on the absolute size of the coefficients and shrinks the coefficients of useless features to exactly zero.
from sklearn.feature_selection import SelectFromModel
from sklearn.linear_model import LogisticRegression
from sklearn.preprocessing import StandardScaler
Xs = StandardScaler().fit_transform(X_train.fillna(0))
l1 = LogisticRegression(penalty="l1", solver="liblinear", C=0.5)
l1.fit(Xs, y_train)
print(dict(zip(X_train.columns, l1.coef_[0].round(3)))) # zeros = dropped features
For regression use Lasso; ElasticNet mixes L1 and L2 penalties.
from sklearn.linear_model import LassoCV
lasso = LassoCV(cv=5, random_state=42).fit(Xs, y_train)
selected = X_train.columns[lasso.coef_ != 0]
print("Selected:", selected.tolist())
2. Tree-based feature importance
Decision trees, Random Forests and gradient boosting compute how much each feature reduces impurity (or error).
from sklearn.ensemble import RandomForestClassifier
rf = RandomForestClassifier(n_estimators=300, random_state=42)
rf.fit(X_train.fillna(0), y_train)
imp = pd.Series(rf.feature_importances_, index=X_train.columns).sort_values(ascending=False)
print(imp)
imp.plot(kind="barh")
3. Permutation importance
Shuffle one feature at a time and measure how much the score drops. It is less biased toward high-cardinality features than built-in importances.
from sklearn.inspection import permutation_importance
result = permutation_importance(rf, X_test.fillna(0), y_test, n_repeats=10, random_state=42)
perm = pd.Series(result.importances_mean, index=X_test.columns).sort_values(ascending=False)
print(perm)
4. SelectFromModel
Wraps any model that exposes coef_ or feature_importances_ and keeps only features above a threshold.
sfm = SelectFromModel(rf, threshold="median") # keep features above the median importance
sfm.fit(X_train.fillna(0), y_train)
print("Selected:", X_train.columns[sfm.get_support()].tolist())
Pros and cons
| Pros | Cons |
|---|---|
| Good balance of speed and accuracy | Tied to the specific model type |
| Captures interactions (trees) | Tree importances can be biased toward high-cardinality features |
| Selection happens during training | L1 needs scaled features and a tuned penalty |
Notes
- Scale features before using L1 or L2 regularization.
- Tune the regularization strength (
Coralpha) with cross-validation. - For correlated features, Lasso tends to keep one and drop the others.