Feature selection is the process of choosing the most relevant input variables for a model and discarding the irrelevant or redundant ones. Instead of feeding every column to the algorithm, you keep only those that genuinely help predict the target.
Why select features?
- Reduces overfitting — fewer noisy features means less chance to memorize noise.
- Improves accuracy — removes misleading and redundant inputs.
- Speeds up training and prediction.
- Simplifies models and makes them easier to interpret.
- Fights the curse of dimensionality — in high dimensions data becomes sparse and distances lose meaning.
- Lowers cost of data collection and storage.
Types of irrelevant features
| Type | Example |
|---|
| Constant / near-constant | A column that is 1 for 99.9% of rows |
| Irrelevant | A random ID number |
| Redundant | height_cm and height_inch |
| Highly correlated | total_rooms and total_bedrooms |
| Leaky | A column computed from the target |
Three families of methods
| Category | How it works | Speed | Examples |
|---|
| Filter | Scores features with statistics, independent of the model | Fast | Variance threshold, correlation, chi-square, ANOVA, mutual information |
| Wrapper | Trains a model repeatedly on different feature subsets | Slow | Forward selection, backward elimination, RFE |
| Embedded | Selection happens during model training | Medium | Lasso (L1), tree-based importance, Elastic Net |
Feature selection vs. feature extraction
| Selection | Extraction |
|---|
| Idea | Keep a subset of the original features | Create new features from the originals |
| Original meaning kept? | Yes | No |
| Examples | RFE, Lasso, chi-square | PCA, LDA, autoencoders |
Feature extraction is covered in its own chapter.