Feature extraction transforms raw or high-dimensional data into a smaller set of new, more informative features while preserving the essential information. Instead of picking existing columns, it builds new ones — often as combinations of the originals — or converts unstructured data (text, images, audio) into numbers.
Why use feature extraction?
- Dimensionality reduction — fewer features, faster training and prediction.
- Removes noise and redundancy — correlated features are merged into one.
- Fights the curse of dimensionality — models generalize better with fewer, richer features.
- Enables visualization — high-dimensional data projected to 2D or 3D.
- Handles unstructured data — turns text, images and audio into numeric vectors.
Feature selection vs. feature extraction
| Feature selection | Feature extraction | |
|---|---|---|
| Idea | Choose a subset of existing features | Create new features from existing ones |
| Original features kept? | Yes | No, replaced by new components |
| Interpretability | High | Often lower |
| Examples | Chi-square, RFE, Lasso | PCA, LDA, autoencoders, TF-IDF |
| Typical goal | Remove irrelevant features | Reduce dimensions, capture structure |
A simple picture
Suppose two features, height_cm and height_inch, are almost perfectly correlated. Extraction can merge them into a single feature that keeps nearly all the information.
import numpy as np
from sklearn.decomposition import PCA
rng = np.random.default_rng(0)
cm = rng.normal(170, 10, 200)
inch = cm / 2.54 + rng.normal(0, 0.1, 200) # almost the same information
X = np.column_stack([cm, inch])
pca = PCA(n_components=1).fit(X)
print("Variance kept by 1 component:", pca.explained_variance_ratio_.round(4)) # ~1.0
Families of feature extraction techniques
| Data type | Techniques |
|---|---|
| Tabular (numeric) | PCA, LDA, Kernel PCA, ICA, Factor Analysis, autoencoders |
| Visualization | t-SNE, UMAP |
| Text | Bag of Words, TF-IDF, n-grams, word and sentence embeddings |
| Images | Pixels, color histograms, HOG, SIFT, CNN features |
| Audio / signals | MFCC, spectrograms, FFT features |
The next pages cover these techniques.