PCA is not the only option. Different techniques suit different goals.
Linear Discriminant Analysis (LDA)
LDA is supervised. It finds axes that maximize the separation between classes while minimizing the spread within each class. It can produce at most n_classes - 1 components.
from sklearn.discriminant_analysis import LinearDiscriminantAnalysis
lda = LinearDiscriminantAnalysis(n_components=1) # 2 classes -> max 1 component
X_train_lda = lda.fit_transform(X_train_s, y_train) # needs labels
X_test_lda = lda.transform(X_test_s)
print(X_train_lda.shape)
| PCA | LDA | |
|---|---|---|
| Supervised? | No | Yes (uses labels) |
| Goal | Maximize variance | Maximize class separation |
| Max components | min(n_samples, n_features) | n_classes - 1 |
| Best for | General compression | Classification preprocessing |
Kernel PCA
A non-linear version of PCA that uses kernels (RBF, polynomial) to capture curved structure.
from sklearn.decomposition import KernelPCA
kpca = KernelPCA(n_components=2, kernel="rbf", gamma=0.02)
X_kpca = kpca.fit_transform(X_train_s)
t-SNE and UMAP (for visualization)
Non-linear methods that keep similar points close together in 2D or 3D. Use them to explore data, not as model inputs.
from sklearn.manifold import TSNE
X_tsne = TSNE(n_components=2, perplexity=30, random_state=42).fit_transform(X_train_s)
plt.scatter(X_tsne[:, 0], X_tsne[:, 1], c=y_train, cmap="coolwarm", alpha=0.7)
plt.show()
# import umap
# X_umap = umap.UMAP(n_components=2, random_state=42).fit_transform(X_train_s)
- t-SNE: excellent local structure, slower, distances between clusters are not meaningful.
- UMAP: faster, preserves more global structure (
pip install umap-learn).
Truncated SVD (LSA)
Works directly on sparse matrices such as TF-IDF, where PCA cannot be used because it needs centering.
from sklearn.decomposition import TruncatedSVD
from sklearn.feature_extraction.text import TfidfVectorizer
docs = ["machine learning is fun", "deep learning is a part of machine learning",
"cooking pasta is easy", "pasta and pizza are tasty"]
X_tfidf = TfidfVectorizer().fit_transform(docs) # sparse matrix
svd = TruncatedSVD(n_components=2, random_state=42)
X_lsa = svd.fit_transform(X_tfidf)
print(X_lsa.round(2))
Autoencoders
A neural network that compresses the input into a small bottleneck layer and then reconstructs it. The bottleneck activations are the extracted features. Useful for non-linear compression of large data, images and anomaly detection.
Other methods
| Method | Type | Notes |
|---|---|---|
| ICA | Linear | Separates independent signals, for example mixed audio sources |
| Factor Analysis | Linear | Models latent (hidden) factors |
| NMF | Linear, non-negative | Interpretable parts-based features (topics, images) |
| Random Projection | Linear | Very fast reduction for huge data |
Which technique to choose?
| Goal | Technique |
|---|---|
| General compression, linear data | PCA |
| Better class separation for classification | LDA |
| Curved / non-linear structure | Kernel PCA, autoencoders |
| 2D/3D visualization | t-SNE, UMAP |
| Sparse text matrices | Truncated SVD |
| Non-negative, interpretable parts | NMF |