Principal Component Analysis (PCA)

PCA is an unsupervised, linear technique. It finds new orthogonal axes called principal components that capture the maximum variance in the data. The first component captures the most variance, the second the next most (and is perpendicular to the first), and so on.

How PCA works

1. Standardize the data (mean 0, std 1).
2. Compute the covariance matrix.
3. Find its eigenvectors (directions) and eigenvalues (variance along each direction).
4. Sort components by eigenvalue and keep the top k.
5. Project the data onto those k components.

Key terms

Term	Meaning
Principal component	A new axis that is a linear combination of the original features
Explained variance ratio	Share of total variance captured by each component
Loadings	Weights of original features in a component
Scree plot	Plot of explained variance per component

PCA in scikit-learn

from sklearn.datasets import load_breast_cancer
from sklearn.decomposition import PCA
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
import numpy as np

data = load_breast_cancer()
X, y = data.data, data.target                      # 30 features
X_train, X_test, y_train, y_test = train_test_split(X, y, stratify=y, random_state=42)

scaler = StandardScaler().fit(X_train)             # PCA needs scaled data
X_train_s = scaler.transform(X_train)
X_test_s = scaler.transform(X_test)

pca = PCA(n_components=0.95)                       # keep 95% of the variance
X_train_p = pca.fit_transform(X_train_s)
X_test_p = pca.transform(X_test_s)

print("Original features:", X_train.shape[1])
print("Components kept  :", pca.n_components_)
print("Explained variance ratio:", pca.explained_variance_ratio_.round(3))

Choosing the number of components

Use the cumulative explained variance (commonly 90-99%) or a scree plot.

import matplotlib.pyplot as plt

pca_full = PCA().fit(X_train_s)
cum = np.cumsum(pca_full.explained_variance_ratio_)

plt.plot(range(1, len(cum) + 1), cum, marker="o")
plt.axhline(0.95, color="red", linestyle="--")
plt.xlabel("Number of components")
plt.ylabel("Cumulative explained variance")
plt.show()

print("Components for 95%:", np.argmax(cum >= 0.95) + 1)

Inspect what a component means

import pandas as pd

loadings = pd.DataFrame(
    pca.components_[:2].T,
    index=data.feature_names,
    columns=["PC1", "PC2"],
)
print(loadings.abs().sort_values("PC1", ascending=False).head(5))

Visualize in 2D

pca2 = PCA(n_components=2)
X_2d = pca2.fit_transform(X_train_s)

plt.scatter(X_2d[:, 0], X_2d[:, 1], c=y_train, cmap="coolwarm", alpha=0.7)
plt.xlabel("PC1")
plt.ylabel("PC2")
plt.show()

PCA in a pipeline

from sklearn.pipeline import Pipeline
from sklearn.linear_model import LogisticRegression

pipe = Pipeline([
    ("scale", StandardScaler()),
    ("pca", PCA(n_components=10)),
    ("clf", LogisticRegression(max_iter=1000)),
])
pipe.fit(X_train, y_train)
print("Test accuracy:", pipe.score(X_test, y_test))

Pros and cons

Pros	Cons
Reduces dimensions and noise	Components are hard to interpret
Removes correlation between features	Only captures linear structure
Speeds up training, aids visualization	Sensitive to scaling
Works without labels	Ignores the target, so components may not be the most predictive

Rules

• Always scale before PCA.
• Fit PCA on the training set only, then transform test data.
• PCA is not feature selection: every component mixes all original features.