Dimensionality Reduction: PCA, t-SNE and UMAP

Datasets often have dozens or thousands of features. Dimensionality reduction compresses them into fewer dimensions while keeping the most important structure. Benefits:

• Visualisation - we can only draw 2D or 3D.
• Noise removal - small, noisy directions are dropped.
• Speed - fewer features means faster models.
• Fighting the curse of dimensionality - in high dimensions distances become less meaningful and data becomes sparse.

from sklearn.preprocessing import StandardScaler
from sklearn.decomposition import PCA
from sklearn.manifold import TSNE
import umap   # pip install umap-learn

X = df.select_dtypes("number").drop(columns=["quality"])
X_scaled = StandardScaler().fit_transform(X)     # ALWAYS scale first

# PCA
pca = PCA(n_components=2)
p = pca.fit_transform(X_scaled)
print("Explained variance:", pca.explained_variance_ratio_)

# t-SNE
t = TSNE(n_components=2, perplexity=30, random_state=42).fit_transform(X_scaled)

# UMAP
u = umap.UMAP(n_neighbors=15, min_dist=0.1, random_state=42).fit_transform(X_scaled)

fig, ax = plt.subplots(1, 3, figsize=(18, 5))
for a, emb, name in zip(ax, [p, t, u], ["PCA", "t-SNE", "UMAP"]):
    a.scatter(emb[:, 0], emb[:, 1], c=df["quality"], cmap="viridis", s=12)
    a.set_title(name)
plt.show()

Always scale first. PCA, t-SNE and UMAP are distance / variance based, so a column measured in thousands would dominate one measured in fractions.

1. PCA - Principal Component Analysis

A linear technique that rotates the data into new, uncorrelated axes called principal components. PC1 captures the most variance, PC2 the next most, and so on.

• Best for: reducing correlated numeric features while retaining most variability, e.g. summarising many correlated stock returns into a few components; also for noise reduction and as a preprocessing step.
• Not suitable: strongly non-linear structure (it only captures linear relationships) or categorical variables that cannot be represented in continuous space.
• Example: an e-commerce platform with dozens of customer-behaviour features uses PCA to derive a few components capturing the main behaviour patterns.

How many components to keep?

pca_full = PCA().fit(X_scaled)
cum = np.cumsum(pca_full.explained_variance_ratio_)
print(np.argmax(cum >= 0.95) + 1, "components explain 95% of variance")

plt.plot(range(1, len(cum) + 1), cum, marker="o")
plt.axhline(0.95, color="red", linestyle="--")
plt.xlabel("Number of components"); plt.ylabel("Cumulative explained variance")
plt.show()

Look at loadings to interpret components:

loadings = pd.DataFrame(pca.components_.T, index=X.columns, columns=["PC1", "PC2"])
print(loadings.round(2))

2. t-SNE

t-Distributed Stochastic Neighbour Embedding is a non-linear technique that places points in 2D/3D so that neighbours in the original space stay neighbours.

• Best for: visualising clusters in complex high-dimensional data, revealing patterns hidden in high dimensions.
• Not suitable: very large datasets (slow); it preserves local but not global structure, so distances between clusters and cluster sizes are not meaningful; results vary with perplexity and random seed; do not use the coordinates as model features.
• Example: customers described by age, income and purchase history are shown as 2D clusters, hinting at customer segments.

Tips: try perplexity 5-50; run with more than one seed; for large data, first reduce to ~50 dimensions with PCA.

3. UMAP

Uniform Manifold Approximation and Projection is a non-linear technique similar to t-SNE but faster and usually better at keeping global structure. It builds a neighbourhood graph and finds a low-dimensional layout that preserves it.

• Best for: large datasets; visualisation, clustering, anomaly detection, genomics, image features. It can also transform new data, so it can be used inside a pipeline.
• Not suitable: tasks that need exact distances; results depend on hyper-parameters (n_neighbors, min_dist), which need tuning.
• Example: a data scientist maps high-dimensional customer-interaction data into 2D to find trends and marketing segments.

UMAP parameter	Effect
n_neighbors (small)	focuses on local detail
n_neighbors (large)	captures broader structure
min_dist (small)	tighter clumps
min_dist (large)	more spread-out points

Comparison

Feature	PCA	t-SNE	UMAP
Type	Linear	Non-linear	Non-linear
Preserves	Global variance	Local neighbourhoods	Local and much of global structure
Speed	Very fast	Slow	Fast
Deterministic	Yes	No	Mostly no (set seed)
Can transform new data	Yes	No (standard)	Yes
Axes interpretable	Yes (loadings)	No	No
Typical use	Preprocessing, noise reduction, first look	Cluster visualisation	Visualisation of large data, features for clustering

Recommended workflow

1. Scale the data.
2. Run PCA first: check explained variance and loadings.
3. If structure looks non-linear, try UMAP (and t-SNE for comparison).
4. Colour the embedding by the target or by known groups to see whether classes separate.
5. Treat embeddings as exploration; confirm findings with real statistics or models.