Beyond PCA's practical mechanics, it's worth understanding why reducing dimensionality often helps in the first place — a phenomenon known as the curse of dimensionality.
import numpy as np
# As dimensions increase, data becomes sparse and distances become less meaningful
for dims in [2, 10, 100, 1000]:
points = np.random.rand(100, dims) # 100 random points in 'dims' dimensions
distances = []
for i in range(10):
d = np.linalg.norm(points[i] - points[i+1])
distances.append(d)
print(f"dims={dims}: distance range {min(distances):.2f} to {max(distances):.2f}")
# in high dimensions, all pairwise distances become surprisingly similar
As the number of features (dimensions) grows, the volume of the space those features occupy grows exponentially — meaning that a fixed number of data points becomes exponentially sparser, with points spreading further and further apart relative to the space around them. This has a counterintuitive but well-documented consequence: in very high dimensions, the distance between the nearest and farthest points from any given point tends to become nearly equal, which quietly breaks any algorithm (k-means, k-nearest neighbors, and others) that fundamentally relies on distance being a meaningful measure of similarity.
Practically, this means that simply adding more features to a model isn't automatically better, and can actively hurt performance once the number of features grows large relative to the number of training examples available — more dimensions require exponentially more data to maintain the same data density, and without it, models become more prone to overfitting on spurious patterns that appear meaningful only because the data is so sparse. This is the deeper reason feature selection (covered earlier) and dimensionality reduction techniques like PCA matter beyond mere convenience — they're often a necessary countermeasure against a real, well-understood statistical problem, not just an optional nicety.