Features in a dataset often live on very different scales — for example Age (0-80) versus Income (0-1,000,000). Feature scaling transforms numeric features to a comparable range so that no feature dominates the learning process just because its numbers are larger.
A simple example
Two people are compared using age and income:
import numpy as np
person_a = np.array([25, 50_000]) # age, income
person_b = np.array([60, 52_000])
print(np.linalg.norm(person_a - person_b)) # ~2000.3 -> dominated by income
Without scaling, the distance between them is almost entirely decided by income, and age is practically ignored.
Why scaling matters
- Distance-based algorithms (k-NN, k-Means, SVM, DBSCAN) compute distances, so large-scale features dominate.
- Gradient-based algorithms (linear/logistic regression, neural networks) converge faster and more stably with scaled inputs.
- Regularized models (Ridge, Lasso) penalize coefficients, which depend on the feature scale.
- PCA finds directions of maximum variance, so unscaled features distort the components.
Which models need scaling?
| Need scaling | Usually do NOT need scaling |
|---|---|
| k-NN, k-Means, SVM | Decision Trees |
| Linear / Logistic Regression (esp. regularized) | Random Forest |
| Neural Networks | Gradient Boosting (XGBoost, LightGBM) |
| PCA, LDA | Naive Bayes (mostly) |
Tree-based models split on thresholds one feature at a time, so the scale of a feature does not change the splits.
Scaling techniques covered next
| Technique | Idea |
|---|---|
| Min-Max scaling (Normalization) | Rescale to a fixed range such as [0, 1] |
| Standardization (Z-score) | Mean 0, standard deviation 1 |
| Robust scaling | Use median and IQR to resist outliers |
| Max-Abs scaling | Divide by the maximum absolute value |
| Unit-vector normalization | Scale each row to length 1 |
| Power / quantile transforms | Make skewed features more Gaussian |