Introduction
Do students who study more score higher? Do houses with more rooms cost more? Is a feature useful for predicting the target?
Correlation measures the strength and direction of the relationship between two variables. It is a single number between −1 and +1:
- +1: perfect positive relationship (when one goes up, the other goes up in exact proportion).
- 0: no relationship of the type being measured.
- −1: perfect negative relationship (when one goes up, the other goes down).
Two correlation coefficients are used most often: Pearson (linear relationships) and Spearman (monotonic, rank-based relationships).
Reading a scatter plot
Before calculating anything, plot the data.
| Pattern | Meaning |
|---|---|
| Points rising left to right | Positive correlation |
| Points falling left to right | Negative correlation |
| Shapeless cloud | No correlation |
| Tight around a line | Strong correlation |
| Wide scatter around a line | Weak correlation |
| A curve | Relationship exists but may not be linear |
Covariance
Covariance measures how two variables vary together:
cov(X, Y) = Σ (xᵢ − x̄)(yᵢ − ȳ) / (n − 1)
- Positive: they tend to move in the same direction.
- Negative: they tend to move in opposite directions.
- Problem: its size depends on the units (centimetres versus metres change it), so it is hard to compare. Correlation solves this by standardising.
Pearson correlation coefficient (r)
cov(X, Y) Σ (xᵢ − x̄)(yᵢ − ȳ)
r = ───────────────────────── = ───────────────────────────────
σₓ · σᵧ √[ Σ(xᵢ − x̄)² · Σ(yᵢ − ȳ)² ]
Properties
- Always between −1 and +1.
- Unit-free, and unchanged if you rescale or shift either variable.
- Symmetric: r(X, Y) = r(Y, X).
- Measures only linear relationships.
- Sensitive to outliers: one extreme point can change r a lot.
Interpreting the size (rule of thumb)
| Absolute value of r | Strength |
|---|---|
| 0.00 to 0.19 | Very weak |
| 0.20 to 0.39 | Weak |
| 0.40 to 0.59 | Moderate |
| 0.60 to 0.79 | Strong |
| 0.80 to 1.00 | Very strong |
Conventions differ between fields, so treat these as guidelines.
Coefficient of determination r²
Squaring r gives the proportion of the variance in Y that is explained by a linear relationship with X. If r = 0.8, then r² = 0.64, so 64% of the variation in Y is linearly associated with X.
Assumptions
- Both variables are numerical (interval or ratio).
- The relationship is approximately linear.
- No extreme outliers.
- For significance tests: approximately bivariate normal data.
Spearman rank correlation (ρ)
Spearman's ρ is Pearson's r calculated on the ranks of the data instead of the raw values.
Steps
- Replace each X value by its rank (1 = smallest) and do the same for Y.
- If values tie, give each the average of the ranks they occupy (for example values 10, 20, 20, 30 get ranks 1, 2.5, 2.5, 4).
- Compute Pearson's r on the ranks.
When there are no ties, there is a shortcut:
ρ = 1 − [ 6 Σ dᵢ² ] / [ n (n² − 1) ]
where dᵢ is the difference between the two ranks of observation i.
Properties
- Measures any monotonic relationship: consistently increasing or consistently decreasing, not necessarily along a straight line.
- Works with ordinal data (ratings, rankings).
- Robust to outliers and does not require normality.
A related coefficient, Kendall's τ, also measures rank agreement and is preferred for small samples with many ties.
Pearson versus Spearman
| Pearson | Spearman | |
|---|---|---|
| Relationship measured | Linear | Monotonic |
| Data used | Actual values | Ranks |
| Outliers | Sensitive | Robust |
| Data types | Numerical | Numerical or ordinal |
| Normality needed? | Ideally | No |
Correlation is not causation
A strong correlation does not prove that one variable causes the other.
- Confounding variable: ice-cream sales and drowning deaths both rise in summer because of hot weather.
- Reverse causation: does exercise improve mood, or do people in a good mood exercise more?
- Coincidence: with enough variables, some will correlate by pure chance.
- Simpson's paradox: a trend that appears in several groups can reverse when the groups are combined.
Other pitfalls
- Zero correlation does not mean independence. If y = x² and x is symmetric around 0, Pearson's r ≈ 0 even though y is completely determined by x.
- Restricted range reduces observed correlation.
- Anscombe's quartet: four very different datasets with almost identical r values, which is why you must always plot the data.
Correlation matrix and heatmap
For a dataset with many features, compute the correlation matrix: a square table where the entry (i, j) is the correlation between features i and j. Plotting it as a colour-coded heatmap quickly reveals strongly related features.
Real-life applications
- Economics: relationship between unemployment and inflation.
- Medicine: association between smoking and lung disease.
- Education: study hours versus exam scores.
- Marketing: advertising spend versus sales.
- Climate science: temperature versus CO₂ concentration.
Where it appears in machine learning
- Feature selection: keep features that correlate strongly with the target.
- Multicollinearity: when two features are highly correlated (for example |r| > 0.9), they carry redundant information and can make linear-model coefficients unstable. Drop one, combine them, use PCA, or apply regularisation. The variance inflation factor (VIF) quantifies this.
- Exploratory analysis with correlation heatmaps.
- Evaluation: comparing model predictions to true values with Pearson or Spearman correlation (for example in ranking or similarity tasks).
- Data leakage detection: a feature with an unbelievably high correlation to the target may be leaking the answer.
Python implementation
import numpy as np
import pandas as pd
from scipy import stats
x = np.array([1, 2, 3, 4, 5])
y = np.array([2, 4, 5, 4, 5])
r, p = stats.pearsonr(x, y)
rho, p2 = stats.spearmanr(x, y)
print("Pearson r =", round(r, 4)) # 0.7746
print("Spearman ρ =", round(rho, 4))
# Nonlinear but monotonic relationship
x2 = np.arange(1, 11)
y2 = np.exp(x2)
print(stats.pearsonr(x2, y2)[0]) # about 0.72
print(stats.spearmanr(x2, y2)[0]) # 1.0
# A perfect but non-monotonic relationship
x3 = np.linspace(-3, 3, 101)
print(stats.pearsonr(x3, x3 ** 2)[0]) # about 0
# Correlation matrix
df = pd.DataFrame({"study": [1, 2, 3, 4, 5, 6],
"sleep": [8, 7, 7, 6, 6, 5],
"score": [50, 55, 65, 70, 72, 80]})
print(df.corr(method="pearson"))
print(df.corr(method="spearman"))
Heatmap
import seaborn as sns
import matplotlib.pyplot as plt
sns.heatmap(df.corr(), annot=True, cmap="coolwarm", vmin=-1, vmax=1)
plt.show()
Solved examples
Example 1: Covariance and Pearson r
Compute the covariance and Pearson correlation for X = 1, 2, 3, 4, 5 and Y = 2, 4, 5, 4, 5.
Solution.
- x̄ = 3, ȳ = 4.
- Deviations of X: −2, −1, 0, 1, 2. Deviations of Y: −2, 0, 1, 0, 1.
- Products: 4, 0, 0, 0, 2. Sum (Sxy) = 6.
- Sxx = 4 + 1 + 0 + 1 + 4 = 10. Syy = 4 + 0 + 1 + 0 + 1 = 6.
Covariance = 6 / 4 = 1.5.
r = 6 / √(10 × 6) = 6 / 7.746 ≈ 0.775. So r² = 0.60, meaning about 60% of the variation in Y is linearly associated with X.
Example 2: Spearman with the shortcut formula
X = 10, 20, 30, 40, 50 and Y = 3, 1, 4, 2, 5. Find Spearman's ρ.
Solution. Ranks of X: 1, 2, 3, 4, 5. Ranks of Y: 3, 1, 4, 2, 5.
Differences d: −2, 1, −1, 2, 0. Squares: 4, 1, 1, 4, 0. Σd² = 10.
ρ = 1 − (6 × 10) / (5 × 24) = 1 − 60/120 = 0.5.
Example 3: Handling ties
Assign ranks to 10, 20, 20, 30.
Solution. The two 20s occupy positions 2 and 3, so each receives (2 + 3)/2 = 2.5. Ranks: 1, 2.5, 2.5, 4.
Example 4: Nonlinear but monotonic
For X = 1, 2, ..., 10 and Y = eˣ, why is Spearman ρ = 1 but Pearson r ≈ 0.72?
Solution. Y always increases as X increases, so the ranks match perfectly and ρ = 1. But the relationship is strongly curved, so a straight line fits poorly and Pearson r is lower.
Example 5: Interpretation
A study finds r = −0.85 between weekly hours of video gaming and exam scores in a sample of students. Interpret this.
Solution. The relationship is strong and negative: more gaming hours go with lower scores. r² = 0.7225, so about 72% of the variation in scores is linearly associated with gaming time. But this does not prove gaming causes lower scores; sleep, motivation or study time could be confounders.
Example 6: Multicollinearity
Two features, "height in cm" and "height in inches", have correlation 1.0. What should you do?
Solution. They carry identical information. Keep one and drop the other; otherwise a linear model will have unstable, hard-to-interpret coefficients.
Common mistakes
- Concluding causation from correlation.
- Using Pearson on clearly nonlinear data, or on data with strong outliers.
- Assuming r = 0 means "no relationship".
- Computing correlation on categorical (nominal) variables.
- Forgetting to plot the data first.
Practice problems
- Find Pearson's r for X = 1, 2, 3 and Y = 2, 4, 6.
- Find Pearson's r for X = 1, 2, 3, 4 and Y = 8, 6, 4, 2.
- Ranks of X are 1, 2, 3, 4 and ranks of Y are 2, 1, 4, 3. Find Spearman's ρ.
- If r = 0.6, what is r²? What does it mean?
- Compute r for X = 2, 4, 6, 8 and Y = 1, 3, 2, 4.
- Which coefficient would you use for customer satisfaction ratings on a 1-to-5 scale versus the number of support tickets?
Answers
- +1 (a perfect positive linear relationship)
- −1 (a perfect negative linear relationship)
- d = −1, 1, −1, 1 so Σd² = 4; ρ = 1 − 24/(4 × 15) = 0.6
- r² = 0.36: 36% of the variance in Y is linearly explained by X
- Means 5 and 2.5; Sxy = 8, Sxx = 20, Syy = 5; r = 8/√100 = 0.8
- Spearman (ordinal ratings and a possibly nonlinear relationship)
Key takeaways
- Correlation ranges from −1 to +1 and measures the strength and direction of association.
- Pearson captures linear relationships; Spearman captures monotonic relationships using ranks and is robust to outliers.
- r² tells you how much variance is explained.
- Correlation is not causation, and r = 0 does not rule out a nonlinear relationship.
- In ML, correlation guides feature selection and warns about multicollinearity and leakage.