Correlation Analysis in Python

The five-step procedure

1. Identify the variables. Choose the variables to correlate. For Pearson they should be quantitative (numeric).
2. Collect the data. From surveys, experiments, sensors, databases or existing records.
3. Choose the coefficient. Pearson for linear numeric data; Spearman / Kendall for ranks or curves; Phi / Cramer's V for categories.
4. Calculate it. Use a formula or, better, a library.
5. Interpret it. Look at sign, size, significance, the scatter plot, and think about confounders.

Using NumPy

import numpy as np

x = np.array([1, 2, 3, 4, 5])
y = np.array([5, 7, 3, 9, 1])

print(np.corrcoef(x, y))
# [[ 1.  -0.3]
#  [-0.3  1. ]]

np.corrcoef returns a correlation matrix. The diagonal is always 1 (a variable is perfectly correlated with itself). The off-diagonal value -0.3 is the correlation between x and y - a weak negative link.

Using pandas

import pandas as pd

data = pd.DataFrame({"X": [1, 2, 3, 4, 5], "Y": [5, 7, 3, 9, 1]})

print(data["X"].corr(data["Y"]))                        # -0.3  (Pearson)
print(data["X"].corr(data["Y"], method="spearman"))     # rank based
print(data["X"].corr(data["Y"], method="kendall"))      # Kendall tau

Series has .corr() for one pair; DataFrame has .corr() for all numeric pairs, returning a matrix.

Correlation matrix and heatmap for the whole dataset

plt.figure(figsize=(14, 9))
corr = df.corr(numeric_only=True)
sns.heatmap(corr, annot=True, fmt=".2f", cmap="coolwarm", center=0, linewidths=1)
plt.title("Correlation Heatmap")
plt.show()

# Which features relate most to the target?
print(corr["quality"].sort_values(ascending=False))

# Show only the upper-triangle so each pair appears once
mask = np.triu(np.ones_like(corr, dtype=bool))
sns.heatmap(corr, mask=mask, annot=True, fmt=".2f", cmap="coolwarm", center=0)
plt.show()

Reading the heatmap

• Values close to +1 = strong positive correlation; -1 = strong negative; 0 = no linear correlation.
• With cmap="coolwarm", center=0, red = positive, blue = negative, pale = near zero.
• The matrix is symmetric, so the mask (upper triangle) removes repeated information.
• Look at the target row (quality) first: which features correlate most with it? In the wine data, alcohol usually shows the strongest positive link and volatile acidity a negative one.
• Then look for feature-feature pairs with |r| above about 0.8 (for example, fixed acidity and density, or free and total sulfur dioxide). These are redundant and cause multicollinearity in linear models.

Finding the strongest pairs programmatically

corr = df.corr(numeric_only=True)

pairs = (
    corr.where(np.triu(np.ones(corr.shape), k=1).astype(bool))   # upper triangle only
        .stack()
        .rename("r")
        .reset_index()
)
pairs["abs_r"] = pairs["r"].abs()
print(pairs.sort_values("abs_r", ascending=False).head(10))

Statistical significance

from scipy import stats
r, p = stats.pearsonr(df["alcohol"], df["quality"])
print(f"r = {r:.2f}, p = {p:.4g}")

A small p-value (for example, below 0.05) says the correlation is unlikely to be pure chance. It does not say the relationship is important - with a large dataset even a tiny r can be "significant".

Pitfalls to remember

• Drop identifier columns such as Id before computing correlations (an ID has no real meaning).
• Handle missing values first (corr skips pairs with NaN, which can change sample sizes silently).
• Encode categorical columns properly before correlating.
• Always confirm surprising numbers with a scatter plot.