Correlation, causation & regression basics

Analysts are constantly asked "does X drive Y?" - does ad spend drive sales, does discount depth drive repeat visits? Correlation and regression are the first tools for answering, and also the easiest to misuse.

Correlation coefficients

Both range from -1 to +1. Values near 0 mean no linear (or monotonic) pattern - but there could still be a curved one, so always plot first.

import numpy as np
import pandas as pd
from scipy import stats

rng = np.random.default_rng(5)
n = 120
store_size = rng.uniform(800, 4000, n)                     # sq ft
ad_spend = 5 + 0.004 * store_size + rng.normal(0, 3, n)    # thousand INR / month
staff = np.round(store_size / 400 + rng.normal(0, 1, n))
sales = 40 + 0.05 * store_size + 2.0 * ad_spend + rng.normal(0, 25, n)   # thousand INR
df = pd.DataFrame({"store_size": store_size, "ad_spend": ad_spend, "staff": staff, "sales": sales})

r, p = stats.pearsonr(df["staff"], df["sales"])
rho, p_s = stats.spearmanr(df["staff"], df["sales"])
print(f"Pearson r={r:.2f} (p={p:.3g}), Spearman rho={rho:.2f}")

Staff count correlates strongly with sales. Should the chain hire more staff everywhere?

Why correlation is not causation

• Confounding: a third variable drives both. Here, bigger stores have more staff and more sales.
• Reverse causation: busy stores get more staff because they are busy.
• Coincidence: with enough variables, some correlate by chance.
• Selection effects: only surviving stores are in the data.

Establishing causation needs a randomised experiment (A/B test) or careful designs such as difference-in-differences - not just a strong r.

Simple and multiple regression

Regression estimates how Y changes with X, and lets you control for other variables.

import statsmodels.formula.api as smf

simple = smf.ols("sales ~ staff", data=df).fit()
multi = smf.ols("sales ~ staff + store_size + ad_spend", data=df).fit()
print(simple.params.round(2))
print(multi.params.round(2))
print(multi.conf_int().round(2))
print(f"R-squared: {multi.rsquared:.2f}")

In the simple model the staff coefficient is large. Once store size is included it shrinks towards zero, while the ad-spend coefficient lands near its true value of 2 - "each extra ₹1,000 of ads is associated with about ₹2,000 more sales, holding size and staff constant".

Reading the output

• Coefficient: expected change in Y for a one-unit increase in X, other variables fixed.
• Confidence interval / p-value: how precisely it is estimated.
• R-squared: share of variance in Y explained - high R-squared does not prove causation either.
• Residuals: plot them against fitted values; patterns suggest a missing curve or variable.

Common mistakes

• Reporting r without a scatter plot (outliers can create or hide correlation).
• Extrapolating far beyond the observed range of X.
• Treating a significant coefficient as a causal effect in observational data.
• Adding highly correlated predictors and then trusting their individual coefficients.