Hypothesis testing

Hypothesis testing answers one question: could the difference I am looking at have appeared by chance alone? It does not tell you whether the difference matters. That is your job, using effect sizes.

The five steps

1. H0 (null): no difference. "The new packaging does not change average basket value."
2. H1 (alternative): there is a difference (two-tailed), or specifically an increase (one-tailed).
3. Fix the significance level, usually alpha = 0.05, before looking at the data.
4. Compute a test statistic and its p-value.
5. If p < alpha, reject H0. Otherwise you failed to find evidence - which is not proof that H0 is true.

Choosing the test

from scipy import stats

# Basket value (INR) at 2 store formats, unequal variance and unequal n
t, p = stats.ttest_ind(group_a, group_b, equal_var=False)
print(round(t, 3), round(p, 4))

# Same 30 stores before and after a shelf redesign
t2, p2 = stats.ttest_rel(before, after)

Use Welch (equal_var=False) by default for two independent groups; it is safer than the classic Student version and costs almost nothing in power.

p-values, honestly

A p-value is the probability of seeing data at least this extreme if H0 were true. It is not the probability that H0 is true, not the probability your result will replicate, and not a measure of effect size. With 50,000 rows, a ₹3 difference in average bill can be "highly significant" and commercially worthless.

Report an effect size and an interval

import numpy as np

diff = np.mean(after) - np.mean(before)
pooled = np.sqrt((np.var(before, ddof=1) + np.var(after, ddof=1)) / 2)
cohens_d = diff / pooled
print(round(diff, 1), round(cohens_d, 2))

Roughly, d of 0.2 is small, 0.5 medium, 0.8 large. Pair the effect size with a confidence interval and a plain sentence: "average basket rose by ₹84 (95% CI ₹31 to ₹137), about 6%."

Two ways to be wrong

Power = 1 - beta, the chance of detecting a real effect. It rises with sample size and effect size. Compute the sample size you need before running the experiment, or a null result means nothing.

Common mistakes

• Reading "p > 0.05" as "no difference". Absence of evidence is not evidence of absence.
• Running 20 tests and reporting the one that worked. Correct for multiple comparisons (Bonferroni, or Benjamini-Hochberg).
• Choosing one-tailed after seeing the direction of the result.
• Testing on non-independent rows, such as many invoices from the same customer.
• Reporting only p, never the size of the difference or its interval.