A/B testing for business, with a Bayesian view

Hypothesis testing gives you the machinery; this lesson is about running an A/B test that a business can actually trust - from planning to the final recommendation - and about an alternative, Bayesian way of reading the result.

Step 1: Plan before you launch

Decide these up front and write them down:

Item	Example
Primary metric	Checkout conversion rate
Baseline	4.5%
Minimum detectable effect (MDE)	+0.6 points (to 5.1%)
Significance level (alpha)	0.05
Power	80%
Guardrail metrics	Refund rate, page load time

Then compute the sample size, so you know how long to run:

from statsmodels.stats.power import NormalIndPower
from statsmodels.stats.proportion import proportion_effectsize

effect = proportion_effectsize(0.051, 0.045)
n_per_group = NormalIndPower().solve_power(effect_size=effect, alpha=0.05, power=0.8, ratio=1)
print(f"Need about {n_per_group:,.0f} visitors per variant")

Roughly 20,000 per group: with 3,000 daily visitors split 50/50, that is two weeks. Run whole weeks so weekday and weekend behaviour are both covered.

Step 2: Randomise properly

Assign users (not page views) randomly and consistently, e.g. hash of user id. Check the split: if you planned 50/50 but got 52/48 on large traffic, investigate - a sample ratio mismatch usually means a bug.

Step 3: Analyse once, at the planned end

from statsmodels.stats.proportion import proportions_ztest, confint_proportions_2indep

conv = [624, 540]           # variant B, control A
visitors = [12_000, 12_000]
z, p = proportions_ztest(conv, visitors)
low, high = confint_proportions_2indep(conv[0], visitors[0], conv[1], visitors[1])
print(f"B {conv[0]/visitors[0]:.2%} vs A {conv[1]/visitors[1]:.2%}, p = {p:.3f}")
print(f"Lift 95% CI: [{low:.2%}, {high:.2%}]")

Report the lift with its interval and translate it into money: +0.7 points on 90,000 monthly visitors at ₹650 average order is about ₹4 lakh extra revenue a month.

Common business pitfalls

• Peeking: checking daily and stopping when p < 0.05 inflates false positives dramatically. Fix the duration, or use sequential methods designed for it.
• Too many metrics: test 20 metrics and one will "win" by chance. Pick one primary metric.
• Novelty effects: users click a new button because it is new; look at trends over the test.
• Ignoring segments... or over-slicing: check major segments (app vs web) but do not hunt for a winning sub-group.

A Bayesian view

A p-value answers "how surprising is this data if there is no difference?" Stakeholders usually want "what is the probability B is better?" Bayesian analysis answers that directly. With a flat Beta(1, 1) prior, each conversion rate has a Beta posterior:

import numpy as np

rng = np.random.default_rng(42)
a = rng.beta(1 + 540, 1 + 12_000 - 540, 200_000)
b = rng.beta(1 + 624, 1 + 12_000 - 624, 200_000)
print(f"P(B > A) = {(b > a).mean():.1%}")
print(f"Expected lift = {(b - a).mean():.2%}")
print(f"P(lift > 0.3 points) = {((b - a) > 0.003).mean():.1%}")

Bayesian thinking in general means: start with a prior belief, update it with evidence, and get a posterior. Priors can encode past experience (e.g. most checkout tweaks move conversion by under 1 point), which protects against over-excitement over small samples.

Key takeaways

• Fix metric, MDE, sample size and duration before launch.
• Do not peek-and-stop; watch for sample ratio mismatch.
• Report effect size with an interval and a rupee impact.
• Bayesian posteriors give intuitive statements like "93% chance B is better".