A/B Testing

A/B Testing is a controlled experiment that compares two versions of something - a webpage, an email subject line, an app feature - to see which one performs better on a specific metric. It's one of the most practical, widely used applications of hypothesis testing in industry.

How A/B Testing Works

1. Split users randomly into two groups - Group A (control) and Group B (variant)
2. Show each group a different version of the thing being tested
3. Measure a specific outcome, such as click rate, purchase rate, or time on page
4. Use a hypothesis test (often a proportion test or t-test) to check if the difference is statistically significant

Setting Up the Hypotheses

# H0: Version A and Version B perform equally well
# H1: Version B performs differently (better or worse) than Version A

A Complete Example

from statsmodels.stats.proportion import proportions_ztest

# Version A: 120 purchases out of 2000 visitors
# Version B: 150 purchases out of 2000 visitors
successes = [120, 150]
totals = [2000, 2000]

z_stat, p_value = proportions_ztest(successes, totals)
print("P-value:", p_value)

alpha = 0.05
if p_value < alpha:
    print("Version B performs significantly differently from Version A")
else:
    print("No statistically significant difference detected")

Common Pitfalls

• Stopping the test too early, before enough data has accumulated
• Running too many simultaneous comparisons without adjusting for it
• Treating statistical significance as automatically meaning practical importance

You've Completed This Section

This wraps up the hypothesis testing toolkit - from comparing means and proportions between groups, to running full A/B tests to make confident, data-driven business decisions.