Analyse an A/B test in Python: rates, z-test and confidence interval

An A/B test shows version A to some visitors and version B to others, then asks whether B converts better. The danger is mistaking luck for improvement. This tutorial analyses a result properly using only Python's standard library.

The result to analyse

A page change was tested on 8,000 visitors, split evenly:

• Control: 4,000 visitors, 480 conversions
• Variant: 4,000 visitors, 540 conversions

That is 12.0% against 13.5%. Is the variant really better?

Step 1: Compute the difference and test it

The standard tool for comparing two proportions is a two-proportion z-test. It asks: if both groups truly had the same underlying rate, how surprising would a gap this large be?

from math import sqrt
from statistics import NormalDist

normal = NormalDist()


def analyse(control_users, control_conversions, variant_users, variant_conversions, confidence=0.95):
    p1 = control_conversions / control_users
    p2 = variant_conversions / variant_users

    # test: is the difference bigger than chance would explain?
    pooled = (control_conversions + variant_conversions) / (control_users + variant_users)
    se_test = sqrt(pooled * (1 - pooled) * (1 / control_users + 1 / variant_users))
    z = (p2 - p1) / se_test
    p_value = 2 * (1 - normal.cdf(abs(z)))

    # estimate: how big is the difference, give or take?
    se_diff = sqrt(p1 * (1 - p1) / control_users + p2 * (1 - p2) / variant_users)
    margin = normal.inv_cdf(1 - (1 - confidence) / 2) * se_diff
    return {
        "control_rate": p1, "variant_rate": p2, "difference": p2 - p1,
        "relative_lift": (p2 - p1) / p1, "z": z, "p_value": p_value,
        "ci_low": p2 - p1 - margin, "ci_high": p2 - p1 + margin,
    }

Two standard errors appear, and they are different on purpose. The test assumes no difference (the null hypothesis), so it pools both groups into one rate. The interval estimates the actual difference, so it uses each group's own rate.

Step 2: Run it and read the result

result = analyse(4000, 480, 4000, 540)
print(f"control  {result['control_rate']:.1%}")
print(f"variant  {result['variant_rate']:.1%}")
print(f"difference {result['difference']:+.2%} points  (relative {result['relative_lift']:+.1%})")
print(f"z = {result['z']:.2f}, p = {result['p_value']:.3f}")
print(f"95% interval for the difference: {result['ci_low']:+.2%} to {result['ci_high']:+.2%}")

control  12.0%
variant  13.5%
difference +1.50% points  (relative +12.5%)
z = 2.01, p = 0.044
95% interval for the difference: +0.04% to +2.96%

How to read it:

• The lift is 1.5 percentage points, or 12.5% relative to the control.
• p = 0.044 means that if there were truly no difference, a gap this big or bigger would turn up about 4.4% of the time. Below the usual 5% line, so the result is "statistically significant".
• The interval is the more useful number. It runs from +0.04 to +2.96 points. The data are compatible with a real improvement that is almost nothing, or one that is nearly three points. "Significant" here means "probably better", not "better by 1.5 points".

Step 3: Ask whether the test was big enough

A test should be sized before it runs. This function returns the visitors needed in each group to detect a given absolute lift with the usual 5% significance and 80% power (an 80% chance of noticing a real effect of that size).

def sample_size_per_group(baseline, minimum_effect, alpha=0.05, power=0.8):
    """Visitors needed in each group to detect an absolute lift of `minimum_effect`."""
    p1, p2 = baseline, baseline + minimum_effect
    z_alpha = normal.inv_cdf(1 - alpha / 2)
    z_beta = normal.inv_cdf(power)
    variance = p1 * (1 - p1) + p2 * (1 - p2)
    return int(((z_alpha + z_beta) ** 2 * variance) / (minimum_effect ** 2)) + 1


print("visitors per group to detect +1.5 points from a 12% baseline:", sample_size_per_group(0.12, 0.015))
print("visitors per group to detect +0.5 points from a 12% baseline:", sample_size_per_group(0.12, 0.005))

visitors per group to detect +1.5 points from a 12% baseline: 7758
visitors per group to detect +0.5 points from a 12% baseline: 67493

Our test used 4,000 per group, about half of what a +1.5 point effect needs. That is why the interval is so wide and why the result sits only just under the 5% line. Two practical lessons follow:

1. Smaller effects need far more visitors. Detecting a third of the lift costs roughly nine times the traffic.
2. Decide the smallest improvement worth having, then compute the sample. If you cannot afford the visitors, you cannot answer the question, and it is better to know that before you start.

Common mistakes

• Peeking and stopping early. Checking daily and stopping when p dips below 0.05 inflates false positives badly. Fix the sample size in advance, or use a method built for sequential looks.
• Running many variants or metrics and reporting the winner. Each extra comparison is another chance of a fluke. Decide the main metric first.
• Unequal or broken splits. If the groups are not about the same size, or the assignment is not random, the test is invalid. Check the split before looking at results.
• Reading "not significant" as "no effect". It means the test could not tell. The interval shows what remains possible.

Check your work

Recompute the two rates by hand from the counts (480 ÷ 4,000 and 540 ÷ 4,000). The difference must equal the lift printed, and the interval must be centred on it. If you swap the groups, the sign of the z statistic flips and the p-value stays the same.