Hypothesis testing and p-values without the folklore

Hypothesis testing is a way to decide whether a difference in your data is probably real or probably luck. The p-value is its most quoted and most misunderstood number.

The setup

You start with a null hypothesis: nothing is going on. The new page converts no better than the old one. The drug does no more than the placebo. Then you ask how surprising your data would be if that were true.

The p-value is the answer. Precisely: the probability of getting a result at least as extreme as the one you saw, assuming the null hypothesis is true.

A small p-value says "this would be unusual if nothing were going on", which is evidence against the null. A large one says the data are unremarkable under the null. It does not say the null is true.

A decision rule, and what it costs

Before the test you pick a threshold, alpha, often 0.05. If the p-value falls below it you call the result "significant" and reject the null.

That rule has a built-in price. Even when there is no real effect, 5% of tests will still fall below 0.05, because that is what the threshold means. You can see it by simulation: run many tests where both groups have exactly the same 12% conversion rate, so there is nothing to find.

import random
from math import sqrt
from statistics import NormalDist

random.seed(11)
normal = NormalDist()


def p_value(n1, c1, n2, c2):
    pooled = (c1 + c2) / (n1 + n2)
    se = sqrt(pooled * (1 - pooled) * (1 / n1 + 1 / n2))
    z = (c2 / n2 - c1 / n1) / se
    return 2 * (1 - normal.cdf(abs(z)))


def conversions(n, rate):
    return sum(random.random() < rate for _ in range(n))


TESTS, USERS, RATE = 1000, 4000, 0.12
significant = 0
for _ in range(TESTS):
    # both groups get the SAME 12% rate: there is no real difference to find
    a, b = conversions(USERS, RATE), conversions(USERS, RATE)
    if p_value(USERS, a, USERS, b) < 0.05:
        significant += 1

print(f"{significant} of {TESTS} tests with no real difference came out 'significant' at p < 0.05")

50 of 1000 tests with no real difference came out 'significant' at p < 0.05

Fifty of a thousand, which is the 5% the threshold promised. Run twenty comparisons on pure noise and you should expect about one "discovery". This is why testing many metrics, segments or variants and reporting the one that hit is misleading.

Four common misreadings

1. "p = 0.03 means there is a 3% chance the result is a fluke." No. It is the chance of data this extreme if the null were true. It is not the probability that the null is true.
2. "p = 0.2 means there is no effect." No. It means the test could not tell. With too little data, a real effect usually produces a large p-value.
3. "Significant means important." No. With enough data a trivial difference becomes significant. Size and importance come from the effect, not the p-value.
4. "Significant results will replicate." Not reliably. A result just under 0.05 from a small test is fragile.

What to look at as well

• Effect size. How big is the difference, in the units people care about (percentage points, minutes, money)?
• A confidence interval. The range of effects compatible with the data. A narrow interval around a useful effect is strong evidence. A wide one that includes zero says you do not know yet.
• Power and sample size. Decide the smallest effect worth detecting and collect enough data to detect it. An underpowered test produces both missed effects and exaggerated ones.
• The design. Was the metric chosen in advance, the assignment random, the stopping rule fixed? No calculation rescues a poorly designed test.

A better way to report

Instead of "the variant won (p < 0.05)", write: "The variant converted 1.5 points better (95% interval 0.04 to 2.96) from 8,000 visitors; the test was smaller than planned, so we are repeating it with 8,000 per group." That sentence tells a reader what happened, how sure you are and what comes next, which a bare verdict never does.