Statistics Hypothesis Testing

Setting Up the Two Hypotheses

Hypothesis testing starts by writing down two competing statements about a population. The null hypothesis (H0) is the default, no-effect or no-difference claim we assume is true until data convinces us otherwise. The alternative hypothesis (Ha, sometimes H1) is the claim we're actually trying to find evidence for - that there is an effect, a difference, or a relationship.

• H0 (null hypothesis): the coin is fair, P(heads) = 0.5
• Ha (alternative hypothesis): the coin is not fair, P(heads) is not equal to 0.5
• We collect flip data and check whether it would be implausible if H0 were true

From Data to a Decision

Once the hypotheses are set, we pick a significance level (alpha, commonly 0.05) before looking at the data. Then we collect a sample, compute a test statistic that measures how far the sample result is from what H0 predicts, and convert that into a p-value. If the p-value is smaller than alpha, the result is considered statistically significant and we reject H0 in favor of Ha; otherwise, we fail to reject H0.

import math

def z_stat_proportion(p_hat, p0, n):
    se = math.sqrt(p0 * (1 - p0) / n)
    return (p_hat - p0) / se

# Testing whether a coin is fair: H0: p = 0.5
p_hat = 0.58   # observed proportion of heads in 100 flips
p0 = 0.5
n = 100

z = z_stat_proportion(p_hat, p0, n)
print("Test statistic z =", round(z, 3))

Two Types of Errors

Because we're making a decision from limited data, two kinds of mistakes are possible. A Type I error happens when we reject a null hypothesis that was actually true (a false alarm) - its probability is controlled directly by alpha. A Type II error happens when we fail to reject a null hypothesis that was actually false (a missed effect); its probability is often labeled beta. Shrinking alpha to avoid Type I errors, without changing anything else, tends to increase the risk of Type II errors, which is why sample size and study design matter.

import math
import random

# Simulate the null being true (a genuinely fair coin) many times
# and count how often we would incorrectly reject H0 at alpha = 0.05 (|z| > 1.96)
false_positives = 0
trials = 2000
for _ in range(trials):
    heads = sum(1 for _ in range(100) if random.random() < 0.5)
    p_hat = heads / 100
    se = math.sqrt(0.5 * 0.5 / 100)
    z = (p_hat - 0.5) / se
    if abs(z) > 1.96:
        false_positives += 1

print("Observed Type I error rate:", false_positives / trials)

1. State the null and alternative hypotheses
2. Choose a significance level (alpha), commonly 0.05
3. Collect data and compute the relevant test statistic
4. Compare the resulting p-value to alpha (or the statistic to a critical value)
5. Decide to reject or fail to reject H0, and interpret the result in plain language