Statistics P-values
A p-value tells you how surprising your observed data would be if the null hypothesis were actually true - it is not the probability that the null hypothesis itself is true.
What a P-value Actually Measures
A p-value is the probability of observing a result at least as extreme as the one in your data, calculated under the assumption that the null hypothesis is true. It answers the question: 'if there really were no effect, how surprising would this data be?' It says nothing directly about whether the null hypothesis itself is true.
Reading Small and Large P-values
- A small p-value means the observed data (or something more extreme) would be unlikely if H0 were true - this counts as evidence against H0
- A large p-value means the observed data is quite plausible under H0 - this does not prove H0 is true, it just means the data doesn't contradict it
Example
import math
def normal_cdf(x):
return 0.5 * (1 + math.erf(x / math.sqrt(2)))
def two_tailed_p_value(z):
return 2 * (1 - normal_cdf(abs(z)))
z = 2.14 # observed test statistic
p = two_tailed_p_value(z)
print("Two-tailed p-value:", round(p, 4))Comparing the P-value to a Significance Level
To turn a p-value into a decision, we compare it with a significance level (alpha) chosen in advance, typically 0.05. If p is less than alpha, we call the result statistically significant and reject the null hypothesis; if p is greater than or equal to alpha, we don't have enough evidence to reject it. The threshold is a convention, not a law of nature, so results just above or below 0.05 shouldn't be treated as categorically different.
Example
import random
def simulate_null_statistic():
# Simulate a sample mean difference under H0: no real difference
group_a = [random.gauss(50, 10) for _ in range(30)]
group_b = [random.gauss(50, 10) for _ in range(30)]
return (sum(group_a) / 30) - (sum(group_b) / 30)
observed_diff = 4.2
extreme_count = 0
trials = 5000
for _ in range(trials):
if abs(simulate_null_statistic()) >= observed_diff:
extreme_count += 1
print("Simulated p-value:", extreme_count / trials)- It does not tell you the probability that H0 is true
- It does not tell you the probability that the alternative hypothesis is true
- It does not measure the size or practical importance of an effect
- It is not a guarantee that a result will replicate in a future study
