SciPy Significance Tests

scipy.stats provides hypothesis tests like the independent t-test that quantify whether a difference between two groups is likely real or just random noise.

What a Significance Test Tells You

A significance test gives you a principled way to answer questions like 'did this new teaching method actually improve scores, or could the difference I observed just be random luck?' You start by assuming a null hypothesis - typically that there is no real difference between the groups being compared - then calculate how surprising your actual data would be if that null hypothesis were really true. The test produces two things: a test statistic summarizing the size of the observed effect relative to the natural variation in the data, and a p-value that translates that statistic into a probability.

The Independent t-test

scipy.stats.ttest_ind compares the means of two independent samples - two groups of students, two batches of a product, two versions of an experiment - and tests whether their averages differ by more than you'd expect from random sampling variation alone. It assumes the two samples are independent of each other and come from roughly normal distributions with similar variance; when that variance assumption doesn't hold, pass equal_var=False to run Welch's version of the test instead, which adjusts for unequal spread between the two groups.

Example

import numpy as np
from scipy import stats

# Exam scores from two different teaching methods
group_a = np.array([72, 68, 75, 71, 70, 74, 69, 73])
group_b = np.array([78, 82, 75, 80, 79, 77, 81, 76])

t_statistic, p_value = stats.ttest_ind(group_a, group_b)

print("t-statistic:", t_statistic)
print("p-value:", p_value)
  • statistic - the t-statistic; a larger magnitude means a bigger difference relative to the spread within each group
  • pvalue - the probability of seeing a difference this large (or larger) if the null hypothesis were actually true
  • equal_var - set to False to use Welch's t-test when the two groups may have different variances
  • alternative - choose 'two-sided', 'less', or 'greater' to match the direction of effect you're actually testing for

Interpreting the p-value Correctly

It's worth being precise here, because this is one of the most commonly misstated ideas in statistics. The p-value is the probability of observing a difference at least as large as the one you measured, calculated under the assumption that the null hypothesis is true. A small p-value means your data would be quite surprising if there really were no difference between the groups, which is evidence against the null hypothesis. A common (but arbitrary) cutoff is alpha = 0.05: if the p-value falls below that threshold, the result is typically called 'statistically significant.'

Note: The p-value is NOT the probability that the null hypothesis is true, and it is NOT the probability that your result happened by chance. A p-value of 0.03 means: if there really were no difference between the groups, there would be a 3% chance of seeing a difference this large just from random sampling variation. It says nothing about how likely the null hypothesis itself is to be true.

Example

alpha = 0.05

if p_value < alpha:
    print("Reject the null hypothesis: the difference is statistically significant.")
else:
    print("Fail to reject the null hypothesis: not enough evidence of a real difference.")
FunctionUse When
ttest_indComparing means of two independent, unrelated groups
ttest_relComparing means of two related/paired measurements (e.g. before vs after)
mannwhitneyuComparing two independent groups without assuming a normal distribution
chi2_contingencyTesting whether two categorical variables are associated
Note: Statistical significance isn't the same as practical importance. A very small p-value can come from a tiny, practically meaningless difference if your sample size is large enough. Always look at the actual size of the difference (e.g. the difference in means) alongside the p-value before deciding whether an effect actually matters.

Exercise: SciPy Significance Tests

What does a significance test like a t-test help determine?

Keep Going

SciPy Interview QuestionsPractice real questions asked for SciPyData Scientist SalarySee what Data Scientist roles pay in the marketAuto Apply for JobsApply to open roles related to SciPy and tweak your resume