Hypothesis Testing

The framework

State a null hypothesis (H0: "no effect/no difference") and an alternative (H1: "there is an effect"). Collect data, compute a test statistic, and derive a p-value: the probability of seeing data this extreme (or more) if the null hypothesis were actually true.

What a p-value is not

A p-value is not "the probability the null hypothesis is true." A p-value of 0.03 means: if there really were no effect, you'd see data this extreme 3% of the time by chance alone. Whether that's "significant" depends on a threshold (commonly 0.05) chosen before the test, not after seeing the result.

Two ways to be wrong

• Type I error — rejecting a true null hypothesis (a false positive); its rate is the significance level α.
• Type II error — failing to reject a false null hypothesis (a false negative); its rate is β, and statistical power (1 − β) is the probability of correctly detecting a real effect.

Prerequisite

Requires knowing which distribution your test statistic follows under the null — which is exactly what the Distributions topic establishes.