Hypothesis tests and power
This is where the t, Normal and Binomial distributions earn their keep. See where a test draws its line, how likely it is to catch a real effect, and how many observations you need. Then simulate thousands of experiments to check the theory, including what happens when the data are not normal.
Where the test draws the line
If H₀ is true, the statistic follows the blue curve and lands beyond the critical line(s) with probability α: a false alarm. If the effect is real, it follows the orange curve, and the orange area beyond the line is the power.
Power curve
Power rises with sample size. The horizontal line is your target and the dot is your current design.
Check it by simulation
| Quantity | Theory | Simulated | Check (within 3 s.e.) |
|---|
p-values when H₀ is true
Should be flat: every p-value equally likely, so a fraction α falls below α.
p-values when H₁ is true
Piles up near 0. The shaded share is the power.