Hypothesis testing evaluates whether sample evidence is unusual under a null model. The p value is the probability, assuming the null is true, of observing a result at least as extreme as the sample. Small p values indicate incompatibility with the null, not proof of practical importance.
- State null, alternative, and significance level before seeing results.
- P value is about data under the null, not probability the null is true.
- Statistical significance and practical importance are different questions.
Core testing steps
A standard workflow is: define H0 and Ha, choose alpha, compute a test statistic, find p value, and make a decision in context. The test statistic compares observed effect to expected variation under H0.
Type I error rejects a true null with probability alpha. Type II error fails to reject a false null. Power is 1 - beta, the chance of detecting a real effect under a specific alternative.
Interpreting p values responsibly
A p value does not measure effect size or importance. With large samples, tiny effects can produce small p values. With small samples, meaningful effects can miss significance.
Report an estimate and confidence interval with the test result. This keeps focus on direction, size, and uncertainty instead of a single threshold decision.
- Do not switch one-sided and two-sided tests after looking at data.
- Avoid treating p=0.049 and p=0.051 as fundamentally different evidence.
- Predefine decision rules in analysis plans.
One-sample mean test with known sigma
Historical average handling time is 50 minutes. A new process sample has n=36, mean 47, assume sigma=9.
- Set hypotheses: H0: mu=50, Ha: mu<50.
- Compute z=(xbar-mu0)/(sigma/sqrt(n))=(47-50)/(9/6)=-3/1.5=-2.0.
- Find one-sided p value P(Z<=-2.0)=0.0228.
- Compare to alpha=0.05 and conclude evidence supports lower mean handling time.
Common mistakes
- Interpreting p value as probability H0 is true.
- Using significance alone as proof of practical value.
- Choosing alpha after seeing the p value.
- Failing to report effect estimate with uncertainty.
Try one
If p=0.03 and alpha=0.05, what is the formal decision?
Reject H0 at the 0.05 level, while still interpreting effect size and context separately.
Sources
- OpenStax: Introductory StatisticsOpen textbook covering descriptive statistics, probability, inference, and regression fundamentals.
- NIST/SEMATECH e-Handbook of Statistical MethodsAuthoritative reference on probability models, inference procedures, and practical statistical diagnostics.