Direct answer
A p-value is a tail probability computed under a specified null model; alpha is a preselected Type I error threshold, beta is Type II error under a specified alternative, and power is 1−beta.
Visual explanation
Decision errors under null and alternative models
What this calculation tells you
Hypothesis testing compares observed data with a model-based reference distribution. Error rates describe repeated decisions under specified states; they are not posterior probabilities for the hypotheses.
State the null, alternative, test statistic, tail direction, alpha, planned effect, design, and analysis before collecting or inspecting confirmatory data.
Where it is used
Experiments
Plan decision errors around a declared effect and design.
Quality
Relate detection sensitivity to false-alarm choices.
Research review
Audit whether hypotheses and analyses were prespecified.
Education
Visualize repeated-decision probabilities under two models.
When this guide helps
- Interpreting a reported p-value.
- Choosing alpha before analysis.
- Planning power for a meaningful effect.
- Explaining the trade-off between false positives and false negatives.
Start with the statistical question
State the null, alternative, test statistic, tail direction, alpha, planned effect, design, and analysis before collecting or inspecting confirmatory data.
Overlapping null and alternative curves show the rejection region, Type I area, Type II area, and power. Moving the threshold reveals the trade-off rather than implying both errors can shrink freely.
Worked example
At alpha 0.05, a result with p=0.03 crosses the rule under the stated test. It does not mean a 3% probability that the null is true or a 97% probability that the result will replicate.
Assumptions that carry the result
Reported p-values and power inherit all test assumptions, including sampling, independence, model form, tail choice, stopping rules, and analysis selection.
Interpret the result without overreaching
Threshold decisions do not measure effect size, importance, bias, reproducibility, or truth. Selective testing and repeated looks can invalidate nominal error rates.
- Interpreting p as P(null|data).
- Choosing one- or two-sided testing after seeing direction.
- Calling 1−p the probability of replication.
Worked case: p below alpha
A prespecified test uses alpha 0.05 and returns p = 0.03.
Because 0.03<0.05, the result crosses the stated rule.
The decision is reject under that procedure.
p = 0.03 is not a 3% probability the null is true.
Reproduce this worked caseOpen P-Value Calculator
Worked case: stricter threshold
Use alpha 0.01 for the same p = 0.03.
Now 0.03>0.01, so the result does not cross the rule.
The data did not change; the error-control decision rule did.
Choosing alpha after seeing p invalidates the claimed procedure.
Reproduce this worked caseOpen P-Value Calculator
p-values, alpha and power: compare assumptions, not just answers
Report effect and interval. Power requires effect size, variability, design and analysis—not simply 1-p.
| Case | Calculation focus | Interpretation |
|---|---|---|
| α=0.05 | p = 0.03 | Crosses rule |
| α=0.01 | p = 0.03 | Does not cross |
p-values, alpha and power: calculation checklist
- Hypotheses/tail prespecified
- Alpha set before data
- p interpreted conditionally
- Effect/CI reported
- 1-p not called power
Practical questions
Frequently asked questions
Is p the probability the null is true?
No. It is calculated assuming the null model and measures tail compatibility of the data or something more extreme.
Is power fixed for a study?
Power varies with the specified true effect, variability, design, sample size, alpha, and method.
Why not always use a tiny alpha?
Lowering alpha can reduce false positives but increase false negatives unless design or sample size changes; consequences must guide the choice.
Further reading
Authoritative sources
Use these primary and professional resources to check definitions, conventions, or requirements that may extend beyond this guide.