Statistical Significance vs Practical Significance

A p-value tells you whether a difference is real. It does not tell you whether that difference matters.

Key takeaway

You can have a statistically significant result (tiny p-value) with a trivially small effect size. Conversely, a large, meaningful effect may not reach significance if your sample size is small. Always report and interpret the effect size alongside the p-value.

Why a p-value alone can mislead

A p-value answers: "If the null hypothesis were true, what is the probability of observing a result this extreme (or more extreme) by chance alone?" A small p-value means your result is unlikely under the null. But it says nothing about the magnitude of the difference.

Consider two studies of the same intervention, measured on the same scale, with identical effect sizes. One has n = 50 per group and finds p = .32. The other has n = 1000 per group and finds p < .0001. The effect did not become "more real" with a larger sample—it was always the same. The p-value only got smaller because we had more statistical power to detect it.

Here is a concrete worked example:

Scenario Sample size (n per group) Effect (score change) Cohen's d t statistic p-value (2-tailed) Interpretation
Small sample 50 2 points 0.2 1.00 .3215 Not significant; fail to reject H₀
Large sample 1000 2 points 0.2 4.47 < .0001 Significant; reject H₀

The effect size is identical in both scenarios (d = 0.2, a small effect per Cohen's convention). In the small-n case, you lack the power to detect it. In the large-n case, you have plenty of power, so significance is achieved—but the underlying effect remains small. A practitioner must decide: is a 2-point reduction meaningful enough to implement or care about? The p-value does not answer that question.

Effect size is the missing piece

Effect size quantifies the magnitude of a difference or relationship. Common measures include:

  • Cohen's d (for comparing two group means): the difference between means divided by the pooled standard deviation.
  • η² (eta squared) (for ANOVA): the proportion of variance in the outcome explained by the groups.
  • r (correlation): the strength and direction of a linear relationship.

Cohen's benchmarks are useful rules of thumb conventions when no domain-specific standard exists:

  • d ≈ 0.2: small effect
  • d ≈ 0.5: medium effect
  • d ≈ 0.8: large effect

However, Cohen himself described these as "arbitrary rules of thumb." In some fields, d = 0.1 is a breakthrough; in others, d = 0.8 is unimportant. Always check what practitioners in your field consider meaningful.

Use our effect size calculator to compute Cohen's d, η², or other measures from your data. Then use the sample size calculator to plan future studies with realistic power.

A checklist for "does this matter?"

After you report a p-value and an effect size, ask yourself:

  1. Is there a domain-specific minimum meaningful effect? In clinical trials, a drug that reduces blood pressure by 1 mmHg is statistically significant in a trial of 10,000 people—but clinically irrelevant. What does the field consider the smallest clinically relevant effect? Use that as your benchmark, not Cohen's generic conventions.
  2. How wide is the confidence interval? A 95% CI on your effect size shows the plausible range. If the CI spans from "small" to "large," you have precision uncertainty even if the point estimate looks impressive.
  3. What is the cost or feasibility of acting on this result? A small effect may still justify an intervention if it is cheap, easy, and harmless. A large effect might not be worth implementing if the intervention is expensive or carries side effects.
  4. Does the effect replicate? Statistical significance in a single study, especially with a questionable design or selective reporting, is weak evidence. Look for independent replication or meta-analytic pooling.

Worked vignette: "Significant but irrelevant"

An online tutoring platform tests a new algorithm for generating practice problems. They randomly assign 5000 students to the new algorithm or the control. After one semester, they measure test anxiety on a 30-point scale.

  • Control group (n = 2500): mean anxiety = 15.0, SD = 4.5
  • Tutoring group (n = 2500): mean anxiety = 14.55, SD = 4.5
  • Difference = 0.45 points

The statistics:

  • Cohen's d = 0.45 / 4.5 ≈ 0.10 (very small effect)
  • t = 0.45 / (4.5 × √(2/2500)) = 0.45 / 0.127 ≈ 3.54
  • df = 4998; p < .001

The conclusion: The result is highly statistically significant. But a 0.45-point reduction on a 30-point scale is undetectable by any individual student and does not change their lived experience of test anxiety. The effect is real (not due to chance), but it is practically negligible. The company should report both the p-value and the effect size, and be honest that the intervention, however statistically significant, is not worthwhile.

FAQ

Can a result be statistically significant but not practically important?

Yes—this is extremely common with large sample sizes. A tiny effect, when detected with enough power, will yield a small p-value even if the effect is trivial. Always report the effect size alongside the p-value and ask whether the magnitude justifies the effort or cost of implementation.

Can a result be practically important but not statistically significant?

Yes. If your sample size is small, a large and meaningful effect may not reach the p < .05 threshold by chance. This is a Type II error (failing to reject a false H₀). Use a power analysis before data collection to choose a sample size that can detect the effect size you consider meaningful.

How do I decide what effect size matters in my field?

Look for published standards or meta-analyses in your discipline. Talk to experienced practitioners and ask, "What is the smallest effect you would actually care about?" This is often called the "minimum clinically relevant difference" or "minimum practically relevant effect." Use that benchmark instead of relying on Cohen's generic conventions.

Why does a huge sample size make almost anything statistically significant?

Because the standard error of the mean decreases as n increases (proportionally to 1 / √n). With a very large n, even a tiny difference will yield a large t-statistic and a tiny p-value. This is correct—statistical significance is about the likelihood of the observed difference under the null, not about whether the difference is practically interesting.

Should I report effect size even when my result is not statistically significant?

Absolutely. A non-significant result can occur either because there is no true effect or because your sample size is too small. Reporting the effect size and its confidence interval tells the reader which is likely. A large effect with a wide CI and a non-significant p-value suggests underpowering, not absence of effect.

What is the relationship between effect size and sample size?

They are inversely related: to reliably detect a small effect, you need a large sample; to detect a large effect, a small sample suffices. Use our sample size calculator to plan the required n for your desired effect size and power (typically 80%).

Sources