Statistical Inference

What Is a P-Value?

A p-value is the probability of seeing data at least as extreme as yours, calculated under the assumption that the null hypothesis is true. It is not the probability that your hypothesis is true.

The single most common misreading

Students often interpret a p-value of 0.03 as "there is a 3% chance the null hypothesis is true" or "there is a 97% chance my hypothesis is true". Both are wrong. A p-value of 0.03 means: if the null hypothesis were true, there would be only a 3% chance of observing data this extreme or more extreme. The null hypothesis is either true or false in reality—it is not a random variable with a probability. What varies is the data, and the p-value quantifies how surprising your particular data are under a specific assumption (that H₀ is true).

§1A worked example: building intuition

Imagine a medication manufacturer claims their new painkiller works no differently than aspirin—that is, the null hypothesis H₀ is "both drugs are equally effective". You run a randomized controlled trial with 60 participants: 30 get the new drug, 30 get aspirin. You measure pain relief on a 0–100 scale an hour after dosing.

The true population means are unknown, but your sample means are: new drug = 72 points, aspirin = 66 points. That is a 6-point difference. The question is not "Is there really a difference?" (there is—you measured it). The question is: "If both drugs were truly equally effective, how likely is it that random sampling would produce a difference of 6 points or larger?"

To answer this, you calculate a test statistic—say, a t-statistic of 2.1—and then ask the t distribution: "What fraction of the area under the curve lies beyond t = 2.1?" (in both tails, if you are doing a two-tailed test). The answer is your p-value. If it is 0.04, you report: t(58) = 2.1, p = .04.

This p-value of 0.04 means: "If the drugs were truly equal and I ran this study over and over on new random samples, I would see a difference this large or larger only about 4% of the time". That is a relatively unusual outcome under H₀. Many researchers choose a threshold α = .05 and decide: "4% is rarer than 5%, so I reject H₀; I conclude the evidence supports a real difference." But that decision is a choice, not a law of nature. Some fields use α = .01 or .10. The threshold is arbitrary.

§2The visualization: where the p-value lives

A two-tailed standard normal distribution curve centered at z=0. The rejection regions are shaded beyond z=-2.3 and z=2.3, each labeled 0.53%. The curve shows how the p-value is the total area in both tails beyond the observed test statistic. −3 −2 −1 0 1 2 3 z statistic z = −2.3 z = 2.3 0.53% 0.53% Two-tailed p-value = 0.0106
Fig. 1 — The distribution of a test statistic under the null hypothesis. If your observed statistic is z = 2.3 or z = −2.3, the shaded areas represent the p-value: the proportion of the distribution that is this extreme or more extreme. Here, both tails together make up 1.06% of the total area, so p ≈ .0106 for a two-tailed test.

§3A brief history: why 0.05?

The 0.05 threshold is a historical convention, not a law of nature. In the 1920s, statistician Ronald Fisher chose 0.05 as a practical boundary between "surprising" and "unsurprising" results when testing whether data fit a hypothesis. He described it informally as "a standard which we can accept as efficient" and noted that results meeting this threshold were rare enough to warrant attention. Fisher never intended 0.05 to be universal or absolute.

By the mid-20th century, the threshold had calcified. Researchers, journal editors, and dissertation committees adopted α = .05 as an unquestioned rule of thumb. This was reinforced by the decision-theoretic framework of Neyman and Pearson, who formalized hypothesis testing as a binary choice (reject or fail to reject H₀ at a chosen α). The two frameworks—Fisher's probabilistic p-value and Neyman-Pearson's error-control approach—were often conflated, and the 0.05 boundary became dogma.

Today, many fields are stepping back from this orthodoxy. Threshold-dependent thinking (p < .05 means "publishable", p > .05 means "not significant") has contributed to the replication crisis, p-hacking, and cherry-picking of analyses. More recent guidance emphasizes reporting the exact p-value, the effect size, the confidence interval, and the study's power—not just a yes-or-no verdict at α = .05.

§4Myth versus reality

Common misinterpretations of p-values and the correct understanding
MythReality
"A p-value of 0.03 means there is a 3% chance the null hypothesis is true." A p-value is not the probability that H₀ is true. It is the probability of observing data this extreme if H₀ is true. H₀ is either true or false; probability does not apply to fixed propositions.
"A p-value of 0.03 means there is a 97% chance my hypothesis is true." No. A small p-value means your data are unusual under H₀, not that your alternative hypothesis is probably true. The alternative hypothesis could be false, and the data could be unusual by chance, or due to a confounding variable.
"A non-significant p-value (p > .05) proves there is no effect." Failing to reject H₀ is not the same as accepting it. A large p-value could mean no effect exists, or it could mean your sample was too small or your measurement was too noisy to detect a real effect. Always report the effect size and confidence interval.
"A smaller p-value means a bigger effect." No. A smaller p-value means your data are more surprising under H₀, but effect size is separate from statistical significance. A large effect in a small sample might yield p > .05; a trivial effect in a huge sample might yield p < .0001. Use Cohen's d or another effect size measure to quantify the magnitude of an effect.
"If I run the same study 100 times, about 5 times I will get p < .05 just by random chance (when H₀ is true)." Approximately correct by design, but misleading in practice. This statement assumes you are running the exact same test with the same sample size each time. In practice, researchers stop experiments early, run many tests, or modify their analysis plan based on the data—all of which change the actual false-positive rate.

§5P-values and their companions

A p-value alone is incomplete. Always report three things together:

1. Effect size

Cohen's d, Hedges' g, or the correlation coefficient r tells you how large the effect is in your data, independent of sample size. A p-value of 0.04 could come from a huge effect in a small sample or a trivial effect in a large sample. You need the effect size to know which. Use the effect size calculator to convert between d, r, and other standardized measures.

2. Confidence interval

A 95% confidence interval gives you a range of plausible values for the true population parameter. It directly answers "How much uncertainty is left?" and works even when the p-value is non-significant. If the confidence interval is wide, the effect is imprecisely estimated; if it is narrow, you have precise evidence for the range. See the confidence interval calculator.

3. Study design and power

When you fail to reject H₀, the p-value does not tell you whether the effect is absent or whether your study was underpowered. Report how many participants you planned to include and whether you achieved your target power. See the sample size calculator to plan power before you begin.

§6P-hacking and related risks

P-hacking (also called "fishing", "researcher degrees of freedom", or "HARKing"—Hypothesizing After Results are Known) is the practice of manipulating analyses or selectively reporting results until p < .05 is achieved, then presenting the result as if it were planned. Common tactics include:

  • Running many tests and reporting only the ones that are significant.
  • Deciding whether to collect more data, remove outliers, or add covariates after seeing the results.
  • Testing multiple outcome variables but reporting only the significant ones as if they were pre-specified.
  • Running analyses on many subgroups and reporting only the subgroup where p < .05 appeared.

Each of these inflates the false-positive rate. If you run 20 independent tests at α = .05, about 1 will be significant by chance alone, even if all null hypotheses are true. P-hacking exploits this. The solution: pre-register your analysis plan before data collection, distinguish confirmatory tests (planned beforehand) from exploratory analyses (okay, but labeled as such), and adjust your significance threshold if you are testing multiple hypotheses (e.g., Bonferroni correction).

§7Frequently asked questions

Is a smaller p-value always better?

No. A smaller p-value indicates that your data are more extreme under the null hypothesis, but it does not mean your finding is more important or that the effect is larger. A p-value of 0.001 from a study with 10,000 participants might reflect a trivial effect size, while a p-value of 0.04 from a smaller study might reflect a large, meaningful effect. Always compare p-values only within studies of similar design and sample size, and always report the effect size alongside the p-value.

What does "statistically significant" actually mean?

Statistical significance means the observed data are sufficiently rare under the null hypothesis (usually p < α = .05) that you reject H₀ and conclude there is evidence for a real effect. It does not mean the effect is large, important, or practically meaningful. A result can be statistically significant and practically trivial, or practically important but not statistically significant (if the sample is small). Always interpret significance in context: consider the effect size, the confidence interval, and whether the finding makes theoretical or practical sense.

Why do some journals now discourage p-values?

Because over-reliance on p-values has contributed to widespread problems: the replication crisis (many published findings do not replicate), p-hacking (selective reporting), and binary thinking (results are either "significant" or worthless). Journals like PLOS Biology, the American Journal of Clinical Nutrition, and others now de-emphasize p-values and require effect sizes, confidence intervals, and pre-registration of analysis plans. The American Statistical Association has also issued guidance cautioning against using p < .05 as an automatic threshold for publication or policy decisions.

What is p-hacking, and how do I avoid it?

P-hacking is the practice of running multiple analyses, deciding on your statistical approach after seeing the data, or selectively reporting results until you achieve p < .05—then presenting the finding as if it were planned. This inflates false-positive rates. To avoid it: pre-register your analysis plan in advance (many journals now require this for empirical studies); designate which tests are confirmatory (planned beforehand) versus exploratory; adjust your significance threshold if testing multiple hypotheses; and report all results, including non-significant ones.

Can I use the p-value calculator on this site to check my own results?

Yes. The p-value calculator converts a test statistic (z, t, chi-square, or F) into a one-tailed or two-tailed p-value, and displays exactly which part of the distribution the p-value represents. You can use it to verify calculations from your own analysis, check textbook examples, or understand what a given p-value means visually. Results are computed in your browser with no data transmission.

When is a p-value the wrong tool for my question?

P-values test whether data are consistent with a null hypothesis, but they do not answer all questions. Use effect sizes and confidence intervals when you want to estimate a parameter (e.g., "What is the average improvement from this treatment?"). Use Bayesian methods if you want to incorporate prior knowledge or compute the probability that a hypothesis is true given the data. Use prediction intervals if you want to forecast future individual values. Use machine-learning evaluation metrics if you are building a classifier. P-values are most useful for yes-or-no causal questions ("Does this treatment work?") in a frequentist framework, but always pair them with effect sizes and uncertainty intervals.

§8Sources

References and resources consulted for this guide on p-values and statistical inference: