T-Test vs ANOVA: Which Should You Use?

T-test: Compares the means of exactly two groups. Answers: "Are these two group means significantly different?" There is one null hypothesis (H₀: μ₁ = μ₂) and one test statistic.

ANOVA (analysis of variance): Compares the means of three or more groups simultaneously. Answers: "Is there a significant difference among all these group means?" There is one global null hypothesis (H₀: μ₁ = μ₂ = … = μk) and one test statistic (the F ratio).

Both test whether observed differences in means are larger than we'd expect by chance alone. Both assume the data within each group follow a normal distribution and have equal variance across groups (homogeneity of variance).

Suppose you have four treatment groups and want to know if any pair differs. There are C(4,2) = 6 possible pairwise comparisons. If you run six separate t-tests, each at α = .05, your actual false-positive rate is not 5%—it's much higher.

Why? Each test has a 5% chance of producing a false positive *if the null is true*. But when you run multiple tests, at least one of them is more likely to give a false positive just by chance. This is called family-wise error rate (FWER) inflation.

The formula is: FWER = 1 − (1 − α)^c, where c is the number of comparisons and α is your significance level.

ANOVA tests the global null hypothesis (all means are equal) in a single test. If you reject this null, you know at least one mean differs from at least one other—but you don't yet know which pairs differ. Importantly, this single global test maintains your α = .05 false-positive rate for the entire family of comparisons.

The ANOVA test statistic, F, is the ratio of between-group variance to within-group variance:

If groups have truly equal means, between-group differences are just random noise, and F ≈ 1. If one or more groups differ, between-group variance swells, and F grows large. The p-value tells you the probability of seeing an F this large *if all means were actually equal*.

ANOVA assumes:

• Normality: Each group's data is approximately normally distributed. If you have large samples (n > 30 per group), this is fairly robust to violations.
• Homogeneity of variance: All groups have roughly equal variance. Violating this can inflate Type I error. Test with Levene's test; if variances differ greatly, use Welch's ANOVA instead.
• Independence: Observations within and across groups are independent (not paired, not repeated measures).

If your ANOVA p-value is less than .05, you reject the null and conclude that at least one pair of means differs. But ANOVA does not tell you which pairs. That's where post-hoc tests come in.

A post-hoc test (also called a pairwise comparison test) controls the FWER while testing all possible pairs. Common choices:

• Tukey's HSD (Honestly Significant Difference): The most popular. Assumes equal sample sizes and equal variances. Balances power and strictness.
• Bonferroni: Very conservative. Divides α by the number of comparisons. Good when you want to be very sure, but loses power.
• Scheffé: Conservative but flexible. Works even with unequal sample sizes.

Do not run post-hoc tests unless your ANOVA p < .05. If ANOVA fails to reject the null, pairwise tests are unnecessary and will only inflate error.

• Exactly 2 groups, continuous outcome, independent samples: Use independent-samples t-test.
• 3+ groups, continuous outcome, independent samples: Use one-way ANOVA (or Welch's ANOVA if variances are unequal).
• 2 groups, same subjects measured twice (or paired): Use paired t-test.
• 3+ groups, same subjects measured at each level: Use repeated-measures ANOVA (not covered here; consult a statistics textbook or specialist).
• Outcome is categorical (e.g., pass/fail, disease/healthy): Use chi-square test, not t-test or ANOVA.