Hypothesis Tests for Experimental Research
A test statistic measures how far an estimate lies from a null value in standard-error units:
\[\text{test statistic} = \frac{\text{estimate}-\text{null value}} {\text{standard error}}.\]The reference distribution determines how unusual that standardized distance is under $H_0$.
One mean
When the population standard deviation is known,
\[Z=\frac{\bar X-\mu_0}{\sigma/\sqrt n}.\]When it is unknown and estimated by $s$,
\[T=\frac{\bar X-\mu_0}{s/\sqrt n}.\]Under a normal population, the second statistic follows $t_{n-1}$ under the null.
One proportion
For $H_0:p=p_0$, the null standard error uses $p_0$:
\[Z=\frac{\hat p-p_0} {\sqrt{p_0(1-p_0)/n}}.\]This differs from the usual Wald confidence interval, which estimates the standard error using $\hat p$.
Two independent means
Welch’s statistic does not assume equal population variances:
\[T= \frac{(\bar X_1-\bar X_2)-\Delta_0} {\sqrt{s_1^2/n_1+s_2^2/n_2}}.\]Its degrees of freedom are approximated by the Welch-Satterthwaite formula. It is generally the safer default for independent groups.
If equal population variances are substantively justified, the pooled variance is
\[s_p^2= \frac{(n_1-1)s_1^2+(n_2-1)s_2^2} {n_1+n_2-2},\]and, for a zero null difference,
\[T= \frac{\bar X_1-\bar X_2} {s_p\sqrt{1/n_1+1/n_2}}, \qquad df=n_1+n_2-2.\]Do not pool merely because a preliminary variance test failed to reject equality.
Paired test
For paired measurements, first define a difference with a consistent direction:
\[D_i=X_i-Y_i.\]Then perform a one-sample test on the differences:
\[T=\frac{\bar D-\mu_{D,0}}{s_D/\sqrt n}, \qquad df=n-1.\]Pairing changes the unit of analysis from individual observations to within-pair differences. This connects directly to Blocked and Paired Randomized Experiments.
Two proportions
For $H_0:p_1=p_2$, let
\[\hat p=\frac{x_1+x_2}{n_1+n_2}\]be the pooled estimate under the null. Then
\[Z= \frac{\hat p_1-\hat p_2} {\sqrt{\hat p(1-\hat p)(1/n_1+1/n_2)}}.\]Classical tests and randomization tests
These tests use sampling-model reference distributions. Fisher Randomization Inference instead derives the reference distribution from the known treatment assignment mechanism under a sharp causal null. The numerical statistic may be the same while the source of the reference distribution differs.