Neyman Repeated-Sampling Inference
Neyman inference treats the potential outcomes of the experimental units as fixed and treatment assignment as random. It studies how the estimator varies across repeated assignments from the design.
Estimator
For a completely randomized two-arm experiment,
\[\hat\tau=\bar Y_1-\bar Y_0.\]This estimator is unbiased for the finite-sample average treatment effect
\[\tau=\frac{1}{N}\sum_{i=1}^N[Y_i(1)-Y_i(0)].\]Finite-population variances
Define
\[S_w^2=\frac{1}{N-1}\sum_{i=1}^N\left(Y_i(w)-\bar Y(w)\right)^2, \qquad w\in\{0,1\},\]and
\[S_\tau^2=\frac{1}{N-1}\sum_{i=1}^N(\tau_i-\tau)^2.\]The exact randomization variance is
\[\operatorname{Var}(\hat\tau) = \frac{S_1^2}{N_1} + \frac{S_0^2}{N_0} - \frac{S_\tau^2}{N}.\]The negative term reflects the fact that treatment and control samples partition the same finite set of units.
Why the exact variance is not identifiable
$S_\tau^2$ depends on both potential outcomes for each unit, but only one is observed. It is therefore generally impossible to estimate this term without additional assumptions.
Dropping the nonnegative term gives the conservative estimator
\[\boxed{ \widehat{\operatorname{Var}}(\hat\tau) = \frac{s_1^2}{N_1} + \frac{s_0^2}{N_0} },\]where $s_1^2$ and $s_0^2$ are the observed within-arm sample variances.
The estimated standard error is
\[SE(\hat\tau)= \sqrt{\frac{s_1^2}{N_1}+\frac{s_0^2}{N_0}}.\]Large-sample confidence interval
Using the upper-tail critical-value convention,
\[\hat\tau\pm z_{\alpha/2}SE(\hat\tau).\]For smaller samples, a $t$ critical value and a suitable degrees-of-freedom approximation are often used in practice.
Constant treatment effects
If $\tau_i=\tau$ for every unit, then $S_\tau^2=0$ and the conservative variance expression equals the exact design variance. Constant effects are stronger than merely having a zero average effect.