Randomized Experiments Exam Workshop
The defining information in a randomized experiment is the assignment mechanism. Keep potential outcomes fixed and ask what would change under another allowed assignment. That perspective separates causal estimands, estimators, and randomization-based uncertainty.
Worked problem 1: potential outcomes and assignment imbalance
Consider six units with the following fixed potential outcomes:
| Unit | $Y_i(0)$ | $Y_i(1)$ | $\tau_i$ | Assignment $W_i$ |
|---|---|---|---|---|
| 1 | 10 | 14 | 4 | 1 |
| 2 | 12 | 15 | 3 | 0 |
| 3 | 9 | 11 | 2 | 1 |
| 4 | 15 | 18 | 3 | 0 |
| 5 | 11 | 13 | 2 | 0 |
| 6 | 13 | 17 | 4 | 1 |
The finite-sample average treatment effect is
\[\tau_S=\frac{4+3+2+3+2+4}{6}=3.\]The observed outcomes are selected by
\[Y_i^{obs}=W_iY_i(1)+(1-W_i)Y_i(0).\]For the displayed assignment,
\[\bar Y_1=\frac{14+11+17}{3}=14, \qquad \bar Y_0=\frac{12+15+11}{3}=12.67,\]so
\[\hat\tau=1.33.\]The realized estimate differs from the finite-sample ATE because the assignment happened to place units with relatively high $Y(0)$ in control. This is randomization error, not evidence that random assignment failed. Across all allowed complete randomizations, the difference in means is unbiased for $\tau_S$.
Worked problem 2: Neyman repeated-sampling inference
A completely randomized trial assigns 50 units to each arm. The observed mean difference is $4.8$, with $s_1^2=100$ and $s_0^2=144$.
The conservative variance estimator is
\[\widehat{\operatorname{Var}}(\hat\tau)=\frac{100}{50}+\frac{144}{50}=4.88,\]and
\[SE(\hat\tau)=\sqrt{4.88}=2.21.\]The large-sample statistic is $4.8/2.21=2.17$, and a 95% interval is
\[4.8\pm1.96(2.21)=(0.47,9.13).\]The exact finite-population variance also contains $-S_\tau^2/N$. Because each unit reveals only one potential outcome, $S_\tau^2$ is not identified. Omitting the nonnegative term makes the reported variance conservative.
Worked problem 3: Fisher’s tea experiment
In Fisher’s classic design, a participant receives eight cups: four prepared milk-first and four tea-first. She is told that exactly four belong to each group and identifies four as milk-first.
Under the sharp null, the labels have no effect on her judgments. There are
\[\binom{8}{4}=70\]equally likely sets of four cups she could select. Only one selection identifies all four milk-first cups, so the one-sided randomization probability of a perfect result is
\[p=\frac{1}{70}=0.0143.\]This is an exact statement about the randomization mechanism under a sharp null for every cup. It is not a large-sample $t$ approximation and it is not a test only of a zero average effect.
Worked problem 4: blocked estimator
A trial blocks participants by site. Site A contains 40% of the sample and has an estimated effect of 8; site B contains 60% and has an estimated effect of 2. The overall blocked estimate is
\[\hat\tau=0.4(8)+0.6(2)=4.4.\]An unweighted average, $(8+2)/2=5$, answers a different question because it gives the smaller and larger sites equal influence. Analysis must use the design weights unless the estimand explicitly weights blocks equally.
Exercises
- For $N=10$ and $N_1=4$, calculate the number of allowed complete randomizations and the probability of each. Compare this with Bernoulli assignment using $p=0.4$.
- Using the six-unit potential-outcome table, enumerate or program all $\binom63$ assignments. Verify that the mean of $\hat\tau$ equals 3 and inspect the randomization distribution.
- A trial has $N_1=80$, $N_0=120$, $\hat\tau=-2.4$, $s_1=7$, and $s_0=9$. Compute the conservative standard error, test a zero average-effect null, and construct a 95% interval.
- In 12 matched pairs, the treated-minus-control differences have mean 1.8 and standard deviation 2.4. Calculate the paired standard error and test statistic. Explain why analyzing 24 units as independent discards the design.
- Explain how Fisher’s sharp null and Neyman’s zero-average-effect null can lead to different questions and potentially different conclusions.
- A blocked experiment has stratum weights $(0.2,0.3,0.5)$ and effects $(5,-1,4)$. Compute the overall effect, then explain why an unweighted average is inappropriate.
Solution checks
- $\binom{10}{4}=210$, so each allowed vector has probability $1/210$. Bernoulli assignment permits every vector and does not fix the treated count.
- The randomization-distribution mean is 3; individual realized estimates vary around it.
- $SE=\sqrt{49/80+81/120}\approx1.135$, $z\approx-2.11$, and the interval is approximately $(-4.62,-0.18)$.
- $SE=2.4/\sqrt{12}=0.693$ and $t=2.60$ with 11 degrees of freedom.
- Fisher imputes every missing outcome under no effect for any unit; Neyman concerns an average and allows heterogeneous unit effects.
- $0.2(5)+0.3(-1)+0.5(4)=2.7$.