Probability and Sampling Distributions
A sample statistic varies from sample to sample. Its sampling distribution describes that variation and provides the basis for standard errors, confidence intervals, and hypothesis tests.
Sample summaries
For observations $X_1,\ldots,X_n$, the sample mean is
\[\bar X=\frac{1}{n}\sum_{i=1}^n X_i.\]The sample variance is
\[s^2=\frac{1}{n-1}\sum_{i=1}^n(X_i-\bar X)^2.\]The divisor $n-1$ makes $s^2$ unbiased for the population variance under independent, identically distributed sampling.
Expectation and variance
For a discrete random variable,
\[E[X]=\sum_x xP(X=x),\]and for a continuous random variable with density $f$,
\[E[X]=\int_{-\infty}^{\infty}xf(x)\,dx.\]Variance can be written in either form:
\[\operatorname{Var}(X)=E[(X-E[X])^2]=E[X^2]-E[X]^2.\]For constants $a$ and $b$,
\[E[aX+b]=aE[X]+b, \qquad \operatorname{Var}(aX+b)=a^2\operatorname{Var}(X).\]| Adding a constant changes location but not spread; multiplying by $a$ multiplies standard deviation by $ | a | $ and variance by $a^2$. |
Standard error of the mean
If the observations are independent with common variance $\sigma^2$,
\[\operatorname{Var}(\bar X)=\frac{\sigma^2}{n}, \qquad SE(\bar X)=\frac{\sigma}{\sqrt n}\approx\frac{s}{\sqrt n}.\]The $1/\sqrt n$ rate means that reducing the standard error by half requires four times as many observations.
Central limit theorem
Under regularity conditions, the standardized sample mean approaches a standard normal distribution:
\[\frac{\bar X-\mu}{\sigma/\sqrt n}\overset{d}{\longrightarrow}N(0,1).\]For a sufficiently large sample this is used as the approximation
\[\frac{\bar X-\mu}{\sigma/\sqrt n}\approx N(0,1).\]The CLT concerns the distribution of the sample mean, not necessarily the distribution of the raw observations.
Important distributions
If $X$ counts successes in $n$ independent Bernoulli trials,
\[X\sim\operatorname{Bin}(n,p), \qquad E[X]=np, \qquad \operatorname{Var}(X)=np(1-p).\]Standardizing a normal random variable gives
\[Z=\frac{X-\mu}{\sigma}\sim N(0,1).\]For a normal sample with unknown variance,
\[T=\frac{\bar X-\mu}{S/\sqrt n}\sim t_{n-1}.\]The sample variance has the exact normal-sample relationship
\[\frac{(n-1)S^2}{\sigma^2}\sim\chi^2_{n-1}.\]These last two statements are exact when the observations are normally distributed. Large-sample approximations can remain useful outside exact normality.
Connection to experiments
The same logic applies to treatment-control differences. The estimator $\bar Y_1-\bar Y_0$ has a distribution across repeated assignments, and Neyman Repeated-Sampling Inference uses its variance to form a standard error.